Speeding Up your Application with Javascript Templates (and cjson)

The web application project that I am currently working on (the one for which I’ve written TGFusionCharts) has some pages with huge tables, generated using the Kid Template Language. Page loading times became more and more notable as the table sizes went up. On some pages, the rendering time could be as long as 2 seconds.

As I already had methods that return the table data as a list of dictionaries, I thought that I could try sending that data to the browser as JSON, and then process it there using Javascript.

I then came across JST, which is a small Javascript component that renders JSON data with a given template. The template itself can be embedded in the HTML document inside a hidden TEXTAREA element. This seemed like an easy solution. Converting the table part of the Kid template to a JST template was straight forward. Looking at the page loading times now, the speed improvement is remarkable.

After doing that I was able to gain another performance improvement by converting my data to JSON using python-cjson, which is a fast JSON encoder/decoder module implemented in C.

I’ve collected some measurements, which are of course very specific to my application, and should be treated as such.

1st Run 10.96s 2.19s
2nd Run 11.14s 2.28s
3rd Run 11.02s 2.28s

Each run consisted of generating the page (i.e. calling an exposed function) ten times. The data structure was preloaded to a global variable before the run, so this time is not included. However, some code that is executed by TurboGears’ expose decoration is included in the measurement.

As you can see, using this technique the page generation times is now about 5 times faster.

This entry was posted in python, turbogears. Bookmark the permalink.

12 Responses to Speeding Up your Application with Javascript Templates (and cjson)

  1. guillaum says:

    If i well understood, your render your tables client-side and not server-side ?

    But what if:

    – your client have not javascript (such as web robot that are able to parse html table, but per haps not json syntax ?)
    – your client have a small CPU. It seem that your measures is render time ‘server-side’. But how much time it take ‘client-side’ ?

    This seem normal that JSON take less time than kid on the server-side, the second one does processing server-side when the first does not.

    Two last questions, do you try the cElementTree implementation of Kid and do you try to gzip your HTML output for minimizing the amount of needed bandwith.

    At last, i’m really interested for an example of what you call “huge table”.

    (and sorry for my bad writing skill in your nice language)

  2. thesamet says:

    Good questions. In my case the relying on Javascript for rendering is acceptable. I can safely assume that my human clients have Javascript enabled, and web robots can not access this site anyway as it requires authentication.

    If necessary, one can overcome this limitation by detecting if the client supports javascript, and if not output the usual html template. I am not sure about how to make a single template for that, however.

    As for CPU requirements on the client side – perhaps this can be an issue on certain configurations. As for my tests, the template processing time was unnoticable.

    I think that the issue here is not that the data set is too large, but Kid is too slow. My tables are about 1000 rows and 5 columns, each containing numbers or short strings.

    My system has cElementTree installed, so I guess Kid detects this. The measurements taken above do not take bandwidth into account, only data generation time on server. I haven’t decided yet if I am willing to take the trade of gzipping the HTML output (bandwidth over server’s cpu). Probably will have to conduct more experiments on that.

  3. guillaum says:

    And if Kid is really so slow, why not generate the code server-side, but with a traditionnal for loop ?

    I think that tools such as kid must not remove for our brain that there is still the “old way”.

    I ran some tests, with a data set such as:

    NB_LINES = 1000
    a = “hello i’m a french guys”.split()
    data = [tuple(a) for i in range(NB_LINES)]

    I use that function to generate the data:

    def XML_from_data(data):
    out = ”
    for line in data:
    out += ”
    for element in line:
    out += ” + element + ”
    out += ”
    return out

    This function is really what we must not do if we want speed 🙂

    The function take 0.00541855292977 in my hardware for one transformation of data.

    I did 3 tests, one with full kid.
    The second with Kid which call XML on the output of XML_form_data
    the third one with just a replace() or string formating call on the output of kid and a placeholder for the table:

    kid: 1.00006444319
    kid with XML(): 0.995648732246
    replace methode: 0.0171378728814

    So if the XML() method is not interesting, the replace is. I don’t know if it’s difficult or not to use that solution on TG, but I think that can save you from the use of that JSON big and horrible hack 🙂

    The code, note I love the *enlarge your textarea now* feature 🙂

    import kid
    import timeit

    NB_LINES = 1000

    # data table
    a = “hello i’m a french guys”.split()
    data = [tuple(a) for i in range(NB_LINES)]

    src = “””




    def XML_from_data(data):
    #return ” + ”.join(”.join(line) for line in data) + ”
    out = ”
    for line in data:
    out += ”
    for element in line:
    out += ” + element + ”
    out += ”
    return out

    solution1 = src%”’$elem”’
    solution2 = src%”’${XML(data)}”’
    solution3 = src%”’%(data)s”’

    print timeit.Timer(‘kid.Template(source=solution1,data=data,title=”test1″).serialize()’,
    ‘from __main__ import solution1, data;import kid’).timeit(1)
    print timeit.Timer(‘kid.Template(source=solution2,data=XML_from_data(data),title=”test2″).serialize()’,
    ‘from __main__ import solution2, data,XML_from_data;import kid’).timeit(1)
    print timeit.Timer(‘XML_from_data(data)’,’from __main__ import XML_from_data,data’).timeit(1)
    print timeit.Timer(‘kid.Template(source=solution3,title=”test3″).serialize()%dict(data=XML_from_data(data))’,
    ‘from __main__ import solution3,data,kid,XML_from_data’).timeit(1)

  4. guillaum says:

    I made a comment, but it don’t appear ? (when i try to resubmit it, i have a warning about duplication, so my comment is on your database, but not displayed)
    Where is my comment ?

  5. thesamet says:

    Your previous comment was waiting in Akismet moderation queue. Probably it suspected it because of its length.

    As your results suggests, Kid is 100 times slower than a manual string crafting string. Well, this means that I’ll have to write my own custom server-side templating language. But hey, why should I invent a new server-side templating language if I have kid? 🙂 Back to square 1.

  6. guillaum says:

    I don’t suggest you to rewrite your own server-side templating language, but to stop using kid for specials cases and replace it by a plain for loop, just for critical parts.

    I can imagine that in your TG apps, you have a place where your write some code (per haps a plain for loop) to create your JSON output and then insert it in your HTML output ?

    Replace it by a loop which generate HTML and insert it in your HTML output ? it’s the same job, and you can remove the client part ?

    I dont’ know how TG really work, but you may have something like that.

    output = mykidtemplate.serialize()

    replace it by

    output = mykidtemplate.serialize()
    output = output.replace(‘PLACEHOLDER_FOR_MY_TABLE’,

    No, this juste add some few lines in your application, increase speed and remove the need for client side javascript.

    Is there is something that i missunderstand ? (I’m not TG user and not a web develloper 🙂

    And can I remember you that your had write your own custom client-side templating language 🙂

  7. thesamet says:

    I agree that this approach could be right for simple for-loop templates.
    On the other hand, some applications, may become simpler and faster (and less lynx or bot compliant) if they outsource the rendering to the client. It is a trade-off.

  8. florian says:

    I don’t like JST syntax and I don’t like parsing template markup in the browser. Hence I wrote Genshi2Js
    which converts Genshi templates to javascript functions, to which you then feed your data (deserialized json) and out comes html fragments intended for insertion in innerHTML.

    You could use it as a compiler on the command line, generating your javascript file, or you could use it as a compiler for handling web-requests for your javascript directly.

    (sorry, meant to comment this here, somehow ended up in another post)

  9. Asking questions are really good thing if you
    are not understanding something completely, but this piece of writing offers pleasant understanding yet.

  10. Thankfulness to my father who shared with me regarding this webpage, this web site is in fact remarkable.

  11. dosial.com says:

    I’m rеally enjoying tÒ»e design and layout оf ï½™Ö…ur
    site. Ιt’Ñ• a very easy on the eyes whicÒ»
    maқeѕ itt much more enjoyable ffor mee tο come
    here and vvisit more often. Did yоu hire out ɑ developer tо create your theme?
    Excellent ԝork!