15Gammons is an online realtime backgammon website. Its backend is written entirely in Python using TurboGears and Twisted.

In fact, it is the first TurboGears project I’ve started (over 18 months ago); and the last one I ended… I finally had the time to give it the final touches and open it to the public.

Back in 2005 I learned, together with everyone else, that web browsers are capable of doing much more than they were originally designed to do. Javascript libraries were popping out every few days giving drag and drop and animation effects. Everyone were talking about AJAX. I couldn’t stop myself from starting to code a game using the latest tools at the time.

15Gammons utilizes the so-called Comet technique of streaming events to the browser over a long-lived HTTP connection for achieving low latency. The Comet part is written using Twisted. I am planning to share some useful code in a later post.

The user interface was written using the Script.aculo.us library. Flash is not required, but is optional if you like sound effects.

Check it out – play backgammon online in 15Gammons.com!

As I already had methods that return the table data as a list of dictionaries, I thought that I could try sending that data to the browser as JSON, and then process it there using Javascript.

I then came across JST, which is a small Javascript component that renders JSON data with a given template. The template itself can be embedded in the HTML document inside a hidden TEXTAREA element. This seemed like an easy solution. Converting the table part of the Kid template to a JST template was straight forward. Looking at the page loading times now, the speed improvement is remarkable.

After doing that I was able to gain another performance improvement by converting my data to JSON using python-cjson, which is a fast JSON encoder/decoder module implemented in C.

I’ve collected some measurements, which are of course very specific to my application, and should be treated as such.

Each run consisted of generating the page (i.e. calling an exposed function) ten times. The data structure was preloaded to a global variable before the run, so this time is not included. However, some code that is executed by TurboGears’ expose decoration is included in the measurement.

As you can see, using this technique the page generation times is now about 5 times faster.

TGFusionCharts is a TurboGears widget built around InfoSoft’s FusionCharts, which lets you easily add beautiful animated flash charts to your web application.

The package support a wide varierty of chart types, including pie charts, column charts, and line charts. Some of the charts comes with a 2D and a 3D version.

Of course, every good thing must have some drawbacks. Your visitors must have the Flash plugin installed (98% of US users have it installed, according to a study stated in Wikipedia). Moreover, the FusionChart component has a commercial license, but comes with a free trial. If you are not too bothered by that, take a look at TGFusionCharts.

The vast popularity of the Python Challenge had pushed the creative juices in my head and a new puzzle with actual rewards is being “cooked” while you are reading this post.

This is the right moment to announce who were the great winners of the Python Challenge. The following ten were the first to successfully complete all the 33 levels of the puzzle. The list is sorted by completion time from the first to the last. Next to the names you will find the first thought which occurred in their minds at the finishing line.

Each of the winners get the prize of being mentioned in this list (I can link to your site if you wish).

But before we’d go on with the list of winners, I’d like to thank to all those who participated in the Python Challenge and those who suggested levels. I’d like to thank my girlfriend Avital who gave me the inspiration. Special thanks go to Tim Peters and Martijn Pieters who helped moderating the forums.

1. bit4: I finally had time to code and run the etch-a-s[ck]etch solver and get past the smiling python. Lovely!
2. wo: Good Game!!I’m learning python since 2005-05-05.
3. University_of_Ulm_ms55_gh5: We needed 3 hours to fully reveal that smiling python image by hand… (and 4 fields had been wrong).But of course it was fun – even as there was the possibility that the next image would be of size 100×100 *g*.
4. cosimo: Three weeks of challenge. Really great time for me! (excluding one or two levels
5. gbrandl: Was fun, again! Loved especially 32…
6. DJohn The <spoiler removed> one required too much external knowledge. But the rest have been fun.
7. mfuzzey: Great fun – very addictive.
8. Hemlock: Loved it!
9. acw1668: I really enjoy the journey and I hope there are more levels soon.
10. Fraxas: Thanks so much for doing this puzzle for us all, thesamet. it’s been great fun.Pharaohmagnetic: I’ve been working on this challenge with Fraxas for the past several weeks. In that time, it has intrigued, frustrated, and thrilled us to no end. We finally made it to his page. Congratulations on making an awesome puzzle.

Here are few more quotes from people who completed the challenge later:

• ajohnson: Wow, that was great. I’m sure my work suffered while I devoted much time to this. I do feel I needed more help than I should have, but it was still much fun!
• confused: Thanks for some really great challenges, it has been a lot of fun.
• sdsmith: At the end at last! It’s been a fun couple of days… ahem… weeks… ahem… well, anyway, thanks for all the superb problems!
• Pinzo: Hey! I’ve done it!!! I’ve done it!!!Continue the great job!
• jazzgeorge: This was a lot of fun. I didn’t know python at all when I started, and this was a great way to learn!
• divergentdave: These puzzles were a great way for me to learn more about Python, such as stuff like string[::-1].
• Franio: more levels please!!!
• Xofon: Guess I have not been first, but this was not why I struggled through all of this. A pleasant way to learn Python.
• Asi.tak: Thank you for this wonderfull game. I really liked it. I’ve never done anything in Python before and I feel that this Challenge was the best way to learn it and start to like it. It was a lot of fun. So thanks a lot. I am looking forward to new levels.
• nageroc: Can it really be the end? Thanks for the challenge!
• nmcserra: This was great fun! Thanks for creating this challenge.
• arnache: Great site.

Could never have done more than half of them without the hints forum: some of them are just to hard without indication (don’t pretend the hints you give are helpful), and easy once we have it. But that’s the nature of riddles isn’t it ?

• interesse: Solved the first level on July 30th 2006. Now we have December 11th. What a time burner…
• grzes: that was fun. took me a long time though. when i started a little less then a year ago all i knew about python was the wikipedia entry on it… this challenge served me as a python tutorial of sorts…
• tomk: Yes!!!! Finally made it, thanks for a really great way to waste loads of time
• newfweiler: I made it! Wow! Thanks for a fun way to learn Python. I’m going to go send you a second Paypal donation just because this was worth more than buying another Python book.

And finally, here are some quotes from quick fellows who have completed the challenge before it had 33 levels:

• jtauber: This challenge is fantastic. Clever, addictive and really gets your mind working. I feel like I’m playing Myst.
• japyh: Please, stop, no more. I beg you.
• Hang: What was fun! Very good brain exercises.It ‘forced’ me to discover something new python stuff, and make some new friends too.
• birkenfeld: Yay! Has been fun till here, hope there’ll be 100 soon! <wink>

To be continued.

Original:

Translated:

Related posts:

• TurboGallery: Making a Flickr Killer in 30 Minutes Flat.
• Intoduction to TurboGears and Comparison with Other Web Frameworks.

At this point, I closed OpenOffice Impress and said that I would demonstrate how quickly you could get a functional Web application up and running with TurboGears.

Setting Things Up

I’ve quick-started a new project called TurboGallery…

…and gone inside its directory to run it:

I started Firefox and browsed to http://localhost:8080. “There is already something to see on the site,” I said. The TurboGears’ default welcome page was showing up.

“The next thing I always do is to delete the entire body of the welcome template.” So, I opened welcome.kid in my favorite text editor and did just that. I then refreshed the page in Firefox. All the contents were gone except for the TurboGears’ header and the footer, which said, “TurboGears under the hood.” Someone asked if there was any way to remove those two remaining items. “Absolutely,” I said. It was time to mention master.kid. I opened the file and explained that it is used to render the layout that is common to all pages. As such, you do not have to duplicate it in all your templates. I next removed the TurboGears’ footer but left the header in its place.

I then opened model.py and added a class that would represent a single photo:

A photo will have a title and a date field, which indicates the time it was uploaded. The field’s default argument makes the date of a newly created photo object be the current date. That way, you don’t need to specify this information every time you create a new photo. In the above definition, we don’t give the date field the current time. Rather, we give it a function that returns the current time. Whenever a photo is created, this function will be called to determine the value for this field. We store the images inside the database together with a thumbnail. BLOB stands for binary large object. The thumbnail part was not actually used in the tutorial.

Next, I created a database with

I was then asked how TurboGears knows where and how to access the database. I replied that the default TurboGears’ setting uses SQLite, which creates a database-in-a-file. It is very convenient to have your database readily available as a normal file while your project is in its development and testing phases.

To save time, I have prepared in advance a database that contains four photos. I replaced the database file that was just created with my copy. The next thing to do will be to display the photos on the main page. I modified the index() method into…

…and uncommented the “from model import *” line. The index() method is called whenever the front page of the Web site is viewed. I explained the expose() decorator, which makes a method available to the outside world. Without it, the method could not be accessed from the Web. The template argument specifies which template should be used to render the page.

The index() method gets a list of all photos from the database and returns it inside a dictionary. TurboGears passes this dictionary to the template. I then moved back to welcome.kid and added the following to the body:

I refreshed the page, and the list of photos was displayed. It is educational to view the source of the resulting page, but the people who were listening to my lecture wanted to see the photos straightaway.

Seeing Some Photos

To see the photos, we’ll have to add an IMG element to the list. The IMG element will get the photo file from another URL. I’ve changed the welcome.kid list into…

…and then added the following photo_info() method to the controller:

I next browsed to /images/world, and the browser displayed “Hello world.” The goal of this step was to demonstrate to the audience how TurboGears (CherryPy) maps URLs to methods. It was also intended to illustrate how easily a part of the URL can be transformed into positional arguments.
The right thing to put in this function would be a query that would obtain the photo from the database and return the associated jpeg file. Here is the code:

I refreshed the page, and the photos I’d prepared were now showing. (I also changed the CSS file when no one was looking to have this layout.)

In the second picture, you can see me evangelizing for the use of Firefox in Thailand.
Since this is just an example application to make things simple, the image is sent in full size and is resized by the browser.
Clicking on the caption below the images sends us to a page we haven’t yet created. This page will display the photo in full size together with some statistics on it. I saved welcome.kid as photo_info.kid and changed the body contents to:

The corresponding method in controllers.py() would be:

Share Your Photos

An online gallery application is quite useless if its users can’t upload photos from their own computers. As such, we need to create an image upload form. In TurboGears, creating forms and validating user input is very easy. I’ve added a form definition to the top of controllers.py. The first part defines the fields:

The form will have a text input field for the title of the photo as well as a field that enables the selection of an image file from the user’s computer. The second part of the definition is the validation schema for this form:

“The touchiest issue with any Web application is its users,” I said. “Without them, there is no need to worry about bugs or invalid input.” The validation makes sure the input your Web application receives is a sound one. In many cases, further validation is needed. The above validator makes sure the title is not empty and is no longer than 16 letters. One audience member asked if the validation can be specified inside the fields definition. Yes, it is possible to specify a validator as a keyword argument to a field definition, but making the validation schema exterior to the fields definition renders it possible to define a more complex schema that involves field dependency or logical operators. A common instance where such a schema is useful is in the validation of a registration form, where you have to check whether or not the entered password field text and the “reenter password field” match.

The last part of the form definition ties the previous two parts together, with text for the submit button and a URL that will handle the form data:

I’ve created a template named add.kid that will display just the form:

I’ve also added a controller method to make this page accessible…

…and I’ve linked to it from welcome.kid.

Here is a screenshot of this page:

As you can see, TurboGears gives us a nice form without us having to type in any HTML at all. Clicking the Upload! Button posts the data to the /upload URL. To test the form, I’ve added the corresponding method to the controller:

The decorators that are attached to this method make TurboGears validate its input using the add_photo_form. If a validation error occurs, the response is handled by the add() method, which just displays the form again along with the validation errors. So if, for example, we type in a too-long title, we will get the following:

I would now really like to make the method save the image in a new photo object. The following code will do:

Yes, handling a file upload in TurboGears is just a matter of reading from a file-like object. It is that simple. The next line creates a photo object with the title and the image data. The flash() method makes the given message appear on the next page, and we redirect to the main page. I filled out the form to upload an image that was downloaded from my camera. Here’s what I got:

Damn! I hate it when my camera decides to rotate an image that has already been uploaded. So, let’s add an AJAX image-rotation tool. A click on the rotate link will rotate the image in place without requiring the entire page to be reloaded. We first add a method to rotate the image to our controller:

The method uses PIL – Python Imaging Library. It loads the image from the database, rotates it, and then stores it again in the database. The method returns a dictionary containing the photo_id and the new photo dimensions. It is set to return the data in JSON format, which makes it extremely easy to use in Javascript. I directly entered http://localhost:8080/rotate/2 to show what the JSON object looks like. I then refreshed the main page to verify that the photo had been rotated. Next, I rotated it three more times until it was straight again.

I then went back to welcome.kid and added a rotate link for each photo:

Clicking on the “rotate” text located next to a photo will call a Javascript function that receives the corresponding photo ID. I’ve added the implementation of rotate_photo() to the top of the welcome.kid template. It uses MochiKit, which must be enabled in config/app.cfg (it is explained how to do so in that file).

The function makes an asynchronous call to the rotate URL, providing it with the photo ID. Once the response arrives, update() is called and is provided with the JSON object we returned from the controller’s rotate() method. In Javascript, you can use dot notation to access the keys in that dictionary. When update() is called, the image has already been rotated at the server. Therefore it is the right time for the browser to fetch it again. To perform this re-fetching, update() sets the image src attribute to the image URL, but in order to prevent the browser from displaying the cached old image, we add a random argument to the URL. You can see a related post describing how to prevent browsers from caching. To make the image() controller method accept this argument and ignore it, its definition becomes:

I then refreshed the main page and rotated the photos a few times to demonstrate how slick this functionality is (especially when you’re working on the server). You can see it in the video below:

Next, I uploaded another image I had prepared in advance, this one called questions.jpg, using the upload form. Here it is:

It says “Questions?” in Hebrew, signifying that the lecture was over and it was time to ask questions.

Download the full source code of TurboGallery.

I hereby give the lecture again, in a written format. These are the original slides used in the lecture. Click on a slide to enlarge it.

Because it was only the second time I had attended one of these meetings, I briefly introduced myself. The two slides above show photos that were taken in my home town, Adi, which is a small town in Northern Israel. In the right photo, you can see my cat Uzi watching the cows in our backyard. If you have taken part in the Python Challenge, you might have already met Uzi.

TurboGears is a framework that enables the rapid development of Web applications in Python. But what is a Web application? Great question!

Wikipedia provides this explanation:

In software engineering, a Web application or webapp is an application that is accessed with a Web browser over a network such as the Internet or an intranet.
Web applications are popular due to the ubiquity of the browser as a client, sometimes called a thin client. The ability to update and maintain Web applications without distributing and installing software on potentially thousands of client computers is a key reason for their popularity. Web applications are used to implement Webmail, online retail sales, online auctions, wikis, discussion boards, Weblogs, MMORPGs, and many other functions.

What makes TurboGears stand out from the crowd is its unique philosophy of reusing. Instead of developing its own Web server, HTML-templating language and database-to-object relational mapper, it integrates existing mature and best-of-breed components. The main actors are CherryPy, which makes adding a page as easy as writing a method; Kid, which is a designer-friendly XHTML-templating language; and SQLObject, which allows you to use an SQL database without writing a single line of SQL. MochiKit is a pill library that eases your pains while coding in Javascript…

Why should you choose TurboGears as your Web-application development framework for your next project? Well, the short answer is that PHP is the devil – it tempts you to mix HTML, logic, and database queries into the same piece of code. Rails, as great as it may be, is written in Ruby, and we like Python better. For Django, I borrowed (err, stole) Adrian Holovaty’s port of Mark Pilgrim’s piece:

I was walking across a bridge one day, and I saw a man standing on the edge, about to jump off. So I ran over and said, “Stop! Don’t do it!”
“I can’t help it,” he cried. “I’ve lost my will to live.”
“What do you do for a living?” I asked.
He said, “I work with computers.”
“Me too!” I said. “What do you do with computers?”
He said, “I’m a Web developer.”
“Me too!” I said. “Design, client-side programming or server-side programming?”
He said, “Server-side programming.”
“Me too!” I said. “Do you use dynamically typed languages or statically typed languages?”
He said, “Dynamically typed languages.”
“Me too!” I said. “Do you use a Web framework, or do you roll things on your own?”
He said, “I use a Web framework.”
“Me too!” I said. “TurboGears or Django?”
He said, “Django.”
“Die, heretic scum!” I shouted, and I pushed him over the edge.

The moral of the story is that TurboGears and Django have much more in common than they have in the way of differences. It has been said that Django is like a Macintosh—it simply works—while TurboGears is more like Unix—its power comes from its being made of many smaller building blocks, each one of them designed to do one thing well.

TurboGears and Django try to solve different types of problems. Django was born in a publishers’ newsroom environment, where Web applications needed to be launched within hours. By its very nature, Django is geared toward the development of content-oriented Web sites. TurboGears, on the other hand, evolved while Kevin Dangoor was developing Zesty News, a news-reader application. As a result, it is more suitable for Web applications (see earlier definition).

Having said that, I found it very easy to develop content-oriented Web sites in TurboGears. Currently, TurboGears’ CRUD features and administration panel are somewhat lacking in comparison to Django’s. As such you have to manually code these aspects of your application if you want a non-tech author to be able to fill your project’s database with content or change various aspects of the Web site.

TurboGears goes a long way in encouraging you to design your application in the so-called MVC paradigm. MVC stands for Model, Viewer, and Controller. The model consists of the classes that represent the information on which the application operates. That information is usually stored in an SQL database. The view is composed of the HTML page templates that are used to render the model. The controller responds to user input (HTTP requests) by operating on the model and returning a view for subsequent interaction.

In a nutshell, MVC is all about separating metadata, logic, and presentation.

The above is an example of how you would model a user with SQLObject. It has first_name and last_name fields as well as a date-time field, which represents the time at which the user was last logged in to the system.

The above class definition can automatically generate the SQL table creation statement:

The “Person” class gives you a very comfortable interface with the database objects:

In the next installment, I will show you how easy it is to create a Flickr killer application with TurboGears in 37 minutes.

Make sure you won’t miss it by subscribing to the RSS feed.

Xapian to the Rescue

Xapian is an excellent open source (GPL) search engine library. It is written in C++ and comes with bindings for Python as well as many other languages, and it supports everything you’d expect from a modern search engine:

• Ranked probabilistic search – The results that are returned are ranked according to their relevancy, with the most relevant occurring first.
• Boolean search operators – You can use AND, OR, NOT, XOR in your searches.
• Phrase and proximity searching – For example, “used books” will look for occurrences of these words as an exact phrase, but you can also search for “used NEAR books” to find occurrences of the words “used” and “books” that are within 10 words of each other. You can even write “used NEAR/3 books” to change the proximity threshold to three
• Stemming of search terms – If, for example, you search for “programmer,” you can find articles that mention “programmers” or “programming.” Xapian currently supports stemming in Danish, Dutch, English, Finnish, French, German, Italian, Norwegian, Portuguese, Russian, Spanish, and Swedish
• Stopwords – Xapian already knows which common words to ignore (like ‘are,’ ‘is,’ and ‘being’)
• Simultaneous update and searching – Xapian allows to index new articles while the database is being searched. New articles can be searched right away.
• Relevant Suggestions – Xapian can automatically suggest documents that are relevant to a given document. As such, you can add a list of “similar items” to each page.
• Value-associated results – You can associate values like word-count, date, page views, diggs, and so on with each document. Xapian can return results that are sorted by any of these criteria
• Document metadata – You can add metadata tags to each document (Xapian calls these terms). These tags can be anything you desire, like author, title, date, tags, and so on. Users can then search within the metadata by typing “author:john”.

The above diagram shows the main participants in a typical search-enabled use case. We assume that the data to be searched is stored in a relational database (the blue SQL server jar), but it doesn’t really matter where the data comes from. The indexer is a Python program that is executed periodically (as a cron job). Its function is to retrieve new or changed documents from the database and to index them. The Xapian library handles the actual read/write operations on the Xapian database (in the purple jar).

Since the Xapian library is not thread-safe and because Web applications are usually multithreaded, you need to implement a locking mechanism if you want access to a Xapian database to be safe. My preferred way for accomplishing this aim is to use a separate process (the orange Search Server box). This process will be a single-threaded RPC server that will handle all searches. The benefit of this strategy is that you can move the search server process (together with the Xapian database) to a different machine. In so doing, you can free up a lot of the resources on that server that runs your application. That makes the system very scalable. In general, you can expect any bottlenecks to be more IO and memory related than CPU related.

Alternatively, your search operations can be directly initiated from your application process. This alternative will work as long as you use a mutex to govern access to your database. However, I wouldn’t recommend doing this in a production environment. Why? Because you’ll never be sure what’s consuming so much memory—the Xapian library or your application.

The application (red box) gets a very clean search API. It simply connects to the XML RPC server (one line of code) and obtains access to a search() method which gets the search query and how many results are needed as arguments. It then returns a dictionary with the total number of available results and the results themselves.

In this tutorial, we’ll create a searchable article database. We assume that you already have Xapian and the Xapian bindings installed. We’ll start with the indexer.

The Indexer: The Golden Retriever

Following is the indexer code. It is tailored to TurboGears and SQLAlchemy, but it can be easily adapted to suit any ORM. It accepts three command line arguments: the configuration file, which helps it find the database (either dev.cfg or prod.cfg); the Xapian database location, which is simply a directory name; and a number of hours (such that all documents that were created or modified within this number of hours will be indexed). If you run the indexer every hour, you can safely give 2 as the third argument. If you’d like to re-index all articles, pass in 0 as the third argument.

All strings that are passed to Xapian functions must be encoded in UTF-8. The create_document() function splits the article’s text into words, stems them, and then adds them one by one to the Xapian document. Next, a term with the username of the article’s author, prefixed by the letter ‘A,’ and the article’s subject, prefixed by the letter ‘S’ is added. Xapian gives special treatment to the first character of each term (i.e., it gives the terms its meaning). You’ll see how we use these terms when we code the search server.

Another term, this one prefixed by the letter ‘I’,’ is now added to render a unique article ID. The article ID is also assigned to the document as a value. This number relates a Xapian document to its authentic source in the SQL server.

The index() method function simply selects the relevant articles and builds a Xapian document object for them. Instead of using add_document(), which can cause an article to be indexed multiple times in the database, we use replace_document(), which is given a unique term. If a document is already indexed by the given term, it will be replaced with the given document; otherwise, a new document will be added to the database.

After the data is indexed, it is time to make it searchable.

The Search Server: Seeing Results

The role of the search server is to obtain queries from the application and to then return results. As we strive for a single-threaded implementation, the Twisted framework makes it extremely easy to write the code for this server. If you are not familiar with Twisted or XML RPC, don’t worry; just imagine we’re writing a controller with only one method exposed: xmlrpc_search().

The search server constructor opens a database and initializes a query parser. We tell the query parser that the keyword ‘author’ refers to terms that are prefixed with the letter ‘A’ and that the keyword ‘subject’ refers to terms that are prefixed with the letter ‘S.’ This specification makes it possible to search for “author:john” or “subject:xapian.”

We then instruct Twisted to call reopen_db() every ten minutes. This reopening renders the latest changes in the database available to the search server. Each time Xapian’s library opens the database, it works against a fixed revision of it.

The xmlrpc_search() is the only method that is exposed (since its name is prefixed by xmlrpc_). The offset (zero-based) and limit arguments allow for an efficient way to split the search results into several pages. The call returns a dictionary with the total number of available results and a list with the selected subset of results. Each item in the list contains a unique article_id and a percent, which indicates each document’s relevancy score.

The program receives the port number to listen on and the Xapian database path. Unless you’d like to expose your search functionality to the world, it is suggested that you block outside access to this port.

By now, you’re probably eager to try searching your own database. Here’s a quick way to do so. First, start the search server:

From another terminal, start a Python shell:

In the same manner, you can use the search server from your application.

Working with Smaller Databases

Search engines are optimized to return results that are sorted according to their relevancy. If you need your results sorted by another criterion, such as date or diggs, it might be useful to run the query over a smaller database. For example, you might try running it over a database that contains only articles from the previous month. This strategy can significantly increase your overall performance.

Alternatives to Xapian

While I haven’t tried working with search engines libraries other than Xapian, you can try the Java-based Lucene which can be accessed from Python using PyLucene. The TurboLucene library eases using PyLucene from TurboGears.

SQL Injection Attacks

The joy of using an ORM like SQLAlchemy or SQLObject—in addition to the benefit of not having to write a single SQL statement yourself—is its ability to protect you from SQL Injection attacks. Although this built-in security measure affords you protection, it is important to understand how SQL injection attacks work.

If an application contains code that looks like this…

…then you are vulnerable to the stinging strikes of a malicious user. That’s because he or she can craft a cookie with the value of "someonebad';
TRUNCATE TABLE users; SELECT '". The SQL statement that will be executed will then be:

…which is not good, not good indeed!

To see just how bad the situation really is for our fellows in PHP land, have a look at these Google code search results, or for our little brothers in ASP land, have a look here. Please act responsibly and don’t hack into the sites listed there.

Luckily, ORMs escape all the strings we send to the database engine, and as a result, we are protected from this kind of harmful attack.

XSRF: Cross-Site Request Forgery

This exploit is very common in Web applications, especially in those that provide AJAX API. It is best to describe this type of attack with an example. Suppose that an imaginary project, TGBank (which is a Web interface for a bank), has a send_money() method:

The bank site using this application might contain a page with a “send money” form, where the user fills in information such as to whom he is transferring the money and how much money he is transferring. Everything’s find and dandy there. But what if the user is connected to the bank site, and in another browser tab, he is simultaneously browsing a malicious site, one that contains the following img tag?

Then the user’s browser will trigger that operation on behalf of the unsuspecting and innocent Web visitor. And because it’s such and inconsequential amount, he probably won’t even bother to check about those four cents.

Allowing only POST requests to go into send_money() will not help the matter. That’s because it is easy to send POST requests using Javascript. On the other hand, checking that the HTTP referrer header of the request is within one’s domain is too restrictive. That’s because many browsers often do not send this header.

A possible solution is to add a hidden field that only your application can generate and validate. For example, you might consider that the application will process the request only if received a query argument with the value of a sha1 digest of a string that is composed of the user id and a secret word. This string can be easily validated by your application, but it will be hard for a malicious site to generate.

Utterly Ridiculous!—More XSRF: Stealing Information with Scriptaculous

In addition to doing serious damage, this type of attack can be used to steal information. Suppose the bank application previously mentioned has a URL that returns Javascript code that defines a list with your monthly statement (list of expenses). It may be used on the bank site for doing client side sorting. A malicious site might contain something like this:

Here, that dastardly attacker placed code on his site, that executes the monthly statement script from the bank’s site. Once that script is executed, we have an object named statement containing a list of monthly expenses. Then, after the document finished loading, it is transmitted right into the waiting attacker’s hands.

Recently, an XSRF flaw was discovered (and fixed) in GMail. This vulnerability allowed an attacker to steal the user’s contact list precisely as just mentioned.

Assault–Take Two! XSS: Cross-Site Scripting

Cross-site scripting vulnerability occurs when a Web application generates output that contains user-supplied data without HTML encoding. For example, if we allow a post in a forum to contain HTML tags, then we can use KID to display it:

Raising the ax yet again, a vile attacker can embed javascript code into his post. Then, when an innocent user visits the page, his or her browser runs the script, unbeknownst to him or her. This script can, for example, send the contents of the page back to the attacker. It might also post a comment to a blog, in the unsuspecting user’s name, or make new friends for him or her in MySpace.

If we do not use the XML() function in KID, then HTML entities are escaped and we are safe. The users will see just the script text and the browser will not interpret it as code. That said, if XML() is to be used, then it is suggested one check whether the string contains '<script' (case insensitive) before actually sending it. You should also beware of spaces between the script and its surrounding < and >, although I haven’t tested which browsers allow it. Note that most HTML elements allow attributes that can contain javascript code like onmouseover or onclick. The safest thing would be to escape all < to < (Python has a cgi.escape function for this). If HTML tags are to be allowed, then the application should carefully check that they contain only permitted attributes. Also the href attribute of <a> tag should be checked that it does not contain something like “javascript:do_something()”. That way, those malicious attackers can be forced to drop their weapons before they have time to draw them.

Now that you’ve learned how to preempt attacks before they strike, you’re well on your way to a more enjoyable Web application development experience. Comments and questions about the strategies outlined here are welcomed. We also encourage suggestions on how you’ve successful waged war against security attackers.

Moving around quickly

The most basic part is to move to the place you want to edit quickly. Here are the most useful shortcuts:

• Ctrl-a – Move to the start of the line.
• Ctrl-e – Move to the end of the line.
• Alt-f – Move forward a word.
• Ctrl-f – Move forward a char.
• Alt-b – Move backward a word.
• Ctrl-b – Move backward a char.

Another very common shortcut is Ctrl-L which clears the screen and reprints the current line at the top. This is useful if you are organizing your movies library and somebody enters the room.

Cut, Paste and Undo

Here is the list of most useful shortcuts for deleting text:

• Ctrl-w – Deletes from cursor position to the previous whitespace.
• Ctrl-k – Deletes from cursor position to the end of the line.
• Ctrl-d – Deletes from current character.
• Ctrl-u – Deletes from whole line.
• Alt-d – Deletes from cursor position to the end of the current word.
• Alt-DEL – Deletes from cursor position to the start of the current word.

The deleted text can be pasted (yanked) using Ctrl-y. You can use Ctrl-_ or Ctrl-X-U to undo the last editing command.

Summary

This covers the essentials. On the next parts of this series we’ll explore the more powerful abilities of readline. Feel free to leave here questions, suggestions, or perhaps your own tips and tricks.