$ telnet -l "-froot" server

where server is the server name. I was able to confirm this on a Solaris server nearby.

While the telnet port is usually blocked to servers on the internet, it is quite common that it is left open inside local networks, and especially in universities. So go ahead and look for Solaris rootkits — the exam period is just over the corner

Source: Errata Security blog.

Xapian to the Rescue

Xapian is an excellent open source (GPL) search engine library. It is written in C++ and comes with bindings for Python as well as many other languages, and it supports everything you’d expect from a modern search engine:

• Ranked probabilistic search – The results that are returned are ranked according to their relevancy, with the most relevant occurring first.
• Boolean search operators – You can use AND, OR, NOT, XOR in your searches.
• Phrase and proximity searching – For example, “used books” will look for occurrences of these words as an exact phrase, but you can also search for “used NEAR books” to find occurrences of the words “used” and “books” that are within 10 words of each other. You can even write “used NEAR/3 books” to change the proximity threshold to three
• Stemming of search terms – If, for example, you search for “programmer,” you can find articles that mention “programmers” or “programming.” Xapian currently supports stemming in Danish, Dutch, English, Finnish, French, German, Italian, Norwegian, Portuguese, Russian, Spanish, and Swedish
• Stopwords – Xapian already knows which common words to ignore (like ‘are,’ ‘is,’ and ‘being’)
• Simultaneous update and searching – Xapian allows to index new articles while the database is being searched. New articles can be searched right away.
• Relevant Suggestions – Xapian can automatically suggest documents that are relevant to a given document. As such, you can add a list of “similar items” to each page.
• Value-associated results – You can associate values like word-count, date, page views, diggs, and so on with each document. Xapian can return results that are sorted by any of these criteria
• Document metadata – You can add metadata tags to each document (Xapian calls these terms). These tags can be anything you desire, like author, title, date, tags, and so on. Users can then search within the metadata by typing “author:john”.

The above diagram shows the main participants in a typical search-enabled use case. We assume that the data to be searched is stored in a relational database (the blue SQL server jar), but it doesn’t really matter where the data comes from. The indexer is a Python program that is executed periodically (as a cron job). Its function is to retrieve new or changed documents from the database and to index them. The Xapian library handles the actual read/write operations on the Xapian database (in the purple jar).

Since the Xapian library is not thread-safe and because Web applications are usually multithreaded, you need to implement a locking mechanism if you want access to a Xapian database to be safe. My preferred way for accomplishing this aim is to use a separate process (the orange Search Server box). This process will be a single-threaded RPC server that will handle all searches. The benefit of this strategy is that you can move the search server process (together with the Xapian database) to a different machine. In so doing, you can free up a lot of the resources on that server that runs your application. That makes the system very scalable. In general, you can expect any bottlenecks to be more IO and memory related than CPU related.

Alternatively, your search operations can be directly initiated from your application process. This alternative will work as long as you use a mutex to govern access to your database. However, I wouldn’t recommend doing this in a production environment. Why? Because you’ll never be sure what’s consuming so much memory—the Xapian library or your application.

The application (red box) gets a very clean search API. It simply connects to the XML RPC server (one line of code) and obtains access to a search() method which gets the search query and how many results are needed as arguments. It then returns a dictionary with the total number of available results and the results themselves.

In this tutorial, we’ll create a searchable article database. We assume that you already have Xapian and the Xapian bindings installed. We’ll start with the indexer.

The Indexer: The Golden Retriever

Following is the indexer code. It is tailored to TurboGears and SQLAlchemy, but it can be easily adapted to suit any ORM. It accepts three command line arguments: the configuration file, which helps it find the database (either dev.cfg or prod.cfg); the Xapian database location, which is simply a directory name; and a number of hours (such that all documents that were created or modified within this number of hours will be indexed). If you run the indexer every hour, you can safely give 2 as the third argument. If you’d like to re-index all articles, pass in 0 as the third argument.

All strings that are passed to Xapian functions must be encoded in UTF-8. The create_document() function splits the article’s text into words, stems them, and then adds them one by one to the Xapian document. Next, a term with the username of the article’s author, prefixed by the letter ‘A,’ and the article’s subject, prefixed by the letter ‘S’ is added. Xapian gives special treatment to the first character of each term (i.e., it gives the terms its meaning). You’ll see how we use these terms when we code the search server.

Another term, this one prefixed by the letter ‘I’,’ is now added to render a unique article ID. The article ID is also assigned to the document as a value. This number relates a Xapian document to its authentic source in the SQL server.

The index() method function simply selects the relevant articles and builds a Xapian document object for them. Instead of using add_document(), which can cause an article to be indexed multiple times in the database, we use replace_document(), which is given a unique term. If a document is already indexed by the given term, it will be replaced with the given document; otherwise, a new document will be added to the database.

After the data is indexed, it is time to make it searchable.

The Search Server: Seeing Results

The role of the search server is to obtain queries from the application and to then return results. As we strive for a single-threaded implementation, the Twisted framework makes it extremely easy to write the code for this server. If you are not familiar with Twisted or XML RPC, don’t worry; just imagine we’re writing a controller with only one method exposed: xmlrpc_search().

The search server constructor opens a database and initializes a query parser. We tell the query parser that the keyword ‘author’ refers to terms that are prefixed with the letter ‘A’ and that the keyword ‘subject’ refers to terms that are prefixed with the letter ‘S.’ This specification makes it possible to search for “author:john” or “subject:xapian.”

We then instruct Twisted to call reopen_db() every ten minutes. This reopening renders the latest changes in the database available to the search server. Each time Xapian’s library opens the database, it works against a fixed revision of it.

The xmlrpc_search() is the only method that is exposed (since its name is prefixed by xmlrpc_). The offset (zero-based) and limit arguments allow for an efficient way to split the search results into several pages. The call returns a dictionary with the total number of available results and a list with the selected subset of results. Each item in the list contains a unique article_id and a percent, which indicates each document’s relevancy score.

The program receives the port number to listen on and the Xapian database path. Unless you’d like to expose your search functionality to the world, it is suggested that you block outside access to this port.

By now, you’re probably eager to try searching your own database. Here’s a quick way to do so. First, start the search server:

From another terminal, start a Python shell:

In the same manner, you can use the search server from your application.

Working with Smaller Databases

Search engines are optimized to return results that are sorted according to their relevancy. If you need your results sorted by another criterion, such as date or diggs, it might be useful to run the query over a smaller database. For example, you might try running it over a database that contains only articles from the previous month. This strategy can significantly increase your overall performance.

Alternatives to Xapian

While I haven’t tried working with search engines libraries other than Xapian, you can try the Java-based Lucene which can be accessed from Python using PyLucene. The TurboLucene library eases using PyLucene from TurboGears.

The idea is to ask the spam bot a question which it does not expect, but it will be no problem for the users to answer. I’ve added to the registration form the question “How much is 5+2 ?”. Most of the new forum members were able to answer it on the first attempt. But spam bots had no clue.

So until someone bothers to write a spam bot specifically for my forums – I am okay. When it happens, I’ll just change the question. It can be many things: “What was the color of the white horse of Hammurabi?” or “How long did the six-day war lasted?” and so on. You got the point.

Here is how to do it.

In the template directory, edit profile_add_body.tpl, and add a new row the the form:

Browse to the registration page on your forum to see that it looks right.

In includes/usercp_register.php, look around line 260, and add the condition that checks if the question was answered properly:

Our application will associate sites with tags (many to many relationship), like delicious does, but in a much simplified manner. For instance, delicious keeps tracks of which user gave which tag to which URL. We will only associate sites with tags. But it will be very easy to add this functionality later.

We’ll quickstart a new project (the -s argument tells tg-admin that the project will use SQLAlchemy and not SQLObject)

Defining The Model

We are going to have a table for the sites, a table for the tags and a table that associates sites with tags (many-to-many). Here’s the code which defines the tables (which goes in model.py):

We will now create the Python classes that correspond to these tables:

Note the link() method in the Tag class. You might wonder what it does there. It just a little habbit that I wanted to share with you. I’ve found myself many times hard-coding URLs inside my templates. Then, if you want to make a tag linkable in many different places in your app, you have to hard-code the link every time. In this way, you can just pass the tag object to your template and do something like:

Ok, now we continue with mapping the classes to the tables:

Great. Now we can construct the database and start populating it:

Handling tags

So we got the model right. The next step is to allow the users to provide tags for the site. The easiest way (for you and your users) is to ask them to enter the tags in a space-separated list. Suppose you are given this kind of space-seperated string of tags from a user, then you have to:

• convert all tags to lower case, in order to avoid case senstivity issues
• check if the string contains the same tag twice
• find which tags are already in the database and which are new
• recover from some nonsense that users might throw at you

and then get a list of Tag objects that you can assign to a site. So here’s a function that does just that:

So you can now easily do something like:

Tag Search

It is straightforward to just list a site together with its tags:

Search is a bit more tricky. It took me few attempts until I got the search queries right. Here’s how to fetch all sites that are tagged by ‘google’:

the magic is mostly inside the join_to method – it stands for the SQL statements that makes sure that the Tag clause is associated to the sites. Without it, the query runs over the entire cartesian product of Sites x Tags.

You can make the query simpler (for MySQL; not you), if you fetch the tag_id of ‘google’ first. Then, the query uses only 2 of the 3 tables:

To search for google|photo:

To search for sharing+photos:

The idea is that sites that are tagged both with ‘sharing’ and ‘photos’ will appear twice in the select, then after grouping by site_id and getting all which appear twice, we get the desired result.

There are many more things that can be done from this point, like: associating with the tag-site relationship which user added the tag, rendering a tag cloud and so on. Feel free to leave comments!

Expires: Sun, 19 Nov 1978 05:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache

Here’s how I do it in TurboGears. To each method that I don’t want IE to cache, I add the strongly_expire decorator, like this:

    @expose()
    @strongly_expire
    def cant_cache_me(self, position):
        ...

The code of the decorator, which is responsible to set the headers propertly:

def strongly_expire(func):
    """Decorator that sends headers that instruct browsers and proxies not to cache.
    """
    def newfunc(*args, **kwargs):
        cherrypy.response.headers['Expires'] = 'Sun, 19 Nov 1978 05:00:00 GMT'
        cherrypy.response.headers['Cache-Control'] = 'no-store, no-cache, must-revalidate, post-check=0, pre-check=0'
        cherrypy.response.headers['Pragma'] = 'no-cache'
        return func(*args, **kwargs)
    return newfunc

Before you start, it will be helpful (but absolutely not required) if you also read about the BrowserSession object that is used in this post.

Tip 1: Set up the test configuration right.

None of your configuration files (dev.cfg, prod.cfg or app.cfg) is not loaded when you run your tests. Therefore, at the beginning of test_controllers.py, set your configuration options:

turbogears.config.update({
    'visit.on': True,
    'identity.on': True,
    'identity.failure_url': '/login',
    })

Another possibility is to create a configuration environment for testing, test.cfg and manually load it, using something like:

turbogears.update_config(configfile='project_dir/test.cfg',
                         modulename='package_name.config')

Tip 2: Initialize your database right.

testutil will set the default database to in-memory sqlite database. This looks like a good choice to me (you want something that will not be affected by previous runs and is fast). If you want to change that, make sure you do it after you import testutil. I never tried this.
In any case, your tables may not exist, and it is a good time to create them once the test module is imported. You can also use this opportunity to fill the tables with some data you want to use in your tests later:

def init_database():
    # hack follows: to create a TG_User, a request
    # environment must be around...
    testutil.create_request('/')
    if TG_User.selectBy(user_name='thesamet').count() == 0:
        TG_User(user_name='thesamet', password='password',
                display_name='Nadav', email_address='spam@me.not')

    Table.create(ifNotExists=True)
    Table.create(table_number=3, table_location='far away')

Note the comment at the beginning of the function. Unless there is a cherrypy request around, you can’t create a TG_User record. This is due to some unfortunate coupling.

Tip 3: Identity likes your login buttons.

When you emulate a user filling the login form, don’t forget to pass also the login button value (which is ‘Login’). If you login through the /login url (as opposed to give a url of some identity-protected method), do not forget to pass the forward_url argument. Example:

user = BrowserSession()
user.goto('/login?user_name=thesamet&password=password' \\
          '&login=Login&forward_url=/')
assert user.status != 403     # not forbidden

Tip 4: Never forget to call stopTurboGears()

You must call turbogears.startup.stopTurboGears() at the end of each test. I won’t pretend that I understand why it is needed. It is also done in TurboGears’ own identity tests. If you don’t do that you’ll get nice exceptions from the VisitManager thread. I like to factor out the stopTurboGears() call to the tearDown part of my test case.

The following is an example of a test case objects that verifies that the protected resource /secret is well-protected:

class SecretPageTest(unittest.TestCase):
    def setUp(self):
        self.user = BrowserSession()

    def test_anonymous(self):
        self.user.goto('/')
        assert 'Login' in self.user.response
        self.user.goto('/secret')
        assert 'You must provide your credentials' \\
                  in self.user.response

    def test_bad_credentials(self):
        self.user.goto('/secret?user_name=thesamet&password=incorrect'\\
                       '&login=Login')
        assert 'The credentials you supplied were not correct' \\
                           in self.user.response

    def test_successful_login(self):
        self.user.goto('/secret?user_name=thesamet&password=password' \\
                       '&login=Login')
        assert 'This is a secret page' in self.user.response

    def test_logout(self):
        self.user.goto('/login?user_name=thesamet&password=password' \\
                       '&login=Login&forward_url=/')
        # not forbidden
        assert self.user.status != 403
        self.user.goto('/')
        # display name should appear and no suggestion to login
        assert 'Nadav' in self.user.response
        assert '<A HREF="/login">Login</A>' not in self.user.response
        self.user.goto('/logout')
        self.user.goto('/')
        # no display name should appear and a suggestion to login
        assert 'Nadav' not in self.user.response
        assert '<A HREF="/login">Login</A>' in self.user.response

    def tearDown(self):
        turbogears.startup.stopTurboGears()

You can download the full source code of a minimal identity testable project.

I have been a customer of BlueHost (a very friendly web hosting provider) since I’ve created the Python Challenge, about a year ago. A short time later, I’ve added to my account the domain thesamet.com which I use as my personal homepage and blog. Recently, I’ve started working with TurboGears and wondered whether BlueHost could host my application. Fortunately, it was quite easy to do so.

UPDATE: BlueHost does not officially support TurboGears. Furthermore, FastCGI is much slower and difficult to work with than mod_rewrite. Therefore, I prefer to spend a few more extra bucks a month and get an hosting plan in WebFaction, where deployment of a TurboGears application is a matter of point and click.

and there is a performance penalty for using FastCGI

For the benefit of newcoming users, I’ve written a small shell script that does most of the boring installation chores. By following the instructions on this post, you’ll have your TurboGears applications installed behind Apache using FastCGI. I personally tested these instructions on BlueHost servers, but it is very likely that they will work for other hosting providers. Please let me know if it worked for you. Most of this work is based on the TurboGearsOnDreamHost page in TurboGears trac.

Installing TurboGears in your account

Connect to your account via ssh. If you are a Windows user, you can use PuTTY for that. Once you are logged into your account, type:

wget "http://www.thesamet.com/tg_scripts/tg_inst.sh"
sh ./tg_inst.sh

The scripts execution takes a while. It downloads and compiles Python in your home directory and then installs latest TurboGears preview, docutils and MySQL-python. If installation succeeds, you can skip the following section.

Installation problems?

Failures of tg_inst.sh can be due to many reasons. As the script tries to download files from remote servers, it is very likely to fail because of networking issues. Maybe trying it again can help. You can also try to delete the directory named build by typing

rm -rf ~/build

before you retry. tg_inst.sh logs all output to ~/build/stdout and ~/build/stderr. These files will contain more information about any failures.

Installing your TurboGears application

In this section we will configure your application to work behind Apache/FastCGI. The code examples below assume that your application name is wiki20 and you want it to be accessible through http://mydomain.com/wiki/
Upload your application to any directory which is not accessible from the web, for instance to /home/username/wiki20. The toplevel directory contains the file prod.cfg. Start editing it by typing:

nano ~/wiki20/prod.cfg

You have to set the line that starts with server.webpath to point to the url of your application relative to the site root. In our example, we will have:

server.webpath="/wiki/"

We also need to create a MySQL database for your application. Login to the cPanel and choose “MySQL Databases”. Create a new database and a user, and the user to the database. Go back to prod.cfg, and uncomment the MySQL dburi string. It should look like:

sqlobject.dburi="mysql://username:password@localhost/dbname"

Configuring Apache

Now we create the root directory for your application:

mkdir ~/www/wiki
cd ~/www/wiki
wget http://www.thesamet.com/tg_scripts/tg_fcgi.tar.gz
tar xzvf tg_fcgi.tar.gz

If you want the application to be on the root of the site, just omit the wiki part. The last command will create two files in this directory. You have to edit the file tg_fastcgi.fcgi (you can use nano as before) to set few pieces of information. On the first line, change the username to your real user name. Look inside for “START USER EDIT SECTION”. You probably will only have to change wiki to your real project name. To find out your code_dir, you can cd to the folder you uploaded your application and type pwd.You can now try to start your application from the web. Just navigate to its root directory: http://mysite.com/wiki/tg_fastcgi.fcgi. It may take it up to 30 seconds to load. Even if it gives you an error message, try to reload the page, maybe it will work. See also the last section.
The next thing we are going to do is to setup rewrite rules, so your visitors will see only clean urls. Create (or edit) the file .htaccess in the application site root directory by typing

nano .htaccess

Make sure it contains the following:

RewriteEngine On
RewriteRule ^(tg_fastcgi.fcgi/.*)$ - [L]
RewriteRule ^(.*)$ /wiki/tg_fastcgi.fcgi/wiki/$1 [L]

Note that the application webpath appears twice in the last line. Then change the permission of this file:

chmod 755 .htaccess

Try now to open a web browser to http://mysite.com/wiki/.

Congratulations! Your web application is now deployed.

If anything goes wrong

If you can’t access your application, check the log files created in its code directory. They may contain some hints.

If they are not even created, try to start tg_fastcgi.fcgi manually and check the logs. After you change any configuration file, it is advisable to kill all fastcgi processes by typing pkill fastcgi
If you want to check if fastcgi is running, type

ps uax | grep fastcgi

To see the list of fastcgi processes.

Conclusion

You can clean up and some some diskspace by deleting the build directory and the installation script by typing rm -rf ~/build ~/tg_inst.sh. I hope this information was helpful to you and you’ll have joy and success with your TurboGears application.