Comments on: Pumping Up Your Applications with Xapian Full-Text Search http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/ The Ramblings of a Freelance Software Developer Sun, 12 Oct 2008 13:41:11 +0000 http://wordpress.org/?v=2.5.1 By: dperales http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-6804 dperales Sat, 13 Sep 2008 00:25:53 +0000 http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-6804 hey, i got a problem, it seems that when i do "author:john wayne", "john wayne" is not treated as a whole term, i tried " author:'john wayne' ", but nothing. what could be the possible problems? . i have to add that when i retrieve all terms, "john wayne" is showed as a whole,and "author:john" is not returning anything. hey, i got a problem, it seems that when i do “author:john wayne”, “john wayne” is not treated as a whole term,
i tried ” author:’john wayne’ “, but nothing. what could be the possible problems? .

i have to add that when i retrieve all terms, “john wayne” is showed as a whole,and “author:john” is not returning anything.

]]> By: david petar novakovic: attempted axiomatisation http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-6713 david petar novakovic: attempted axiomatisation Fri, 04 Jan 2008 10:20:57 +0000 http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-6713 <strong>Installing xapian and its bindings on OS X 10.5 Leopard...</strong> Well I’ve had an interesting few days where I got so frustrated with Leopard that I switched back to Tiger. Upon trying to install many libs that I need in Tiger I realised that Leopard is actually great for developers and had to switch back to i... Installing xapian and its bindings on OS X 10.5 Leopard…

Well I’ve had an interesting few days where I got so frustrated with Leopard that I switched back to Tiger. Upon trying to install many libs that I need in Tiger I realised that Leopard is actually great for developers and had to switch back to i…

]]> By: Sridhar Ratnakumar http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-6695 Sridhar Ratnakumar Fri, 26 Oct 2007 08:02:02 +0000 http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-6695 <code>stem_word</code> is <a href="http://www.xapian.org/docs/deprecation.html" rel="nofollow">deprecated</a>. Replace that line with <code>stemmer(term.group()),</code>. stem_word is deprecated. Replace that line with stemmer(term.group()),.

]]> By: Richard Boulton http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-5675 Richard Boulton Tue, 22 May 2007 06:28:53 +0000 http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-5675 Just to add to the comments about thread safety - while a Xapian database can indeed only safely be accessed from one thread at a time, we (the Xapian developers) have ensured that the cost of creating a Xapian database object is very low (it basically involves opening a few files, and no complex processing), and there are no global variables used or anything like that, so it is perfectly reasonable to open the database for each search request and handle each request in a separate thread. I've built many multi-threaded applications in Python which search Xapian databases. You are restricted to having only one instance of a database open for writing at a time, but as many instances of a database for reading as you like can be open (from a single process, or from many). We intentionally removed thread handling from the innards of Xapian because it was error-prone and imposed an overhead on all searches, whether single or multi threaded. Just to add to the comments about thread safety - while a Xapian database can indeed only safely be accessed from one thread at a time, we (the Xapian developers) have ensured that the cost of creating a Xapian database object is very low (it basically involves opening a few files, and no complex processing), and there are no global variables used or anything like that, so it is perfectly reasonable to open the database for each search request and handle each request in a separate thread. I’ve built many multi-threaded applications in Python which search Xapian databases.

You are restricted to having only one instance of a database open for writing at a time, but as many instances of a database for reading as you like can be open (from a single process, or from many).

We intentionally removed thread handling from the innards of Xapian because it was error-prone and imposed an overhead on all searches, whether single or multi threaded.

]]> By: Chris Lu http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-4210 Chris Lu Sun, 22 Apr 2007 05:17:45 +0000 http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-4210 I would argue Lucene is faster, because it is thread-safe. The concurrent searching threads Xapian can handle is limited by the number of process you can create before-hand, while Lucene was not. And the mis-conception that Java == slow is so last century... Chris Lu http://www.dbsight.net I would argue Lucene is faster, because it is thread-safe. The concurrent searching threads Xapian can handle is limited by the number of process you can create before-hand, while Lucene was not.

And the mis-conception that Java == slow is so last century…

Chris Lu
http://www.dbsight.net

]]> By: sally http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-4206 sally Sun, 22 Apr 2007 02:12:35 +0000 http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-4206 Hello Nadav, Thanks for doing a great job of Xapian explanation. Can you please tell me what is the merit of using Xapian as against say lucene or some other full text search engine. I'm in the process of looking for a robust full text serach engine, that is very scalable and my product will be using the engine using apis. Do you have any opinions? I was leaning towards lucene, but may be I should look further (My concern for Lucene is that since it is Java based it may not be fast enought?) Thanks Sally Hello Nadav,

Thanks for doing a great job of Xapian explanation.

Can you please tell me what is the merit of using Xapian as against say lucene or some other full text search engine.

I’m in the process of looking for a robust full text serach engine, that is very scalable and my product will be using the engine using apis.

Do you have any opinions? I was leaning towards lucene, but may be I should look further (My concern for Lucene is that since it is Java based it may not be fast enought?)

Thanks

Sally

]]> By: Nadav Samet http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-4114 Nadav Samet Thu, 19 Apr 2007 05:58:07 +0000 http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-4114 Hi Mark, Great question. The index() gets a period (say 1 hour) argument and then operates on all documents changed in that period. So it is meant to run as a cron job every hour. I believe that on a high load system, indexing documents in batches is faster than indexing on every change. If we would run the indexer in the same process of the search server, it will block searches while it indexes (and vice versa). But setting the indexer on its own process, can be an alternative to cron, but I'm not sure what the benefits are. Hi Mark,

Great question. The index() gets a period (say 1 hour) argument and then operates on all documents changed in that period. So it is meant to run as a cron job every hour.

I believe that on a high load system, indexing documents in batches is faster than indexing on every change.

If we would run the indexer in the same process of the search server, it will block searches while it indexes (and vice versa). But setting the indexer on its own process, can be an alternative to cron, but I’m not sure what the benefits are.

]]> By: Mark http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-4088 Mark Wed, 18 Apr 2007 18:04:59 +0000 http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-4088 Is there any reason why you didn't implement the indexer on Twisted? The way you implemented it, there is no incremental update to the index. I am trying to index as objects are created/updated, thinking of using Twisted. Any thoughts on your part about how I may approach this. Thank you. Is there any reason why you didn’t implement the indexer on Twisted?
The way you implemented it, there is no incremental update to the index. I am trying to index as objects are created/updated, thinking of using Twisted. Any thoughts on your part about how I may approach this.

Thank you.

]]> By: thesamet http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-3602 thesamet Sun, 01 Apr 2007 22:18:19 +0000 http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-3602 Hi Peter, Great question. The short answer is: Yes, only one search query is handled in one time, but <a href="http://www.thesamet.com/multitasking.png" rel="nofollow">sometimes multiple threads don't help too much</a>. Here is the long answer. Let's start with a word about Twisted servers. In general, single-threaded servers can handle many requests at the same time. The basic idea is that their execution flow is event-driven, instead of thread-driven (or system scheduler driven). All events are handled on the same thread, and the assumption is that handling an event is a short operation. As Xapian is not thread-safe by design, there is no way to interleave accesses to the database, and each query is handled from start to finish, regardless of the server's implementation. So if needed, this solution can be scaled by starting several instances of the search server (possibly even on several machines) and load-balancing. As experience had taught me, you never become Google overnight. It's best to stick with the simplest thing as long as it works. I use <a href="http://txfx.net/code/wordpress/subscribe-to-comments/" rel="nofollow">Subscribe to Comments plug-in</a>. Nadav Hi Peter,

Great question. The short answer is: Yes, only one search query is handled in one time, but sometimes multiple threads don’t help too much. Here is the long answer.

Let’s start with a word about Twisted servers. In general, single-threaded servers can handle many requests at the same time. The basic idea is that their execution flow is event-driven, instead of thread-driven (or system scheduler driven). All events are handled on the same thread, and the assumption is that handling an event is a short operation.

As Xapian is not thread-safe by design, there is no way to interleave accesses to the database, and each query is handled from start to finish, regardless of the server’s implementation.

So if needed, this solution can be scaled by starting several instances of the search server (possibly even on several machines) and load-balancing.

As experience had taught me, you never become Google overnight. It’s best to stick with the simplest thing as long as it works.

I use Subscribe to Comments plug-in.

Nadav

]]> By: Peter http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-3600 Peter Sun, 01 Apr 2007 21:19:40 +0000 http://www.thesamet.com/blog/2007/02/04/pumping-up-your-applications-with-xapian-full-text-search/#comment-3600 When you say: "single-threaded implementation, the Twisted framework makes it extremely", do you mean only one search query can be handle at any one time? So there is no concurrency, how does this scalable in internet world? Also, what plugin are you using for this blog, that allows "subscribe to comments via email"? Thank you When you say: “single-threaded implementation, the Twisted framework makes it extremely”, do you mean only one search query can be handle at any one time? So there is no concurrency, how does this scalable in internet world? Also, what plugin are you using for this blog, that allows “subscribe to comments via email”?

Thank you

]]>