Porting Twitter Digest to Google App Engine #

I've been meaning to play around with Google App Engine for a while, and as a quick project, I decided to port Twitter Digest to it (not as exciting as Kushal's Millidunst Calculator). This looked to be pretty straightforward: the original version was already in Python, and wasn't very complicated (just a single CGI script). It did indeed end up pretty easy; the whole process took a couple of hours.

The first step was to port the script from CGI-style invocation to the App Engine webapp framework. Then I looked into what it would take to get Python Twitter (the library I used for fetching data from Twitter) running. Switching it from urllib2 to urlfetch was pretty painless (though I don't use the posting parts of the API, so I didn't check if those work too). The other part of the library that I was relying on was its caching mechanism (since the digests are daily, there's no point in querying Twitter more often). DeWitt (the library's author) had thoughtfully put the caching functionality into a separate class, so it was easy to replace it with another one that implemented the same interface but was backed by App Engine's datastore.

The result (complete with App Gallery entry) is not that exciting, in the sense that it functions identically to the original. The only issue that I've run into so far is that when there are several cache misses, the URL fetches can take long enough that the request hits App Engine's deadline. However, since the successful fetches are cached, repeating the request will eventually succeed (so if consuming the digest via a feed, this shouldn't be a big deal). Ideally the urlfetch functionality would also support asynchronous fetches, since it would be easy to adapt the code to fetch all user timelines in parallel.

Update on 11/23/2008: Since I've gotten some requests for the modifications to twitter-digest that I made to get it to run on App Engine, here's a patch.

4 Comments

This is a very nice tool for digesting Twitter properly :)

However, it would be even greater if the subscriber list was auto-generated based on my Twitter subscriptions (following). Instead of rewriting the feed everytime I want to add (or remove) someone, I would simply change the following list on Twitter and the TwitterDigest URL would be the same.

Something like:

http://twitter-digest.appspot.com/generate?following=mihai

Would generate a feed including all users in Mihai's following list.
Would you consider posting the modified python-twitter code somewhere so others could download it? That would be very helpful.
I wonder if this might be modified to create a weekly twitter digest? I don't really want to clutter my blog with daily twitter digests, and I don't post enough in a day to warrant it anyway...

I haven't been able to find anything that will do this for blogger--only for WordPress.
Twitter Digest seems to be broken--I'm not getting any results.

Post a Comment