plusplusbot: Karma for anything and anyone #

The recent release of Foamee reminded me that I had never blogged here about plusplusbot, a little toy of a site that I started to work on this past summer. The idea behind plusplusbot is that you can express your (dis)pleasure with something or someone over Twitter, and the site will keep track of the target's score over time. All of this is done over Twitter, in a barnacle-like fashion. The thinking being that Twitter handles message delivery over a variety of mediums (Jabber, SMS, Web) and provides a ready-made social network to piggyback on (you can view the activity of you and those you follow, for example, here's mine). This concept should be familiar to those that have enountered "karmabots" in chatrooms.

The targets can be anything. Companies, food, people, even Twitter itself. For whatever reason, the site has caught on most with Japanese Twitter users, so the homepage is often incomprehensible.

Technically, the site is not too exciting. Like Twitter Digest, it also uses python-twitter to talk to Twitter and templet for simple templates. The site itself is not dynamic at all, instead a script running on one of my machines polls Twitter every two minutes (since it has to fetch both friend updates and direct messages, fetching both every minute would go over Twitter's 70 requests/hour API rate limit). If it determines that a new plusplus or minusminus has been sent, it re-generates those pages (the user's and the target's) and uploads them. The simple design means that the site can be hosted nearly anywhere.

Macros for the new version of Gmail #

As those of you on the new version of Gmail might have noticed, nearly all Greasemonkey scripts that used to work on the old version no longer do. Even though it looks pretty similar, the new Gmail is entirely different from a JavaScript, HTML and CSS perspective, so this isn't surprising. Some of the scripts are no longer necessary. For example, saved searches aren't really needed, since searches now get their own URLs and can be bookmarked.

However, my Macros script is still needed; while the new version of Gmail does have additional keyboard shortcuts, it's still not possible to do everything from the keyboard. I've therefore ported it for the new version of Gmail, to install it, click below.

Install Gmail Macros

More specifically, the following keyboard shorcuts have been ported over:

  • g: Go to label
  • l: Apply label
  • b: Remove label
  • e: Archive (regardless of view, unlike "y")
  • d: Discard (mark as read and archive)

There is also a new keyboard shortcut, which the old script didn't have:

  • f: "Focus" the current view (only show unread, starred or inbox messages)

You may remember the script supporting other keyboard shorcuts. Since the new version of Gmail supports additional shortcuts, those have obsoleted. The new ones are:

  • shift + i: Mark as read
  • shift + u: Mark as unread
  • shift + 3: Move to trash (not actually new, but not many people seem to know this one)
  • shift + 8 followed by a, n, r, u, s, or t: Select all, none, read, unread, starred, or unstarred

For those of you adventurous enough to look at the script source, you'll notice that it uses a gmonkey object that is present on the window, which in turn gets you a gmail object with methods like getNavPaneElement() and getActiveViewType(). What this means is that the version of Gmail, in addition to being faster, also has semi-official support for Greasemonkey scripts. I'm pretty sure docs for this API will be out soon, but in the meantime, feel free to look at the script and use a tool like Firebug to investigate the properties of the gmonkey and gmail objects and play around.

Update on 11/06/2007: And here are the semi-official docs.

Feed Proxy: View Raw RSS/Atom Feed Data in Firefox 2.0 #

Firefox 2.0 has a nice RSS/Atom feed preview feature, which pretty-prints the feed and allows you easily subscribe to it in a feed reader. While this is nice from a user's standpoint, it can get in the way when trying to debug or otherwise see a raw feed as a developer. There is no option to turn this mode off, which seems to be making some people unhappy.

To help with this use-case, I made Feed Proxy, a simple service that gives you back the raw XML output (by adding a 512 byte dummy header that makes Firefox not invoke its feed preview code). Its page has more details on how to invoke it and what its caveats are.

Valleywag, brought to you by....Google? #

Gawker Media's use of virtual hosting for its sites was being discussed in PartyChat, so to verify this, I ran host valleywag.com to compare it with host fleshbot.com. Imagine my surprise when I saw this output:

$ host valleywag.com
valleywag.com has address 69.60.7.199
valleywag.com mail is handled by 1 aspmx.l.google.com.
valleywag.com mail is handled by 5 alt1.aspmx.l.google.com.
valleywag.com mail is handled by 5 alt2.aspmx.l.google.com.
valleywag.com mail is handled by 10 aspmx2.googlemail.com.

If I am interpreting this correctly, Valleywag's email is being handled by Gmail/Google Apps, which is slightly ironic.

Twitter Digest #

There are quite a few people on Twitter that I'd like to follow but can't, since they update way too often. For example, I'd like to know how Craig Hockenberry's MobileTwitterrific work is going. John Gruber often posts advance tidbits from Daring Fireball. However, since Twitter doesn't seem to allow different notification levels for contacts, I'd rather not get overwhelmed with their updates on a regular basis.

Twitter does have an API though, and a pretty complete one at that. Additionally, python-twitter makes it pretty easy to code against it. The result is that it was pretty easy to make Twitter Digest. You can give it a bunch of usernames and it will update a page once a day with all of those people's tweets. Feeds can also be generated, so that you can subscribe to your digest in your favorite feed reader. To give an example, here's my digest (and its feed.)

Since python-twitter already has the the concept of caching API replies, this was especially easy to develop. Simply by juggling with cache timeout values, API replies are kept around for up to a day, thus I don't have to persist anything in my code. All digests are refreshed at GMT day boundaries because supporting timezones seemed like too much work.

Digests can be thematic too. Here's one for the Twitter team. I'm sure there are other uses too. I'm pretty happy with how this turned out, and the only thing missing is better reply support (if a message is in reply to another, then (optionally) show that message inline too). Right now the API is missing support for this (even though the regular UI surfaces this information), but I'm sure something can be hacked together.

Update on 9/3/2007: Adding support for "in reply to" messages turned out to be easy enough (look for the first message from the other username that's before the one that's replying, but only up to an hour earlier, in case it's not actually connected).

iPhone Game Development: A Tale of Failure #

Having acquired an iPhone, I've been playing around with it to see just how much can be done with the "SDK" that is the MobileSafari browser. I thought it would be fun to see if I could make a simple game for it, since that's different enough from the kind of UI coding I do at work.

Most of the games for the iPhone that I've encountered thus far were of the puzzle/card variety, and there wasn't anything too exciting/button-mashing/mindless. It seemed like this was mainly caused by the limitations of the events that the browser exposes. With no real mousemove events, and with mousedown events actually firing when the cursor was lifted (i.e. at the same time as mouseup), it was hard to get anything to happen on the screen quickly in response to a user action. One avenue that I tried was to have the screen covered in narrow vertical <div>'s, and to listen for click events on each of them. Unfortunately, while this worked when each was clicked separately (piano key-style), it didn't work when the user dragged his finger from one to another, since this would switch the browser into panning mode, even if the page/viewport was set to be only 320 pixels wide (i.e. no scrolling could actually occur).

Since the viewport panning does not seem to expose any events, I thought that maybe there could be other ways of catching this user action. When scrolling quickly, the iPhone renders newly-visible areas with a checkerboard pattern before actually displaying their contents. I wondered if this meant that images contained in these areas would not be loaded until they were scrolled into view and rendered. If so, it would be possible to use a load event listener as a notification that the user had scrolled to a particular part of the screen. Unfortunately this didn't seem to be the case, images are loaded as soon as the document is displayed, even if they are off the screen.

I then remembered that the iPhone supports an additional gesture, a two-fingered scroll that can be used to scroll text-areas, elements with overflow: auto and possibly other nodes. It would appear that scroll events trigger for this scrolling motion, meaning that it's possible to detect and get continuous updates for this. Though having to use two fingers seemed a bit clumsy, this seemed pretty promising insofar as it allowed for quick responses to user input.

I had lofty goals for my iPhone game (I thought a vertical scrolling shooter was a good fit for the dragging input method and the vertical screen layout), but I decided that I should do something simple for my first attempt. I thought I would recreate aa racing game that I hard first played on my TI-82 calculator, where a car that can only be moved left or right moves at a constant vertical speed through a winding, narrowing road, with the objecting being to avoid hitting the sides. I thought this would map well to a horizontal scrolling motion that I could get with a scroll event handler as mentioned above.

After writing a simple random road generator and the animation code, I was about to add collision detection. Out of curiosity, I tried to move the "car" left/or right as road moved by, even without collision detection, to see if the speed was too high. To my surprise, the road stopped moving as soon as I put my two fingers on the screen. It appeared that intervals (set up with window.setInterval) do not fire when two-finger scrolling mode is engaged. I thought that maybe I could refresh the animation from the scroll event handler, but that only seems to fire when there is actual motion, thus smooth animation can't be ensured.

My failed attempt is visible here. Before pressing the "Start" button, use the two-fingered scrolling motion left and right to confirm the the "car" (the blue square) moves as expected. Then press "Start" and obverse the reasonably smooth animation. Unfortunately it all stops when placing two fingers on the screen again.

This is rather disappointing, since there doesn't seem to be any fundamental technical reason why this should be the case. We're not talking about Mac OS 9 here (there, due to the way UI events were processed, normal event loop execution would be suspended while a menu was held open or a scrollbar thumb was dragged). It's quite clear, given what native iPhone apps can do, that the platform has quite a lot of CPU and graphics leeway. It's therefore unfortunate just how constrained the MobileSafari browser is, since just a few additional hooks would allow much richer interactions with the user.

Bad Headlines #

Headlines seen over the past few weeks on the start page that Verizon forces on its users, and what they refer to in parentheses:

  • "Daily Life is a Struggle" (life in Baghdad)
  • "All Sides Deny Deals" (Iran/UK sailors hostage situation)
  • "Panel Seeks Documents" (Alberto Gonzales' federal prosecutors firings)
  • "Cheruiyot Wins Marathon" (Robert Kipkoech Cheruiyot won the Boston Marathon)
  • "Gates: Clock is ticking" (Defense Secretary Robert Gates on the situation in Iraq)
  • "Donkey Becomes Witness" (Dallas man is sued for noise disturbance caused by a donkey)
  • "Dems Predict a Win" (House Democratic leaders predict that they can win a vote on a bill calling for a withdrawal from Iraq)
  • "Candidates to Debate" (The Republican presidential candidate debate)

Someone forgot to read up on microcontent headlines.

Tiled Windows Are the Way of the Future #

Sorry Bill Atkinson. I know you worked really hard to make regions and thus overlapping windows work. But since I now have 30 inch displays at both home and work, I've been tiling my windows and using Desktop Manager (and soon Spaces?) to switch between workspaces. Not having to deal with z-ordering makes mental context switches faster.

Communicating through screenshots #

Two Web Development Tips #

Memory Leaks

Fighting memory leaks in fancy AJAXy apps is a way of life for most UI developers. Besides Internet Explorer-specific issues, leaks can also occur due to inadvertent remaining references to objects you don't actually need (frequent culprits are global event registries and history/state management stacks). Safari/WebKit turn out to have some useful built-in hooks for tracking these down. By enabling Safari's debug menu and choosing the "Show Caches Window" command from there, you can see JavaScript object counts, force a garbage collection and see the types of objects that still have references pointing to them (note that garbage collection has been improved in WebKit nightly builds so you'll probably want to use that).

Drip can provide some similar information for IE, and Leak Monitor finds some Firefox-specific leak types, but it would be good for browser/add-on creators to provide even more debugging information to authors.

Browser Bugs

WebKit and Firefox/Mozilla both have publicly visible bug tracking systems. As they work on their respective new versions, regressions will occur that will break web sites, possibly including yours. Often, this is a genuine bug in the browser code, and hopefully it'll be fixed before the final release. However, it can sometimes be due to incorrect assumptions made by your code which no longer hold true when browser code is tightened up. Both WebKit's and Mozilla's Bugzilla instances support generating RSS feeds from search results. By subscribing to feeds for bugs that mention your product name, you can stay on top of such bug reports. For example, here's a shared page (with feed) for the bugs that mention Reader on either site.

Determining Twitter's Growth #

Opinion on Twitter is divided. What seems to be undisputed is that right now it's growing very quickly. I was curious just how "quickly" quickly was, preferably going beyond just anecdotal "my network doubled in size in the last 5 seconds" kind of observations. It seems like Twitter assigns globally unique, incrementing IDs to all messages it receives. By looking at the values of these IDs over time, it's possible to see how many status messages Twitter is keeping track of. I've generated a logarithmic graph of this.

I'm not sure why there was an inflection point in early November. It's also possible that this is affected by technical changes on Twitter's side. Still possibly interesting. Also, Joshua's post on autoincrement considered harmful is related and an interesting read.

Update: As it was pointed out to me in the comments, Andy Baio had the same idea except he executed it more throughly.

Twitter Message IDs

Intern on the Google Reader team #

The Reader team is hoping to have a student intern or two this coming summer. We're fast moving and always have more ideas than manpower, so an internship can be quite rewarding as far as the "working on real stuff" factor goes. For example, our intern last year, Brad, worked on the subscribe and feed search functionalities of the new Reader that launched last September. You can intern in Google's New York or Mountain View offices, working on either Reader's frontend/UI or backend.

If you or anyone you know is interested in this internship, contact me at mihai at persistent dot info. This page also has more general information about interning at Google.

Blogger Migration Part II: Getting Data Into Blogger #

Blogger doesn't have any built-in entry importing facilities. My plan for dealing with this was to use their API to re-post all of the entries I had exported from Movable Type. A quick test showed that such entries could be back-dated, which was my main concern.

Using some sample code that I found as a starting point, it was pretty easy to write a simple importing script for entries. Since I needed to parse the Atom responses from the API (e.g. to get at entry IDs), the Universal Feed Parser came in handy. Since I had used the Convert Line Breaks option on quite a few posts, I had to HTML-ify some post bodies before sending them to Blogger (I've turned off Blogger's similar setting. In addition to being only at the blog level, instead of a per-post setting, I've decided that the closer to regular HTML my posts are, the easier it will be to migrate again in the future).

The Blogger API allows for posting of comments too, but unfortunately without the ability to impersonate others (even when anonymous comments are enabled). The solution was to impersonate the regular (non-API) comment posting form, which does allow for authors to be specified (but no backdating, which is why all imported comments are dated February 10th). This was made slightly trickier since a security token is required (to avoid XSRF attacks), so the posting page had to be scraped first to extract it.

Finally, since Blogger proved to be occasionally flaky when doing many API operations in a row, I had to add some simple checkpointing so that if the process failed I could restart it and it would continue from where it left off. Once I did all that, importing 350+ entries and 600+ comments took around an hour, but worked flawlessly.

I've uploaded an archive of my code that I used for the importing. It's not the cleanest it could be, but others may find it useful too. Additionally, in the time that has passed since I began my importing, it looks like a similar script has appeared that does a similar import operation from WordPress, which may be worth a look too.

Understanding feed reader marketshare numbers #

Update on 2/25: FeedBurner has published a post discussing this same issue but providing numbers for their whole userbase, which makes it even more interesting.

Ever since the Reader team announced that we were making public subscriber counts (thanks Justin), bloggers have been excitedly posting about the bumps they're seeing in their subscriber stats. I'm obviously very happy that Reader is getting all this attention, and that we turn out to be quite popular when compared to other feed readers. However, these statistics need a bit of interpretation. Most people post charts of their subscriber counts, like this one for this blog:

FeedBurner subscribers

For web-based readers where feeds are fetched on behalf of multiple users, the subscriber number is based on what the site reports. To the best of my knowledge, with the exception of My Yahoo!, these number are total subscribers, even if an account is inactive. Unless the site is aggressive about cleaning up inactive accounts, these numbers are only upper bounds on the number of actual readers that you have.

A more interesting number to look at is how many viewers each item gets from each feed reader. FeedBurner provides this as part of their TotalStats package. By embedding a small tracking image in your burned posts and looking at referrers, it's possible to see these item-specific views. Here are how many views and clicks my post from yesterday got in various feed readers:

FeedBurner item use

From this it would appear that Reader has an even bigger lead over Bloglines (though given the biases in this blog's readership, I'm not reading too much into this). There are other factors involved here too. The user bases for feed readers are not identical, if an item appeals more to one population than another, that may skew things. Additionally, some readers (especially homepage-style ones like My Yahoo!, Google Personalized Homepage and Netvibes) don't have to display the item body and allow users to jump straight to the post page. These would show up in the "Clicks" column but not in the "Views" one.

What becomes apparent is that none of these statistics provide a complete picture of your readership, but that when used together they can still give you broad trends and help you tailor your content to your audience.

Blogger Migration Part I: Getting Data Out of Movable Type #

The first step in migrating away from Movable Type was to get all of my entries and comments in a structured format that could be parsed and uploaded to Blogger*. MT doesn't hold data hostage, there is a documented import/export format. Six Apart considers the format "lossy", in that it doesn't save a complete snapshot of your blog. I decided that what it did contain was good enough, though it turn out that what it lacked (entry IDs and permalinks) did make things slightly more difficult. A search on code.google.com for Python code to parse the format turned up Transfusion which does just that (searching for one of the magic strings in the format, CONVERT BREAKS specifically, was the easiest way to track this down).

As I was skimming through the exported entries, I saw that they weren't quite HTML. I had used MTMacro to create various shorthand tags for linking to entries, reference to images, etc. Similarly, I used MTCodeBeautifier to pretty-print code samples. None of these were getting evaluated when exporting, and even if they had been, I probably would have wanted to tweak their output anyway (e.g. to change URLs). Generally, it seemed like the time I had spent customizing my Movable Type installation with cruft-free URLs, plug-ins, etc. would be directly proportional to the time I would have to spend migrating away from it.

One of the more prevalent macros I had used was one of the form <entryLink id="NNN">foo</entryLink> so that I could link to my past entries. Unfortunately, since entry IDs were not included entry IDs issue, there was no easy way to turn these into actual links, since the exported information did not contain entry IDs or URLs. In the end, I ended up converting these by hand.

That's it for the exporting part, part II will contain the Blogger import process and part III the template/design reasoning.

* the other migration option that I was considering was WordPress. However, the idea of having to do SQL queries to serve traffic didn't seem that appealing given my current provider's slow SQL performance. WordPress.com would have been a hosted option, but if I was going to relinquish control of the installation, it might as well be to a Google product.

Switched to Blogger #

Partly because I was fed up with Movable Type's rebuild times, but also for dog food reasons, I've moved my blog over to Blogger (custom domains was the final missing piece). Redirects should be in place and no links should break, but feed readers will most likely see a bunch of new entries (I didn't see an option in FeedBurner to suppress duplicates). Please leave a comment if you see anything amiss.

I'll be writing up more details this coming week about the work that was necessary to migrate.

More Efficient FeedBurner Visitor Tracking #

I was quite happy to see that FeedBurner had integrated site visitor tracking in addition to their already great feed usage statistics. Actually using it with my blog was just a matter of cutting and pasting some code into my template. Here's the snippet that I was offered for Movable Type:

<script src="http://feeds.feedburner.com/~s/PersistentInfo?i=<$MTEntryPermalink$>" type="text/javascript" charset="utf-8"></script>

This seems reasonable, but it bothered me that the tracking URL varied for each entry. This meant that for my front page, 7 different (but nearly identical) pieces of JavaScript would have to be fetched, slowing down the rendering of my site. Others have complained too. I think a better way of implementing this would have been to have something like:

<script src="http://tracking.feedburner.com/tracking.js" type=text/javascript" charset="utf-8"></script>
<script type="text/javascript">FBTrack("PersistentInfo", "<$MTEntryPermalink$>");</script>

That would have meant that the same tracking code can be used across all sites thus most users would have it cached (I think that's what Google Analytics relies on). Unfortunately, that's not something that I can change on my own.

However, when running into a related problem (a FeedFlare was stuck on) it was pointed out to me that the i=... part of the script URL can be removed if all that's desired is the visitor tracking and not per-entry features like FeedFlares. This means that at least across my entire site there is only one tracking script URL that visitors have to load. It turns out FeedBurner documents this, but only in the "Others..." category (the bottom "If you want to use StandardStats only" section).