Evan recently shared shared a link to Bret Victor's CUSEC presentation (which is worth watching, but not what this post is about). In the (internal) thread, Ami Fischman mentioned that, in addition to being more reliable, Vimeo's HTML5 player allows the use of the playbackRate attribute to speed up playback without affecting its pitch (at least with the codecs used by Chrome). I'm a big fan of listening to podcasts at higher speeds (especially via Downcast, which supports up to 3x, though I haven't made my way that high yet), so the idea of being able to do this for any video was very appealing.

YouTube's HTML 5 player already makes use of this, but for those sites that don't, I've made Chrome extension (source). It adds a context menu that allows you to control the playback rate for any <video> or <audio> element. Note that Chrome's audio implementation currently has a bug which may result in the 1.25 and 1.5 rates not working, but it should be fixed soon. Also, Vimeo turned out to the tricky, since it overlays other nodes on top of its <video> element, and buffers very slowly. Your best bet is to let the video buffer a bit, and then right-click just above the controller.

Bonus tip: If you are viewing a video in QuickTime Player (the QuickTime X version) and miss the old playback rate controls from the A/V Control palette, you can instead option-click on the fast-forward button to increase the speed.

I recently spent an enjoyable Sunday morning tracking down a Chrome extension-related bug. Though not as epic as some past bugs, it was still interesting, since it involved the interaction of four distinct codebases (wix.com's, Facebook Connect's, the extension, and Chromium's).

The reason why I'm writing about it here (or rather, just pointing it out), is because it seemed like a bit of waste to have that experience live only on Stack Overflow. It's not that I don't trust Stack Overflow (they seem to have good intentions and deeds). However, I'm no Jon Skeet, Stack Overflow isn't a big enough part of my online life that I feel like I have an enduring presence there. The test that I apply is "If I vaguely recall this piece of content in 10 years, will I be able to remember what site it was on? Will that site still be around/indexed?" The answer to the latter is most likely yes, but to the former, I'm not so sure (I already have a hard time searching across the mailing lists, bug tracker and Stack Overflow silos).

On the other hand, a blog post is too heavyweight for every piece of (notable) content. The lifestreaming fad of yesteryear also doesn't seem right, I don't want this aggregation to be necessarily public or a destination site. ThinkUp (and a plugin) seems like what I want, if only I could get over the PHP hurdle.

My earlier stance on Stack Overflow was based on being a pretty casual user (answering the occasional Reader question, using it as a reference when it turned up in search results). Now that it's an official support channel, I've been using it more, and the Kool Aid has started to wear off. For every interesting question, there are repeated ones that end up being about the same topic. For Chrome extension development, a recurring issue is not understanding that (nearly) all APIs are asynchronous.

Is the right thing there to mark them as duplicates of each other, since they have the same underlying cause? I tried that recently, and the moderator did not agree (relatedly, I'm amused that Eric Lippert's epic answer about local variables is on a question that was later closed as a duplicate). Even if closing as dupes were OK, what should the canonical answer look like? Presumably it would have to be something generic, by which point it's not all that different from the section in the official documentation that explains asynchronous functions. Is it just a matter of people not knowing the right terminology, so they will never search for [chrome extension api asynchronous]? Is the optimal end state a page/question per API function asking "Why does chrome.<API name here>() return undefined?" with a pointer to the documentation?

PuSH Bot is a PubSubHubbub-to-XMPP service that I created a couple of years ago. It had been running pretty much attended for a while, but in the past couple of months I started to get a complaint or two that it was flaky. It turned out that the new App Engine pricing model was making it not fit into the free quota anymore, so by late afternoon the service would start to report errors (it's not the only unattended App Engine app that this has happened to).

I decided to dust off the code, move it to its own repository and see if it could be brought back to life. Between some basic optimizations and kicking out some (unintentionally?) abusive users, it now fits in the free quota on most days. I also took this opportunity to modernize the code a bit.

With Google Reader shared items gone, some of the initial use cases aren't there anymore (I don't want to subscribe to random blogs with it; that's what Reader is for). However, there are still plenty of other sites that support PubSubHubbub. One recent addition that I'm experimenting with is Stack Overflow. With per-tag feeds, I can now get notified via IM when someone asks a question in the two topics that I tend to answer.

With the upcoming transition of social features in Google Reader to Google+, I thought this would be a good time to look back at the notable social-related events in Reader's history. For those of you who are new here, I was Reader's tech lead from 2006 to 2010.

Late 2004 to early 2005: Chris Wetherell starts work on "Fusion", one of the 20% projects that serve as prototypes for Google Reader. Among other neat features, it has a "People" tab that shows you what other people on the system are subscribed to and reading. There's no concept of a managed friends list, after all when the users are just a few dozen co-workers, we're all friends, right?

September 2005: Ben Darnell and Laurence Gonsalves add the concept of "public tags" to the nascent Reader backend and frontend. There are no complex ACLs, just a single boolean that controls whether a tag is world-readable.

October 2005: A remnant of the "People" tab is present in the HTML of the launched version of Google Reader, and an eagle-eyed Google Blogoscoped forum member notices it and speculates as to its intended use.

March 2006: Tag sharing launches, along with the ability to embed a shared tag as a widget in the sidebar of your blog or other sites. On one hand, tag sharing is quite flexible: you can share both individual items by applying a tag to them, and whole feeds (creating spliced streams) if you share folders. On the other hand, having to create a tag, share it and manually apply it each time is rather tedious. A lot of users end up sharing their starred items instead, since that enables one-click sharing.

Summer of 2006: As part of Brad Hawkes's summer internship, he looks into what can be done to make shared tags more discoverable (right now users have to email each other URLs with 20-digit long URLs). He whips up a prototype that iterates over a user's Gmail contacts and lists shared tags that each contact might have. This is neat, but is shelved for both performance (there's a lot of contacts to scan) and privacy (who exactly is in a user's address book?) concerns.

September 2006: Along with a revamped user interface, Reader re-launches with one-click sharing, allowing users to stop overloading starred items.

May 2007: Brad graduates and comes back work on Reader full-time. His starter project is to beef up Reader's support for that old school social network, email.

Fall of 2007: There is growing momentum within Google to have a global (cross-product) friend list, and it looks like the Google Talk buddy list will serve as the seed. Chris and I start to experiment with showing shared items from Talk contacts. We want to use this feature with our personal accounts (i.e. real friends), but at the same time we don't want to leak its existence. I decide to (temporarily) call the combined stream of friends' shared items "amigos". Thankfully, we remember to undo this before launch.

December 2007: After user testing, revamps, and endless discussions about opt-in/out, shared items from Google Talk buddies launches. Sharing is up by 25% overnight, validating that sharing to an audience is better than doing it into the void. On the other hand, the limitations of Google Talk buddies (symmetric relationships only, contact management has to happen within Gmail or Talk, not Reader) and communication issues around who could see your shared items lead to some user stress too.

Spring of 2008: With sharing in Reader picking up steam, a few aggregators and leaderboards of shared items start to spring up. Louis Gray comes to the attention of the Reader team (and its users) by discovering the existence of ReadBurner before its creator is ready to announce it.

May 2008: Up until this point sharing has been without commentary; it was up to the reader of the shared item to decide if it had been shared earnestly, ironically, or to disagree with it. "Share with note" gives users an opportunity to attach a (hopefully pithy) commentary to their share. Also in this launch is the "Note in Reader" bookmarklet (internally called "Tag Anything") that allows users to share arbitrary pages through Reader.

August 2008: Incorporating the lessons learned from Reader's initial friends feature, the preferred Google social model is revamped. Instead of a symmetric friend list based on Google Talk buddies, there is a separate, asymmetric list that can be managed directly within Reader. The asymmetry is "push"-style: users decide to share items with some of their contacts, but it's up those contacts to actually subscribe if they wish (think "Incoming" stream on Google+, where people are added to a "See my Reader shared items" circle). This feature is brought to life by Dolapo Falola, who injects some much-needed humor into the Reader code: the unit tests use the Menudo band members to model relationships and friends acquire a (hidden) "ex-girlfriend" bit.

March 2009: After repeated user requests, (and enabled by more powerful ACL supported added by Susan Shepard) comments on shared items are launched. Once again Dolapo is on point for the frontend side, while Derek Snyder does all the backend work and makes sure that Reader won't melt down when checking whether to display that "you have new comments" icon. The ability of the backend and user interface to handle multiple conversations about an item is stress-tested with a particularly popular Battlestar Galactica item.

May 2009: Bundles are launched, extended sharing from just individual tags to collections of feeds.

July 2009: Continuing the social learning process, the team (and Google) revamps the friends model once again, switching to a asymmetric "pull"-style (i.e. following) model. This is meant to be "pre-consistent" with the upcoming Google Buzz launch. Also included in this launch are better ties to Google Profiles and the ability to "like" items. In general there are so many moving parts that it's amazing that Jenna's head doesn't explode trying to design them all.

Also as part of this launch, intern Devin Kennedy's trigonometry skills are put to good use in creating an easter egg animation triggered when liking or un-liking an item after activating the Konami code.

August 2009: Up until this point, one-click sharing had mainly been for intra-Reader use only (though there were a few third-party uses, some hackier than others). With the launch of Send to (also Devin's work), Reader can now "feed" almost any other service.

February 2010: The launch of Google Buzz posed some interesting questions for the Reader team. Should items shared in Reader show up in Buzz? (yes!) Should we allow separate conversations on an item in Buzz versus Reader? (no!) With a lot of behind the scenes work, sharing and comments in Reader are re-worked to have close ties to Buzz, such that even non-Reader-using friends can finally get in on the commenting action.

March 2010: Partly as a tongue-in-cheek reaction to social developments within Google, and partly to help out some Buzz power users who were complaining that all the social features in Reader were slowing it down, I add a secret (though not for long) anti-social mode.

May 2010: Up until this point, it was possible to have publicly-shared items but only allow certain friends to comment on them. Though powerful, this amount of flexibility was leading to complexity and user confusion and workarounds. To simplify, we switch to offering just two choices for shared items, and in either case if you can see the shared item, you can comment on it.

As you can see, it's been a long trip, and with the switch to Google+ sharing features, Reader is on its fourth social model. This much experimentation in public led to some friction, but I think this incremental approach is still the best way to operate. Whether you're a sharebro, a Reader partier, a Gooder fan, the number 1 sharer or someone who "like"-d someone else, I am are very grateful that you were part of this experiment (and I'm guessing the rest of the past and present team is grateful too). And if you're looking to toast Reader for all its social stumbles accomplishments, the preferred team drink is scotch.

One of the big assignments in my 7th English class was to write an autobiographical composition entitled "Me, Myself & I". This being 1994, "multimedia" was a buzzword, so students were also given the option of doing the assignment as an "interactive" experience instead*. I had been playing around with HyperCard, so I chose that option (it also meant extra computer time while the rest of the class was writing things out long-hand). I recall the resulting HyperCard stack being fun to work on, and it featured such cutting-edge things as startup 3D animation rendered with Infini-D (albeit, with the trial version that was limited to 30 frames only).

I'm a bit of a digital packrat, so I still have the HyperCard stack that I made 16 years ago. I recently remembered this and wanted to have a look, but lacked a direct way to view it. Thankfully, there are many options for emulating mid-90s 68K Macs. Between Basilisk II, a Quadra 650 ROM, Apple distributing System 7.5.3 for free, and a copy of HyperCard 2.2, I was all set. I was expecting to have more trouble getting things running, but this appears to be a pretty well-trodden path.

I was pleasantly surprised that the full stack worked, including bits that relied on XCMDs to play back movies and show custom dialogs. The contents are a bit too embarrassing personal to share, but it contains surprisingly prescient phrases like "I will move to California and work for a computer company".

This stack also represents one of my earliest coding artifacts (outside of Logo programs that unfortunately got lost at some point), so I was also curious to look at the code. Unfortunately whenever that stack was loaded, all of the development-related menu commands disappeared. I remembered that stacks have user levels, and that lower ones are effectively read-only. I tried changing the user level in the Home stack, but to no effect: as soon as my stack was brought to the foreground, it was reset back to the lowest level. Hoping to disable script execution, I engaged in a bit of button mashing. Instead I rediscovered that holding down command and option shows all field outlines, including invisible fields. 13-year-old me was clever enough to include a backdoor – a hidden button in the lower right of all cards that when pressed reset the user level back to the development one.

Code-wise, 13-year-old me did not impress too much. There was a custom slider that moved between different events in my life, showing and hiding text in a main viewing area that was awfully repetitive:

on mouseDown
  repeat while the mouse is down
    set location of me to 118, mouseV()
    if mouseV() < 91 then
      set location of me to 118, 91
      walkfield
      exit mouseDown
    end if
    if mouseV() > 238 then
      set location of me to 118, 238
      stmarysfield
      exit mouseDown
    end if
  end repeat
  if mouseV() >= 91 and mouseV() <= 103 then
    set location of me to 118, 91
    walkfield
  end if
  if mouseV() > 103 and mouseV() <= 127 then
    set location of me to 118, 115
    talkfield
  end if
  ...and so on
end mouseDown

on walkfield
  play "Click"
  show card field walk
  hide card field talk
  hide card field beach
  hide card field garden
  hide card field school
  hide card field japan
  hide card field stmarys
end walkfield

on talkfield
  play "Click"
  hide card field walk
  show card field talk
  hide card field beach
  hide card field garden
  hide card field school
  hide card field japan
  hide card field stmarys
end walkfield

With Rosetta being removed from Lion, PowerPC-only Mac OS X software is next on the list of personally-relevant software to become obsolete (Iconographer in this case). Thankfully, it looks like PearPC is around in case I get nostalgic about 18-year-old me's output.

* I was initially going to have a snarky comment about the teacher** not realizing that the web was the way of the future, but after thinking about it more, having this level of flexibility was great, regardless of the specific technologies involved.

** Hi Mr. Warfield! Good luck with whatever is next!

This past weekend I took part in the Foursquare Global Hackathon. I used this opportunity to implement an idea that I had while having dinner with Ann and Dan at Sprout:

The premise of the show How I Met Your Mother is that in the year 2030 the narrator (Ted) is telling his kids how he met their mother. He starts the story 25 years earlier (i.e. in 2005), and thus far (after 6 years), we've gotten a lot of hints, but we haven't met the mother yet. However, it's quite apparent that Ted has in fact been at the same venue as the mother several times. In a hypothetical world where Ted and the mother use Foursquare, I thought it would be neat if they could compare checkin histories and see all the near-misses that they had over the years.

Intersquares does exactly that: you can sign in with your Foursquare account, and then once your checkin history is processed, another user can sign in with their account, and you'll both be told where you were together (whether you knew it at the time or not). This can be great for remembering first dates or for finding close calls.

To see it in action, feel free to see if you have run into me anywhere. There's also a screencast that demos the the site.

This was my first hackathon, and I feel like it went pretty well (i.e. I managed to finish something). I definitely tried to keep the hack part in mind, as far as getting something together quickly. The code has a few dodgy technical decisions (keep all checkins in one entity property? what could go wrong?). It was also helpful to build on the same stack as Stream Spigot (App Engine, Python, Closure, Django Templates) so that I could lift a lot of utility code and patterns. Next time, I'd like to be a part of a team though: while being a one man band is satisfying (check out that logo), it does tend to limit the scope to toy-like apps such as this one.

The hackathon has prizes, so if Intersquares intrigues you, your vote is appreciated. Definitely check out some of the other entries too. I haven't gone through all of them yet, but so far Near, Magic Muggle Clock and Plan Your Next Trip all seem really neat.

Update on 9/28/2011: Intersquares was a finalist in the hackathon!

As Jonathan has blogged, "What is the hardest bug you've ever tackled?" is an interesting conversation starting point with engineers, one that I often use to start (phone) interviews. I usually rephrase it as "Describe an interesting or difficult bug that you ran into", since "hardest" often causes people to freeze up as they ponder whether the bug they have in mind is actually the hardest. In any case, most bugs become interesting if you ask "why?" enough.

Along these lines, here's a bug that I ran into in mid-2007 while I was working on Google Reader: Soon after a production push, we noticed that some users were complaining that Reader wasn't loading properly when they reloaded the page. Stranger still, others said that it wasn't working properly initially, but after a few reloads it would start working. Checking things in the office revealed similar inconsistent results: Reader would load for some but not for others. For those for whom Reader hadn't loaded successfully, it turned out to be because of a 404 that was returned when trying to load Reader's main JavaScript file.

This happened soon after Gears support was added to Reader, so we initially suspected some interaction with offline support. Perhaps an old version of the HTML was being used by some users, and that contained a link to a version of the JavaScript file that we didn't serve anymore. However, some quick Dremel-ing showed that we had never served the URLs that triggered 404s until the push began. Stranger still, not all requests for those URLs resulted in 404, only about half.

At this point a bit of background about Reader's JavaScript infrastructure is necessary. As previously mentioned, Reader uses the Closure Compiler for processing and minimization of JavaScript. Reader does runtime compilation, since it supports per-user experiments that would make it prohibitive to compile all combinations at build or push time. Instead, when a user requests their JavaScript file, the set of experiments for them is determined, and if we haven't encountered it before, a new variant is compiled and served. JavaScript (and other static resources) are served with a checksum of their contents in the filename. This allows each URL to be served with a far-future cache expiration header, and makes sure that when its content changes users will pick up changes by virtue of having a new URL to fetch.

The JavaScript URL is used in two places, once embedded as a <script src="..."> tag in the HTML, and once when requesting the file itself. The aforementioned compilation and serving steps happen once for each (identical) frontend machine, and some machines had one idea of what the URL should be, and others had a different expectation. Since the frontends are stateless, it was quite likely for users to request the JavaScript from a different one than the one they got the HTML with the URL from. If there was a mismatch, then the 404 would happen. However, if the user reloaded enough times, they would eventually hit a pair of machines that did think the JavaScript URL was the same.

I said the users were getting "seemingly" identical JavaScript, but there was actually a slight difference when doing a diff (which explained the difference in checksums). One variant contained return/^\s*$/.test(str == null ? "" : String(str)) while the other had return/^\s*$/.test((str == null ? "" : String(str))) (note the extra parentheses in the test() argument). The /^\s*$/ regular expression was distinctive enough that it was easy to map this as being the compiled version of the Closure function goog.string.isEmptySafe, which is defined as:

goog.string.isEmptySafe = function(str) {
  return goog.string.isEmpty(goog.string.makeSafe(str));
};

The goog.string.isEmpty and goog.string.makeSafe calls get inlined, hence the presence of the regular expression test and String() directly (note that the implementations may have changed slightly since 2007).

Now that I knew where to look, I began to turn compiler passes off until the output became stable, and it became apparent that the inlining pass itself was responsible. The functions would not be inlined in the same order (i.e. goog.string.isEmpty and then goog.string.makeSafe, or vice-versa), and in one case the the compiler decided to add extra parentheses for safety. Specifically, when inlining the compiler would check to see if the replacement AST node was of lower precedence that the one it was replacing. If it was, a set of parentheses was added to make sure that the meaning was not changed.

The current compiler inlining pass is very different from the one used at that point, but the relevant point here is that the compiler would use a HashSet to keep track of what functions needed to be inlined. The hash set was of Function instances, where Function was a simple class that had a couple of Rhino Node references. Most importantly, it didn't define either equals() or hashCode(), so identity/memory address comparisons and hash code implementations were used.

When actually inlining functions, the compiler pass would iterate through the HashSet, and since the Function instances corresponding to goog.string.isEmpty and goog.string.makeSafe had different addresses depending on the machine, they could be encountered in a different order. The fix was to switch the list of functions to inline to a List (Set semantics were not necessary, especially given that Function instances used identity comparisons so duplicates were not possible).

The inlining compiler pass had used a HashSet for a long time, so I was curious why this only manifested itself then. The explanation turned out to be prosaic: this was the first Reader release where goog.string.isEmptySafe was used, and there were no other places where there were nested inlineable function calls. (This bug happened around the time we switch to JDK6, which changed HashSet internals, but we hadn't actually switched to JDK6 at that point, so it was not involved).

None of this was reproducible when running a frontend locally or in the staging environment, since all those setups have a single frontend instance (they're of very low traffic). In those case, no matter which version was compiled and which URL was generated, it was guaranteed to be serveable. To prevent the reoccurrence of similar bugs, I added a unit test that compiled Reader's JavaScript locally several times times, and made sure that the output did not change. Though not foolproof, it has caught a couple of other such problems before releases made it out into production.

The main reason why I enjoyed fixing this bug was because it involved non-determinism. However, unlike other non-deterministic bugs that I've been involved in, the triggering conditions were not so mysterious that it took months to solve.