Feed Playback and Stream Spigot #

Feed Playback

A few weeks ago I came across sysadvent, an advent calendar-style blog with a new tool or tip each day. However, I only stumbled across it on day 15, and I wanted to read it from the beginning (and perhaps the previous years too). I could Instapaper all of the older posts but that would be 1) tedious 2) make it unlikely that I would ever actually read the entries, since seeing a pile of 36 long posts would be daunting.

Ann needed something similar a few years ago when she first came across cooking blogs like Chocolate & Zucchini and thepassionatecook, so I ended up writing a few scripts that scraped their archive and then generated a feed that would have a new post every day. Instead of one-off scraping scripts, I wondered if I could use Google Reader's archive of feeds to provide the backlog. Reader's been around for five and a half years by now, so this should work quite well for a lot of blogs.

Feed Playback is the resulting tool. You give it a blog or feed URL, and it generates a new feed for you that updates once a day (or more rarely if you'd like) with a new item from the original feed's archive. The use cases are learning a new skill or language, or reading a long-running comic from (nearly) the beginning. I also noticed that this playback idea seems to be a bit in vogue right now; as I was working on the tool I was reminded of Disunion (The New York Times' playback of the Civil War) or the Orwell Diaries (a playback of George Orwell's World War II diaries).

For the actual implementation of the tool, I decided to use Reader's shared tag functionality. For each playback, a corresponding shared tag is created. As the playback is advanced, items from Reader's archive of the original feed are tagged into it. This has a couple of advantages: serving of playback feeds is offloaded to Reader, instead of being the responsibility of my app. Perhaps more importantly, this means that copies of items are not made; the item that's in the playback feed is the one that was originally published. This means that metadata like who shared or liked it is preserved. In case any of this is interesting to you, source is available.

Stream Spigot

Working on this tool reminded me of my earlier work on Twitter Digest. Both are tools that enable me to more efficiently consume content. I decided to put both of them under the umbrella of Stream Spigot, and to (hopefully) collect more such tools at that site. Ben (or was it Chris?) would make TiVo analogies when describing Reader, and I think Stream Spigot can use that comparison too. Real-time content may be popular right now, but I think there's also a need for time-shifting tools that let you consume such content at your own convenience.

Though I'm sure I'll blog about big Stream Spigot changes here, if you'd like to keep up with more day-to-day tweaks, you can either follow the project on GitHub or its Twitter account.


Mihai, great idea to read an electronic book in installment chunks. Can you shed some light on what requirements web sites need to meet in order for the Stream Spigot feed auto-discovery mechanism to work? I notice Stream Spigot works fine with both your own site and my own, but not, for example, with TechCrunch or ReadWriteWeb, resulting in a "could not look up URL" error message - though both publish their feeds in their HTML headers. Just curious...

like usually make a awesome tools...its so usefully master.
@Marjolein: TechCrunch and ReadWriteWeb had more posts that I expected, and I was running into a App Engine exception. That should be fixed with https://github.com/mihaip/streamspigot/commit/424f5d1cf29abe39492b27c91979e7e3f53a9795.
I'm frustrated looking for tool that would enable to read a blog posts in chronological order. It seems obvious that many people would like such option, but it's simply not taken into consideration :(

Perhaps it is possible to modify Stream Spigot to read a blog backwards in one session instead to get it bit by bit?
This is a fantastic tool! I know many people are frustrated about Google Reader's inability to "Mark all as unread" to make reading from the beginning of the blog easier.

Is it possible to increase the frequency of the playback, or even give total access to the entire feed archive initially? I can understand how if you're learning a language, one post a day would be sufficient, but for something like a webcomic, or for people like me who like to skim, this would be great.

Thanks so much for Feed Playback, I love it!
@Anonymous: Thanks for the feedback. I've added more frequent updates to my todo list.

Post a Comment