Click here to show or hide the menubar.

Home >  Archive >  2010 >  September >  30

Previous / Next

Feedhose -- a firehose for feeds
By Dave Winer on Thursday, September 30, 2010 at 7:33 PM.

A picture named hose.jpgA strange confluence of events. I was writing a series about rebooting RSS. Then I had a bike accident that left me dazed and in pain. One evening I woke up and found I could sit for a few hours in front of the computer without pain. That was pretty cool, because even lying in bed was excruciating. My poor body was in need of some rest. Programming could provide it.  permalink

So there I was in front of the machine. Now what? I could write some code. But what? permalink

I had been playing with long-polling for instant outlining, with excellent results. I wonder if the same could be done for feeds. I had found an OPML file on the NY Times site with all their feeds, so I hooked it up to River2. Now I had a relatively realtime flow of every story from the Times. So I wrote a minimal server that would wait until something new showed up and then return it. (Later I wrote a filter that eliminated duplicate titles, so even if a story appeared in two feeds it would only be shot out through the hose once.) permalink

A small embellishment, let the caller give me a "seed" that indicated where he left off. If any new items came in while he was processing the last ones I returned, he wouldn't miss any. Every packet includes a seed which you can send back to me. permalink

Then I wrote a client. Then I added another stream, and broadened my client so it could handle multiple streams.  permalink

The result is a nice sweet little protocol that behaves like two familiar things: permalink

1. A long-poll server. permalink

2. RSS. permalink

It's pretty cool. Let me know if you're writing some scripts to test it out. The format is described below. permalink

Here's an example of a query:  permalink

http://hose.scripting.com/?name=nytimes&timeout=3. permalink

Click the link. It'll just sit there till it times out after 3 seconds or, if a new item comes in before the timeout, it will return with that item. permalink

Here's an example of a response with some items. There are two forks to the XML, items and metadata.  permalink

items are just a series of RSS items (or Atom entries, for feeds that are in Atom format). However we add three elements to each item, feedUrl, feedTitle and feedLink. My app, on the other end of the hose, needed this info. Seemed reasonable to provide it since I had it there. permalink

In metadata, there are two elements, now -- might add others in the future. seed is the one you need to include as a parameter in your next call. If the seed returned is xxx, then your next call would be http://hose.scripting.com/?name=nytimes&seed=xxx. permalink

The seed is supposed to be opaque, that is -- you shouldn't expect to understand its structure, but you can see mine is pretty straightforward. It's a walk of the hierarchy in my object database. Year-month-day-serialnum. So 2010-09-30-00651 is talking about the 651st news item on Sept 30 (today). permalink

One more thing, I wrote a handler that returns the current seed in case you want to start off with a seed. You don't have to, but your code might be a little simpler if you have it to start out with. http://hose.scripting.com/seed permalink

Update: I've released the source code for the Frontier tool that implements both sides of the protocol. permalink

RSS feed for Scripting News
This site contributes to the scripting.com community river.


© Copyright 1997-2012 Dave Winer. Last update: Saturday, October 02, 2010 at 7:51 AM Eastern. Last build: 8/26/2012; 5:57:31 PM. "It's even worse than it appears."

RSS feed for Scripting News

Previous / Next