A strange confluence of events. I was writing a series about rebooting RSS. Then I had a bike accident that left me dazed and in pain. One evening I woke up and found I could sit for a few hours in front of the computer without pain. That was pretty cool, because even lying in bed was excruciating. My poor body was in need of some rest. Programming could provide it.
I had been playing with long-polling for instant outlining, with excellent results. I wonder if the same could be done for feeds. I had found an OPML file on the NY Times site with all their feeds, so I hooked it up to River2. Now I had a relatively realtime flow of every story from the Times. So I wrote a minimal server that would wait until something new showed up and then return it. (Later I wrote a filter that eliminated duplicate titles, so even if a story appeared in two feeds it would only be shot out through the hose once.)
A small embellishment, let the caller give me a "seed" that indicated where he left off. If any new items came in while he was processing the last ones I returned, he wouldn't miss any. Every packet includes a seed which you can send back to me.
Here's an example of a response with some items. There are two forks to the XML, items and metadata.
items are just a series of RSS items (or Atom entries, for feeds that are in Atom format). However we add three elements to each item, feedUrl, feedTitle and feedLink. My app, on the other end of the hose, needed this info. Seemed reasonable to provide it since I had it there.
In metadata, there are two elements, now -- might add others in the future. seed is the one you need to include as a parameter in your next call. If the seed returned is xxx, then your next call would be http://hose.scripting.com/?name=nytimes&seed=xxx.
The seed is supposed to be opaque, that is -- you shouldn't expect to understand its structure, but you can see mine is pretty straightforward. It's a walk of the hierarchy in my object database. Year-month-day-serialnum. So 2010-09-30-00651 is talking about the 651st news item on Sept 30 (today).
One more thing, I wrote a handler that returns the current seed in case you want to start off with a seed. You don't have to, but your code might be a little simpler if you have it to start out with. http://hose.scripting.com/seed
Update: I've released the source code for the Frontier tool that implements both sides of the protocol.