Introducing "Cloud Pipe" 

This is the first version of the Cloud Pipe docs, written on Dec 8, 2009, archived on Dec 27.

This is the last major piece of the rssCloud architecture, the only piece not specified in 2001.

With Cloud Pipe, it will be possible to flow realtime updates from the cloud to desktop and mobile devices, even if they are not "net-accessible," that is, even if they are behind a firewall or NAT.

The technology here is not new, in fact the design is modeled after the realtime part of the FriendFeed API. I chose this approach because: 1. It's known to work. 2. It's easy to implement. 3. The server-side is known to scale. 4. Many other developers are familiar with it. 5. There aren't many ways to solve this problem.

Narrative 

The server implements an API that allows a client to register an OPML file with it, containing the feeds that the user follows. The server subscribes to any feeds that it does not already subscribe to. If the feeds are realtime it registers a notification request.

The client simulates a "long poll." On startup, it enters a loop. At the top of the loop it opens an HTTP connection to the cloud server with a timeout of X minutes. The call completes either by receipt of an XML packet, or by error (communication problem, timeout). If it's an XML packet, the client parses it and acts on it. In either case, it goes back to the top of the loop.

When the server detects a change in a feed the client is subscribed to, either through polling or notification, it sends a packet to the client indicating the feed has changed. The client then reads the feed, and processes it the same way it would if it had detected the change through polling.

Discussion 

This protocol works equally well for all kinds of feeds, whether they're RSS, Atom, OPML or a format that has yet to be invented.

No doubt there will be proponents for "fat" packets. I believe in keeping the protocol as simple as possible and since the methods for reading and detecting new elements in feeds are very well known, there's no need to centralize the functionality. Further, it gives too much power to the server, it won't be long vefore they start "deprecating" feed formats. I choose to implement as little functionality as possible to achieve the result to keep as much power as possible out of the hands of big companies and away from their motives (patents, lockin, centralization, competing with other big companies). That said, if other groups want to form to create a fat packet format, and a consensus develops, and it's not too hard to implement, and it's not too evil, we will implement it.

Of course people will ask how this compares with PubSubHubBub. As far as I know it does not allow for communication with clients that are behind firewall and NAT. Further, there's no reason the server can't handle PSHB subscriptions as well as rssCloud subscriptions.

I think Facebook's Tornado open source release, which I understand is the back-end software of FriendFeed, is a likely environment for implementing this functionality. I plan to do an implementation in the OPML server environment, alongside rssCloud.root.

I'm not designing a specific XML packet format at this time. I want to give this idea a chance to settle in first. However I think a format will be very easy to design and specify.

Dave Winer
New York, NY