Introducing "Cloud Pipe" 

This is the last major piece of the rssCloud architecture, the only piece not specified in 2001.

With Cloud Pipe, it is possible to flow realtime updates from the cloud to desktop and mobile devices, even if they are not "net-accessible," that is, behind a firewall or NAT.

The technology is modeled after the realtime part of the FriendFeed API. I chose this approach because: 1. It's known to work. 2. It's easy to implement. 3. The server-side scales. 4. Other developers are familiar with it. 5. There aren't many ways to solve this problem.

Document status 

This is the second revision of the document, the first was written on Dec 8, 2009.

This is the first version that specifies the actual protocol used in the demonstration client and server I've put together.

This should not be considered frozen, and anyone who implements this protocol should be prepared for it to change, perhaps radically. Caveat implementor!

The previous paragraph is the one that people will pretend they didn't read. So please read it. Thanks.

Basics 

The server defines an endpoint, specified by a URL.

The server is net-accessible. It is not behind a firewall or NAT.

It can receive rssCloud or PubSubHubBub notifications, perhaps others. At this time, the server I have implemented only supports rssCloud.

Users are identified by a username and password.

The protocol is REST. Basic HTTP authentication is used.

Telling the server what feeds to follow 

To tell the server what feeds the user is following, the client sends a POST to the endpoint. The body of the POST is an OPML subscription list.

The feeds may be realtime feeds or require polling. For realtime feeds, the server registers a notification request.

The server returns immediately, with only a <system> packet as a response. (See Formats, below.)

The client learns of updates 

The client simulates a "long poll."

On startup, it enters a loop.

At the top of the loop it opens an HTTP connection to the cloud server by making a GET request, with a long timeout, say five minutes.

The server checks to see if there are any updates for the client. If so, it returns them as a sequence of XML elements, one for each update. (See Formats, below.)

If there aren't any updates, the server goes to sleep for a short period (on my server, three seconds) then checks again. After 180 seconds it times out and returns without any updates, with only a <system> packet.

The client is expected to call again. By having the server timeout after a few minutes we keep it clear of clients that may have died. It assures us that there is someone on the other side of the connection waiting for a response, or at least there was a few minutes ago.

Formats 

The server response is in XML, a single <packets> element.

If there were N updates, there are N+1 sub-elements; one <fatPing> element for each update and a <system> element, with information about the server. As usual with XML data, the client ignores information it is not interested in.

Each <fatPing> element has a single attribute called "feed" which is the URL of the feed the update came from. The value of the fatPing element is encoded XML text, the contents of a single RSS <item> or Atom <entry>.

The server has processed the feed and determined that this is a new element. Further, it's valid XML. It's transmitted as CDATA because it may use elements from namespaces that are not declared by in the <packets> element. It's encoded because it may itself contain CDATA.

A <system> element has no attributes and (as of this writing) three sub-elements: <serialnum>, <when> and <secs>. serialnum is a number that is incremented for each packet. when is a RFC822-formatted string representation of the time the system packet was generated on the server. secs is the number of seconds the server thread was running when it returned.

Examples 

A response that represents a single item from the NY Times home page feed.

A response that only has a <system> packet.

The most recent response from my development server to my desktop client.

An OPML subscription list sent to my server.

Source listings of my client and server.

A server to test against 

I've set up a server for people to test CloudPipe apps with.

The endpoint: http://rpc.cloudpipe.org:5337/river2/cloudPipeServer

To use the service you must have an account on Identi.ca. Your identi.ca username and password identify you. I call them once to verify that the username and password are valid, then I retain a copy of the password and use it to authenticate you from then on. I do not under any circumstances use the identity to post to their service or to read any of the data in your account.

There's a limit to 15 feeds you can follow on this server. That's just to keep things manageable for testing.

If you want, you can follow the same feeds I'm testing with. It's a good mix of realtime and static feeds that update fairly frequently. By using the same feeds you really keep the server load down. Here's the OPML text.

My server only supports rssCloud. If your feed uses another realtime protocol, I'll poll the feed, once an hour.

If you have any questions post a comment here. Also if you get an implementation running, please let us know by posting a comment!