Home > Archive >  2009 >  December >  26

Update on CloudPipe work

Saturday, December 26, 2009 by Dave Winer.

As I noted in a post on Wednesday, I'm using the holiday to build out the CloudPipe protocol that I explained in a post on December 9. 

I'm getting pretty close to the point where I'll write docs and put it up for review. 

But I wanted to post a heads-up now that I've made a change in how it works, a difference between the actual protocol and the December 9 narrative.  

It says: "When the server detects a change in a feed the client is subscribed to, either through polling or notification, it sends a packet to the client indicating the feed has changed. The client then reads the feed, and processes it the same way it would if it had detected the change through polling." 

The client may read the feed but I implemented and will specify a "fat ping." Along with the URL of the feed, we send the content of the item. If there are two items that changed, then two complete fat ping packets are sent. (This should answer in advance the question of how many items may be contained in a ping -- one and only one.) 

I went this way for two reasons: 

1. Competitive -- It's often mentioned as the reason people prefer PubSubHubBub over rssCloud. By providing for a fat ping, we erase that advantage. Now the two methods are at least at parity, with an edge to rssCloud because, once CloudPipe is done, it will work for clients that are behind firewalls and NAT, and PSHB does not. 

2. Efficiency -- I learned when I met with the Tumblr guys in NY earlier this month that their feeds get pounded. The number one most referenced pages on their sites are the feeds. When I was running UserLand, it never was an issue, but we had a very small fraction of the traffic that Tumblr, WordPress, Blogger, Typepad etc have. Now that I know it is an issue, I want to address it. By sending the content along with the ping, in the realtime layer that's building now there will be fewer feed reads. 

Now there's the question of whether to normalize the contents of an item. I can do it, but it might be a politically sensitive issue. Right now I have bigger fish to fry, so I won't normalize items, what you get in the fat ping is exactly what the feed broadcast. You'll have to deal with all the variability that can appear in the various flavors of RSS and Atom. I could make it so you didn't have to, but... We'll get to that later. 

I expect to have a server for people to test against and docs for the protocol before January 1. Knock wood, praise Murphy, etc etc. 

Update: Here's a 5-minute podcast that explains what these fat pings are all about.