Home > Archive >  2009 >  December >  26

Update on CloudPipe work

Saturday, December 26, 2009 by Dave Winer.

A picture named ninja.gifAs I noted in a post on Wednesday, I'm using the holiday to build out the CloudPipe protocol that I explained in a post on December 9. Permalink to this paragraph

I'm getting pretty close to the point where I'll write docs and put it up for review. Permalink to this paragraph

But I wanted to post a heads-up now that I've made a change in how it works, a difference between the actual protocol and the December 9 narrative.  Permalink to this paragraph

It says: "When the server detects a change in a feed the client is subscribed to, either through polling or notification, it sends a packet to the client indicating the feed has changed. The client then reads the feed, and processes it the same way it would if it had detected the change through polling." Permalink to this paragraph

The client may read the feed but I implemented and will specify a "fat ping." Along with the URL of the feed, we send the content of the item. If there are two items that changed, then two complete fat ping packets are sent. (This should answer in advance the question of how many items may be contained in a ping -- one and only one.) Permalink to this paragraph

I went this way for two reasons: Permalink to this paragraph

1. Competitive -- It's often mentioned as the reason people prefer PubSubHubBub over rssCloud. By providing for a fat ping, we erase that advantage. Now the two methods are at least at parity, with an edge to rssCloud because, once CloudPipe is done, it will work for clients that are behind firewalls and NAT, and PSHB does not. Permalink to this paragraph

2. Efficiency -- I learned when I met with the Tumblr guys in NY earlier this month that their feeds get pounded. The number one most referenced pages on their sites are the feeds. When I was running UserLand, it never was an issue, but we had a very small fraction of the traffic that Tumblr, WordPress, Blogger, Typepad etc have. Now that I know it is an issue, I want to address it. By sending the content along with the ping, in the realtime layer that's building now there will be fewer feed reads. Permalink to this paragraph

Now there's the question of whether to normalize the contents of an item. I can do it, but it might be a politically sensitive issue. Right now I have bigger fish to fry, so I won't normalize items, what you get in the fat ping is exactly what the feed broadcast. You'll have to deal with all the variability that can appear in the various flavors of RSS and Atom. I could make it so you didn't have to, but... We'll get to that later. Permalink to this paragraph

I expect to have a server for people to test against and docs for the protocol before January 1. Knock wood, praise Murphy, etc etc. Permalink to this paragraph

Update: Here's a 5-minute podcast that explains what these fat pings are all about. Permalink to this paragraph

   Recent stories

A picture named dave.jpgDave Winer, 54, pioneered the development of weblogs, syndication (RSS), podcasting, outlining, and web content management software; former contributing editor at Wired Magazine, research fellow at Harvard Law School, entrepreneur, and investor in web media companies. A native New Yorker, he received a Master's in Computer Science from the University of Wisconsin, a Bachelor's in Mathematics from Tulane University and currently lives in Berkeley, California.

"The protoblogger." - NY Times.

"The father of modern-day content distribution." - PC World.

One of BusinessWeek's 25 Most Influential People on the Web.

"Helped popularize blogging, podcasting and RSS." - Time.

"The father of blogging and RSS." - BBC.

"RSS was born in 1997 out of the confluence of Dave Winer's 'Really Simple Syndication' technology, used to push out blog updates, and Netscape's 'Rich Site Summary', which allowed users to create custom Netscape home pages with regularly updated data flows." - Tim O'Reilly.


Dave Winer Mailto icon

© Copyright 1994-2009 Dave Winer Mailto icon.

Last update: 12/26/2009; 6:30:22 PM Pacific. "It's even worse than it appears."

RSS 2.0 feed.