Archive >  2009 >  February >  4 Previous / Next

Scripting News, the weblog started in 1997 that bootstrapped the blogging revolution.

A metadata race? Permanent link to this item in the archive.

Steve Gillmor perceives a race between Twitter and FriendFeed, which I find interesting, even if I don't think it's as much a race as he does. At this point there is so much distance between the services they provide, it's hard to see them as competitors.

However, there is one area where all the current providers of social networks are in competition, and I'm not sure they're all aware of it, and so far no one has entered the arena, but I get a sense that at least a couple are poised to -- Twitter and FriendFeed.

A picture named car.gifThere's an interesting story that came up that's very much in line with the Where's Your Data? and What Happened To NakedJen? threads here on Last week, a bookmarking service, ma.gnolia, lost all its data. All of it. It was bound to happen sooner or later. Everyone who operates a public service back-end knows how fragile these things are, I sure do. At UserLand we came close to melting down a few times, and every one of those times we urged our users, as best we could, to keep local copies of their data.

Anyway, ma.gnolia crashes, and someone there realizes that FriendFeed has been archiving their data for users that established the connection between the two services. And FriendFeed has an API. So voila -- write a web app to pull the data out of FF and re-populate their database, at least for some of their users.

This suggest a buddy system for web service providers. Not every service can rely on the same back-end for safety, what happens when that service goes down? And who knows all the dependencies we're creating -- what if FF is using Amazon's back-end and Amazon gets attacked by an asteroid or a crazy terrorist, or whatever.

All of this leads me to a wish that one of these companies would allow, through their API, for us to store arbitrary data on a per-user basis. I'm working with a small group of users on a new build of the Instant Outliner, and am using FriendFeed's "room" structure to great advantage. It's almost at the point where I don't have to write a back-end at all, I could almost completely depend on theirs. If only. If only they allowed me to store a relatively small amount of XML-based data with each user. Less than a megabyte per user. Probably way less.

Something like Amazon's SimpleDB only even simpler, would do the job. The equivalent of a Perl hash or a Python dictionary. The same data we pass around in XML-RPC would be very good.

I know all this sounds super-technical, but the architects at Twitter and FriendFeed know exactly what I'm asking for, and it wouldn't surprise me at all if one or both had this facility almost ready to go.

Archiving Twitter in OPML Permanent link to this item in the archive.

In mid-January I started a project to archive the Twitter posts of the people I follow. At first I experimented with rendering the archives in an XML-compatible form of HTML, but decided the point would largely be lost, so I decided to go with OPML.

You can find the folder of archives here:

At the top level of each sub-directory is calendar.opml and today.opml. The former links to every OPML file for that user, and the latter contains all the twits from today, or the last day that person posted something to Twitter.

There's also a folder called 2009, and under that two sub-folders: 01 and 02 for January and February. And under each of those is a file for each day. In March there will be a folder called 03, and so on.

I do not archive posts for people whose Twitter accounts are private.

The archive follows the whims of my follows and unfollows. If I started following someone on January 29, their archive would start on that day. If I unfollowed and then followed, there will be a gap in their archive.

The archive is updated once a minute.

The server is in Amazon's EC2 cloud.

No guarantees are made that this service will remain up, I'm doing it entirely as an experiment, to learn what the issues and perhaps what the opportunities are.


Last update: Wednesday, February 04, 2009 at 2:12 PM Pacific.

A picture named dave.jpgDave Winer, 53, pioneered the development of weblogs, syndication (RSS), podcasting, outlining, and web content management software; former contributing editor at Wired Magazine, research fellow at Harvard Law School, entrepreneur, and investor in web media companies. A native New Yorker, he received a Master's in Computer Science from the University of Wisconsin, a Bachelor's in Mathematics from Tulane University and currently lives in Berkeley, California.

"The protoblogger." - NY Times.

"The father of modern-day content distribution." - PC World.

One of BusinessWeek's 25 Most Influential People on the Web.

"Helped popularize blogging, podcasting and RSS." - Time.

"The father of blogging and RSS." - BBC.

"RSS was born in 1997 out of the confluence of Dave Winer's 'Really Simple Syndication' technology, used to push out blog updates, and Netscape's 'Rich Site Summary', which allowed users to create custom Netscape home pages with regularly updated data flows." - Tim O'Reilly.

Dave Winer Mailto icon

My most recent trivia on Twitter.

My Wish List

On This Day In: 2008 2007 2006 2005 2004 2003 2002 2001 2000 1999 1998.

February 2009
Jan   Mar

Click here to see a list of recently updated OPML weblogs.

Click here to read blogs commenting on today's Scripting News.

Morning Coffee Notes, an occasional podcast by Scripting News Editor, Dave Winer.

Click here to see an XML representation of the content of this weblog.

Click here to view the OPML version of Scripting News.

© Copyright 1997-2009 Dave Winer.

Previous / Next