Scripting News: Sitemaps, day 3

Scripting News, the weblog started in 1997 that bootstrapped the blogging revolution.

Sitemaps, day 3

Since I've been playing with sitemaps, of course I created one for the RSS 2.0 site.

And I've checked to see that the maps I deployed for scripting.com are properly updating, and they are.

But when I checked, I realized that I would have done it differently, so that the sitemaps, in adition to helping search engine crawlers, might be interesting things for human beings to read as well.

I refer back to sitechanges.txt, a simple project I was doing in 1997 that was like sitemaps. It was also before I did XML. ";->"

The idea was that the content server was responsible for providing a daily reverse-chronologic list of pages that had changed. Then a crawler would keep track of when it had last visited my site, and only suck down the files that had changed since then. This would enable search engines to be more efficient, and provide more current content. It was nice because you could read it yourself and see what had changed. Contrast this with sitemaps, where you have to go hunting for the changes, it's no better a user interface for finding the new and newly updated stuff than the file sytstem is. I was kind of disappointed.

Another thing I would have done differently is allowed sitemaps to include other sitemaps. There really is no need for two file types, just let me link to an index from an index, much like inclusion in OPML 2.0. This added an extra layer of complexity for everyone implmenting sitemaps on moderately large sites, or old ones where some content changes frequently and other content not so frequently (like scripting.com).

However, on balance, it's a great thing that all these companies got together and did something to make the web work better. We need more of that!

If anyone is working on more stuff like this, I am available to review it before it's cast in stone.

Last update: Thursday, June 3, 2010; 4:00:40 PM

~About the Author~

Dave Winer, 55, is a visiting scholar at NYU's Arthur L. Carter Journalism Institute. He pioneered the development of weblogs, syndication (RSS), podcasting, outlining, and web content management software; former contributing editor at Wired Magazine, research fellow at Harvard Law School, entrepreneur, and investor in web media companies. A native New Yorker, he received a Master's in Computer Science from the University of Wisconsin, a Bachelor's in Mathematics from Tulane University and currently lives in New York City.

"The protoblogger." - NY Times.

"The father of modern-day content distribution." - PC World.

One of BusinessWeek's 25 Most Influential People on the Web.

"Helped popularize blogging, podcasting and RSS." - Time.

"The father of blogging and RSS." - BBC.

"RSS was born in 1997 out of the confluence of Dave Winer's 'Really Simple Syndication' technology, used to push out blog updates, and Netscape's 'Rich Site Summary', which allowed users to create custom Netscape home pages with regularly updated data flows." - Tim O'Reilly.

Mail: scriptingnews1mail at gmail dot com.

Mar May