Sitemaps, day 3

Friday, April 13, 2007 by Dave Winer.

Since I've been playing with sitemaps, of course I created one for the RSS 2.0 site. 

And I've checked to see that the maps I deployed for scripting.com are properly updating, and they are. 

But when I checked, I realized that I would have done it differently, so that the sitemaps, in adition to helping search engine crawlers, might be interesting things for human beings to read as well. 

I refer back to sitechanges.txt, a simple project I was doing in 1997 that was like sitemaps. It was also before I did XML.  

The idea was that the content server was responsible for providing a daily reverse-chronologic list of pages that had changed. Then a crawler would keep track of when it had last visited my site, and only suck down the files that had changed since then. This would enable search engines to be more efficient, and provide more current content. It was nice because you could read it yourself and see what had changed. Contrast this with sitemaps, where you have to go hunting for the changes, it's no better a user interface for finding the new and newly updated stuff than the file sytstem is. I was kind of disappointed. 

Another thing I would have done differently is allowed sitemaps to include other sitemaps. There really is no need for two file types, just let me link to an index from an index, much like inclusion in OPML 2.0. This added an extra layer of complexity for everyone implmenting sitemaps on moderately large sites, or old ones where some content changes frequently and other content not so frequently (like scripting.com). 

However, on balance, it's a great thing that all these companies got together and did something to make the web work better. We need more of that!  

If anyone is working on more stuff like this, I am available to review it before it's cast in stone.