Sitemaps, day 3
Friday, April 13, 2007 by Dave Winer.
The idea was that the content server was responsible for providing a daily reverse-chronologic list of pages that had changed. Then a crawler would keep track of when it had last visited my site, and only suck down the files that had changed since then. This would enable search engines to be more efficient, and provide more current content. It was nice because you could read it yourself and see what had changed. Contrast this with sitemaps, where you have to go hunting for the changes, it's no better a user interface for finding the new and newly updated stuff than the file sytstem is. I was kind of disappointed.
Another thing I would have done differently is allowed sitemaps to include other sitemaps. There really is no need for two file types, just let me link to an index from an index, much like inclusion in OPML 2.0. This added an extra layer of complexity for everyone implmenting sitemaps on moderately large sites, or old ones where some content changes frequently and other content not so frequently (like scripting.com).