NewsSearch.UserLand.Com
Monday, March 15, 1999 by Dave Winer.
A big news week 
This is already turning out to be a big news week. Today Netscape announced a new syndication format called RSS. Also today, we are rolling out a new service, a special-interest search engine. Tomorrow Apple is announcing something related to operating systems and on Thursday Microsoft is shipping Internet Explorer 5.0 for Windows and various flavors of Unix.
Netscape's announcement
Earlier today Netscape announced a new low-tech format for syndicating web content called RSS, for RDF Site Specification. It's a more realistic spec than the one proposed by Microsoft several years ago, called CDF, or Channel Definition Format, because it reflects the dynamic nature of news-oriented sites.
It's not as far-reaching or complex as the ICE spec promoted by Vignette. It's most comparable to the scriptingNews format we've been promoting since late 1997, and the "ultramode.txt" format being implemented by Slashdot.org, and presumably other sites.
We are supporting RSS with our content. Over the weekend we put together a script (of course!) that generates a Netscape-compatible rendering of our site.
http://news.userland.com/scriptingNewsToRss.xml
Compare this to scriptingNews format, which includes more contextual information:
http://news.userland.com/mostRecentScriptingNews.xml
Congratulations to Netscape for getting this out. There are still improvements we'd like to see in RSS, they should be obvious in this comparison, and we've communicated them to the people working on this stuff at Netscape.
Further, we'd like to see how people are using Netscape's format. If you have a RSS file on your server, please send me a link so we can have a look. Thanks!
Specialized search engines
These syndication formats are mildly interesting as ways of flowing my content thru your website, but where this gets really exciting is when you use this information to seed the contents of a search engine. Did I fail to mention that except for Slashdot's format, all the syndication formats are XML-based? They are!
Like many others, we felt that when critical mass was achieved, there would be an opportunity to build an infinite number of XML-based special-subject search engines. Now we're there and today we're announcing the first such search engine, one that indexes three sites that support scriptingNews format:
http://newssearch.userland.com/
A search engine with an editorial voice
This search engine is different from any you've used before. Think of it as a search engine with an editorial voice, the combined voices of Lawrence Lee who edits Tomalak's Realm, Wes Felter of Hack-The-Planet, and mine, wearing the hat of editor of the Scripting News website.
Every night between midnight and 1AM Pacific, our crawler reads the news for the previous day on each of the three sites, and breaks it up into individual bits. Each bit is separately indexed. Usually the bit also links to an article on another site. We index that as well, but our crawler stops there. That's where the editorial voice comes in. If Wes, Lawrence or I thought it was worth presenting to our readers, it makes it into the search engine.
What do these three sites have in common? We all cover the web for web enthusiasts, people who develop for, invest in, or make decisions regarding the web. We have different perspectives, but when you put the three of them together, as we have in the search engine, you get a good cross-section of what's happening in web technology, marketing and politics.
Vertical search engines
This raises the next question. What about the search engine for intellectual property attorneys? Or the search engine for high school math teachers? Or the search engine for people who love Grateful Dead music?
Now that we have learned how to cover web developers and investors, we're ready to start branching out and provide tools and server software, editorial tools and web hosting for people whose expertise lies outside the web. We think this is a multi-billion-dollar opportunity, so the vision is keeping us *very* busy!
We're hitting milestones on a daily basis, our editorial and indexing software is easly deployed on other people's servers, and the workstation software runs on Windows and Mac, which is where the people with the knowledge work. Whew. Now I have to get back to work on my software..
Still diggin!