A Bright Future for Syndication

Friday, September 3, 1999 by Dave Winer.

Zig-zag  

The thing that's so wonderful about the web is that it's so easy to put up a website. And the thing that's so frustrating about the web is that it can be hard to get people to come look at your stuff. 

Concentrators  

That's why, in the early days of the web, so much attention was placed on "concentrators", or web sites that sucked in the web, and then directed people to the places they wanted to go. 

There are two basic forms of concentration -- search engines like AltaVista and directories like Yahoo. Directories provide a structure for the web, and search engines suck the web into a soup. One is like the table of contents in front of a book and the other is like the index in the back of the book, but neither is exactly like their print counterpart.  

The concentrators are far from ideal because the search engines don't pick up new stuff quickly and only cover a small portion of the web; and the directories must be edited by human beings, who often lack what usability expert Jakob Nielsen calls "domain expertise" or even an interesting opinion. Further, directories rarely address the issues of cross-referencing, offering a single hierarchy and listing a site only once. And it can be difficult to get a site listed at Yahoo. I suspect you have to pay them money, but where is this discussed or disclosed on the Yahoo site? 

These are the thorny issues that web entrepreneurs and technologists spend a lot of time thinking about. There have been a wide variety of attempts to tweak and recombine the search engine and directory formula, including Alexa, About, Ask, Deja, Google, GoTo, Netscape, Third Voice, to name just a few.There are dozens if not hundreds of well capitalized companies trying to nail this one -- to find the formula that makes the web coalesce around a single site. Some have had stunning success but most have struggled to find an audience.  

I think the smartest business strategy is to embrace the distributed nature of the web, and build businesses that profit from this. Otherwise it's like fighting City Hall, swimming upstream, or competing with Microsoft. Longterm you end up in jail, being eaten, or testifying in Washington. 

It's the old win-win thing. ;-> 

Why concentrators suck  

All concentrators have one thing in common, they like to point in, not out. Economically they are incentivized to capture flow, not distribute it. It's against their interest to point outside of their sites, but guess what, that's their whole purpose! So they're at odds with their purpose? My philosophical brother Jakob Nielsen calls this the "dark side of the web" and I agree! 

Here's a paragraph Jakob wrote when I asked what he thought of my website: "The Scripting News homepage helps fight the Dark Side in the struggle for the soul of the Web. Many other sites try to entrap users inside their proprietary content and rarely link to the outside world: if this continues, the Web will stop being a web and turn into a set of disconnected medieval fortresses, each behind its own moat." 

Feeding the concentrators  

Dark side or not, the concentrators have grown enormously, at least in stock value, so much so that there's a software and services company, Vignette, that has grown to serve them. Initially Vignette offered a content management system, StoryServer, that's somewhat like the software my company sells, Frontier. They helped large content sites like News.Com, ZDNet, Pathfinder, build systems to pour lots of stories out to the web. But the portals, Yahoo, Excite, etc, generally cooked their own content management software. And they bought stories from sources that didn't use Vignette, sources like AP and Reuters. 

So Vignette went the next step, and took the lead in what was then a new technology for the web, syndication. Last year they released a specification thru the W3C called ICE. With an ICE-compatible server, ZDNet could now publish their content, without ads, and sell republishing rights to portals. It was a clever business for Vignette, because it furthered dependence on their software, and if it caught on, it would create new customers for their software, the portals. 

Focus on the flow here. A story begins its life on the desktop of an editor. It enters the StoryServer system. It appears on the home page of the site that the editor works for. And shortly after that it appears on a dozen other sites, repurposed, with different ads and different links, but otherwise the story is exactly the same. 

Now focus on the cash flow. The portal puts an ad on the page. They collect revenue for the ad. A percentage of the revenue flows back to the company that the editor works for. Money flows from the source of the hit to the source of the story. 

What if value flowed the other way?  

My company, UserLand, has been working with Netscape and others on a reversal of the Vignette philosophy. Instead of being a flow concentrator, we propose to be flow distributors, with value flowing in the opposite direction, from the source of the content to the source of the click. 

Our aggregation engine, My.UserLand.Com, gathers new links from the web and flows them for free to anyone who wants to pick them up. We are working on open source "affiliate" software that makes it easy for anyone who operates a web site to tap into this flow, offering a wide variety of news sources to readers. Everything you need to participate in the syndication flow will be free and open source. It's designed to run on the LANs behind the firewalls at all the major web sites, on all flavors of Unix and eventually on NT and Mac OS. It'll be tough competition for Vignette. 

Focus on the flow. A story begins its life on the desktop of an editor. It appears on the site the editor works for, linked into the home page, and it is also linked to from the XML version of the home page. The aggregation engine reads the file every hour, notices that there's a new story, and pumps it out to all subscribing affiliates. It appears on the home page of an affiliate. A reader clicks on the link, opening the page on the editor's site. A payment is generated, from the content source, the editor's site, to the source of the click.  

The more you pay for a click, the more incentive there is for people to link to you. It's a way of turning cash into flow. You're betting that a visitor that comes to you this way will like what they see and become a regular visitor. You're betting on the stickyness of your template, basically. 

In our system, each story has a *single* location, the site where it originated. We think this is the way the web was meant to work. Stories can live and grow while new information is obtained. Comments from readers can add new facts and ideas and link to other related stories. And the portal sites, the ones with the huge flow, can play a big role, because in this model, they get paid for many (but not all) of the hits they deliver. It's a micro-payment form of what they already do so well on a much larger scale. 

In fact Excite, one of the more innovative portals, already has such a program in place, affiliate.excite.com. They pay sites that deliver flow in "award points". We'd like to see flow delivery sites to be paid in real currency, because people who run sites that direct flow, also known as weblogs, need to buy groceries, and for that you need cash. ;-> ">

The role of content management  

Regardless of whether our approach is better than Vignette's, we're incentivized to make it work. Vignette sells a high-priced content management system for big publishing organizations. UserLand makes content management software aimed at a larger market, with an under-$1000 pricetag, for everything, not just to get started. (Vignette's price isn't publicly posted, but installations commonly go for between $50K and $2 million.) 

Content management is key to the next architecture of the web, on both sides of the syndication connection. If processes aren't automated, the job quickly becomes too complex for human beings to manage. However, if you have stories flowing thru content management, you can hook in payment systems in lots of different places, and the web can grow economically, without sacrificing the thing that makes the web unique -- a low cost of entry into the publishing business. 

Dave Winer  

PS: ICE is an acronym for Information and Content Exchange.