Scripting News

Two-way Search

Sun, 26 Jul 2009 18:41:38 GMT

When I started DaveNet in 1994 I had a bunch of ideas for products that I hoped one day to develop. But I had been waiting so long -- it was becoming apparent that I would never get to develop them. So I wrote emails about them, and sent them to all the people I knew from various industry events. What came back often were ideas that built on them. And eventually some of the products did get built. So the idea of dumping ideas publicly, ones that aren't doing you any good, is solid.

In that spirit, here's another -- I call it Two-way Search.

Here's the idea -- if the search engine knew a little about me, it could give more relevant answers. But it's too much trouble to enter demographic info, and I might not want to share that with the search engine company. But... There's a single piece of data that unlocks a vast trove of preference information -- the address of my weblog. From that it would be obvious that I live in the Bay Area and am involved in tech. So when I ask about New York style pizza, you might include places in Berkeley in the search results. When I search for a driver, I'm probably not looking for someone who drives a car. It goes on and on.

I call it two-way because like most things that show up on the Internet, at first search was a one-way thing. I ask questions, the search engine provides answers. By using information on my weblog to provide context, now data flows both ways.

BTW, it's conceivable that Google knows where my blog is, but I don't think they incorporate that knowledge in search results.

PS: Of course this is the solution to the Suggested Users List problem as well, if the new user has a blog. You don't need to know anything but the address of the blog to make intelligent non-random recommendations of people to follow. For one, you'd know what language the user speaks, so you wouldn't recommend 20 English-speaking celebs if they only know Portuguese. For example.

rssCloud design issue

Sat, 25 Jul 2009 18:19:52 GMT

Bryan Field-Elliot raises an interesting question in the comments on the rssCloud walkthrough.

"The automatic expiration after 25 hours seems rather arbitrary. If a cloud server is going to have a policy of expiring subscriptions after X hours, I suggest that the value X be published somewhere. Within the <cloud> element perhaps, or, as part of the return value from the pleaseNotify call. That way different implementations of cloud servers can vary this value as they feel appropriate, and different aggregators will have a means to know how often they need to resubscribe."

I responded as follows (and reposted here because I wanted to make sure it gets proper consideration):

It's an interesting question.

On the pro side -- it would add flexibility.

On the con side -- it would also add complexity. Another thing to test, another thing to break.

The subscriber should poll anyway, periodically. If it detects a change that it wasn't notified of, it can resub. No harm if already subbed. In my implementation of rssCloud that gives you another 25 hours.

I'm not sure I shouldn't give you another 25 hours anytime I detect that you're alive, for example, you respond to a notification.

The point is to give the cloud a rule for when it's okay to clean out garbage. It's really hard to imagine why someone would want a different value.

Loosely-coupled 140-char reading lists

Sat, 25 Jul 2009 07:00:00 GMT

I believe we will get beyond Twitter's very simplistic and limited Suggested User List, which I have written about so many times.

What sealed it for me was reading Glenn Greenwald's piece in Salon about Cheney's plan to use US troops inside the US.

Here was my thought process:

1. I'm going to pass this link on to my readers.

2. A very small number compared to the overall size of the Twitter base.

3. There are probably a few hundred thousand people who use Twitter who would want to know about this story, maybe 60K who would read it, as I did.

4. So how will this gap be filled? How will they find out about this story?

Well -- they won't.

But -- if there was a service they could subscribe to that alerted them to stories that would be of interest to them, based on their profile, a lot of people would give it a try.

In other words, I'm sure there's a place for editorial products delivered via the loosely-coupled 140-character network.

I tried this with NewsJunk, but it was either too early or there was something wrong with the way we did it.

It will happen.

Meetup in NYC, next Thurs

Fri, 24 Jul 2009 16:50:47 GMT

When I have new tech to talk about I love to do what I call a "roadshow." I take the idea to some other city and have a meetup with anyone who's interested. Last time, when I lived in Florida, I did meetups in NYC, Toronto and Berkeley.

Now with rssCloud beginning a bootstrap, it's time for another roadshow, this one in NYC, at the same place as the 2005 meeting.

Next Thursday, July 30 at 6:30PM at the offices of Ritchie Capital, 747 Third Ave at 46th, 38th floor. We have a big conference room with a midtown view and a rooftop patio, the meetup should go for a couple of hours, then if people want to go out for a bite, let's go.

Already have confirmation that some of the most interesting people from NY tech will be there. If you know you'll be there, please post a comment below.

BTW, thanks to Thane Ritchie for allowing us the space. The investment world has been through some hard times, as has tech, and it's great to see them still out there. His latest interest, I hear, is urban farming.

rssCloud developments

Fri, 24 Jul 2009 16:00:39 GMT

Yesterday I posted the Implementor's Guide. It walks a coder through the components of rssCloud -- the authoring tool, cloud, and aggregator.

I also released the source code for my implementation of the cloud. My aggregator, River2, was updated to support the cloud. I have not released my authoring tool yet.

I started a FriendFeed group to facilitate communication among implementors.

A feed that will help you test, if you're developing an aggregator. It updates every 15 minutes, notifies the cloud server, which in turn will notify any registered subscribers.

My personal "LifeLiner" feed is also available for testing. I wouldn't recommend subscribing in a feed reader like NetNewsWire or Google Reader. It's designed for the loosely coupled 140-character network. For which there aren't any readers. Yet! (Gotta start somewhere..)

Meetup in NYC next Thurs.

Anil Dash has written an excellent white paper on recent developments in the realtime or "pushbutton" web, including of course rssCloud.

rssCloud isn't "recent" --> it was defined in January 2001.

My two cents on Gates

Sat, 25 Jul 2009 00:01:30 GMT

I just posted a tweet saying I don't view the Gates matter through a racial lens, I view it through a Harvard lens. I want to explain, and I'll try to do it briefly.

First, I've had my share of run-ins with cops. When I was young, I liked drugs and street politics, and that meant lots of confrontations with NY cops. Never got arrested because I was lucky and because I was fully submissive when stopped by cops. They scared the shit out of me. I knew they had lethal force, and they must teach them how to be terrifying, because they are. Very.

Later, as a college student, I'd occasionally get drunk (in New Orleans, everyone does it) and once even got picked up and put in jail for a night to sober up. No record, but I'll never forget it.

Hitch-hiking in Calif, I got stopped by police in Tracy, and was fined exactly the amount I had in my pocket. I paid it, and wondered where the money went, but I didn't make a stink. I just left and never went back.

I once had two cop cars come to my house in Woodside, and I had to prove that I lived there. I did. Kept my mouth shut except to say "Here" as I handed over my driver's license. I owned the place. Didn't matter. Until they left they owned my ass.

A couple of years ago driving from Calif to Denver, I got stopped in Winnemucca. I was speeding. Really speeding. Stupidly speeding. I accepted my ticket graciously. Drove slowly the rest of the way and got stopped in every major town in Nevada. Never expressed any irritation. Just begged for them to let me leave the state. Promised never to return.

I say all this to point out that I have some experience with cops, and I'm white, and I would never in a million years think of yelling at a cop. Never have, and if I ever do, I expect to be arrested.

Now Gates says he didn't yell at the cop. I don't believe him. Too many other people who were there say he did. I'm pretty sure when you're yelling at a cop who's doing his job he's supposed to arrest you for disorderly conduct. I think that's more or less what disorderly conduct is.

I also think yelling at a cop is stupid. He's got a gun. If it's so much worse being black with cops (and I believe it is) you'd think blacks would be 100 times more careful about it than a white guy.

Now, the second part of the story...

In addition to being hassled by cops, sometimes deservedly and other times not, I also spent 1.5 years at Harvard as a research fellow. I was not at the level of Gates, but I had an office in Harvard Yard and a very nice ID card that got me into all kinds of great places. Being in Harvard gives you an Ivy League feeling, you're one of the Special people. It's very nice.

Imho what we're seeing here is not black outrage, but Harvard outrage. As a piece in Salon pointed out today, if he were anyone else, white or black, no one would have cared, and he probably wouldn't have been so vocal in his rage. What he's saying over and over is "Hey I'm a tenured Harvard professor. I just got back from China where I was on a PBS show. I'm a big dude. You don't treat me this way."

But I'd like to say to the Harvard prof what the Salon guy said to him. Shut up Prof Gates. You're just like the rest of us. When a cop gives you an order, you do what he says. If you have a beef with it, that comes later. And let your lawyer speak for you, and be sure you're right.

Another way RSS works

Sat, 25 Jul 2009 01:27:36 GMT

JY Stervinou had a great idea today.

He started by posing a question...

"If a decentralized microblog is represented by an RSS feed, how to represent the microblog avatar?"

Ahhhhh...

There is already a tag for this.

I added an <image> to both of my test feeds.

Great work JY!!

Implementor's guide to rssCloud

Thu, 23 Jul 2009 15:02:40 GMT

I have a cloud server running now.

I have feeds that are connected with the cloud and more on the way. I have an aggregator that is wired into the cloud.

Now, if you're a developer of a Twitter authoring tool or want to start a small community of your own (emphasis on small), it's time to start at least thinking about your role in the bootstrap.

Here's the new implementor's guide, first cut:

http://rsscloud.org/walkthrough.html

We need implementations in all environments. My code runs in the OPML Editor. We need everything else.

Dumb XML Question answered

Thu, 23 Jul 2009 18:07:41 GMT

I just stumbled on an answer to Tuesday's Dumb XML Question.

If you recall, I wanted to provide a way for users to view the XML of a feed that contains a <cloud> element. I didn't want them to have to do anything like View Source, for a simple reason. I wanted to make it one-click to refresh, so you could quickly see the effect of a change on the XML. I really missed this from the days before the browser vendors hacked up the viewing of RSS in the browser.

Here's what I did...

1. Add a button to the LifeLiner editing window called View.

2. When the user clicks it, I read the feed XML from the server.

3. Write it out to the local file system with the name preview.xml. Turns out the extension is significant.

4. Send the OS an openDocument message.

5. Voila -- you see the XML as God intended.

Let's hope they don't "fix" this. :-)

What worked for HBO won't work for news

Thu, 23 Jul 2009 13:53:50 GMT

First a couple of upfronts.

1. I love The Wire. It's the best TV series ever. I've paid for it twice, once on HBO and once on DVD. So I not only believe in paying for content I love, I practice it. Redundantly!

2. I'm drinking coffee. There's no spittle in the corner of my mouth. Writing about something I've spent my whole career working on and thinking about. And I'm no kid. I'm five years older than David Simon, former Baltimore Sun journo and co-writer of The Wire.

Simon wrote a remarkable piece in the Columbia Journalism Review, saying that the NY Times and Washington Post must charge for their work the way HBO charges for shows like The Wire.

With all due respect, putting up a "pay wall" is exactly what these organizations don't need. They need to decentralize, get further out into the world, not hole-up behind a wall and try to tough it out.

What worked for HBO won't work for the news because HBO is ficition, and news is not. You can take years writing and developing a story on HBO, polish it, cut out parts that don't support the plot you've devised, even drop the series in the middle if you lose interest. That doesn't happen with the news. News is happening all the time, on its own schedule, all over the place, including many places you don't have reporters. (Think about the arrest of Prof Gates in his own house in Cambridge, this week. Sounds like something that would happen on The Wire.)

And often, the stories are far more complex than reporters can comprehend. This is something I know many people in the news business disagree with. I just don't think the reporter model is working. All it does is inflate the self-importance of these people, turn them into gatekeepers, and often bullies. People who behave like the power brokers they're supposedly covering, when they're forced into playing footsie with them if they want access. Usually under the table but sometimes in plain sight.

As a user of news, I'm sure the future is in shortening the distance between the sources and the readers. Yes, there was a time when, if you wanted to get a story on the wire you had to call a reporter. But that's less true every year, as new channels of news have developed, channels that the NYT (not so much the Post) are just starting to participate in. Watch out as that develops, because it's a potent combination. News people immersed in a sea of news makers. I don't know what news will look like coming out the other end, but it won't look like the system Simon and I grew up in.

This must run its course. The idea of putting up a paywall will just force more reporters outside of it if they want to do their jobs and shrink the publications further. It's no solution.

PS: I don't believe in citizen journalism, which has amateurs playing the role of reporters. I think news is being refactored, unbundled -- broken into components, much the same as other aggregators like travel agents and stock brokers. Music A&R people. I believe in the Sources Go Direct model, the disintermediation of journalism. I also think there's a need for aggregation, but it's a practice people like Simon often mock. In fact reporters base their work on generous people who contribute their knowledge for free -- sources.

One of our heroes dies

Wed, 22 Jul 2009 19:03:25 GMT

MSNBC: "Gidget, the Chihuahua best known for her Taco Bell ad campaign, died from a stroke on Tuesday night at age 15."

5/6/98: "The dog is cool, and Taco Bell owns [her], for a while. Then some ad guy at some agency realizes that he could get a dog too and that dog could eat dog food and like all dogs that we love, the dog farts. Yay!"

Snappy retarded answers

Wed, 22 Jul 2009 18:44:46 GMT

Joseph Smarr asked if I had a snappy answer to why rssCloud is better than The Leading Brand.

I said I do have snappy answers, but like all such answers, they are retarded.

But I gave him a list anyway.

1. Google sux.

Or

2. Feedburner sux.

Or

3. I love RSS, they haven't heard of it.

4. Simple is better.

5. Trade one Big centralized server owned by the tech industry for... another one? You must be kidding.

6. Let's have fun again!

I don't mean any of these things. It's the tech industry way of explaining why the BigCo won't crush your or eat your lunch, or worse, crush you and eat your lunch. In all my years in the tech biz, the only times I've seen the Bigs ever actually crush anyone was when the crushee bought into the crushing.

I don't think Google will crush RSS, any more than TechCrunch will. If they try they're:

1. Assholes.

2. Idiots.

3. Losers.

So they won't try.

Evan Williams vs the Internet

Tue, 21 Jul 2009 16:28:28 GMT

In October 1994, at the dawn of blogging, I wrote a piece that actually shook the software world. At the time, the idea of a mere software developer expressing an opinion in public, unedited, in his own words, without the help of a major publication, was unheard of. It had never happened.

The piece was called Bill Gates vs the Internet. The thought was pretty simple. The tech industry was mired and exhausted. Too many BigCo's struggling to be the one who controls the future. As if a company could control the future. But the headlines in the business press encouraged them to think this way. Much as the leading tech blogs encourage Schmidt, Zuckerberg and Williams today to think of themselves as masters of the universe. They aren't and it's a losing strategy today as it was 15 years ago.

The problem for Bill Gates in 1994, the newly crowned King of Tech, was the Platform Without a Platform Vendor, the Internet. The difference between the Internet platform and the Microsoft platform was this: No Microsoft. No one to hold on to the family jewels. No one to put a developer out of business if they personally offended Bill. No one to keep the personalities of developers under control. No one to cut off their air supply.

In 1994, there was a revolution brewing. Bill didn't believe. But it happened anyway, even though he struggled mightily against it.

Blogging is one of the things that came out of this revolution, and along with it archives. So I can point to a piece I wrote in 1998 and it's still there. It was systematized, in software. This idea didn't come from a BigCo, and it didn't get killed by one. The free Internet solves problems pretty well. BigCo's don't solve problems.

So now instead of Bill Gates it's Evan Williams.

I read the piece on TechCrunch and thought it sounds like the transcripts of conversations from Microsoft in the mid-90s. Both were trying to compete with the Internet. Ev's problem is how is he going to keep his key engineers from defecting to the competition. How are they going to let developers use the "firehose" without using it to kill TwitterCorp. These are problems the Internet doesn't have. It doesn't employ any engineers, and when they leave one company to work for another they still work for the Internet. On the Internet no company owns all the data, so no one can control it. If you don't like the way a service works, use another.

The tech industry keeps having this argument with the Internet. It keeps thinking "this time we gotcha" but nahh, the Internet keeps right on going.

Moral of the story: If you find yourself in competition with the Internet, you should find a way out. Imho.

Frontier bug

Tue, 21 Jul 2009 19:25:47 GMT

Wanted to record this to be sure I get back to it some point.

Problem is with all HTTP requests emanating from tcp.httpClient.

Setting the timeout has no effect when its not possible to open a connection on the server because tcp.openStream doesn't take a timeout parameter. It's always 20 seconds, near as I can tell.

Once I get back into the C source again I'll have to check this out.

You can see the problem on the log page for rssCloud -- when testing the link back from a remote app registering a handler, the timeout is never less than 20 seconds, even when it's unreachable. I have the timeout param in tcp.httpClient set at 180 ticks (3 seconds), which is plenty to find out if there's anyone at the other end.

Dumb XML question

Tue, 21 Jul 2009 18:33:08 GMT

0. I know about View Source.

1. I use Firefox.

2. When I view my site's RSS feed I want to see the XML, not a stylesheet rendering of the XML.

3. To be clear, I want to see the actual XML.

4. Is there some way to force the browser to do this?

5. Please no lectures on how this isn't the way it's supposed to work. TIA.

6. I know about View Source.

7. I know about View Source.

Update: This post seems to explain what's going on. Most browsers do funny stuff with RSS.

Speedbumps and a city's carbon footprint?

Mon, 20 Jul 2009 19:54:22 GMT

We have a mail list for the back channel at InBerkeley.com, and from time to time a question comes up that requires research. If the question is interesting, my first impulse is: Write It Up!

Now, this is the result of 12-plus years as a blogger. I know my community loves interesting questions, and we have an informal approach on Scripting News that I'd like to port to InBerkeley.com.

So, in that spirit -- here's a question posed by my colleague Mark Haas.

Do Berkeley's infamous speed bumps, traffic diverters and other traffic-related policies, like politically-motivated, too-low speed limits raise the city's carbon footprint?

We just need a qualified author. Anyone know any traffic engineers, or perhaps someone at the UC Berkeley Institute for Transportation Studies? Other experts?

Walter Cronkite's 'Cosmic Disaster' editorial

Mon, 20 Jul 2009 14:54:54 GMT

In this week's Rebooting The News podcast, I chose Walter Cronkite as our inspiration of the week.

What a 140-char message looks like in RSS

Sun, 19 Jul 2009 17:37:57 GMT

Here's a screen shot to contemplate.

There are two 140-character messages. Each illustrates features of the new shipwreck I hope to sink, to create a new coral reef for Twitter-like systems to grow on and around.

The first three items in each message are fairly obvious:

<description> holds the 140-character text.

<pubDate> is the timestamp, when the message was created.

<guid> is the identifier for the message, so a reader can tell if they've seen it before.

This makes it possible for the messages to be edited after publication, a common feature requests from writers using Twitter.

After that come optional elements.

<category> works like tags in apps like Flickr or YouTube. You specify them in a dialog, blanks separate them, you can create tags with blanks by putting them inside quotes.

<link> is used to point to web pages. No need to shorten the URLs because they don't take up space in the 140 characters.

<enclosure> is how you attach media objects to messages. Again, no need to shorten the URLs. And since the clients know the media type, they can show a preview, or embed a player.

These all use well-understood elements of RSS 2.0. Nothing new needed to be invented.

Craigslist is progress

Sun, 19 Jul 2009 20:41:10 GMT

I don't think I've ever written about Craigslist here.

Probably because I don't spend much time thinking about it, or worrying about it. But I know that some people do, for example Terry Gross, the host of NPR's Fresh Air. It comes up when people talk about the Internet destroying things that matter, like the classified ads in newspapers. At one point in an interview with Wired editor Chris Anderson she asks, in a bewildered way, what happened. She was saying it was a shame that Craigslist comes along and does what the newspapers were doing, for a fraction of the cost, employing a small fraction of the people who used to support the classified ads in newspapers.

I'm not surprised, and if you think about it, it's very predictable. It's called productivity, and it's what new technology is supposed to do. We used to employ 20 percent of the workforce in agriculture, now it's just 2 percent. That's because of technology. You may say it's bad, but there's also less hunger in the US now than there was then. And there probably are far more classified ads today, now that they're mostly free, than there were when they cost money.

It's productivity. It basically a good thing. And as long as we invest in progress it's inevitable.

Here's an MP3 of the segment quoted above.

Another test post

Sun, 19 Jul 2009 17:09:32 GMT

It's amazing how much discussion these test posts get over on FriendFeed.

I have to do them because my app needs something new to ping about to see if the apps that are subscribed to this get the updates, in real-time of course, via RSS.