Scripting News

How to rank real-time search

Wed, 28 Oct 2009 17:29:51 GMT

Eric Schmidt says he can search real-time stuff, but how to do ranking?

Good question. Would have been easier had Twitter not polluted the follower-count measure of authority. But you can still do it by making it relevant on a personal level. Someone I follow is a lot more relevant than someone I don't. After that people who are followed by people I follow. That immediately cuts down the power of the super-elites with millions of followers (they tend not to follow many).

Google is onto it with their social search. I've been asking for that, but in a different form. I want to tell them that I'm the author of this blog. Now they know a lot more about what my interests are.

7/26/09: Two-way search.

So it would be nice if ranking were a personal thing. Keep going the way you're going Eric.

Twitter feeds stray puppies

Wed, 28 Oct 2009 16:15:40 GMT

A bunch of random notes on returing from the #140conf in Los Angeles.

Next time I go to a conference I'm taking the Asus, not the MacBook. You're always looking for a power outlet with the Mac and that sucks. I'm also going to actively look for a replacement for the Sprint MiFi and my AT&T iPhone, both of which have terribly spotty coverage. I couldn't get online in LAX last night, even though my iPhone can tether and I had the Sprint. If you can't get online in one of the largest airports in the world, what is the point of carrying this thing with you. SFO wasn't much better. And I couldn't use either of them in my hotel room in the biggest hotel in Hollywood in the middle of a shopping mall and convention center. These are places that by now these cell providers should have the best coverage in. The question is -- is Verizon any better?

Great Rebooting The News with Jeff Jarvis as the guest. Lovely rapport betw Jay and Jeff.

At the conference yesterday they explained the vague announcement made by YCombinator and Twitter over the weekend -- YCombinator startups will have access to Twitter's firehose. The audience heard "startups get help from Twitter" which they reacted to as if they said "Twitter feeds stray puppies." I hate to spoil the party, but not all speculative investigations are done by "entrepreneurs" -- and not all entrepreneurs are part of YCombinator. This is just more of the lunacy that comes from building an industry around a company instead of an open format or protocol. Paul Graham hypes it as Twitter having discovered a protocol like SMTP or HTTP. That's pure bunk. When there's a protocol, no one will own the firehose, and no one will be granted access (and no one will not be granted access).

I'm continuing to love my linkblog. I've gotten nothing but complaints from readers. Eventually you all will love it too. I'm sure of it. In the meantime, my work is 100 percent more valuable to me, and my incentive to remember a link by pushing it through Twitter (and my linkblog) is greater than ever. So I'll do more work for you, you'll be better informed, and happier and more productive. I can't promise you'll live longer. We'll feed some stray puppies too.

Reminder to subscribe to this feed not the one that WordPress provides. (Note to Matt and the WP community and company, as I use WP more and more I'm hitting limits we never had in Radio or Manila. You guys should seriously look at stealing some ideas from those products. I'll help you find them, because I'm starting to depend on this software.)

I started to watch the video of my presentation at 140conf, which everyone says went well (it was widely quoted on Twitter, of course). That's good, cause I couldn't stand to watch it because I'm frowning too much because there's a light shining in my eyes. We have to come up with a better way to do this, so it doesn't feel so much like "I'm up here and you're out there." I have to be able to at least abstract the audience when I'm speaking. That's why I much prefer the interview format, because I can talk to another person. It's part of the theme of my talk, we're just people, I'm not Mike Wallace and you're not really an audience. It's a brave new world and we should have the courage to accept it for what it is. And please believe me that I'm smiling as I write this. I wish I had been smiling more while I gave the talk.

And don't forget to feed the stray puppies.

My linkblog is active

Tue, 27 Oct 2009 22:55:48 GMT

If you haven't been following my linkblog, it's time!

http://protoblogger.com/

If you want to follow it, don't subscribe to the wordpress feed, follow this feed.

http://static.lifeliner.org/dave/rss.xml

Dave

Random travel notes

Mon, 26 Oct 2009 22:26:58 GMT

I chose to travel with my newish 13 inch MacBook Pro instead of my newish Asus Eee PC. It's just a one-day trip to LA and I figured I wouldn't be needing the 8-hour battery, but there is a fundamental difference between the two computers. With the MacBook I'm always looking for a power outlet. With the Asus, you know you're going to make it all the way without a charge, so you can relax about power. Apple may think they have the battery issue licked, but they don't. And the fact that you can't carry a spare battery for this computer is a real step backward.

The computer also likes to randomly reboot. It's happened four or five times so far. Just happened a few minutes ago. Luckily I didn't lose any work.

Also the computer just disappears for a minute at random times. Computers have been doing this for 25 years. When will someone make an operating system that's always there for the user, no matter what crazy thing the OS has to do to keep itself running. All the michegas about Macs working better, that's a half-truth and half-lie.

Speaking of lies, the lies caused by the Suggested User List are approaching epicness. CNN ran a piece today that profiles five unknown superstars of Twitter, all with over a million followers. They only mentioned the SUL once, in passing, when they were describing Veronica Belmont. So the myth created by the SUL, that there are superstars and the rest of us, keeps growing. And then you have to wonder how much of a tool the SUL is for Twitter, to keep people in line.

Pierre Omidyar is on the list now, and he wonders how many of his 99K followers have any idea who he is. He has Fuck You Money so there's no way he's controlled. But Anil Dash is now on the list too and has 99K followers, and he's a working man, and I'm sure he can be influenced. I unfollowed Anil when he made a joke about how it feels like being on the Yankees. Exactly. That's what I dislike intensely about the Yankees. Their sense of entitlement. Maybe not so much by the players, but by the fans. Twitter is like blogging, it's best when it's just people. The people with millions of unearned followers must be uncomfortable, wondering when the millions are going to catch on.

Is 20 people enough to get started with? That's what a new user gets by default. I seriously doubt it. My Berkeley page is just starting to get interesting, and it follows a list of 167 people. And they weren't chosen at random. They all have one thing in common, they're neighbors of mine.

The other day I said I was starting a linkblog. It's now visible at protoblogger.com. I really like the way it feels. I'm using the LifeLiner tool so it's hooked into rssCloud and it publishes through wordpress.com and I can route a link to Twitter with a single click.

The idea of restarting our blogs came up on today's Rebooting The News, with our guest this week, Jeff Jarvis. This is how I think we will restart them. By making websites that carry the kind of content we're flowing through Twitter.

I was wrong the other day about what the BuddyPress theme is for. I'm still confused about the layers of WordPress. I'll figure it out.

Who or what will be the BitTorrent of Realtime?

Sun, 25 Oct 2009 21:17:53 GMT

A market develops, a bunch of people get it started, then someone at a big company discovers it, changes its name (sometimes they don't even do that) and relaunches it as if it were something wholly new. The press, many of whom were aware of the earlier efforts, goes for it. "Everyone knows" that it only matters when a big company does it. However, if you look at history that's often not true, it's often the small guy who ends up defining the market, despite what the press thinks.

A classic case is P2P. Ten years ago there were all kinds of early efforts, some remarkably popular (thinking of Napster) and the industry launched a huge hype balloon. Conferences, white papers, press tours, alliances, books, VC, startups, etc etc. Billions of dollars thrown at it. What ends up taking the prize? BitTorrent! An open source project launched by a bunch of nerds, without much PR. I don't know if it was the best technology, but it certainly was good enough. It wasn't glitzy or even particularly easy to to use. It worked, and most important, you could get the music and movies and TV shows you wanted.

It's a good bet that in five years we'll look back and most of the companies staking out realtime today will be forgotten and something like BitTorrent will rule this space, gently of course.

In one way, losing a father is a relief

Sun, 25 Oct 2009 18:14:49 GMT

I've been amusing myself with illogic this last week, at times giggling with the relief I feel at the passing of my father. I finally found a way to explain it in words.

When I was a kid, like every other kid, when the parents went out and left me alone, I'd do stuff that I wasn't normally allowed to do. But I had to be sure to cover my tracks so my mischief wouldn't be discovered. The lessons learned from the failures were incorporated into my future exploits, I'd never get caught the same way twice!

So all through my life I've been preparing for my father to come home and catch me doing whatever it is is I'm doing. Whether I was aware of it or not, I was always covering my tracks. My subconscious can't get rid of this. It's a program I'll be running forever. But now it has a different ending. He's never coming home.

Good API design at Twitter

Sun, 25 Oct 2009 17:49:58 GMT

I've been putting off programming with Twitter lists, but I shouldn't have.

They did a really good job, as usual, with the API.

The project -- convert the page of Berkeley people's tweets to run off a list instead of a special Twitter account. It turns out there's an API call that retrieves the timeline for a list, and it works exactly like the API call that retrieves the timeline for an account. So much so that I didn't even have to change the glue script, I pass in a different URL and it just worked.

It Just Worked™.

That's the Holy Grail of APIs.

I am happy to criticize Twitter when I think they made a mistake, and I'm even happier to applaud when something works incredibly simply, and well.

Thanks!!

My buddy Rex

Sat, 24 Oct 2009 22:20:45 GMT

These days my blog posts are always essays, but it wasn't always so. In the beginning they were all links, with pointers to articles both on this site and off-site. Example.

Then it became a hybrid, at the top of the page were the links and at the bottom were articles. Example.

Then in early 2007 (Jan 6, to be exact) I went all essays, and then a few weeks after that, started using Twitter. It's funny how one event followed another.

Anyway, this article by Rex Hammock is so lovely and so vindicating, I'd do a special post just to link to it.

Rex Hammock: Facebook goes River of News.

And one little thing, I'm going to have a linkblog up in the not-too-distant future. Again. Everything is new again, every few years, it seems.

Also it's sad that my friends, people like Rex, have to hedge so much because of a handful of stinkers who follow me around on the web. I'd like to encourage my buddies to just go through it, and say what you want to say and let the stinkers stink up some other place. Life is too short. With much love, Dave.

Testing BuddyPress

Sat, 24 Oct 2009 16:22:52 GMT

When Matt told me that WordPress was going to support rssCloud that got me started using WordPress with new purpose. I've been learning to use the product through wordpress.com. I haven't yet started my own installation. My attention is focused elsewhere.

Anyway, the point of this post is to get help learning how to use BuddyPress. I don't want a huge hosting obligation. Ideally I want a freemium deal like the one at wordpress.com. However, it doesn't seem to exist anywhere, yet.

I just came across a site that says it lets you test BuddyPress.

http://testbp.org/

I was expecting to have to create an account, but it (apparently) found me on Facebook, and I'm already leaving a trail there. Totally not happy about that, but I suppose my gripe is with Facebook, who somehow has decided that they own the web and can give access to my account to anyone who asks for it? I was never asked to opt into this. Unless I'm missing something this seems just plain bad.

Anyway, I thought BuddyPress was supposed to be like Twitter. It doesn't look anything like Twitter. There's no box at the top of the page that asks What Are You Doing? Without that it's not Twitter-like.

Confused.

Update: I found my "wire" page -- and on that page, there's evidence that I had been here on April 30. So that lets Facebook off the hook. I must have created the connection then.

Update: Some free advice for the BP designers. The home page of my site has to look more or less exactly like the home page of the Twitter site. Any difference is going to equal pain for users, and pain for users means slower adoption. Later, when and if you achieve dominant market share, you can slowly evolve the UI, if you really feel you must. Users are less interested in innovation in the UI than you would think they are.

Our all-you-can-eat lifestlye

Sat, 24 Oct 2009 12:44:38 GMT

First thoughts on our San Quentin field trip.

I went for a late lunch in Sausalito with Scoble after spending most of the day inside the walls at San Quentin state prison. We were sitting on a quiet beautiful street with healthy, well-fed people walking by, driving in to eat at the Indian restaurant, riding bicycles and stopping to ask for directions. Scoble entered our location into Foursquare and a few minutes later a clean, friendly young man showed up with an infant wrapped in a blanket. He greeted us with a smile, Scoble instantly knew he was. We didn't in any way at any time feel we were in any danger. I was pretty sure most of the people with us there had never killed anyone.

That may strike you as an odd way to describe a lunch in the center of high-tech land, because that's our normal reality. We expect so much, and we get it. We live the all-you-can-eat lifestyle. But just a few miles away reality is very different.

We met a man who had never used the Internet, had never seen a cell phone, had no clue what Twitter is, and probably a million other things we talk about all the time. He's been in jail since 1987. He talked to us for a while in the courtyard just inside the entrance gate. He's in a "program" and my guess from the way it sounded, will be paroled in January. He murdered his little sister when he was 18. Blew her head off with a shotgun. He did it because she and her brother and mother hid his money and drugs. He told his brother that he'd kill his sister if he didn't tell him where the stash was. The brother said he'd never kill her. He did.

He didn't tell us this, Rudy Luna, the assistant warden who was taking us on a tour, did.

The warden said that, ironically, that prison is a revolving door for people who commit minor crimes, but for murderers like the guy we were talking to, sometimes they get out and stay out. He says there's a point, usually at 11 years, where they realize that they could change. The guys who get sentenced for smaller crimes don't get there.

The guy we were talking to might not commit another murder, but I don't see how he can live with himself.

Everwhere we went we were being watched.

By everyone.

That may have been the oddest thing. I am accustomed to leading what I think is a fairly anonymous life. Sometimes on BART a stranger is staring at me, I imagine they recognize my face from my blog. But most of the time I move around without anyone paying much attention. Not inside the prison.

And it's not just us they're watching, they're all watching each other, all the time. Because prison is a dangerous place. Everything they do seems to be about keeping from getting slashed or beat up or killed.

We saw thousands of people in tiny cages.

We saw the outside of a building where people are locked up all the time, their crimes so heinous or infamous, or they attract so much attention, or they are people who will try to kill anything that they possibly can.

It's the contrast that is so striking. And what it tells me about who I am.

Having just lost my father, I'm thinking about what death means in a much more real and present way these days. Our guide tells us at one point that most of the people we're looking at, and there are hundreds of them, killed someone. And they're walking around like you and me in a park. Except it's nothing like the way we walk around in a park. Everyone is watching everyone. All the time.

I'm sure there will be other insights. Coming out of it, I think none of us knew what to write. That's the sign that we were doing something very different, something very very outside our normal experience.

Is Google/Microsoft/Twitter in the news business?

Thu, 22 Oct 2009 20:05:49 GMT

Yesterday the earth shook, and at first glance you might think it just shook in Silicon Valley, but I think a few years from now we'll look back and realize that the earth was shaking just as hard in the media industry.

I've had this really strong feeling ever since I got enamored of Twitter in 2006 that it was something the news and entertainment world would jump on if it had leadership that was as bright and ambitious as Steve Jobs and Bill Gates at their prime. Alas, those days must be long gone, because they busily tried to litigate peace in the old war, which they lost long ago, looking for the reparations that are due the loser who manages to make the victor feel guilty. (In other words the crumbs left on the table after the meal is over and after the cleaning people have made their first pass.)

"Look over here!" I've said over and over. "You should be competing here."

"We don't see why," they respond.

Here's why.

Twitter got Google and Microsoft to pay for the content that the media industry should have been hosting instead of Twitter. There was money here. And as we all know, the media industry can't find enough money to keep going. They're looking for handouts from the government. Meanwhile there was money everywhere. They just had to evolve.

I'm not saying this payday means Twitter has it made, they don't. Google and Microsoft are sharks and Twitter may be a goldfish. It could be that Evan Williams and his team have the competitive instincts of a Gates or Jobs, if so, they certainly have a few tricks up their sleeves, or they wouldn't have done these deals. They better, because Google and Microsoft are almost surely executing an Embrace & Extend. What that means for Twitter is that they have clones of Twitter in development. The race is certainly on. Have they cross-licensed their streams? In other words, does Twitter have reciprocal rights to any realtime content generated by users of Google's TwitterLand? Microsoft's? Even if they do, could they handle the load? My guess is that both Google and Microsoft will quickly take the search function away from Twitter. Now everyone has the Twitter stream. What streams can Microsoft and Google add to differentiate theirs?

And what business are they in now? I believe they're in the news business. This isn't tech anymore. This is what the Times and CNN should have become, what CBS, NBC and ABC should be. What Jay calls a pro-am system where everyone collaborates to create the realtime stream of news on all levels, national, international, local, broad coverage and specialized stuff. Everything from newsletters to nightly news. Everything flows through the same pipes, and curators pick off the good stuff and route it to people who are interested. This is the way news is done from here-on. We're not talking about the future, we're talking about now. And the moguls of the media industry, without a single leader thinking in advance of the wave, are sitting on the sidelines, hoping someone will give them some money because they're such great writers or whatever it is they're so great at. Soon they'll be looking for reparations.

They should own the platform.

And it's bad for the rest of us that they don't because the moguls of Silicon Valley have a very crude understanding of what news is. Witness the longevity of the 140 character limit and the inability of Twitter to carry any type of content other than text. The horror of the Suggested Users List. I don't expect Google or Microsoft to do much better, but they'll probably have the sense to hire a few news pros to advise them on how to build a system that works for news. The Twitter guys are fumbling around, and in doing so, holding all of us back.

And FriendFeed. Oh man what a wasted opportunity that was. If they had an ounce of competitive spirit they would have noticed that the news industry wasn't seeing their way into this space, and they would have gotten on a plane and camped out in NY and found someone, anyone, with a good flow of news to partner with, to guide them toward creating the fantastic news system that Twitter wasn't building. They had the technical ability to do it, but they were too much of homebodies, they enjoyed the comfort of other engineers too much.

This is what we still have to do -- create the connections between people with technical knowhow and people who can make the news flow to create a safe harbor for the millions who want to participate in news to do so, without being owned and controlled by the titans of tech.

Flickr in hindsight

Wed, 21 Oct 2009 23:11:39 GMT

You know what they say about hindsight being 20-20. It's that way with Flickr, which was way ahead of everyone else in the social network thing. With a few tweaks three years ago, with active entrepreneurial development and the resources of Yahoo, it could have been everything Twitter or Facebook is today.

I don't know enough about what goes on behind the scenes to know if it really was possible. It could be that the code is a mess and that the last engineer who understands it left five years ago. If so, the previous paragraph is probably nonsense. But if there are still some good people working on it, it may not even be too late for Flickr to act as a backbone for at least part of the future realtime web.

The thing that Flickr does that Twitter doesn't is, as I said a few days ago, payloads. In Flickr a payload can be a picture or a video. Another thing Flickr has going for it is a great API with lots of developer support. And while Flickr does go down sometimes, it's a lot more reliable than Twitter. And there's no Suggested User List.

What it's missing is a Twitter-like timeline. But I honestly don't think that's so hard to do. I think they already do the hard stuff. Maybe when Carol Bartz gets over the flu we could meet to talk about giving Flickr a lot of independence and a bunch of cash and letting it be free to compete in wild, free of the constraints of corporate Yahoo. Then, once it's flying, the various Yahoo properties can latch on to its growth as any other developer could.

It could be a terribly bad idea. But I still love Flickr and use it all the time. I even pay them money every year for privilege. I bet a lot of other people do too.

Bruce Sterling at Reboot

Wed, 21 Oct 2009 21:11:41 GMT

Bruce Sterling gave a wonderful talk at the Reboot Conference this summer in Copenhagen. At the beginning of the talk I wanted to strangle him, but as it progressed, it made more and more sense. By the end I thought it was one of the best speeches I'd ever heard, a story that I think everyone should hear. I've made an MP3 of his talk because I want to make it available to people in my family as a podcast. I hope Bruce and the people at Reboot don't mind.

He talks about clearing your life of posessions, how you should divide everything into four categories: 1. Beautiful things. 2. Things with emotional value. 3. Functional things. 4. Everything else.

Divide each category into the things you keep and the things you get rid of. In category 1, you can keep it if it's on display in your house, if you show it to your friends, if you share it. If not, then you don't need it, it's taking up space and time, which you're paying for with your money, time and health. Take a picture, put it on a thumb drive, take it everywhere with you and get rid of the original. In category 2, if it has a compelling story, one that you actually tell people, you can keep it. In category 3, unless it's very good at what it does and it does something you do a lot of, it goes. And of course everything in category 4 goes.

He says you shouldn't try to do this in normal times. Wait until a spouse dies, a divorce, a child is born or a child leaves home. Wait till you move. It pays to figure out now what you want to do when that time comes.

I know Sterling is right because I've had things like that happen and I've done it both ways. Most of the time I don't clean house, and miss the opportunity to improve my life. But sometimes I do make the changes and it's always, in the end, been a good thing. Most people advise you not to make changes in times of great life turmoil. That's exactly the wrong advice. Those are the only times you can make change.

This is a hot topic in my family because of Father's Day. It just happened, and the shock is just now beginning to set in. It's strange that along with the pain and sorrow, there's also a new sense of freedom, of possibilities. It's palpable. And it doesn't take a second to locate the source -- it's the changes Sterling talked about so eloquently.

Anyway, most of the time most of us are not in position to do anything about the mess in our lives. But listen to Sterling's talk. It's only 43 minutes. It might be the best 43 minutes you've ever spent.

How to learn to love the Fail Whale

Wed, 21 Oct 2009 17:23:44 GMT

What if your Twitter client stayed up when Twitter is down?

Believe it or not -- it's possible. I'm sure it's crossed the minds of the people who run Tweetdeck, Tweetie, Brizzly, Seesmic, et al. What if users could keep communicating with people who used the same tool you use and have all your tweets flow out through Twitter when they're back on the air?

Even better, what if Tweetie and Brizzly got together and worked out a way for their users to stay up and communicate with each other, even if they used the other guy's tool? (See where this is going?)

An rssCloud case study: Brizzly & Seesmic.

Most of the time people don't think this way, except when the Fail Whale is showing up, as it has been for the last couple of days. Then creativity kicks in and you start wondering if it's possible, and if it were, what's in the way?

Maybe Twitter wouldn't like it if the client companies got too independent? Maybe they have some way to punish those who stray and reward those who don't? Some people think they do, with the little ads in the right margin of the Twitter web page. They say those ads really work, and if you don't play ball the way Twitter wants you to -- no soup for you!

I know how the Twitter clients can become free from Twitter, yet still work with it. You might have to give up your tasty Twitter soup, but you might be able to find new users you wouldn't otherwise, if word of mouth started carrying the message that your client doesn't go down when Twitter does.

Innovating outside the 140

Tue, 20 Oct 2009 21:45:26 GMT

I pitched a simple idea at yesterday's GigaOm meetup, one that is easy to explain verbally but I've not yet attempted to explain it in writing. So here goes.

When the Mac came out in 1984 it had a file system that worked much like a Unix or PC file system, it was hierarchic, had volumes, folders and files. But files had two "forks:" 1. Data and 2. Resource. The data fork was like a regular file, but the resource fork was really cool and different. It was like a file system within a file, but not quite.

Resources had a type and an id. The type was a four-character string, and the ID was a number. There were standard system types like WIND and MENU, and in them you put designs for windows and menus. There was a resource editor that shipped with the OS that had tools for the standard system types, the menu editor let you add a command to a menu or delete a command. The window editor let you set the default size of the window, its initial title, and what WDEF routine was responsible for drawing it. A big part of learning how to program the Mac was learning what all the resources were and how to set them up. Then you'd write C or Pascal code to open the windows or draw the menus.

But that wasn't all you could do with resources -- because -- and this is the key point -- you could define your own resources. You didn't have to get anyone's permission (okay theoretically you did, but we never bothered). So if I wanted to write a DAVE resource to my file I could. And then I could tell you what a DAVE resource contains and your app could read and write them, and all of a sudden we've just enhanced the platform. Pretty cool! And we did this kind of stuff all the time.

So why shouldn't tweets also have resource forks? Then if I wanted to attach a picture to a tweet I'd just pack it up in a blob and shoot it up to Twitter as part of a PICT resource along with the 140 characters which would then be a description for the picture. Or why not have a menu go with a tweet? Or a bit of HTML? Or whatever the fuck.

This would get Ev and Biz out of the loop, they could just kick back and run a storage system and stop worrying about what features to add to the platform. You see the users get to innovate inside the 140 characters, but there would be so much more action if the developers could innovate outside the 140.

Random Tuesday notes

Tue, 20 Oct 2009 18:39:35 GMT

Good morning. Getting back to work after a half-day at a conference followed by a five-hour baseball game. The conference was at Om Malik's and was the first tech conference I'd been to since the disaster at Gnomedex in 2007. This one was much better. I got to talk with a number of people I'm working with on RSS-related projects, and met a few developers with interesting projects. Om gathers an interesting group, that's for sure. And the mood at Om's place is respectful and collegial. We got some work done. Nice.

I told Om I'd like to try out the flash conference idea at his new 2nd St office. Great location and a good size. So next time there's a rush to get people's ideas on some new tech development maybe we can get together to talk about it at Om's place.

The baseball game was the third in the ALCS between the Angels and the Yankees. I'm really liking the way the Angels are playing, and of course I'm always up for rooting against the other New York team.

There will be more baseball and I'm speaking at Jeff Pulver's 140 character conference next Tuesday, a week from today. It's in Los Angeles. I can invite the regulars at Scripting News as my guest, so if you'd like to come, send me an email or post a comment, and I'll send you a link that gets you free admission. Jeff was very kind to let us party on his dime in LA.

Meanwhile, two really interesting articles you all should read:

1. Wired: How Users Took Over Twitter.

2. Mediate: We Keep Finding Historically Significant Photos. Will Our Grandchildren?

The latter question was the punchline of yesterday's Rebooting The News podcast, which has yet to appear in the feed. The irony is that we could be doing a better job at archiving our thoughtstream, but we're actually doing a worse job. Our pictures, movies, recordings, thoughts have never been more ephemeral.

Yesterday at Om's we wasted (imho) a time talking about seredipity. The time could have been better used working on more mundane topics like maintaining a memory that lasts more than a year at a time. We love the latest and greatest stuff, but don't recognize the patterns, we've seen this before. It's like the recurring theme in BSG, it's happened before. But we threw out the archive! Oy gevilt.

I refuse to solve the problem only for Scripting News, because I don't want future generations to think I was the only one writing in the early part of the 21st century. One of our bloggers is the next Ernest Hemingway or William Faulkner, Willa Cather or Emily Dickinson. Let's make sure we have their earliest emails, Flickr-ings and tweets.

BTW, this guy at Yahoo is surprisingly funny. I didn't know they were allowed to hire people with a sense of humor. Hate to say it, but I've never seen anyone at a big tech company, Microsoft, Google, Apple, Facebook or whatever that had a sense of humor. I have an idea, they should make being funny one of the requirements to work at the Bigs. Who knows the software might be more er uhhh you know -- fun.

The Internet abhors a funnel

Sat, 17 Oct 2009 18:19:30 GMT

This might be one of those fundamental Internet laws, or it may just turn out to be a way to express another one.

The Internet is a bunch of pipes and tunnels with processors located at the junctions and at the ends of the pipes. Some of the processors are little things like iPhones, routers, blogs or webcams. Some are huge like Google's cloud.

All is good until something comes along that tries to get the whole Internet to flow through it. Compelling things that get more compelling the more stuff you flow through them. Instead of being shaped like pipes, they are shaped like funnels (as illustrated to the right).

Platforms that are owned by companies start out as funnels. They may trick you for a while into believing that they made of pipes, after all it would be suicide for the platform owner to turn the lovely ecosystem into a funnel, but companies can't help themselves. They compete internally to control resources and to the people inside the company, the platform looks like just another resource to fight over. Eventually it gets funnelized, if it wasn't from the start.

This is why my current motto about Twitter is "How I learned to stop worrying and love the Fail Whale." I can love the Fail Whale, even though as a user I hate it, because it says to me that I will soon control my own destiny in this space. That cute little whale says the idea of Twitter as a funnel (it started out as one, always has been one) is not working. As the network grows, it becomes less viable.

The funnel is also why I don't like a lot of the things Google does. Things like SideWiki that are only useful when everyone uses them. It shows funnel-oriented thinking over there. Long-term they can't make the funnel work. But as Microsoft showed us in the 90s, they can slow the rest of us down while we wait for the shakeout. Better if they never tried, imho. Better for them, for us, and esp for the Internet.

Rebooting the reboot of rssCloud

Fri, 16 Oct 2009 17:01:12 GMT

It's been a really rough month for me, personally, and that has stalled some of the forward motion in rssCloudLand. I was able to find a little time between crises to implement both of the proposed changes, and to outline an open discussion page.

Originally I had planned to have four or five days between the implementation and updating the rssCloud walkthrough, but then Father's Day happened, and well, all my cards went up in the air. I'm still not firing on all cylinders, so please check my work carefully, but it is time to clear the space for more deployment of what amounts to a crucial feature for rssCloud implementers.

This morning I updated the walkthrough document to allow for:

1. An optional domain parameter on the REST request for notification.

2. If the notification request included the optional domain parameter, the verification process works differently.

A note about evangelism

Fri, 16 Oct 2009 17:06:11 GMT

If anyone in the rssCloud community is marketing against PubSubHubBub, we will ask them to stop. My position, and I hope that of the community is that these are non-commercial efforts, no one is going to profit (or lose) based on the success of one or the other protocols.

I see adoption of PubSubHubBub as a win for the Internet, and believe strongly their advocates should see adoption of rssCloud the same way. If they feel pressure from rssCloud, it should result in them more fully embracing RSS, which I felt they weren't doing when I first reviewed their efforts. Once that happens the differences will probably melt away and everyone will be happy.

Personal attacks in furthering advocacy are totally unacceptable and should not be tolerated. I hope this goes without saying, but unfortunately it appears it still needs to be said. Please.

Seeing past Twitter's limits

Fri, 16 Oct 2009 15:49:06 GMT

When thinking about the future, something I've spent a lifetime training myself to do, I try flipping the classic question around. Instead of asking "Do you think X will happen?" try this -- "Can you imagine X not happening?" It doesn't always yield a breakthrough, but it often does. It helps you see past the limits of today.

Anyway, I'm giving a 10-minute talk to open Jeff Pulver's Twitter conference in LA on the 27th. The title of my talk, suggested by Carla Casilli, is: "How I learned to stop worrying and love the Fail Whale." It's a ripoff of the sub-title of one of the greatest movies of all time, Stanley Kubrick's Dr. Strangelove. In his version, they replace the Fail Whale with The Bomb. In the 50s, 60s and 70s people were as obsessied with nuclear weapons as they are today with Twitter (that's only half a joke).

Anyway, today a tweet is 140 characters with an ever-evolving cadre of metadata marching alongside. And it's about the metadata that I wish to ask the inverted question. But first, I'd ask you to click on the small picture below, and go to Flickr and look at all the metadata that's assembled around a picture of my aunt and uncle taken by my mother sometime in the late 70s or early 80s.

What a rich collection of information there is. Yet you could imagine more, yes? If you read the paragraph that introduces the picture you'll see that there's some data that isn't reflected in the Flickr database. It was taken by my mother, and the two people in the picture are related to both of us. One by blood (the man, my uncle Ken) and one because she is married to the other (my aunt Dot). That data is missing probably because Flickr stopped actively evolving before social networking had fully gained a foothold in online culture. Same with the web itself. Wouldn't it be cool if a pointer could imply a familial relationship? I know that's what TBL has been talking about. Maybe we're getting closer to actually having it.

Anyway, if you go crazy and try to imagine what Twitter might become, you can see that a lot of what it is is in the Flickr metadata without the thing in the middle -- the picture. I've been urging Twitter to support payloads for years now. I can't imagine why they're not doing it. One piece of metadata is all that's needed, minimally, the URL pointing to the picture. Today we have to cram it into the 140 characters. Meanwhile they're advancing, adding geographic data and lists and retweets, all of which add little bits of data to a tweet, but for some reason they won't add the url. Which gets back to the question. Can you imagine that Twitter will never get this feature? No, of course not. It will someday get it. Why not get it over with?