Scripting News

NewsJunk.com

Mon, 02 Jun 2008 19:42:56 GMT

My next big project is NewsJunk.com.

The name comes from the people it is designed to serve, news junkies.

So we've gone beyond mere users, now we're making tools for serious users.

This is turning out to be a broad project that will involve lots of people, and I will have much more to write about it over the coming weeks. But the news is happening so fast now, and we're bringing tools on as fast as we can, so I'll have to wait for the philosophy.

For now there are five main ways to consume the flow:

1. Refresh the home page periodically.

2. Subscribe to the RSS feed.

3. Follow it on Twitter.

4. Befriend it on FriendFeed.

5. Watch for developments on the weblog.

There will be more ways, for sure, soooon!

Still diggin!

Update: Nicco's post on NewsJunk.

Owen Thomas speaks in Berkeley

Mon, 02 Jun 2008 19:21:57 GMT

Just got an email from Sylvia...

Why does Silicon Valley need its own gossip rag? Come find out Wednesday, June 18, from noon to 1:30 at the Berkeley Rep on Addison between Shattuck and Milvia in Berkeley. It's $12 at the door for a delicious meal and a chance to hear Valleywag editor in chief and founder Owen Thomas dish it out on the dubiously fact-checked doings of our digerati. Thomas will be taking questions, so feel free to ask away.

RSVP to whoisylvia at aol dot com.

NASDAQ goes real-time

Mon, 02 Jun 2008 17:39:17 GMT

As if on command, this press release dropped this morning.

We're getting real-time stock quotes.

Great. Now think about it -- how much like Twitter is that?

I wrote about this yesterday. If NASDAQ and other rich companies like Google, CNBC, etc are willing to take the risk on their technology it must be pretty reliable? Otherwise, imagine the lawsuits that would ensue.

As I wrote yesterday, I bet you can license the technology to run Twitter much more cheaply than you can to build it yourself.

Bill Clinton's Nixon moment?

Mon, 02 Jun 2008 16:56:44 GMT

Bill Clinton: "I want to say also that this may be the last day I'm ever involved in a campaign of this kind. I thought I was out of politics, 'til Hillary decided to run. But it has been, one of the greatest honors of my life to go around and campaign for her for president."

Richard Nixon: "For 16 years, ever since the Hiss case, you've had a lot of -- a lot of fun -- that you've had an opportunity to attack me and I think I've given as good as I've taken. But as I leave you I want you to know -- just think how much you're going to be missing. You won't have Nixon to kick around anymore, because, gentlemen, this is my last press conference."

Update: I think Marla Erwin nailed it. This means WJC won't be campaigning for Obama.

What Twitter did for Scripting News

Mon, 02 Jun 2008 17:30:58 GMT

We spend a lot of time talking about the technology of Twitter here, because tech is the root of Scripting News, it's how the blog got its start in 1997. But it's been largely a male thing, where DaveNet, its predecessor, was closer to gender-balance. The new connection with Twitter has helped bring us more women readers our way, which is appreciated. It isn't much of a party without em. And I might add, it's given us a bit of racial balance too, although I think the all-but-certain Democratic candidate for President had something to do with that as well.

We're angry, uneducated and unhealthy. Now what?

Sun, 01 Jun 2008 17:45:13 GMT

Jeff Jarvis said something that got my fur up: Only Sen. Carl Levin of Michigan had an irrefutable point. 'We've got a totally irrational system of nominating our president,' he said.

It's refutable. Obama looked at how the nominating process was laid out and then built an organization and strategy to win on the terms of the system, rather than close his eyes and try to hit the target and then blame the game for his loss as Clinton and her supporters are doing.

Which kind of President would you rather have -- one who accepts the world as it is and then maps out a way to win, or one that grouses at how irrational it is.

Yeah it's irrational that all the oil is in the Middle East. Now what?

Yeah it's irrational that Bush started a crazy war and that the country's education and health care systems are inadequate to compete in a global economy. Now what?

Our infrastructure is crumbling, our products aren't competitive, we're uneducated, unhealthy, angry and to make matters worse our houses aren't worth shit. Now what?

I want a President who welcomes the chaos and then figures out how we can be smart about the hand we've been dealt. Not one that whines and complains about how irrational the world is.

I can't wait until the Clinton Democrats accept that their time has passed and the world their way worked in has passed too, and let's get on with it.

Update: This piece, cross-posted at Huffington.

Cousin Mikey is taking an interest in Twitter

Mon, 02 Jun 2008 00:22:19 GMT

Mike Arringtoh wrote a post yesterday about architecture issues in Twitter which was mostly pretty good, though his last question is very lawyerly and off the wall, no way one person is responsible for the problems with Twitter, and if there were one person, it would be the CEO not a programmer.

I sent an email to Mike and even tried to call him, but he's not answering, so I'll just get him the info here on Scripting. If you know Mike and he reads your email, please send him a pointer to this piece and say hi for me!

Here's the message: It's pretty likely you can buy or license off-the-shelf software that does more or less what Twitter does. (Probably more.) The problem of reliably sending massive numbers of notifications quickly was solved a long time ago in the financial services industry, apparently. Think about it -- the stock markets and banks have to do this, and if they drop out like Twitter does, billions of dollars would be lost, maybe the whole economy! There would be a lot of good reasons to throw lots of money at this problem.

The pointers came up during an extensive discussion about decentralizing Twitter's architecture here in January.

It made sense to me -- I encountered these kinds of people when I worked in NYC as a programmer after graduating college in the mid-70s. That the software has been commodified since then is not surprising.

I've tried to suggest to the Twitter management that they take this route, but haven't gotten through.

Maybe someone should look into the idea of just adapting technology the enterprise guys are using?

Just a thought.

NewsJunk podcast with Joe Trippi

Sat, 31 May 2008 19:16:09 GMT

We just did a podcast with Democratic consultant Joe Trippi about today's Democratic Rules Committee meeting, and the next steps in the nominating process.

http://newsjunk.com/mp3/nj080531.mp3

Then we switch gears and talk about the new venture I'm starting with Nicco Mele. We're still just covering the edges of the vision, but it's about news, politics and technology, three things close to my heart.

BTW, this is the first NewsJunk podcast. You can subscribe, for now, through the scripting.com RSS feed.

IRC for Democratic Rules Committee Meeting

Sat, 31 May 2008 15:19:24 GMT

Fascinating stuff coming up during testimony at Democratic Party Rules and Bylaws Committee today in Washington.

Since we had IRC for each of the major primaries this year, I thought we'd try the same for this event.

irc://irc.freenode.net/#dncMay31

Hope you can join us!

More Twitter humor

Sat, 31 May 2008 01:59:48 GMT

Comment thread on Flickr.

Photos of downtown Oakland

Fri, 30 May 2008 01:48:40 GMT

Here's a set of photos I took yesterday in Oakland.

Overlooked in the McClellan coverage

Thu, 29 May 2008 13:26:06 GMT

The coverage of the McClellan tell-all book has focused on the White House spin, which amid all the bluster about surprise and how this isn't the Scott they all knew (come on, why should voters care that you're surprised), they aren't really contesting the assertions, or if they are, they're doing it weakly.

Probably some of them want to have jobs in the future, and lying right now wouldn't help them in the careers. Further I think almost everyone who has been paying attention knows that what McClellan says is true. Why didn't he speak out earlier? Why didn't a lot of people? Also consider the possibility that other people in the White House got scooped, the ones trashing McClellan and are jealous that his tell-all book got out before theirs, and others are likely to be tried and perhaps go to jail for their actions. In other words, they all have axes to grind here.

The other point being overlooked, and this is a real problem, is that he says that the press was complicit. This is the more important allegation, and unsurprisingly, it's being swept aside by the press. Had they done their job, and pressed for the truth, it would have been easier for insiders to tell them the truth. But corporate-owned media isn't interested in helping us make decisions as a country, they're only interested in ad revenue. That's why it's so important that we're creating new media that isn't so conflicted, and why the question of whether bloggers run ads or not is far from a trivial issue.

In court, if you have a conflict of interest, you're supposed to disclose it, and if it's serious enough, it disqualifies you. I've recommended many times that professional news media should have relationships with less conflicted bloggers for circumstances like this, so when they become the story, the public can have a discussion about them using the channels they own. They don't have much of a choice here, because the channels are going to develop with them or without them. We could all save a bunch of time if they didn't fight it, and welcomed amateurs into their midst.

Jury Duty, day 2

Thu, 29 May 2008 12:53:32 GMT

I have another day of jury duty. I haven't been selected, but the voir dire is going into a second day because we started so late yesterday. I am not allowed to talk about the specifics of the case as long as I might potentially be a juror, however I can say that I think it's not likely that I will be selected. And I can say is that it is, again, an inspiring process. A few comments follow.

Until I moved to the East Bay in 2006, I had never ridden on BART. It doesn't go down the peninsula to where I used to live, but it's a fixture of life on this side of the bay and in San Francisco. I like riding BART not only because it's usually faster than driving, but also because I get to see my fellow citizens without their cars. It's fascinating to see who my neighbors are. In NY and Boston, where public transit is much more a part of daily life, you get that experience all the time. Not so much on the west coast.

Well, jury duty is like riding BART, only more so. It's as if you were riding on BART, but each passenger, in turn, tells you what they do for a living, who they live with, where they came from, in some cases why they can't serve (always dramatic). What the judge says about jury duty is true, it's what makes America work. There are no professional jurors, just BART people. You just have to be registered to vote or have a driver's license for them to find you.

Which leads to a curious inexplicable fact. First, judging from the jury, the Bay Area is a remarkably diverse place. But if you went by the jury alone, you'd conclude that there are no blacks in Alameda County. The only black person of the 100 or so potential jurors was an African immigrant. However, if you go outside the courthouse in downtown Oakland, or ride BART, you'd see that there are lots of blacks. What happened? Maybe they don't vote or drive? I honestly don't know. It seems very improbable that a random drawing would be so skewed.

I was riveted listening to the stories people told. It was fascinating. I also had opinions of the lawyers, and the parties. But I can't talk about that yet. What I can say and will is that I was struck by the nobility of my fellow citizens. When asked to serve, they all rise to the occasion. People who doubt that there is cause for hope should sign up for jury duty and go through the process. It is something to behold.

From the Mind of Hugh

Thu, 29 May 2008 01:26:49 GMT

An open source Twitter client?

Wed, 28 May 2008 19:19:49 GMT

There have been a lot of back-channel conversations about the future of Twitter, and one subject that keeps coming up (I keep bringing it up actually) is the need for a Twitter client that isn't aligned with any commercial interest. One that, by design, is meant to serve users and users only. I think it's time to start a public discussion about this, not sure if there's interest among developers -- this isn't something I'd write myself. The only way to find out is to ask.

Jury duty

Wed, 28 May 2008 16:34:47 GMT

I got called for jury duty in Oakland.

They have wifi here but it's $5.95 per day.

Can you imagine, here we are doing our civic duty, taxpayers all of us, and they want us to pay to get online. It's a crime I tell you! Put me on a jury to decide this and I'll vote to convict.

I served on a jury in 1996 and wrote up the experience.

Update at 10AM: They called the first group, I was not in it. Is this good news or bad? (I'd rather not serve this time.)

Update at 10:30AM: I'm in the second group. Here we go. But we don't have to be here until 1PM. So now what? Oy.

Update at 10:45AM: All the group 3 people were excused. Envy.

Spewage update

Tue, 27 May 2008 19:19:55 GMT

Update: JR suggests adding a column to the Twitter spew report that shows the percent of total spew each person is responsible for. Good idea! Done.

Should Twitter charge high-spew users?

Mon, 26 May 2008 20:05:24 GMT

Om Malik asks if Twitter should charge users like Scoble who have huge numbers of followers.

It's a fair question because these users are super-expensive for Twitter, much more so than users with modest numbers of followers.

To get an idea, I have a little agent script that counts and ranks people I have followed in the recent past to get a rough idea of how much work they generate for Twitter's system software.

http://twitter.scripting.com/spewage.html

You can see that Scoble tops the list with a "spew factor" of 308,359,436. I'm #5 with a spew of 77,174,172.

Imho, they shouldn't charge these people because they're feeding the growth of Twitter. If you charge them a competitor will come along and might actually pay them to use their system because it will attract so many other users.

Right now Twitter doesn't need more money. They need a design that works, and an implementation of that design. They have lots of money and can get lots more.

How to do data portability

Mon, 26 May 2008 15:03:50 GMT

I've heard a lot about data portability conferences and workshops, I've even been criticized for not going to one which happened on the west coast while I was in the east earlier this month. I don't plan to go to any of them, I don't see what's accomplished by having public meetings about this stuff. People who control users' data can accomplish a lot more by finding ways to give them the power to use it more effectively. Talking about principles of data portability only achieves talk. It gives people a sense of propriety over talking, not data, and people giving up propriety over talking are just yielding the floor, not yielding any power over users.

The best way to achieve data portability is to just do it.

I know that sounds silly, or obvious, but there is so much pretending that there's more to it, that it has to be said.

If you want to accomplish something by talking, call up a friend who works at Netflix or Yahoo and ask them if they'll let users move around their movie rating data. I've been asking about this for years. No one's email addresses are involved. All I want is the power to give Netflix permission to read an XML file on yahoo.com that contains my movie rating data (assuming Yahoo goes first). Anyone can see how much power this would give Yahoo. Why don't they do it? I honestly don't know. If I were them, I would.

Another example -- if Twitter wanted to buy itself some time and growth, and give developers something exciting to do, they would store as much user profile data as they can off twitter.com servers and on Amazon. Simple XML formats, use some of their ability to raise investment capital (which they have proven) to grow the human network while they patch up or rewrite their system software. The more data they can move off their outage-prone systems, the more the network can grow around them, but not dependent on them. Amazon has proven they can keep their servers running. Leverage that.

The discussion about data portability so far has fixed on the hardest most vexing technical, privacy and economic issues, the ones that probably don't have a resolution. My advice is to instead pick a few relatively easy data portability problems and solve them. Flying around the world to go to conferences to talk about the hardest problems won't actually achieve any data portability.

Update: Brad Feld argues for APIs. A few months ago I would have agreed, but today I don't think an API is enough. As we've seen with Twitter, when the service goes down, there is no API and there is 100 percent lock-in. We need more. The most vital data must be stored off-site, so it doesn't go away when the service goes down.

The 16-year rewrite

Mon, 26 May 2008 14:57:46 GMT

In February 1992, I started work a piece of Frontier called the scheduler. It's the equivalent of what they call "cron" in Unix-Land. You can put scripts in four different places: 1. everyMinute scripts, 2. hourly scripts, 3. overnight scripts and 4. threads. It was a simple bit of code that's been running now for 16 years, on every copy of Frontier, Radio, or the OPML Editor.

It was built on the foundation for background processes that existed in 1992. A few years later a better foundation was built, but the scheduler was never adapted to run on that.

It's always had a certain flakiness, and I never had the patience to track it down. It's old code, written before I learned a lot of things about the Frontier environment, what works and what doesn't. I just lived with the flakiness.

Yesterday I got tired of it, and I did what programmers like to do, I rewrote it. It took a few hours, but the new version is *much* cleaner, and already runs much more reliably.

Proving the point that sometimes code rewrites are the way to go.

I've released the new part to OPML Editor users. There's no code that uses it yet, but there will be soon.