Scripting News

New news flows

Wed, 03 Dec 2008 15:12:19 GMT

When we talk about news on the net the conversation is dominated by the interests of news organizations. The stories we tell are from their point of view. The vexing problems we face are their problems, not ours. That's been the point of the series of pieces I've been writing about news. I do care about the people of news, as I care about the people of the car industry and the people who lost their jobs at Lehman Brothers. And the 10K contractors who may be laid off at Google. But for the sake of this discussion, what I really care about is news and how it's going to get from them that have to them that want.

In a comment yesterday I said it's often overlooked that while the Internet makes some things that we used to do diseconomic, if you took the Internet away some things we've come to expect would go away too. All the stuff people call "crowd-sourcing" -- the million eyeballs that are constantly watching, and the thousands of them that are there when news happens.

I watched a bunch of campaign events this year, and one of the things that's largely been unreported is how much reporting goes on at them. I first noticed it when Hillary came out on stage to make her concession speech. Immediately every pair of hands in the room goes up, not in salute, not cheering -- each pair held a digital camera, and they were capturing images of the Clinton family. There's no doubt if you wanted a picture of that event you could get many to choose from.

It was something else at Mile High Stadium for the Obama acceptance event. It seemed everyone there was taking in the history of it, and again, the cameras were everywhere.

Look at this striking picture of the audience at the Obama rally in Berlin, taken from Obama's perspective. This is what he must have been seeing as he went across the country. Recording devices of every kind, all pointed at him. (A fair number of American flags too, which gave me goose bumps.)

Now if there isn't something we can do with the next generation of networking tools that's truly exciting and enabling, then we need to hang it up and let someone else drive for a while. In a couple of years every one of those devices will be replaced (knock wood, praise Murphy) and will they communicate better? I hope so! At the same time, we need to work on software and networking tools that allow us to process millions of pictures of an event and do intelligent things with it. When I was in Boulder in August I saw such a tool.

Update: VentureBeat has an excellent description of Occipital. "If multiple people upload multiple photographs from the same event around the same time, Occiptial will figure out that an event just happened and classify the photographs accordingly. Doing this right is really, really hard, yet with two people, Occipital seems to have done it. This team is scary good."

I've also been playing with a flow of thousands of professional photographs every day. It's really something to wrap your mind around, but after almost a year, I'm beginning to understand what kind of editorial tools you need to make sense of such a flow.

And that's always the tough problem, in my experience, making sense of the information. That's what reporters do. But it's all happening now on such a huge scale, we need new systems to grapple with it.

Do I think there could be money-making ventures built off this flow? Absolutely. What are they? Not sure yet.

TechCrunch federates with Facebook

Wed, 03 Dec 2008 18:59:18 GMT

TechCrunch: "TechCrunch readers can now use their Facebook accounts to sign in before leaving comments."

Interesting. And it integrates with Facebook's news feed.

I left a comment suggesting they do the same for Twitter, FriendFeed and Identi.ca. Very easy to validate a name with any of those services, though the companies didn't make a big deal about it. I'd like to see some of the smaller developers get a chance to play in league with the big guys. They could also share a pointer to your comment in the flow of any of the services, their APIs make it brain-dead simple to do.

Update: There's another reason for a site like TC to federate with the three sites above. Some of us don't use Facebook, but are regular readers of TC. I do have an account on FB, of course, but I almost never check it. I get FB friendship requests from people I haven't seen in years, and care about, and it makes me sad that I don't have the bandwidth to add Facebook to my rotation, I just don't think about it. But... I recently added a connection between Disqus and Friendfeed, and I like what's happening there. I am a constant user of both software tools, so connecting them makes a lot of sense. Any time I post a comment anywhere on Discqus's network, it propogates to FriendFeed. TC is not on the Disqus net, but I would like it to be on the FF net. I think it makes sense for TC to support any site that a significant number of their readers use.

Bathtime in Clerkenwell

Wed, 03 Dec 2008 22:46:23 GMT

XML-RPC update

Wed, 03 Dec 2008 11:13:46 GMT

Its been a long time since I written about XML-RPC, it's one of those things that when I do, the flamers show up and get all personal. I shouldn't let that get in the way, of course; and while I wasn't looking, for example, Mozilla baked-in support for XML-RPC. Not sure what you can do with that, but I'm sure someone will explain.

The other day when I asked about an XML-RPC interface for ImageMagick, Justin Walgran took me up on it, and deployed one on Google AppEngine. Now that makes sense for so many reasons. A perfect application for AppEngine, and since its native language is Python, and Python has great XML-RPC support (we upgraded ours in Frontier based on their inspiration, the highest form of respect), it was not a very large programming project.

Hopefully I'll be able to test it out today, once we know the name of the procedure, what server its running on, and what parameters it takes.

I made a suggestion in the comments that where the procedure calls for the image itself, that it accept the URL of the image. This would work better for my app because by the time it needs the thumbnail it has already uploaded the image to an HTTP-accessible server. It would work better to not have to upload it twice.

So the procedure would look something like this:

imageMagick.createThumb (image, height, width) returns binary

Where image can either be a binary type containing a JPEG, GIF or PNG graphic; or a string that contains an HTTP URL to the graphic. Height and width are numbers that reflect the desired height and width of the thumbnail. It returns a binary type containing a PNG (?) thumbnail.

Of course if there's an error it uses the XML-RPC exception mechanism.

Needless to say this is a very interesting project to me. And if someone wants to create an equivalent REST application, I will promote it alongside the XML-RPC application.

Update: Imagin appears to be exactly what I was looking for. It's fast, flexible, takes a URL as an image parameter. Very nice. What's not clear is how hard you can drive it without pissing him off.

A Plan B for news?

Tue, 02 Dec 2008 19:10:06 GMT

Jeff Jarvis responds to my series of pieces about news after the hypothetical collapse of the news industry. I wrote a comment there, which I'm reproducing here, with some light edits.

Jeff, the stuff you're justifying is the stuff that's going away, that there is no money to support. If we all care about the news, and making sure that it gets from the people who have it to the people who want it, we're going to have to learn how to do it without all the heavy iron. It seems to me the responsible thing for the news industry to do, while it is laying off its reporters and editors and the rest, is to help us come up with a Plan B -- what we will do for news once all that is gone.

An analogy -- imagine a group of doctors knew that the hospitals and pharmacies were about to shut down. What would they do? Might they do something to make sure their client's health needs were at least partially attended to?

The same would presumably apply to many other professions, whose services are in some way necessary for life: police, fire, bus drivers, teachers, garbage collectors.

We're often asked to believe how noble the profession of news is -- now that is about to be tested in a whole new way. Are we just supposed to cry for this industry and throw our hands up and wait for the collapse before starting to put it back together, or would they like to help while they're still here?

Here's a question I ask people privately to help focus their thinking... Suppose there were no NY Times tomorrow, and you heard somewhere, maybe on Politco or Huffpost or Memeorandum that it had gone out of business and was never going to publish again.

1. How would you feel?

2. What would you do?

3. What should the Times have done but didn't do before they shut down?

Food for thought.

It's time to have this conversation Jeff. Imho.

Update: Scott Rosenberg checks in on the thread. "Victimhood is written deeply in the culture of the newsroom."

Newsosaur: "A newspaper that cannot sell enough advertising or cut enough expenses to sustain profitable operations is not likley to make it to the other side of 2009."

An image processing web service

Tue, 02 Dec 2008 17:50:15 GMT

On Sunday, I wished for a web service that would take an image, a height and a width, and return a thumbnail for the image.

Andrew Burton put up a service, I gave it a try, with no luck. Maybe we can get this working. Ideally, I'd like to run it on the same machine as the application that calls it, since the images can be fairly large.

Here's a text file containing the text, and the picture I used.

Pownce we hardly knew ye

Mon, 01 Dec 2008 21:22:17 GMT

I was a Pownce user. (Ack it can't find my page -- and I was a premium, paying user! Oy. When did that happen?)

There were some things I liked about it, but I agree it's time to pull the plug.

I stopped using it when:

1. Twitter got its act together and stopped acting like a Norwegian parrot pining for the fjords.

2. FriendFeed occupied the space above Twitter, as the messaging system with more (than Twitter). FriendFeed has never had trouble staying up.

The biggest problem with Pownce was:

1. It couldn't handle even a modest load. It would get very very slow when anything interesting started happening, therefore keeping anything interesting from happening.

The one thing Pownce got right was:

1. It had payloads!

Three things that slowed adoption of Pownce beyond the inability to handle a load:

1. It was in private beta for a long, long time.

2. It took forever for it to get an API.

3. When the API finally came it wasn't compatible with anything.

Net-net, there were interesting things about Pownce, and we'll remember it with a certain amount of fondness.

Hopefully Leah can take what she's learned and turn out something great at SixApart.

I'd recommend: Twitter-Plus-Plus. (With lots of interop, and do the payloads thing again, they need a kick in the ass over there at Twitter to get it into their product.)

FriendFeed and level playing fields

Sun, 30 Nov 2008 19:10:40 GMT

I've always been a bit puzzled about how FriendFeed does RSS, but I've never (until now) taken the time to find the source of the puzzlement. I've always just fumbled my way around, sort of approximating what I wanted, and when I couldn't get it, falling back to the API. But now I've hit a wall, and taken the time to understand the nature of the wall. Let me explain.

Consider this screen (click on it to see the detail):

Suppose you used a photo site that wasn't one of the ones listed, but you had an RSS feed for your photos and favorites on that site. What are you supposed to do? I always assumed you should just add the feed under "Blog" but then your readers will start asking why your pictures don't do all the neat things that happen automatically with Flickr, Picasa, SmugMug or Zooomr sites. I have such a site, and I don't want them to do anything special for it, I just want to tell FF that it's a photo site and have all the cool special goodies they have for Flickr kick in automatically.

If you pop up a higher level, you'll see that this is actually contrary to the whole idea of feeds, which were supposed to create a level playing field for the big guys and ordinary people. That's why a guy like Mike Arrington was able to start TechCrunch and eventually be a competitor of CNET. The fact that RSS didn't favor the big guys made that possible. In fact the whole web is like that. You don't need a special client to read the NY Times and another to watch videos on YouTube. Any browser will do, for any site, no matter who's writing and who's reading. It's why many of us fell in love with the web, at first sight. In the software world before that, it mattered who you are or who you worked for. Kind of like FriendFeed. :-(

It's also against the level playing field idea to favor people, like me, who can program to the APIs. The point of feeds was to make the technology transparently understandable to people who just had brains, that you wouldn't need to understand anything deeply arcane to make RSS work. Since FF is about feeds, it seems to me that it ought to be consistent with the philosophic simplicity of feeds. Again, this is just another application of a principle of the web -- you could always View Source to see how a website worked, and if you were willing to do a little trial and error, and head-scratching, you could make your site work the same way as any site you could view in your browser. This was a good thing.

Now, don't get me wrong, I like APIs, I even love APIs, but only when a feed won't do. There are cases where the API shows more power than the feeds, where feeds can and should have the same power. For example if I want a description to come along with a picture, I have no choice but to write a program to push the content to FriendFeed. That seems wrong to me. RSS and Atom both have description elements, why ignore them? Also, I can if I want make sure the content arrives in a timely manner, but only through the API. The functionality of a web app shouldn't unnecessarily favor programmers. That's unweblike imho.

Now I wouldn't make these criticisms if I didn't think FF was an excellent web app. But like all technology it can be better. That's why I make the suggestions.

Tech developers and users

Sun, 30 Nov 2008 17:07:56 GMT

It's impossible to tell how tech companies will take feedback or advice, I just give it as it occurs to me. I don't try to sugar-coat it, but then I don't think that there's anything wrong with providing an imperfect or incomplete product or service.

I was the guy who said "We make shitty software" to his developers as he passed them in the hall. To which the standard response, which always got a laugh, was: "With bugs!"

It's a joke, but not really. We know our software sucks. But watch, we'll make it suck less.

Anyway, offering advice to most developers is a waste of time, and only makes them hate you. But what are you supposed to do if you want to build on their product and keep hitting the same brick wall, month after month. Is there a polite way to express frustration? If so, I'd like to know what it is.

In Thursday's piece I said developers are every bit as insistent about ignoring users as news people are. I see it happen every damn day. It's just as bad no matter where it happens.

Medvedev and his Macbook

Sun, 30 Nov 2008 16:04:21 GMT

Update: Obama uses a Mac too.

Fractional horsepower ImageMagick HTTP server?

Sun, 30 Nov 2008 15:42:37 GMT

I love ImageMagick, it's creating thumbs for me, night and day, day in and day out. The feed that's using the thumbnails is up and working.

But I'm still bugged that: 1. It seems slower than it should be. 2. A window flashes every time it creates a thumbnail.

People have suggested that I use PHP to interface to ImageMagick, but no slight to PHP, I already know too many languages, I'm trying to forget some!

Then I got a message from Phil Pearson, a guy who helped a lot in the early days of XML-RPC, and that triggered a thought -- you know what I really want -- an HTTP interface right into the ImageMagick engine. I'd accept a REST interface, but I'd be ecstatic about an XML-RPC interface. Truly.

Then you could put the engine where ever you wanted and call it from anywhere. If it started consuming the whole CPU, then fork off another one. I know you can probably do this with PHP, but I'm picky. I want my XML-over-HTTP. That's my comfort zone.

Update: I hear that Graphicsmagick is faster than Imagemagick.

If you never listen you never learn

Sat, 29 Nov 2008 20:30:50 GMT

I just re-read the Rosen thread over on FriendFeed and another irony struck me. The argument is over things that I didn't say in the piece they're arguing about.

The piece is about listening, and they didn't listen.

Listening is hard. When you respond after listening make sure you aren't responding to something that came out of your head because you're having that argument with yourself, not the other person. And they're likely to get confused, or angry.

You never know what you'll learn if you listen. Maybe the people who want to say something to you might just make the difference between driving off the cliff and finding a new future. Maybe it'll help you find the great idea that cracks the nut. Or maybe what they want is something you can give them, maybe it's something you'll want to give them. Some users are pretty smart, I've found.

Knowing how other people see you can be disturbing, but it can also be eye-opening.

In 1986, I had a meeting with Guy Kawasaki when he worked at Apple. I showed him an early version of one of our products, we had thrown the kitchen sink into it, every half-baked R&D idea, cause our company was failing and this was our last chance. One idea intrigued him. He said everyone at Apple was hand-designing foils to print on Laserwriters (they were new then). He took a piece of paper and drew a box around one of our pages, and asked if we could do that. Of course we could, and we did, and we immediately sold 1K copies of the product for Apple people, but more importantly, they were so excited by it, they in turn sold many more thousands to their customers, and our company went from being in the brink of shutting down to gushing cash. All because (drum roll) we listened to a user. Ask Guy if you don't believe me, he's on Twitter.

One more thing -- when did listening become "listening in the aggregate." If you know anything about me, you know that I don't think of users as couch potatoes, passive participants. At the same company, we designed regcards to solicit original thoughts, not just box-clicking. When a new batch of regcards came in I grabbed them and studied them for interesting comments. They told me how our new stuff was being received, what they liked and didn't like, what was missing that would make the difference for them. When I had a question, I called and asked. It's also good for business if people get that you care what they think, if you really do. They can smell it when you're being patronizing.

It really is long past the time for the news industry to listen to its users. We've been trying to start this conversation since the first blog post, but there's not been much listening. That may turn out to be the epitaph of the news industry, the users did care, but the industry never listened.

Why we don't listen to users, by journalists

Sat, 29 Nov 2008 13:56:43 GMT

Jay Rosen argues with journalists, who explain why they shouldn't listen to users (sources and readers). I'll probably write more about this later, but for now, read the thread, it's fascinating. Here's the piece they're responding to.

House on a hilltop

Fri, 28 Nov 2008 23:03:50 GMT

Happy ImageMagick user here

Fri, 28 Nov 2008 17:52:48 GMT

Everything is such a futz, but it's nice when these things have happy endings. I have my thumbnail creating app up, and while it's not linked into the RSS its going to be used in, that's the easy predictable stuff, the stuff that requires no futzing, at least not for me.

Here's a folder of thumbs that ImageMagick is adding to every five minutes.

I'm doing it by launching the ImageMagick convert app, once for each picture. The fixed width is 100 pixels, I compute the height to be proportional. My script makes sure not to start another conversion until the previous file exists, because the process is unfortunately asynchronous. However it is easily synchronized. I have a 15 second timeout. Then I wait 5 seconds between conversions to let the CPU catch up processing other tasks. Image processing, esp for very large images, is a very CPU-intensive thing, apparently.

I hooted out loud (HOL) when I got it working. This one has been on my to-do list for a very long time!

Netbooks are about the users too, dummy!

Fri, 28 Nov 2008 17:27:59 GMT

CNet: "If you've ever used a Netbook and used a 10-inch screen size -- it's fine for an hour. It's not something you're going to use day in and day out."

To which I say -- well hmm. I think the first part is right. And you will use your netbook every day, for about an hour or so, sounds just right. Inbetween things. Kind of the way you use an iPhone, but for people who like more of a computer.

For real work, I use a full setup with lots of hard drive space, and two big screens and comfortable seating.

A netbook is for the coffee shop or airplane or subway ride. For watching a movie, checking email, updating Twitter, fast, mobile stuff.

But it's good that Intel is checking in with the users. And eventually I think netbooks will evolve into market-expanding machines. We're still in the first year of netbooks. Give it a chance.

It's about the users, dummy!

Thu, 27 Nov 2008 17:52:40 GMT

Michael Fraase tries to explain what I've tried real hard to explain for the last N years (where N might be as many as 15 believe it or not) that news, like any other business, is about the users.

I do this with programmers, with about as little success as with people in the news industry. People who write software invariably think the users' job is to give them "feedback" which they are free to do with as they please. Of course they are free to ignore the users, but eventually that results in the users (er ahem, drum roll please) ignoring them. If you want to keep their interest, you need to be interested in them.

I once went to a lunch at the University of California School of Journalism, where I mouthed off on this subject, more or less, and was greeted with a stunning idea -- it was largely considered unethical for a reporter or editor to know which sections of the paper were most read by users of the paper. If the reporter knew, the story goes, he or she might be influenced by peoples' interests in deciding what to write about.

To which I said (and say) loudly -- OY!

Or another way --> THERE'S THE BUG.

Fraase says I think like a programmer. I suppose. I didn't always. When I was younger I wasn't going to be a programmer. I became one out of necessity. I had ideas that expressed themselves in software, and I couldn't interest any "real" programmers in making the ideas real, so...

But all along I've been a glutton not just for feedback, but to know how the ideas I had would be used. I never create any software that I myself don't want or need, because I wouldn't know how to do it. My method of development depends on me being a user. So do I listen to the users? Yes. If I listen to myself, which I try to (it's harder than it might appear at first).

Listening is hard. But all people who create products for users must listen if they want to do well at making products. That includes doctors, bus drivers, mailmen, entrepreneurs, programmers, and yes, reporters and editors too. Because if you don't listen you might miss a corner-turn and end up going off a cliff, just like the news industry is doing. They see the cliff, they know they're headed for it, but they don't ask how to turn the car. They don't really want to know. I think sometimes what they want is to be missed when they lie dead in a crumpled car at the bottom of the cliff. But we don't want that to happen. Not because we love them, but because life without them is pretty hard to imagine. They should turn the corner, no matter how painful it is. But in order to do it, they're going to have to look out the front window and the mirrors and listen to the person in the passenger seat.

That's why it's not enough for the NY Times to have a Public Editor, they have to have the Public itself on the op-ed page. That should start on Monday. There's no reason to wait for that. The Times should have more branded blogs, given to people whose opinions they value. Want a list of 5000 such people? Make a list of the people the Times has quoted in the last year. I bet it's more like 50,000. At this point, the Times is still reputable enough and alive enough that they would want to be under the Times umbrella. Immediately you have a reason to survive for between 5000 or 50,000 people.

We can save one newspaper that way. One newspaper is infinitely better than zero. We can probably save a few magazines too. The key thing is to incentivize people to make them survive. Open the doors as wide as you can imagine, and let the world flood in. It should have happened slowly and carefully over a decade, I told you so back then but you didn't listen. Now you have to achieve what would have been accomplished in that period in the space of a few months. It isn't going to be easy or anything like pain free. But it can work. I'm sure of it.

Seth Godin: "The only reason to answer the phone when a customer calls is to make the customer happy."

Thanksgiving stuffing

Thu, 27 Nov 2008 15:57:41 GMT

Either Thanksgiving is a mad dash to get 18 different dishes ready, all at the same time (never works, something's always cold, something burned) or, you sit around wasting time until it's time to go to a dinner where you eat someone else's labor of love. This year, for me, it's the latter.

NakedJen has a wonderful tradition for Christmas Day, to spend the day at the movies, watching the new releases. Christmas is always a big day for new movies. In Santa Cruz, where we did it last year, a surprising number of people did it too. We saw Sweeney Todd, the stupid Tom Hanks movie about the politician who supposedly saved Afghanistan, and The Savages, which was, imho, by far, the best of the three. Sweeney starred two of my favorite actors, Helena Bonham Carter and Johnny Depp (with a small part for Sacha Baron Cohen!) and directed by Tim Burton -- it should have been great, but I could hardly stay awake during the movie. This year I'm going to Salt Lake City to join NakedJen in my second NakedJen Film Festival.

BTW, when NJ writes about \"Dave" -- it's not me. I want to be clear about that. It's some other guy named Dave, who wasn't very nice to her.

See this is why NakedJen and I get on so well. She says everyone she knows hated Burn After Reading but she liked it. Well I have news for you -- I really liked it too. Yes it was stupid, but sometimes stupid is just the thing. The Coen Brothers rarely fail to entertain. I had a strong feeling about this movie, that it was the counterbalance to No Country For Old Men, which was very very very serious (and unprecedented for the Coens). The CIA Director and his report seemed to speak for us and for the Coens, asking WTF just happened? No one knows. (And you wouldn't believe it if I told you, and you wouldn't even care.) But as long as everyone who was involved is now dead, why should we care? Seems to me a perfect explanation for NCFOM.

IRC for Mumbai terrorism

Thu, 27 Nov 2008 14:46:58 GMT

In case there's an interest in IRC for news around the Mumbai terrorism:

irc://irc.freenode.net/#mumbaiTerrorism

I'm already there.

Is Twitter journalism?

Thu, 27 Nov 2008 14:28:05 GMT

What you see on Twitter, when:

1. People witness events that others are interested in; and

2. They're posting about it on Twitter; and

3. The interested people are reading their posts...

It certainly is news. Whether it's journalism or not isn't a very interesting discussion, to me.

To the user, both extremes, Twitter and the most vetted pro news, require skepticism. The reader triangulates.

See also: Mike Arrington, Mathew Ingram.