Scripting News

A cloud-enabled podcast feed

Mon, 28 Sep 2009 20:33:53 GMT

In the early days of podcasting, in January 2001, we got the bootstrap started with a feed of Grateful Dead songs. Every day or so I added a song to the feed, in case anyone was interested in writing an application that grabbed audio enclosures from feeds. I needed something to test with because Radio UserLand, which we were working on at the time, had the capability.

It took a few years for podcasting to catch on, but having that example feed made a difference. You have to start somewhere.

Now in 2009 we're trying to bootstrap a network of realtime feeds, and it's going pretty well so far. Podcasts are implemented with RSS too, and while we have excellent examples of realtime photo feeds, we don't yet have a realtime feed with audio.

So a week or so ago I started exploring options, and thought I'd use the Grateful Dead again, until JY suggested using a fast-updating audio feed from the Internet Archive. I took one look and realized this was it. It took a bit of a coding to check it periodically to see if it has updated, add a cloud element and notify one of my cloud servers. Now it's done.

http://static.newsriver.org/archiveOrg/podcastRss.xml

So if you're working on podcatching software give it a try.

Since River2 is a podcatcher, it automatically works with this feed.

TechCrunch is cloud-enabled!

Sat, 26 Sep 2009 13:55:23 GMT

It's a cause for celebration every time a feed is cloud-enabled, but it's a special victory when a large tech news site takes the leap. The first to do so was GigaOm, which became a source of realtime updates on the same day that WordPress got the feature. Now the second big brick falls into place -- TechCrunch.

I've been emailing with their new CTO Nik Cubrilovic, for the last month. This morning I got an email from him saying their service was up.

I added the TechCrunch feed to my River2 aggregator, and it worked exactly as it should!

Here's a screen capture of the Log page from River2.

1. I subscribe to the feed.

2. River2 requests notification from TechCrunch.

3. They check out its notification handler, and apparently like what they see.

4. They accept my request.

5. My River2 logs the registration.

Now when TechCrunch posts a new article my aggregator will be directly notified, will read the feed, and will immediately post the new item to my home page. All this should happen, if everything is working, in a few seconds.

Thanks TechCrunch for supporting rssCloud!

PS: Last night I posted an rssCloud status report.

Where we're at with rssCloud

Sat, 26 Sep 2009 02:46:00 GMT

A brief report on where we're at with rssCloudLand as September winds down.

It's been a great summer. In July, we got busy rebooting the RSS cloud, based on the experience in 2001 and 2002, with lessons learned, and a lot more success than we had seven years ago.

Based on the walkthrough document, the community has shipped multiple implementations of all three sides of the interface.

1. There have been many implementations of the cloud software, including my own rssCloud.root and wordpress.com and the plug-in that turns every standalong WordPress installation into an rssCloud server. Total number of installations: millions.

2. Lots of feeds and authoring systems numbering in the millions. Example real-time podcast feeds and photo feeds.

3. Not so many aggregators. So far only two -- first my own River2 and LazyFeed. There many more on the way. I hope one or two Twitter clients will ship rssCloud support in the month of October.

What next?

I have four immediate priorities:

1. Features that will make it possible for Google Reader and other large scale aggregators to wire into rssCloud. These are technical features, to most users they are details, but nonetheless are important. It must be possible for a server to register a notification request using a domain name, on behalf of another server. And along with this feature, most developers believe a stronger verification process is needed. I am one of them. Yet it must remain simple to implement, so the barrier to entry is as low as possible. There have been proposals in both these areas.

2. Long-polling proxies for clients running on systems behind NATs or firewalls. This technology is well-understood, and a number of developers are working on solutions. I hope when code emerges, there will be an effort to make their interfaces compatible, so maximum interop can be achieved.

3. Conventions for naming, so that URLs can be mapped to names that are managed by DNS, the naming system of the Internet itself. I've done a simple experiment in this area to get myself educated on the issues and learn how users and developers see this.

4. A high-level user-oriented vision statement for rssCloud with a roadmap for developers and marketers. My goal is to create an open loosely-coupled message network that achieves more or less what Twitter does, but in a decentralized manner. It must use the architecture of the Internet to achieve what Twitter achieves in a centralized fashion. No doubt there will be tradeoffs, some things the open network will do better, and other things that the centralized one will. This is similar to previous layers of the Internet, where systems like AOL and CompuServe provided more sophisticated features, where the World Wide Web was more low-tech, more basic. Both ways of doing things have advantages and survive to this day.

Comments, questions and suggestions are welcome.

Arrington in the airport

Fri, 25 Sep 2009 01:23:08 GMT

A question for DNS gurus out there in InternetLand

Thu, 24 Sep 2009 04:55:45 GMT

I bought a cool domain for a project I'm working on, r2.ly.

It takes Libya forever to approve these things, but they finally have, and I've changed the DNS to ns1.slicehost.com and ns2.slicehost.com (where I now have an account, with DNS control) but I can't tell what's going wrong cause I don't really know how to debug DNS.

I just know that it's not resolving on any of my machines.

Can you tell what the problem is with it?

And if you solve the problem you get to call me stupid in the comments and I won't moderate it out.

I want to divorce my iPhone

Wed, 23 Sep 2009 18:02:02 GMT

The iPhone is so totally not my spiritual soulmate.

I refuse to become dependent on apps grown in their environment. To me it's like contributing to the enslavement of my brother and sister programmers. I don't care how sexy the environment is as a user or a developer, the fact that Apple holds up apps and rejects them often because they compete with their own software is to me like buying a coat made of the skins of endangered species. I won't use iPhone apps for ecological reasons.

I use my iPhone as a: 1. Phone. 2: Camera that can communicate (very valuable feature to me). 3. A Bluetooth tethering device for places my Sprint MiFi doesn't work (and that's a lot of places).

For that I pay about $100 per month. I think I'm being ripped off. (Sure of it.)

Okay Scripting News readers -- tell me I'm crazy but I want a divorce. Enough of this bullshit.

But I need a phone that does 1, 2 and 3.

What will I fall in love with?

PS: I have my contacts in GMail. Must be able to synch with them. One of my favorite iPhone features.

PPS: I never use it as an iPod. I prefer my Walkman.

Sharing links in the River2 community

Wed, 23 Sep 2009 12:54:56 GMT

A new feature makes it easy to share links to stories from the River2 news page to followers on Twitter.

http://newsriver.org/r2ly.html

It also maintains a Top 40 list for River2 users.

This feature is itself a bootstrap since one of the key ingredients are the people using it. Just the right number and the right kind (thoughtful, passionate, forgiving, visionary) are needed. The result can be an editorial product in its own right. And the first experiences will ~~probably~~ certainly suggest the second and third level of features.

Bootstrapping is for users too. In fact at some point, without users participating, the bootstrap stops. You have to wait until they show up, or keep trying to figure out what it will take to entice them to participate.

I realized something yesterday, that bootstrapping is hard because you have to use things that don't exist yet.

I often use bridges as metaphors to describe bootstraps in software. Here goes..

People hardly notice driving across the Golden Gate Bridge, they might notice the scenery, or the walkers or bike riders, or think about the oncoming traffic. But one day, a long time ago, someone stood on one side or the other -- in San Francisco or Marin -- and looked at the Golden Gate (which was the waterway before it was the bridge) and thought "I bet we could put a bridge here." Nice thought, but then what? What's the first step? And when that doesn't work, what's the next first step? Long before there's a bridge, someone has to make the trip in a bucket hanging from a cable. Today, River2 users are those kinds of people.

I've been having an on-and-off discussion with Marshall about this. He's been frustrated by the software. I understand. But I need people like him to struggle through it, tell me what went wrong, so I can try to fix it, and then tell me if it worked. The rewards mostly come from within, to know that you played a role in making something new happen. In this case it will be a distributed loosely-coupled 140-character message network that is free from YWFFTMMR. And there might be business opportunities to provide services to users. No one's saying you can't get rich. But it has to be fair, and people should always have lots of choice.

One more note -- it's taking a long time for the domain I'll use, r2.ly, to come through the registration process. So to begin with we'll use a teamrss.com domain to share links. If you see one of these on Twitter it's coming from this part of River2.

What is the real-time web?

Tue, 22 Sep 2009 22:49:44 GMT

Four words: It Happens Without Waiting.

Narrative: Today I wrote a piece about the Berkeley Public Library on InBerkeley.Com. I wanted to find a pointer to the library website, so I switched over to Google. Looked up Berkeley Public Library. My piece, publshed less than a minute earlier. was the first item. Real-time web. (True story.)

The SUL as a tool to control news?

Tue, 22 Sep 2009 12:29:43 GMT

SUL is Twitter's Suggested Users List.

It's a group of approximately 500 Twitter users who are "suggested" to new users when they create an account. The stated purpose is to provide people to watch when you're starting out. But are there other purposes? Could it be used to reward positive coverage and punish negative coverage? I think we now have some data on that.

There's no doubt that Twitter has received a lot of help from the press, and much of it is genuine enthusiasm for a communication tool that at least hints at the future of news.

Many of the suggested users are news organizations, reporters, columnists, marketers, and as a result, most have over a million followers. Almost all of the top tech news organizations are on the list. And TechCrunch was one of them until something happened in July as is evident in this TwitterCounter graph.

Compare this to the graph for Mashable, over the same period.

And plotted on the same graph.

It's pretty clear something happened in July.

We know this much -- TechCrunch was dropped from the Suggested User List, right around the time their follower count started heading down. As to why, we can only speculate that it was because they ran a piece that Twitter didn't like.

7/16/09: Twitter's Internal Strategy Laid Bare: To Be "The Pulse Of The Planet."

People have always questioned whether there was a connection between being on the list and not being too critical of Twitter. At this point, there isn't much doubt that the connection is there.

What became of Radio's POST button?

Tue, 22 Sep 2009 02:34:58 GMT

An interesting story of evolving software.

In 2002, my company shipped a product called Radio UserLand. It was a very popular blogging tool, but it was also the most popular RSS reading tool of the day. And because it was both things, we could do integration that no other product had ever done before, or since.

Adjacent to every item in the aggregator was a button that said POST. When you click it, you flip to the blog post entry screen with the text of the item in the big box. You could add your own words, shorten it, whatever you like. When you were done, hit Submit and you'd have a post that pointed to the original article with your comments.

As an aside, this is where the RSS <source> element came from. We'd embed that, invisibly, in your post so tools could find their way back to the original. This was in response to an outcry from bloggers that we were helping people steal content. Seems like a foreign idea today, doesn't it?

Anyway...

Fast-forward to 2009, and I'm back at work in AggregatorLand, and like it or not, Twitter is where we push links to these days. So now instead of a POST button look what's there in its place.

Now, it is very much more clever than the POST button was back in 2002. Just how much clever -- you'll have to wait to find out, because I'm still working. But when you see my links to test.teamrss.com on Twitter you'll know that I'm testing the new stuff. Murphy-willing it should be released to River2 users tomorrow.

TopTwits tracks your Twitter linkage

Mon, 21 Sep 2009 14:41:11 GMT

When tr.im announced that it was going open source, I said I would also release the code of the app that does my Top 40 page and that of Jay Rosen, Kevin Tofel and Zach Seward.

TopTwits is that app. It runs as an OPML Editor tool. This means you must have the OPML Editor installed on your machine, and then install topTwits.root in the OPML Editor.

YWFFTMMR

Mon, 21 Sep 2009 07:00:00 GMT

When you get something going all of a sudden the ~~parasites~~ opportunists swoop in and want to take control. Funny how they leave you alone when you're not so hot.

I suppose I should see it as a good sign, but they all resort to the same kind of character attacks when I decline their offers. Some nastier than others. The offers amount to me working my ass off to make them rich, for which, in turn -- I get nothing. $0. Bupkis.

Only in the tech industry do people have the audacity to look you in the eye and say You Work For Free To Make Me Rich. YWFFTMMR.

The stupid thing about it is that's at least part of the reason I'm trying to get out of Twitter. I don't like their economic proposition but, I do like microblogging. I figure if I'm not going to make any money off my work, then I'll work in an environment where no one does. It's weird that the people behind Twitter are supposedly capitalists yet seem to not understand that very simple idea. People don't work for free. They don't pour out their passion in the cause of making you wealthy. They might be motivated to do it if they saw some upside.

I'll let you know when someone approaches this space with respect and an offer that isn't usurous.

Today's a big shipping day

Sun, 20 Sep 2009 16:31:28 GMT

I didn't plan it this way, but a lot of stuff is ready all at once.

Over the day I'll link them in here. They will also appear in various other places such as the rssCloud mail list, on Twitter, and I'm not sure where else.

1. A rssCloud walkthrough for publishers. This doc is a shortcut for publishers so they don't have to wade through the michegas that cloud and aggregator developers do.

2. A self-documenting OPML reading list of real-time photo feeds. Lots of new stuff working here. I also added a photo feed for television, in anticipation of the Emmies tonight.

3. DNS for RSS feeds. I have a proof of concept app up, for you to try out and comment on.

N. And last but not least, and I had nothing to do with this one -- the Scobles shipped another Scobleizer last night. Baby and mommy are happy and healthy, and according to daddy, cute. Happy birthday Ryan! It's nice to have you on board for this crazy thing we call life.

Hooking the lizard brain up to the cerebral cortex

Mon, 21 Sep 2009 03:21:15 GMT

This is worth a special post as a followup to the earlier piece, it's so interesting.

I was talking with a friend today, he's an expert in DNS, and I said it's too bad people can't open their supercloud.org sub-domain in a browser. I understood why this wasn't possible:

1. The browser is looking for an A record or a CNAME, and we're setting the TXT record.

My friend who doesn't want me to say his name (and I respect that) said it might not have to be that way. We talked about connecting the DNS with HTTP and it was really intriguing. If you think about it, DNS is like the lizard brain of the Internet, and HTTP, while it is in some ways lower tech than DNS, is the cerebral cortex. To have them integrate is like bringing the Wright Brothers plane they flew at Kitty Hawk on an Apollo moon mission. But that's the way we do things in techland.

Then I thought -- wait a minute, what's to stop me from also registering an A record that points back to me, and then keeping a database locally that associates a name with an RSS URL. Then when you open your supercloud.org subdomain in a browser:

1. It comes to my machine.

2. It looks you up in the database I'm keeping that's a mirror of what's in the TXT records in DNS.

3. From there I get your RSS address.

4. To which I simply do a 302 redirect.

Hah! I had to try it. And guess what! The fcuker works!!

Try it out: http://davewiner.supercloud.org/.

And then go through the registration process, and try it out yourself.

This is actually important because everyone expected taht you'd be able to open the supercloud.org sub-domain in the browser. Now you can.

DNS for RSS feeds

Sun, 20 Sep 2009 17:12:44 GMT

Here is a Unix shell command that gets the address of my RSS feed: dig +short davewiner.supercloud.org TXT

It makes a DNS call to get the TXT record associated with davewiner.supercloud.org. That's different from an A record or a CNAME record. TXT records are used for things like this. That's why when you go there in your browser it doesn't go anywhere.

I have an app up for you all to try out, so you can have a supercloud.org domain for your RSS feed (it works equally well for Atom, or anything that can be parsed as XML).

http://dns.rsscloud.org/

Enter your Twitter username and password and the URL of your feed. It verifies that it is the correct password. (And it doesn't store it or use it for any other purpose.) And then if everything goes well, your username will map to a supercloud.org domain and point to your feed.

This is of course not rocket science.

But it does seem to work.

PS: Why this is interesting, possibly, is that it may give us a way to shorten URLs and make them more flexible if we want to build a loosely-coupled Twitter-like network with feeds distributed around the net. DNS is the little bit of centralization that the Internet, itself a loosely-coupled network, is built on.

Fuck you John Edwards

Sun, 20 Sep 2009 23:53:56 GMT

I read the front page story on John Edwards with delight, as I'm sure many others did.

Yes it is news, to those who ask, because he came close to being VP, and was a serious candidate for President and the press was ignoring the story until they couldn't ignore it any more.

I remember when Edwards came to speak at Gnomedex and everyone said he was great and he should have a webcam everywhere but in the bathroom and bedroom. They wanted the transparent candidate. He thought it was a good idea.

Now we know for sure, but I thought so for sure then, nothing about the guy was real. His story was good, too good. His wife has cancer. Meanwhile, according to the Times he's promising to marry the mother of his out-of-wedlock child as soon as the wife dies, in a rooftop ceremony with Dave Matthews providing the music.

This is the end of John Edwards, who if he could have kept the act up a bit longer might have been the Attorney General of the United States. He's fucked. And maybe we should reflect on just how fucked up we are that we go for that kind of nonsense. Edwards was our ideal of what a candidate should be. A commercial product advertised on TV with the judgement of a con artist. Just the kind of guy we want a heartbeat from the Presidency. Not.

Yes this is a front page story because who Edwards is says so much about who we are. Edwards is a good serious look in the mirror.

DNS progress, another wish for a miracle

Sat, 19 Sep 2009 17:20:47 GMT

There seems to be much progress on the DNS project I asked for a couple of days ago. Two or three developers seem to be approaching the point of deploying and the APIs probably are pretty close. I hope that when they surface we can try to get the APIs into agreement. The differences appear to be cosmetic.

There's another project I'd like to wish into existence.

I fell in love with the realtime API implemented by FriendFeed. Very simple, clever, reliable, functional, a kind of no-brainer that I wish more APIs were. The only problem was it only worked with FriendFeed, which is cool because there are a lot of interesting people there, but it got even more interesting when they released the back-end server, Tornado, as open source.

This is good for two reasons: 1. It's really good interesting technology, and 2. It doesn't come from Google, it comes from Facebook, one of their rivals. Diversify is always a good strategy in the financial markets and in the tech world too.

Anyway here's what I wish for. I'd like someone to show up in the rssCloud communithy with a Tornado server running that either is an instance of rssCloud (preferable) or is connected to an instance (i.e. receiving notifications). I'd then like to hook River2 up to this server through a persistent realtime connection so that it can be notified of updates through a NAT and firewall. Then of course I'd write up a Howto explaining how I did it, ask for feedback and hope that people create more software that interops.

How about it? Anyone up for making some history??

PS: I wasn't really a Deadhead, but I have lots of friends who are, and I admire much of the spririt of the community. One of the things they invented was this idea of needing a miracle. It comes from a John Perry Barlow/Bob Weir song. People talk about wanting "miracle tickets" -- that is, a ticket that gets you into something for free, like a Dead show. But the concept applies to airlines, baseball games, movies, private parties, you name it. It also applies to new Internet bootstraps. To get something like rssCloud booted, like Barlow and Weir, I need a miracle -- ever-ee day!

He has a million followers

Sat, 19 Sep 2009 01:02:19 GMT

Scott Simon is on the SUL and boasts in the first sentence of the description of his interview with Clay Shirky that he has nearly 1 million followers.

"Host Scott Simon has nearly 1 million followers on Twitter."

Scott Simon is an expert on Twitter because he has nearly 1 million followers.

He actually has 1,022,105 followers and follows 56.

I once replied to him on Twitter because he said something dorky about blogs on his NPR show. He said that people talk lovingly about their newspaper, it's "my paper," but no one says "my blog."

I told him that I have a blog and I'm proud of it, and I've never had a newspaper. I give money to NPR every year, but I wonder why. He's one of the smartest people there, and as you can see, he'd not that smart.

But I do love Radio Lab. They're on Twitter too. They don't have a million followers. Thanks!

The RSS channel-level image

Fri, 18 Sep 2009 16:45:07 GMT

A heads-up on something that's going to prove useful down the road, something you might want to start thinking about now. Credit for this observation goes to the brilliant Frenchman, JY Stervinou.

One of the things about Twitter that really works are the 48-by-48 images they call avatars. They quickly become symbols for the person. When someone changes their avatar it's surprisingly important. I changed mine from King Kong to Don Quixote and people started treating me better. Not kidding. People really want me to use my face, but it bothers me to look at my face all the time. When I figure out how to have two views of myself, one for me and one for everyone else... Anyway.

If we're going to bootstrap a Twitter-like network outside of Twitter we're going to need those avatars. And luckily there's a very nice place to put them, the RSS <image> element. It's as if when Netscape spec'd RSS 0.91 they knew that 10 years later we'd need this.

The only problem is that most RSS images aren't 48-by-48 (of course) and most of them aren't square. That's what you might start thinking about, creating a square graphic that looks good. Since many people and organizations are crafting Twitter versions of themselves, this should be a relatively easy thing to do.

BTW, I include the Twitter avatars in the cloud-enabled feeds I maintain for all the people I follow. Here's an illustration.

Radio Lab

Fri, 18 Sep 2009 14:27:57 GMT

Once in a while you come across a gem like the Radio Lab podcast. I listened in delight a few nights ago when an episode about death aired on KQED. An hour of philosophical and scientific stories about death, a subject we all must spend a fair amount of time thinking and dreaming about.

A couple of examples.

The three stages of death: 1. Your body dies. 2. It's buried or cremated. 3. Your name is spoken for the last time. They postulate that your soul is in limbo until you reach stage 3. For most people that day is the day you're joined in death by your last loved one. And this makes the point that fame, which so many peope seek, may not be such a great thing. After you reach stage 2, you have no influence on how your name is used. The poor farmer whose property was turned into a college after he died, and was named after him, now must wait until the college closes. And that might not happen for quite a while.

The next idea is about a dead language, Latin. What does it sound like? No one knows, because the last native Latin speaker died centuries ago. But think about pottery. It's spun on a wheel while wet. Maybe, just maybe while it's being spun, the grooves faintly record people speaking around the potter. So a Roman vase might be an ancient phonograph record and contain echoes of long-dead Latin-speakers.

These ideas aren't useful but they touch something inside me that I like.

Bravo!