Home > Archive >  2009 >  December >  23

How I develop formats and protocols

Wednesday, December 23, 2009 by Dave Winer.

I don't think I've ever written a blog post that explains how I develop formats and protocols, probably because I was afraid of getting flamed. Nowadays I'm not so afraid, so here's how I do it. 

1. First, I have to understand the problem. That might take a month or more of walking around Berkeley or New York listening to music and podcasts. When I finally feel I grok it enough to begin coding, I need to have at least two apps. One that produces the format and one that consumes it. So I start by writing very simple versions of those apps and usually run them on the same machine to make debugging easier. 

2. I connect them with a protocol that I know won't be the one I will use, but models it in some fashion. In the app I'm working on now, I had the server wait a random amount of time before returning the name of one of the 50 states chosen at random. On the client side, I have it open a connection to the server and wait until it gets an answer, which it just records in the database. 

3. Once I have the two apps conversing this way, I start to make the conversation meaningful. I take my time, really slow down and think, try out lots of approaches. I also do a lot of prior art searches to see if I can pick up ideas from people who came before. In this case, I looked at FriendFeed's realtime API, but I came up with something that's a bit simpler. It could be that the previous protocol can do more than mine, but I just wanted to flow updates across firewalls and NATs. (Even so, my protocol has a lot of room for growth.) 

4. Once I've settled on the format, and this could take a few days, I then split the apps between a real server and client and start to test it doing the work it was intended to do. Since I haven't done that yet, I can't tell you how it will go. I might loop back to step 3 and change the format based on what I've learned. I will certainly add logging features and metadata to the payload to enable debugging and performance monitoring.  

5. Once I'm satisfied that it works, I write an informal document for discussion and review. I don't sneak a look to friends or people I like. If I did, that would violate the very important principle of keeping the playing field level. Instead I'll post the document publicly and post pointers in various places, on the rssCloud mail list, on Twitter, probably on the Frontier-Kernel list since I'm trying to reactivate that community after a long period of staying out of it.  

Now some people say this isn't an open process, but in developing technology there has to be a time for thinking. We can't mind-meld like Vulcans. I require solitary thinking time to figure stuff out. So while people have a chance to spot problems in my proposed formats and protocols, there really isn't much room to argue over the minute details. There are always two or more ways to do something. Instead of calling a packet a <packet> I could call it a <blob> but what difference would that make? 

I have participated in the working group style of format definition, what I called the "deliberative approach" in yesterday's piece. But I don't like the results that come from that. I believe in designing these bits of tech as much as Jonathan Ive would design a new iPod or BMW would design a driving machine. To me this is an art, and in art there are always choices, and someone has to make them.  

The deliberative process is all about not making choices, putting all the possible ways of doing something in the spec. This gives an ego boost to the people on the working group, and a lot of grief to the engineers who have to deploy. In the end you have multiple profiles and lots of missed opportunities for interop. Having one person act as designer may hurt some egos, but the format works better in the real world.