Unstalling SOAP

Thursday, March 29, 2001 by Dave Winer.

Introduction 

This has been an unusually heavy week in the politics of SOAP 1.1 interop. As I've been working I've been taking notes in a series of documents that I call Dave's SOAP Journal. This essay is a set of lightly edited excerpts from that journal. I want to help people see the human side of the technical work. If there's something you don't understand, just go right through it, I'll see if I can address people's questions in a followup piece.

Where we're at with SOAP interop 

Imho, it's going to take a miracle and a lot of compromise for SOAP to coalesce into a single protocol that interops. The spec for SOAP, of which I am a co-author, is not the best spec in the world. It's got lots of mistakes and contradictions, and it points to other specs which are moving targets.

There's a mail list where SOAP interop is being discussed among implementors. It's a fascinating place because there are lots of small developers, like my company, UserLand, and two or three really large companies, and more showing up every day. It's like a baseball game where everyone showed up with a mitt, a glove, a ball and a bat, ready to play, but the rulebook was written by an evil non-athletic lunatic. We all seem to want to work together but the blueprint is in the way.

Adding to the confusion, Microsoft and IBM, apparently thinking it was solely up to them to decide what SOAP is, added a complex layer called WSDL, in private, without consulting anyone I know, and there's an idea that it is somehow related to the base layer for interop. To be fair, some independent developers agree with them. I have no interest in spending months writing code to support something whose benefits are so vague. On the other hand, when I ask that they support XML-RPC syntax, they say they don't have the resources.

Anyway, today, there is no single thing called SOAP, when you look at reality there are lots of things people call SOAP, and they don't work together. Now maybe that will change, I hope so, but that's where we are right now.

A loop 

This is a loop for me. In the old days before the Internet we tried playing this game with Microsoft and Apple on the Mac. They would always cut the deck 26 cards for Microsoft, 26 cards for Apple, and we'd always end up saying "Whoooa, where are our cards?" I learned then that the gorillas appear to be saying they're doing things that they're not doing and vice versa.

There's always a moment of surprise when you find out that assuming the worst was the right way to think. But back then I didn't have a website to document this stuff, and I didn't have XML-RPC as a great insurance policy against such misbehavior. All they can do is screw SOAP, they can't screw the independent developers if we act in our own interests. (But that's a big if.)

The gorilla's game 

The road to completion is infinite when you play the gorilla's game. You'll always still need to do One More Thing to be cool. And when you do it, they'll break the deal anyway. Claude Bullard said of a similar process in the W3C "Thou shalt not deploy anything that has not been deployed before you deploy except if said deployment shall slow the deployment of the fastest among you."

Playing the gorilla's game you can't be first. They always get to be first, no matter what.

Now some of the developers involved in SOAP shrug their shoulders and accept the dominance of the gorillas as "reality". It's not reality. Don't believe it. These are struggling companies with no philosophy. It might be possible to make a few bucks doing what they want you to do, but in the end it's a losing strategy.

XML-RPC did not go away, even though they would like to believe it did. Interop is not an elusive thing, it's here now. Just cut bait and switch gears and work with other developers, as Benjamin Franklin observed "We must all hang together, or assuredly we shall all hang separately."

So what's the take-away on this? I'm still working on interop in SOAP. I will not do WSDL. We will do RPC with SOAP, using SOAP syntax as we use XML-RPC syntax. Let's interop at that level. And if there's noise from the BigCo's that somehow this isn't enough, remember that they have infinite time and infinite money and we have neither.

Transparency 

Talking with Paul Andrews the other day I asked if this stuff was going over his head. I'm trying to make it understandable to people with non-technical backgrounds. We have a seat at one of the most interesting tables in 2001, perhaps the most interesting table. I want my experiences to be transparent, I want all the glitches, near-flames and full-flames to be out in the open so we can resolve our differences.

Others are not seeing it the same way, and are working in private, and in doing so disempowering people they don't include. I give you my word that my role in all discussions and my thoughts and pushback will be visible. I can't stop and answer all questions, there isn't enough time, but later when people ask how this happened, you'll be able to see what I did, even if you can't see what others did.

This method has worked well in every stage of SOAP, the only times we got in trouble was when the decisions were made behind closed doors. I believe you can see that result in the SOAP 1.1 spec which is a marvel of compromise and confusion. Our job is to clear that up so we can all move forward compatibly. I want a spec that's a marvel of clarity and ease-of-implementation. I want the barrier to entry for SOAP to be as low as possible, so as not to favor Microsoft, which has many thousands of programmers.

Two choices 

I see two choices. The first is to document the common elements of various SOAP implementations we've been working to interop with. The second choice is to punt on that and define a SOAP envelope for XML-RPC messages.

The advantage of the first approach is that it's kind to SOAP and won't jar the existing SOAP community, which is large and confused.

The advantage of the second is that it opens the gates for all the XML-RPC implementors to participate in the SOAP world, with a minimum investment. I believe our energy is very much needed in the SOAP world.

Microsoft 

Wednesday morning, 7AM.

As a Mac developer, I could admire Microsoft from a distance, even though they played the BigCo game in Apple's space at the expense of the SmallCo's (Apple did it too). I admired Microsoft because they were smarter than Apple about technology. But as I've gotten closer to Microsoft, the same thing has happened with me that has happened with virtually everyone else that's tried to work with them.

I've come to see them primarily as a bully that uses its advantage at every step, whether they need to or not, to achieve their goals, and they play on peoples' ignorance and willingness to believe that somehow they're special, that Microsoft respects them even as they dis the others. Even in an area that's probably of minor significance to them, they don't ever show restraint, repeated calls for them to back off and let people work openly are ignored.

There's lots of offlist activity, and when it emerges Microsoft is always there. "Oh we're just tool vendors," they like to say. Uh huh. A tool vendor with 45,000 employees, the dominant desktop operating system, and $27 billion in cash.

My experience now matches the stories I've heard from others, they must have a playbook for dealing with outsiders, and they stick to it. I don't believe they can participate in an open development process without spoiling it.

I know many people are nodding as they read this, but I had to do the experiment, I'm that kind of guy, I had to get the data.

Survey results 

On Tuesday I ran a survey asking Scripting News readers and members of the soapbuilders and xml-rpc lists what they think the outcome of the SOAP interop work will be.

31 percent chose "SOAP will be a balkanized 'standard' with major (or minor) differences between the implementations. There will be a group centered around Java, one around Microsoft, one around independent developers, or some combination thereof. Interop across the 'clubs' will be very problematic, with all groups blaming the others for the lack of interop." It was the highest-ranked choice.

19 percent chose "Works with Microsoft."

13 percent chose "A simple subset of SOAP will be indentified and documented, much like XML-RPC, but using SOAP envelopes."

My choice 

I voted with the 13 percent and that's where I'm putting my bet. I'll do some work over the next two days to draft A Busy Developer's Guide to SOAP 1.1. If it works, you should be able to adapt an XML-RPC implementation to be a SOAP 1.1 implementation in two sessions, one to get the basic code written, and one to test and debug against a validator.

It's a bet-hedge against the most popular choice. If it's going to be a balkanized standard, I want to be sure that my software is in at least one of the major groups. It would be disastrous for us if we had to go with the "Works With Microsoft" choice. That's a losing proposition. At some point the hoops will get too numerous and we'll have to bow out. Better not to waste any effort on that choice, it's totally unworkable.

Examples 

Last night (Tuesday) I asked Jake to prepare examples for A Busy Developer's Guide to SOAP 1.1 that, like the XML-RPC spec, will show a developer what a request-response looks like. This will be the core around which the spec is developed.

The first draft is ready 

Wednesday afternoon, 2PM.

At first I thought I'd do the writing and then pass it by Jake, but he knows so much more than I do about the twisty maze of SOAP interop, and how big it is, and what the other developers are doing. So we wrote the spec together, me writing and Jake correcting and explaining. It was a great process. We've never worked so closely together, in real-time. Jake is always proving smarter than I thought he was (and I always thought he was smart).

It took about six hours to get the spec done, but in those six hours I think we accomplished more than the soapbuilder process would accomplish in six weeks.

At about 2PM we passed a pointer over to Simon Fell to review. He's also been involved in the interop process on a daily basis, and has been working closely with Jake.

I am now confident that I understand all I need to know to get SOAP interop at least among the developers who want interop, more than they want to be friends with Microsoft. I would love to see Microsoft say they're going to support the Busy Developer's Guide, and then we're done, and we can get back to adding features and fixing bugs (and adding bugs) to our software, which is how we earn our paychecks. I'm sure this is something a Microsoft person could understand.

http://www.xmlrpc.com/aBusyDevelopersGuideToSoap11

Forehead slap 

As I've admonished people for not working in the open, a simple thought occurred to me re the BigCo's and the way they work with the SmallCo's.

Another way of seeing the Internet. Sometimes it distills down to a single sentence and here it is.

"The Internet is the Intranet for The Rest of Us."

Working in the open is the only way to achieve interop. When this is done if all parties are enabled we'll have a strong standard. If the work is done privately we'll have a balkanized standard. I think it's that simple.

So please consider our open proposal for interop in SOAP 1.1. If we can get there quickly, the Internet can be more than a venue for warfare between the BigCo's.

Thanks for listening.

Dave Winer