Scripting News: NYU open source project

Home > Archive > 2010 > August > 2

NYU open source project

By Dave Winer on Monday, August 02, 2010 at 4:46 PM.

A group of four Computer Science grad students will work with Dave Winer, who is a visiting scholar at NYU, on a project to create an open source back-end message system for realtime applications. The work will occur entirely within the Fall 2010 semester.

A picture named nyu.jpg A prototype of the system is already running. The resulting software will be compatible with this server, so that apps being developed as front-ends will be able to switch transparently to the new software. The goal is to build a realtime framework that can support an open-ended set of interesting applications.

The students will not only learn how to code an efficient open source project, but will also learn how to develop and document possible standards, and how to refine protocols for simplicity and wide adoption, as well as work with a global community to promote adoption of the software and the protocols it supports. If the project is successful students will learn how to be leaders in open development. You will also get the satisfaction of knowing that your work made a difference!

The technology

There are two main entry-points:

1. getUpdates (username, password) returns array of struct

This routine doesn't return until: 1. There are one or more updates. 2. It times out.

If there are N updates, the returned array contains N structs.

Each struct has four or five pieces of information:

1. htmltext -- a string that can be displayed explaining in verbiage what the event was.

2. type -- a string that indicates how the update is to be processed. It's something a callback on the client can watch for. For the instant outliner, the type is instantOutline.

3. when -- the date/time the event was posted.

4. username -- the user who caused the update.

5. data -- optional, it's up to the poster of the event what it contains. In the instant outliner, it's the OPML text of the outline that updated.

If there are updates waiting it returns immediately.

The timeout period is up to the server. I have mine set to 180 seconds. When it times out, the array returned has 0 elements.

After receiving an update, whether it was an error or not, the caller should loop back and make another update call.

2. pushUpdate (username, password, htmltext, type, data)

The other side of the updating mechanism. Each user that's listening receives a copy of the update struct, which is returned when they next make a request to be updated.

The identity system is implemented through callbacks, so this system can be hooked into any existing system or new system. We are not trying to implement yet another identity system. There are already many, and lots of discussion of standards here, we don't want to get embroiled in this.

The API is accessible through a simple REST interface or XML-RPC.

The server software will be implemented in a high-level language, the data will be stored in memory or in a database. These design decisions will be discussed in the first weekly meetings, and a decision will be made by the Benevolent Dictator of the project (me). We will only produce one implementation in the fall semester. The focus will be on broadest-possible distribution, and best scaling. We want to create a server that can support thousands of simultaneous users receiving updates.

We will use the tools!

The group will meet weekly, but will communicate in realtime through the week, using the communication tools we are developing, from the beginning. We will only use open source tools for the project. The goal is to create something usable and fun without selling our souls.

We will finish the project with a half-day developer conference on the NYU campus where members of the team meet with members of the community to party and talk about the future and boast of our accomplshments.

Follow @davewiner

Previous / Next