August 03, 2004
Chandler's Network Architecture: Priorities Change

We've changed our priorities for the network architecture we will use to support sharing and collaboration in Chandler. If you follow OSAF's work here or here, this won't come as a surprise, but as it's a pretty significant shift it's worthy of elaboration and broader discussion.

For small groups, the use of an Microsoft Exchange server to support group calendaring has always been burdensome, but it has been the best available (or least bad) choice in many cases. Exchange is designed to be deployed and supported in medium to large enterprises with dedicated IT personnel administering it. The motivation for Chandler came in part as I saw my wife's small consulting firm struggle with the use of Outlook and Exchange to support critical scheduling functions. I surmised that a serverless (or server optional) architecture could be used to coordinate calendars for small groups, and that became a cornerstone of the Chandler vision. My supposition wasn't based on actual experience with peer-to-peer approaches as much as an intuition, which turned out to be less than fully appropriate.

The long and short of it is that initially Chandler users will share collections of items such as calendars with each other via a WebDAV server. Apple uses a WebDAV-based server for its proprietary .Mac service. Lisa Dusseault, who manages the development group working on WebDAV here, is also a long-time IETF participant and author of a WebDAV book.

Initially, and for the first wave of bleeding edge Chandler adopters, OSAF will be hosting servers itself. From an end-user point of view, the goal is to make sharing data painless and transparent. We are working on a product plan for an open source WebDAV-based server which can be freely deployed and which will become a fundamental part of our offerings. More on this as it develops.

This server will evolve into the server contemplated for the Westwood release of Chandler, the one for higher education. One of the benefits of the new plan is that we have a more direct path to bringing Westwood into existence.

We are also looking at incorporating the server into the client software as an additional option. Assuming we can do this, then those Chandler users who do not want any dependence on an external server can share data with other users directly. There will likely be some limitations to this approach, as the client will have to be online in order to perform this function.

True peer-to-peer coordination without this and other limitations is something which I ruefully understand to be a difficult (some would say unsolved) problem. If you have a lot people all trying to coordinate and there is no central "source of truth", when there is a partial loss of connectivity (say it splits into two subnets each of which is fully connected within itself, but which are isolated from each other) it is a challenge to put Humpty Dumpty back together again and make sure everyone has the most current version of everything. While achieving this is still very much in the long-term Chandler vision, our emphasis on not trying to solve research problems and even more on making useable sooner rather than later (our "dog food" approach) made the choice clear, if painful.

It has also occurred to me there may be another overlooked opportunity here. I think I've unfairly maligned servers in the past. It's not the server I dislike, it's the idea that as an end user I am disempowered if the work I want to do depends on the administration of a piece of software I don't control, can't get access to, and plays by a different set of rules. The PC-era pioneer in me says, "get rid of it". Another approach might be, "tame it and make it serve me".

Electricity comes out of the plug in the wall reliably (in the developed nations). Landline telephones have reliable dial tone. Why can't we have utility-level connectivity for user data? And why can't it be open source? This is a big, ambitious vision, and it's not just about servers per se, but operational reliability as an overall system function (think Google with its hundred thousand servers) but maybe there's something here. More on this later too.

A learning: We wound up focusing on the client software first, not the network architecture. Having had my formative software design experiences in the era of stand-alone PC's, it was easy to repeat the pattern of paying more attention to what is going on on the local machine than to what is happening across the network. While the repository is network-aware, and, in fact, very early versions of Chandler supported a simple form of accessing items from a remote repository, I didn't fully appreciate until this year the importance of considering issues of network availability, reliability, and performance as first-rate issues in and of themselves. You're never too old to learn (or to admit mistakes).

Posted by at August 03, 2004 09:46 AM

"No server needed to share information" was one of the most important points Mitch made 18 months ago.

My understanding is that the "dog food" approach is more important than anything else and I share this view.

I assume that everything else, e.g. 3 different operating systems, will be examined in the light of "dog food" as well.

I am looking forward to reading the final "dog food" plan - as announced by Mitch - next month.

Posted by: DaniGro at August 3, 2004 10:26 AM

It would be great if this could work off a fairly dumb server, e.g., Apache with mod_dav. Like that, Apache is basically a simple fileshare, but that's why it's so easy to deploy. It would be even better if it could work off a CGI script or PHP script; that's the kind of server that's become commodity, and a very well connected, very robust commercial web serving package is about the cheapest kind of server you can get.

Among blog software, these kind of dumb servers have been fairly successful. I suppose XMLRPC is another example of this. It's somewhat naive, and somewhat dumb, but it's easy to deploy. WebDAV isn't quite as easy to deploy.

Also with a dumb server, most of the control remains in the client, which means administration remains fairly easy, and many of the political issues of a shared resource are avoided.

Posted by: Ian Bicking at August 3, 2004 11:28 AM

Just as a point of reference, and as a possible way of solving the 'no-server' issue, which is a very compelling feature of Chandler, General Magic solved this problem among their Magic Cap PDAs by creating a scripting language called TeleScript. See for details.

When you wanted to schedule a meeting with someone else, you would use the calendaring application on one PDA, which would use TeleScript to have an intelligent conversation with another PDA and that's how the scheduling worked.

I'm not suggesting that you guys recreate TeleScript, but it appears that it's possible for Chandler to have an intelligent conversation with other Chandler clients in order to do scheduling without a server in the middle. I hope it's possible to do with today's technology that didn't exist in 1995 when TeleScript was created.

BTW, I think using WebDAV is great. Perhaps you could use it in a way that's similar to how Apple uses it from iCal and allowing people to subscribe to people's calendars that are published on WebDAV servers. Mozilla Calendar does the same thing.

-- Al

Posted by: Al Willis at August 3, 2004 01:10 PM

Note that Outlook has included these scripting features, and that has been the source of many of its worst security holes. Billions have been lost because of poorly-thought-out scripting features. That's a dangerous path.

Posted by: Ian Bicking at August 4, 2004 01:58 PM

Some comments in reply to the comments...

Ian, a dumb server would work fine for some use cases, but it wouldn't meet the requirements of demanding users. A plain HTTP server wouldn't allow you to browse a set of collections of shared Chandler objects, and multi-author use cases wouldn't work so well. A plain WebDAV server would be very close to achieving the use cases we plan, except for the ability to grant permissions directly to somebody from a different organization -- for that use case, we are thinking of extending WebDAV to support certificates or tickets.

The next natural question, of course, is whether we can write a client that can do a limited functions with a dumb server, and advanced functions with an advanced server. Yes, but that would require work. There's a tradeoff triangle here between effort required, functionality, and portability. We could have different levels of client functionality depending on how good your server was, but that would be a lot of effort. We could do something with a more reasonable level of work and high functionality but it wouldn't be portable to those dumb servers. We could do something not too hard and portable but it wouldn't have the functionality to meet all the use cases.

My recommendation for now is to continue programming for Apache + mod_dav. Later we can extend the server to be more powerful, or extend the client to talk to dumber servers. For now the base WebDAV functionality seems like a reasonable compromise. And it might turn out that for people who can't get a high-end WebDAV+extensions server, we can figure out ways to increase availability of advanced sharing servers -- or we can develop other ways to meet their use cases besides HTTP.

That brings me to Al's comment: Al, we do envision doing client-to-client item sharing, only instead of a remote programming model, we're thinking of transporting item data using email. Email is really the lowest common-denominator "dumb" server technology available to just about everybody, so it's very likely I can successfully email you an event I wish to share with you. This mechanism would technically have a server in the middle, but since it's only a dumb email server, it feels like peer-to-peer, and it even works when you're offline.

Posted by: Lisa Dusseault at August 4, 2004 02:17 PM

Why not sync through email? Each Chandler *program* could get it's own email address and the body of the email could contain xml updates to them. Then the clients wouldn't have to stay online, they'd hold up, they'd queue, and you could have the recipient return receipt so you know the packet made it out okay.

Have OSAF give out id's to different copies of chandler, and then people can link them to get peer-to-peer sync going/

Posted by: Aubrey xM at August 18, 2004 09:01 AM

Interesting, re: WebDAV. My comment is really a question and some comments together.

First of all I aplaud your effort. I spent a few years on the MeetingMaker board of directors and I can't stand MS' stranglehold on the corporate desktop which I believe can now be traced back to the closed nature of MAPI and Office file formats which has greatly exagerated the cost of groupware and hence the cost of building a small business.

How will your software make it easier on small work groups? If you're using a server based architecture how will your cost in terms of deployment, administration, hardware, learning curve, etc., be lower then exchange or meetingmaker or any other groupware/calendaring software out there? Even if you give your software away these issues must be addressed.

I hope you don't give up on a true serverless architecture.

I hate the fact that MAPI is only truly available via Exchange/Outlook but it does work. My gripes with it are:
1. too expensive
2. too hard to administer for a small company
3. requires a central server which is not cost efficient for small groups in terms of knowhow, hardware costs, time, etc.
4. not cost efficient to deploy in an ASP model (from what I can tell it's really no cheaper to buy Exchange via an ASP)
5. no non-PC support for MAPI which means no calendar/address book sharing beyond Entourage (which uses WebDAV as well). I should note that Goodlink and Blackberry do have client/server solutions for accessing MAPI functions from non-PC clients.

If you can solve the problem that your wife's company was facing you will effectively eliminate a big cost of doing business for the mittelstand of planet earth. That's a problem worth solving.

Good luck.

Posted by: Bill Barhydt at August 21, 2004 09:07 PM