Sean Middleditch

I’ve been thinking a bit about how much today’s collaborative communication systems suck. I’m not refering to real-time chat systems, like IRC or Jabber, but larger systems like mailing lists, newsgroups, and web forums. There are some serious problems to both.

Let’s take a look at web forums first. The biggest advantage of web forums is that anyone with a web browser can use them. The biggest disadvantage is that it’s a web-only medium, and web interfaces tend to suck, unless you work for Google.

Specifically, with web forums, the problems are:

* Threading sucks at best, and is practically non-existant at worst, making following a discussion difficult
* All data is on the server, so a lot of rich per-user data can’t realistically be kept (like which messages you’ve read) without massive disk requirements
* No integration between different forums - every forum must be in its own window/tab, has its own interface, and in general is just completely separate from all other forums.
* No real cross-posting support for when a discussion affects multiple forums

Now, mailing lists are the popular alternative to web forums. Mailing lists themselves have several problems, however.

* They require use of a mail client, so you lose universal access unless you also use web-mail
* All information is stored on the client and never retained on the server (except for lsit archives - see below), so if you somehow miss a message or join the list after an interesting discussion, you never get those messages in your mail client
* Similar to the previous item, once you delete mail from your local storage, you lose it forever and cannot retrieve it - also happens if you must reinstall your machine or switch to a different one and you aren’t using IMAP on a server with lots and lots of storage space
* Thread still sucks, mostly due to clients not supporting certain mail headers necessary, and because clients all provide different UIs that don’t always map well to group discussions

Mailing lists and web forums can be combined in many ways. The most common example is a mailing list archive. Mailing list archives are generally read-only, however, so they don’t really facilitate discussion. Their biggest use is for casual perusal of discussions, and for working around the problem noted above where mail clients cannot ever get access to old messages.

The simplest and best way to improve a mail list archive would be to simply allow the web interface to post messages in addition to read them. Basically, it’s a simple web form that lets you post new messages or reply to old ones, supports all the proper mail headers for handling threading, and uses the email address and password you used to register with the list. Away from your normal mail client/server? You can still be part of the discussion. Missed an older discussion or lost your local archives? Go to the web interface and reply there.

Unfortunately, in this scenario, mail access is still prefered, and it still sucks because you do need to use a second interface on occassion.

The second form of mailing list and web forum integration is when you have what is primarily a web forum, but with mailing list featuees hacked in. The simplest form of this is when you let users subscribe to be mailed whenever a new forum message is posted. Some of these will even use the right mail headers to make the mails appear threaded. Then all you need is to make it possible to reply to these messages and have those mails appear in the forum.

The biggest problem with this approach is that you end up coding against a least common denominator. Most web forums are not truly threaded. Even when you have rich threading and meta-data for forum messages, the mail clients will then not support them, or at least not support them fully - many mail clients can’t even properly handle mailing lists.

If one were to design the ultimate discussion system, what would it look like? Probably something with the following features:

* Rich meta-data support that allows accurate tracking which message a new message is in reply to and which thread each message is in
* Based on the above, allows replying to a message while also making it start a new thread, since discussions tend to branch out in this fashion rather frequently
* Also based on the above, ability to note that a message was in reply to a mesasge from another discussion area, and which one
* Web-based access so anyone can get to the system
* Ability for local clients to access the system, and other systems, providing a universal integrated interface
* Identity and authentication support (anti-spam and forgery)
* File sharing for patches and screenshots and such

Anyone who’s been paying attention can probably already think of a system that does all the above: newsgroups! A system older than both mailing lists and web forums already does exactly what both of those systems fail to do properly.

A good question, then, is how come newsgroups are not in frequent use today? Why don’t big project hosts like gnome.org, kde.org, sourceforge.net, and so on host newsgroups? Why don’t the majority of Internet user utilize newsgroups? Why don’t most ISPs even provide newsgroup features?

In large part, I believe, this has to do with newsgroup architecture. A lot of it has to do with perceived architecture problems. For example, many popular newsgroups are part of Usenet. However, a newsgroup doesn’t have to be a part of Usenet. A site like gnome.org could host its own private newsgroup server, and anyone who wants to use the newsgroups with would have to use that server, or one that explicitly mirrors it.

However, doing this requires you to actually be able to install a newgroup server. A lot of hosts provide mail services, web space, even a MySQL database, but not newsgroups. You’re stuck with using Usenet in this case, which makes it difficult to even get the newsgroup started.

Another real problem with newsgroups is their lack of integration with the web. Aside from Google Groups, you must install and use a local newsgroup client.

Google Groups is actually an important point. Google Groups provides what is definitely the best web-based access to news groups, bar none. It also now provides a service to let anyone register a new group which provides both a good mailing list interface and a usable web-based interface. Unfortunately, this system does not support newsgroups, and suffers from the least common denominator problems mentioned above.

A new, simpler system could be written that takes all of these problems into account. It would be coded with a specific feature set in mind, and only access methods that support that feature set would be fully supported. Let’s define how this system works on the low-levels, first.

The system will keep a list of messages in a database. A message has a body and zero or more file attachments. The meta-data of the message would be the user that posted the message, the parent message id if the message is a reply, the date and time of the posting, and thread id. That’s it.

Threads will also be stored in the system. A thread will contain a subject, the id of its parent message, and the id of the parent thread (for when threads fork). Again, that’s it.

For retrieving messages, users may search the database for a list of message ids that match the search criteria (thread subject, author, content, etc). Messages can be retrieved in whole, without attachments. Individual attachments may also be retrieved.

Posting will only be allowed when following these rules. In addition to a body and attachments, a new message may either have a parent message id, a parent message id and a new thread subject, or just a new thread subject. Those correspond to replying to an existing thread, forking a new thread from an existing thread, and starting a new thread. Submission to the database must use a transaction to ensure that the new message and thread are created together, and that any attachments are entered all at once.

Writing a wrapper API around this database at this point would be the best idea to ease implementation of software that accesses. At this point, we are ready to implement our first access method - XML-RPC.

This system would provide a standardized XML-RPC interface for querying and posting to the message database. This API allows for some very interesting tools to be written - automated server tools that process messages posted to the system, local clients that can query old messages, etc. We then build in what are, essentially, legacy access methods, and the web method.

The web method is the most important, as it is the one that provides the universal access. Both reading and posting must be supported. This will, essentially, be a web forum. One that is designed around a clean message and thread model.

Newsgroups are another obvious access method. The functionality maps quite well. One could even argue that this system is really just a new newsgroup server written with a more modern backend and with no support for replication.

The email interace would likely be the hardest, simply because email on its own does not map well to group discussion at all, and the various “extensions” to email are not widely and uniformly implemented. At the very best, email support would be clunky. However, its important to have as many people want a local client and want to use their existing email client. It’s just important to note that while these users will be able to participate in the discussion, they will suffer with most of the existing list problems. The new system, however, not being designed around email, won’t make all users suffer those problems.

Ideally, new clients will be written using the XML-RPC interface. Clients like Evolution 2.2 and Outlook even make it possible to include features of the new system into those clients, integrating with their existing infrastructure (including email).

Again, as noted above, the system is pretty similar to a newsgroup. However, there are some big advantages. The first, and perhaps best, is that both of the most important interfaces - the web interface and XML-RPC interface - can implemented entirely on a web server with PHP, Perl, or something similar. It’s not easy to get free or cheap Newsgroup hosting, and it’s not at all triviail to get a new Usenet group for small projects or personal discussion. It’s extremely easy to get cheap or even free web hosting with PHP and a database, however. You could drop in a few PHP scripts and setup a database and you can have both the universal web access and rich-client access over XML-RPC.

Integration with newsgroups and e-mail are tougher, but still possible. The web system could probably, at the very least, provide a read-only form of the email interface, and external services similar to Google Groups can be written that integrate external newsgroup and email interfaces to the web-based server. Simply writing an Evolution plugin, and/or a client library for mail clients to use, would help facilitate adoption of the new system. Granted, wide-scale adoption would likely be years in the coming, if it ever does come - but the system can still be useful now even without widespread popularity. People that want mainstream access can stick with their legacy mailing list or web forums or newsgroups, of course.

The local clients would be interesting to use. Even plugins to existing mail clients. The interaction is slightly different than a mailing list. Mail clients generally know how to ‘Reply to Author’, ‘Reply to List’, and ‘Reply to All.’ Many are missing the list reply. This system is designed to support three kinds of replies which are slightly different - ‘Reply to Author’, ‘Reply to Message’, and ‘Reply in New Thread’. The first method would be a normal mail message, sent to the e-mail address of the message author, using normal mail interface. Specialized clients would invoke the user’s mail client, for example. The next two are different in several key ways. First, when posting, they obviously have different outcomes - one adds the reply to an existing thread, while the second starts a new thread but retains the meta-data noting the original thread and replied-to message. Also, in the case of forking a thread, you’d be able to enter a new subject, while replying to an existing thread would not allow a new subject.

I’m not sure that Outlook plugins could handle the above interface changes, although I do believe that Evolution 2.2 plugins could do it. Overall, it would be around as difficult as adding newsgroup support to an existing mail client.

Over-all, is this a worthwhile project? In all honesty, I expect that, for the most part, it would result in little more than a new web forum, albeit one that is possibly much cleaner and better suited to real discussion than the existing popular forums. With even just an Evolution plugin, though, the project could be useful to enough people to be worth implementing. I doubt it’ll take over the world of group discussions anytime soon, if ever, but then who knows? Who expected wikis, with all their many interface warts, to become so popular?

On the topic of Evolution integration and wikis… A wiki board is, generally, entirely web based. Imagine, for moment, that you had a local client that was designed solely for wiki editing. The wiki engine in use would have to be standardized to use a particular markup language; HTML would work as well as anything else if you have a local client with rich-editing. Imagine having what is, essentially, a word processor window open that accesses a document stored online using WebDAV (that is, with the actual Versioning part of DAV active), but which is also viewable and editable online using nothing more than a normal web browser. WYSIWYG wiki editing with a rich local client, local real-time notification of edits and updates… that would be mighty cool.

David is right. SPF is fundamentally flawed, or at least the part of it that handles sender authentication. It simply has no way to correctly validate mails from forwarders or other services, not without using the monstrous (and broken) SRS protocol.

Unfortunately, there aren’t any decent alternatives out there yet. I’m going to, for the time being, continue to use SPF on the receiving end, and if people are using -all commands in their SPF records, too bloody bad for them if they want to send me mail. I am removing my SPF records for my domains, though, since they really are pretty useless overall. Of the “actually out there in the wild” anti-forgery systems, I think DomainKeys looks like the most promising. Unfortunately, the way its implemented makes it pretty difficult for me to use it, since I can’t (freely) setup a _domainkey subdomain on my DNS providor, DynDNS. It would be a lot nicer if I didn’t need a subdomain in order to configure DomainKeys. Of course, I also use DynDNS’ outbound mail server, so the best bet is to just wait for them to use DomainKeys. The early-adatper/geeky-hacker in me thinks that solution is no fun, though. ;-)

I patched gnome-vfs to support GSSAPI, and it works. See bug #159663 for the patch against gnome-vfs 2.9.91.

I got the original GSSAPI patch for Dovecot from the Dovecot maintainer. Unfortunately, it doesn’t apply against either 0.99.14 or 1.0-test. With the gnome-vfs patch, Dovecot is the last piece of the single sign-on puzzle. I looked into other IMAP servers, but the only others that support GSSAPI are UW-IMAP (yuck on a stick) or Cyrus (and its weird, proprietary mail store). I’ll stick with Dovecot. I’m going to see if the maintainer is interested in getting GSSAPI in by 1.0; if not, I’ll give it a try myself.

Woot! I got Exim 4.44 to authenticate using GSSAPI. Yay me!

Here’s what was necessary. First, you need the cyrus_sasl authenticator. This is not compiled in by default in the Debian packages (I’ve filed a bug on this), so you must compile it in.

Next, you need the usual Kerberos configuration. Create the principal (service smtp) and make a keytab for Exim. I put mine in /etc/exim4/keytab. Make sure its only readable by Exim.

Now you must configure the libsasl2 for Exim by making the file /usr/lib/sasl2/exim.conf. You need to specify the srvtab to use. I also told the library to saslauthd for authenticating passwords when the PLAIN mechanism is used (since I need to send mail from afar when I don’t have Kerberos tickets). My exim.conf looks like:

srvtab: /etc/exim4/keytab
pwcheck_method: saslauthd

You need to configure saslauthd if you use that. In my case, I needed to add the kerberos5 method in /etc/default/saslauthd.

Now you must configure Exim. Aside from TLS configurations (won’t detail those), I use these two authenticators in my /etc/exim4/exim.conf

plain:
  driver = cyrus_sasl
  public_name = PLAIN
  server_advertise_condition = ${if eq{$tls_cipher}{}{no}{yes}}
  server_set_id = $1

kerberos:
  driver = cyrus_sasl
  public_name = GSSAPI
  server_hostname = %host%
  server_realm = %realm%
  server_advertise_condition = ${if eq{$tls_cipher}{}{no}{yes}}
  server_set_id = $1

Replace %host% and %realm% as appropriate.

Now, that is mostly what I had done months ago. Why wasn’t this working?

The problem is that Exim limits SMTP commands to 512 characters, but GSSAPI authentiction requires longer lines. At least, it did on my network.

In Exim’s source, in src/smtp_in.c:37, there is an option to set the SMTP command line length. I increased mine to 2048 and things work fine. I sent a bug report to the Exim folks noting this problem and asking for a real fix.

In any event, GSSAPI now works for sending mail. Yeah!

Only two things left for complete GSSAPI usage on my network: getting Dovecot to handle it, and getting gnome-vfs to use it. There’s a patch for Dovecot that exists, but I can’t, but am looking for, and gnome-vfs will apparantly need a newer neon (with API changes) so I’ll have to wait on that.

I finally got Mozilla (and Firefox/Epiphany) to work with GSSAPI authentication to my web server. The trick is that a special configuration key, network.negotiate-auth.trusted-uris, must be set with a list of the servers that may use GSSAPI. It’s a security issue, basically. One I’m not exactly sure is warranted, but one that’s there none-the-less.

I’m currently testing a patch on gnome-vfs bug #159663 to enable GSSAPI support there.

… and no, it doesn’t work. Drat.

Down to owing only $6,500 on my Ranger. A chunk of money back from the government helped with that, as has not spending a whole lot of money on anything unnecessary.

Something really nice, however, is that my insurance payments have dropped. By a lot. Before, I paid around $3,500/year in insurance. This year, it’s only $2,200. Yay me. ^_^

I figure that, based on how quickly I’ve been paying off on the truck, it’ll be paid off in a little over six months. I expecting some additional pay, but also expecting to splurge a little on stuff like chainmail for KANAR, so I figure for now that’ll mean that I break even in the extras department.

That leaves me at my original estimation of having the truck paid off entirely by the end of summer. That pleases me muchly. :)

Was just informed that a fellow player at KANAR has passed away. Jay Abu-Awad. Far too young to die.

I didn’t know him very well, but he was definitely… definitely a part of the entire atmosphere of KANAR. You couldn’t be around Jay without having fun, one way or another. He wasn’t just a great player of a game, he was an all around awesome guy. The loss to his close friends and family must be staggering.

Death never feels right. Goodbye, Jay.

I have found the maker of some of the most beautiful instruments I’ve ever seen. I am particular fond of The Device Formerly Known as Ethel.

Unfortunately, there are two problems with these instruments, from my personal perspective. The first is that the maker, Stephen Owsley Smith, is not taking orders at the moment. Second, even if he was, instruments of this quality cost many, many thousands of dollars more than I have, or will have anytime soon. I’ve seen instruments of lesser quality cost over $10k. I don’t know what Steve would charge, but I’d imagine it being too much no matter what.

When I do eventually get a bouzouki, it’d looking a lot like I’m going to be getting a Trinity College instrument. I’ve been reading up on customer reviews, and they seem to all be pretty positive.