Sean Middleditch

Another rant. I must be extra cranky lately or something.

This time, Ubuntu’s installer is under fire. Several releases back, Ubuntu switched to using a LiveCD as both their demo platform and as their primary installation platform.

What a fucking disaster.

LiveCD’s in general take a while to boot. They’ve got a huge OS to load, a ton of applications which aren’t even that useful on a LiveCD (the amount of configuration a heavyweight mail client like Evolution needs is just ridiculous in a LiveCD environment), CD drives aren’t particularly fast, and RAM is often a lot tighter than than most people would prefer even when the filesystem isn’t copied into it. Even on a nice dual-core machine with 2GB of RAM, the LiveCD can take a while to boot. A painfully long while On an old Pentium II with 256MB of RAM? You’re going to be waiting a good long while. I’ve downloaded and installed FreeBSD on a weaker machine in less time than it’s taking Ubuntu’s LiveCD to finish booting on some not-too-old machines, and booting is just the first step of many to get a usable system actually installed.

If you’re lucky enough for the damn thing to ever finish booting at all.

It has gotten down to the point where the LiveCD installer is useless to me. I can’t get Ubuntu to install on the vast majority of machines I have access to using the LiveCD. It takes forever to boot, it locks during booting, it locks after booting, it locks during installation, it takes forever for installation to finish, or some other problem or issue pops up that didn’t exist in the old installer or the current “alternative installer.” The alternative installer has its own set of issues, though, which may be due to the less extensive testing it receives during development periods. I’m staring at a lockup on one machine right now from the alternative installer, and I’ve seen similar lockups on completely different machines installed from completely different CDs burned from independently downloaded ISOs from at least the last two releases.

Ubuntu is a nice OS, once you get it going. The major fuckup of switching to a slow, flaky, and mostly pointless LiveCD installer is killing any love I had for the distribution very quickly, and the pathetic state of the actual usable (when it doesn’t lock) alternative installer is making it difficult for me to even TRY to love Ubuntu, since I can’t get the damned thing installed on half the machines I’ve try to put it on.

What is the point of the LiveCD installer? It’s a neat toy, and that’s about it. I haven’t once ever had a need to use the LiveCD to actually do anything useful (a text-mode rescue CD on the other hand is an invaluable tool that nobody should be without… those usually have more of the actual tools you need than the Ubuntu LiveCD, and they don’t take 3 minutes to boot on a several-year-old hardware), and I’ve never found a need to wow and impress friends or family with a running Linux desktop on a LiveCD. If anything, I’d be embarrased to show them a LiveCD, since it’s just going to make them think Linux is slow and buggy.

I see LiveCDs as a fad, and not one of the more useful kinds. They are not useful to most people, they certainly aren’t ideal installer platforms, and yet they’re everywhere these days.

Let the madness end! Please ship the next version of Ubuntu with an installer that actually bloody works… please!

/me goes to reboot a machine that locked up 85% into the Ubuntu install process.

Linux packaging sucks. I’ve written about this before, and I’ve planned many times to tackle the issue with a new Ultimate Package Manager That Doesn’t Suck(tm), but the constant realization that it’ll never get used by all mainstream distros and thus never actually solve anything worthwhile keeps coming around and killing any motivation I had for the project. Every few months the bug bites me again and I pick up work on the design for a new system, and I start by identifying the problems of the current Linux packaging landscape. Thankfully, even with all of the projects out there trying to tackle this, Linux packaging sucks the same as it ever did, so I don’t have a lot of work to do to reidentify the problems.

Appliances

Linux distributions are almost invariably built using the appliance model. You get a distribution, at some specific version, which comes with some set of packages (many of which may not be installed by default, or even on the install media), and that is all you get. Your version of Red Hat or SUSE or Ubuntu has a specific set of packages at specific versions. If you install software outside of that package set, or you install a newer version of one of those packages, you lose ALL support from the official packaging system: you will not receive any security updates for the software you install. If you didn’t use a third-party packaging repo, then you even lose all the support of the packaging system mechanics themselves, such as easy removal, or even the guarantee that your custom-installed apps and libraries won’t be over-written.

This really isn’t about proprietary software. Excellent Free Software gets screwed by this setup. A new version of Abiword comes out? You have to upgrade your whole OS to get the new version. If there’s even a new version of your OS out yet that has that new version - which isn’t likely to be true for another 3-12 months after that version of Abiword comes out. Some softwre, like Abiword, tries to offer packages for a variety of distributions, but they can’t cover every distribution, or even every recent version of the distributions they do support, and they usually don’t have the solid update services for their packages that the distribution has. They can’t be expected to, since each distribution needs a totally different update mechanism.

I really can’t stress this enough. So long as Linux distributions are built as appliances and not as compatible platforms that allow for one package from the upstream authors to work on a wide variety of distributions, Linux packaging is going to keep on sucking.

Best of all, this is already known to work. Packaging add-on solutions like AutoPackage have already solved many of the pure technical hurdles. The only problem left is the social and beaucratic hurdles imposed by the distributions. (I’m not saying that the AutoPackage software should be used. I don’t really care for their system. But their tools like apgcc, binreloc, and other research has shown that it is quite possible to make binaries of Free Software run on multiple Linux distributions with no problems.)

Terrible User Interfaces

Let’s get off the appliance topic. Let’s pretend we’ve got the perfect Linux distribution that packages every single Free Software project in the world, and includes automatic updates indefinitely for all of them for every version of the distribution.

Let’s say you’re trying to OpenOffice.org. You fire up your nice easy to use graphical package manager - Pirut, Synaptic, whatever. You search for OpenOffice.org. You are cheerily returned 20+ something packages (in most distributions) with cryptic names like openoffice.org-common, and are expected to figure out which ones you want. Sure, dependencies are marked for you, but do users really have ANY need to even see packages like *-common in the package manager, really? No, they don’t. Those are internal implementation details of the packaging structure. The *-common packages are useless by themselves are are installed automatically if you ask for the actual application. Many distros have a lot of these pacakges, and Debian-derived ones (eg Ubuntu) have more than any other. These are noise. They don’t need to be shown. At all.

So we add some package flags or rules or something else clever to make sure that *-common and other “internal” packages are never shown. Then what? We get a nice list of OpenOffice.org packages that may or may not be organized in a nice way. If we’re lucky, the matched package list will show:

OpenOffice.org
OpenOffice.org Writer
OpenOffice.org Calc
OpenOffice.org Presenter

Wait, what’s the first package? If I just install that, do I install need to install the Writer? Most of you reading this will recognize it as a “meta package,” a package which depends on other packages to make them “easy to install.” Other than the fact that they’re really only useful for people on the command line who want to type less and in a graphical package manager it’s just noise. So we go ahead and add meta-data or rules to hide those in the graphical package manager, too.

Now we still have some 20+ packages left, because software like OpenOffice.org has some bazillion add-ons. Language packs, clipart, templates, and so on. Complete pain in the butt to scan through when it’s all in a nice big useless flat list. It would be so much easier for most people to just have one entry for OpenOffice.org that installs everything, because let’s face it, most of us have the disk space. But not all of us, and some of us have other legitimate reasons to not install everything, so we can’t just turn OpenOffice.org into one big package.

Thankfully, this would be pretty easy to solve with a little more package meta-data. Users of installation engines from the Other OS having likely seen what I call the “component” model to packaging. There is a single installer (ie package) for each piece of software, such as OpenOffice.org. That package then has a number of optional components (also packages). The installer will display all of the components, usually in a nice tree format if there are many of them, with some marked for installation by default and others not. This would be easy to add conceptually to almost any graphical package manager, once the underlying packages contain enough meta-data.

For example, the OpenOffice.org meta packages on most distributions could be marked explicitly as a meta-package, and its dependencies then becomes components. Package formats like dpkg that offer Suggested or Recommended packages already even have a means of marking install-by-default components vs purely optional components.

The graphical package manager, using the new meta-data, knows to only show the OpenOffice.org package, and none of the other dozens of packages that make up a complete OpenOffice.org install. Either by using an expander, a pane, a popup, or whatever other UI element the designers happen to prefer, the user can view the list of components of OpenOffice.org before installing it. Most users are likely to just check the meta-package and run with it, which should get them all the standard components and most useful extras. The UI is as simple as you can make it for average users of OpenOffice.org (which, btw, also includes highly technical skilled users who simply have no reason to waste time with largely inconsequential sub-package selections) and still has flexibility for those who actually need it.

The user could later open the package manager to view his list of installed applications. He can look at the OpenOffice.org application and look at its list of components and add or remove some if he wants. Compare this to opening up the package manager to look for “uninstalled” packages to find add-ons for the “installed” OpenOffice.org. Sure, that makes sense to us techies (the add-on package is uninstalled, after all) but it’s an illogical complication for people who are, quite justifiably, thinking about _applications_ instead of about RPMs or DPKGs or whatever.

Updates

A new security update comes around. This is an update to KDE, which affects a number of packages. Instead of the updater showing a nice, “There is a security vulnerability (CVE-XXXXX) patch available. Please download.” the user instead sees:

There are updates available:
kde-konquerer
kde-kioslaves
kde-foo
kde-bar

What is that all about? Are there new features? Security updates? What the hell is kde-kioslaves? I’ve never run that program!

I’m rather surprised this one hasn’t been fixed already. Developers are already used to this excellent technology called “version control” and are used to writing these awesome documents called “changelogs.” With package update services available now, developers slap a new version of a package into a repository, updates check to see which new packages are available, and give the user no more information than that list of updated packages.

Instead, the update repository could include each updated set of packages as a cohesive changeset. Along with the list of packages could be something as simple as a changelog entry indicating why those packages are being updated. The user now gets much more informative update notices, and updaters could still allow individual package selection if they wanted (as they’d be able to compare package updates along with the changesets as a whole - they’d even be able to see which changesets apply to a single package update if they haven’t updated that particular package in a while). Some distributions already include per-package changelogs, such as Debian, but this information is sometimes overly technical and boring, even to technically skilled users, and per-package changelogs aren’t always the best fit for updates that span multiple source packages.

Of course, if the component meta-data is available, the update notices can be even nicer, by just showing “OpenOffice.org” as the updated package, which is all that most of us really give a hoot about, even if all 20+ OpenOffice.org packages are being updates. Those of us interested in seeing which components are affected could still look at that information if the updater UI wants to provide it.

Summary

- Packages need to be able to be distribution and version independent so users are not locked-in to their distributions’ package sets
- Large software suites that are split into multiple packages need meta-data to associate the packages logically with the application
- Update services need to group related updates and include proper update notices

Trojans (the malware, not the happy hats) are sometimes cited by groups such as PMK and alternative installer frameworks as a problem for systems like autoconf or common Linux package formats.

The argument is that a configure script or a installer script could easily contain malware, and that this malware will infect the system when the installer is run as root.

I can’t figure out what is wrong with these people.

It is absolutely no more likely or easier to hide malware in an install script than it is to hide it in the application’s binary or source code. Sure, an installer for Foo can’t break my system, but as soon as I run Foo itself, it could do just as much damage.

The usual counter argument here is that the installer runs as root, and the application does not. Avoiding the fact that most the alternative installer systems allow for running a binary, and also the fact that many sysadmins are not going to shirk away from running a config/setup tool as root if the app “requires” it in order to work after install (because the install itself couldn’t, eg, update an application-specific DB after install), is avoiding root malware really much of a benefit?

Let’s say an installer, running as root, trashes my system and deletes all my files. Reinstalling any decent Linux distro only takes 15 minutes, tops, and getting it tweaked and configured shouldn’t be more than an hour for any reasonable person. The real loss is to all my important files. Records, source code, email, and anything else a person might keep on his computer. If the Foo application itself has malware, and I run it as my user, it can do all of that important damage just as easily as the UID 0 installer script could. I see no benefit from refusing to run arbitrary scripts at install time, because my actual valuable data is still at risk, and all I gain is the need to do extra manual work to install many apps since they can’t configure themselves at build/install time.

The second counter argument in this debate is that we have multi-user systems. In theory, root is more dangerous on such systems. In truth, it is, because one mistake as root wipes out multiple persons’ valuable data, not just one. However, this doesn’t much hold for installers.

There are three general setups, broadly speaking: there are single-user machines, like the majority of home desktop Linux users, in which there is only one real user of the system, and thus usually only one user account and the root account. There are then the rarer true multi-user home and small office desktop Linux users, in which several people share a computer. One or more of them are the “Linux guy” who knows how to work the thing and he has the root password. Finally, there are the large deployment setups, where there are many machines, many users, and a few sysadmins.

The first setup - single-user machines - are already covered. They’re just as at risk even if installers can’t have trojans.

The second setup - standalone multi-user machines - are barely more protected. If the sysadmin installs a new app, it’s likely that all users of the machine will use it. Furthermore, such systems rarely have things like file-system quotas and the like in place, so one bad user (or one user running a bad app) can still render the disk full, the machine unusable, etc. Most systems grant read access to user home dirs, and most office sweets don’t set modes on documents for users, so users still end up with sensitive data in a world-readable state. These very, very common setups are still ripe for data-theft trojans, and can still with some (ill-)luck allow for destruction of work or data.

The final setup - large deployments - are hopefully no more protected. Sysadmins for such networks should be testing applications before deploying them. They should be going through levels of care and scrutiny that the average home user - even one who’s an experienced sysadmin - usually doesn’t bother with. They should only be installed software from trusted sources, checking signatures, etc. If they don’t, then it is possible that a trojan in an installer could wreck havoc after infecting an NFS volume or the like. However, many careless sysadmins are just as likely to run the program itself to test that it functions, or just run random scripts or utilities as root from the web.

It’s fairly clear that there is little real, practical benefit to efforts to remove installer trojans.

While projects like PMK have many fine qualities, the oft-cited advantage of being free of configure script trojans is just hot air. These projects should focus on advertising their real strengths instead of phantom ones.

I’ve signed up for a virtual server with TekTonic, and moving all of my server bits off of Moonweaver, my old server I had here at the apartment. The new server, Cometsweeper, will soon (by the end of tomorrow at the latest) be hosting all of my services, as well as a few new ones that I couldn’t host before as Moonweaver is on a Comcast connection.

This blog, if you’re reading it, is already hosted on Cometsweeper. :)

I have only two regrets about this move. First, my favorite hostname is definitely Moonweaver, and it is now being retired. I may resurrect the name with a future server at some time. Second, I have to lose all of my cool single sign-on Kerberos setup. Ah well.

It seems that everytime some developer makes a decision that a user doesn’t like, he gets the “but Linux is all about choice!” argument thrown into his face.

This line is selfish and foolish.

Linux is not about choice. Linux is about creating an open source UNIX clone. Likewise, a distro like Ubuntu is not about choice. It’s about creating a kick ass open source operating system. Neither is a Free Software project like GNOME about choice. It’s about creating a quality desktop environment.

Open source does give the user a lot more choice than proprietary software. It gives the user the choice to take all of his data and move it to another product. It gives the user the choice to modify the source to customize the software to his tastes and needs.

Open source does not have some underlying goal of “providing choice to users” that requires that developers stuff every conceivable configuration option in the UI, to write the huge chunks of code necessary to support all of those options, or to work on projects that allow users to do things the developers don’t care about.

So GNOME won’t offer an option you want. Don’t even think about saying “but Linux is about choice!” If you believe that, then make the choice to use something else. That or modify the source and add the damn option yourself. Those are the *only* choices open source gives you.

So Ubuntu won’t add some huge, complex feature you “need” to get work done like multi-arch. Don’t bore us with “but Linux is about choice!” You have the choice to use a distro that supports multi-arch or to do the work to add it yourself. Those are your two options.

If you really think that “choice” is a core part of open source, then think about this: the developers are allowed to decide what they want to spend their time on, and they aren’t bound to serve your whims against their choice.

Dovecot’s GSSAPI support is perfect. I needed to add only a single word to a single line in dovecot.conf to get it working. That is pure beauty.

It’s worth noting that I did already have a key generated for it and placed in the keytab file, so that is one step others will need to do which I did not.

Once Jabberd2’s GSSAPI-enabled release is out and in Debian unstable, I should have every single service on the network kerberized.

Finally got my SATA drive back from Maxtor after it broke. Sadly, the mounting brackets that my case requires for disks are all missing except for the one already in use. After much looking, I think I just need to buy a new case. Whcih isn’t too bad since I wanted a new one anyhow. Of course, I had planned on buying a Shuttle XPC case/mobo and with that a new CPU and memory, all of which is just right out of the question given the cost, so I will have to find a bearable ATX case.

My Maxtor drive in my work station at work died on Monday. Have it back already and most of the system working again (running Ubuntu instead of Fedora, as a bonus) except for the install of Windows in VMWare, which I had just gotten done last week and which took many many hours.

I hate hardware. Software only ever fails because it was told to do something wrong. Hardware fails just because the universe hates me. ;-)

So I’ve been having this funny trouble after installing OpenSSH 4.2p1 where I could SSH into my host moonweaver, but not my host stargrazer. SSHing from stargrazer was fine, just not into. I chalked it up to some silly misconfiguration and put it off for another day.

Well, a couple days ago, I took a look at it. I couldn’t find a misconfiguration problem anywhere. I even went so far as to put identical ssh_config and sshd_config files on both hosts. Still no change. I wasn’t getting any useful errors out of sshd, and my ssh -vvv gave me nothing more useful than, “No key exchange for context.” Like I know what the hell that means.

Anyways, about an hour ago, I decided to up the logging level of sshd to DEBUG3. Sure enough, that gave me something useful to know: the host principal wasn’t found in the keytab.

The funny thing was, the host principal was most definitely in the keytab. I tried regenerating the keytab with no effect. The Kerberos configuration just seemed dandy.

Out of desperation, I tried typing hostname -f. Bingo. On the moonweaver, the proper fqdn was returned. On stargrazer, I was getting localhost.localdomain. Which meant that Kerberos on that host was looking for a host principal of host/localhost.localdomain, not the proper host.

Then the trick was figuring out what was going on. After much digging in every init script, configuration file, and man page I could find, I finally found a website which told me that a common error was putting a line like the following into /etc/hosts:
127.0.0.1 localhost.localdomain localhost

Sure enough, I changed the line to read:
127.0.0.1 localhost

And hostname -f started returning the proper fqdn and SSHing into stargrazer started working fine.

I don’t think I added that line in there. I’m fairly sure I didn’t. I’m fairly sure it’s an Ubuntu default file. Once my SATA gets back from the shop and I reinstall the OS, I’ll have to double check that. If it is an Ubuntu default, I will most definitely raise a bug report on it.

Dovecot gets GSSAPI support!

I think this makes me very close to the happiest man alive right now. After a new Dovecot alpha is released and installed, I’ll only be missing an IM server with GSSAPI support.

Also still missing SSH support as Ubuntu still has OpenSSH 4.1p and Debian unstable is using OpenSSH 4.2p1 which has a different GSSAPI mechanism or some such. I see openssh-4.2p1 sources in the Ubuntu archives, but it doesn’t seem to be building as it depends on openssl-0.9.8a which is missing. That package just came across in a Debian update, though, so I’m hoping it’ll be in Ubuntu within a day or two, and then the new openssh package will be ready, and everything will be awesome.

Anyone know of any GSSAPI-capable, Open Source Jabber servers?

Oh, there’s also the issue of the copy of Neon in gnome-vfs being old and having broken GSSAPI support, despite Red Hat having a patch and my cleaning it up and posting it to GNOME Bugzilla. Hopefully that’ll get resolved sometime in the next two decades or so.

At work, my boss installed Groupwise 7, which includes the SOAP access method for Evolution. For the first time in five years, I can finally access the contacts and calendar of the township’s groupware system. The LDAP entries exported by Groupwise were never useful to Evolution (the users had no names, just email addresses and their login ids) and the calendar just wasn’t accessible at all.

Now we just need the server to have an updated iFolder system that works with the iFolder client available on Mono/Linux, and I’ll finally be all up-and-up with the township’s network. :-)

I installed OpenLDAP at home to run the user accounts. I’ve been poking at LDAP a bit lately as I’ve really wanted an object-oriented database, a very fast and efficient one that isn’t Java or CLR based, and LDAP is (although they don’t admit it) an object-oriented database system. it is, unfortunately, not particularly well optimized to the use I have in mind, which has rather frequent changes to the stored data. Still, LDAP would be a good reference for building system a database, and it’s even possible that, although LDAP isn’t designed for efficient-writes, that some particular implementation of an LDAP server (or, preferably, an embedded LDAP engine) would pull it off nicely. Samba4’s ldb may be the answer, if and when it’s ever stable, documented, and released.

Anyways, OpenLDAP is up. I migrated the user accounts and setup my machines. Authentication works over GSSAPI. It’s all rather nice, I think.

I had two big problems, amid a number of smaller ones, while setting things up. The first was that Dovecot, in its infinite perfection, eschewed cyrus-sasl in favor of its own SASL library. Which doesn’t support GSSAPI. Nor does it support binding to an LDAP server with SASL to authenticate. It (shudder) directly queries the userPassword attribute of the LDAP server if configured to authenticate with LDAP. Thankfully, because PAM and NSS both work with LDAP and GSSAPI, I didn’t really have to change much in the Dovecot configuration, although it took me a while to realize that. I just told it to stop using /etc/passwd for the user database and to instead use LDAP, while continuing to use PAM for authentication. Still no single sign-on with my IMAP server, but at least it still works. Oddly, Dovecot doesn’t seem to be able to use NSS for querying user account properties. Either that or I’m missing something.

The second problem was that GDM on my desktop refused to work. I made many changes to the configuration files, some of which looked quite broken but which didn’t seem to inhibit login from working (the user accounts in question had been completely removed from /etc/passwd, so it had to have been using LDAP properly). I still have no idea what exactly I did to fix it. After making another batch of changes and getting a different error message (I received no less than four different error messages, randing from “invalid credentials” to “could not create groups” to “administrator has disabled logins”), I ran /etc/init.d/gdm restart and GDM started working.

Confusing, but hey, at least is works now!

My next system administration task is getting DNS SRV records setup for the various services I run. With a decent DNS-SD (service discovery) implementation, it should then be possible find any service on the network without manually entering any information. That is, the DNS-SD implementation should know the domain the host is in and query that domain for services. Current DNS-SD implementations that I know of all are limited to using M-DNS (multicast DNS), which while useful for ad-hoc networks, is not that useful for engineered/administrated networks.