Sean Middleditch

Just Sick

moray: That’s just awesome. Now everyone else can pretend they’re l337 h4×0r5 like the Gentoo folks as well. Thanks for the smile. ;-)

Sane Strings

I’ve been rather irritated with C++ strings lately, mostly because they aren’t designed for my specific needs. Few things ever are, of course, but in the case of strings, I can fix it.

So I’ve started yet another string library, this one named Sane Strings. It isn’t going to be overly fancy, and probably won’t be usable to most people. For one thing, it’ll depend on the Boehm GC (which I use in Scriptix and AweMUD). For two, it’ll be optimized for the kind of usage I see most in those projects.

My needs are rather simple, to be honest. First, I copy strings around a lot, so I want to reduce duplicating strings needlessly. Second, I rarely change/modify string data, so I don’t need to worry about the performance there-of. Finally, I mix three sources of strings: string already in the String class, static strings, and string buffers (from those rare cases where I do modify/hack strings up).

The standard C++ string class works either by always copying string data or by using reference counting. The later is used to avoid copying the string data by allowing efficient copy-on-write semantics. I don’t modify strings a lot, as I said, so the latter is fairly useless, which makes reference counting rather expensive; that’s a lot of reference count checks and accesses (more on this in a bit) which I don’t need. I definitely don’t want to copy around strign data all the time, however. Thus, garbage collection, which isn’t much overhead given that I already use it for other objects.

The problem with reference counting performance is that it requires active work. Every time you copy a string you have to increment the counter. Every time you stop using a string you have decrement the counter, check if it’s now zero, and then potentially free it.

Accessing a reference counter can be expensive, as well. There are generally two ways to store the counter. The first is to allocate the string memory with extra space in the front for the counter. The second is to allocate another block to hold the counter. The first will force me to make way more string data copies than I need (explained below). The second doesn’t have all that great of cache behaviour, since accessing the string data (usually done just before or after you decide you need to copy it) won’t pull in the reference counter, so you have to grab that too. You also then end up needing to pass around two pointers, which is less efficient.

So far as embedding the counter in the string memory block, the problem here then is that you end up with a different memory layout for strings which are part of the String class/system and static strings or character arrays/buffers. i.e., if you have some code that uses the “manages” string class and try to pass in a static string, you’ll be forced to copy the string just to get the counter in the memory block, and that copy will most likely just end up being unused very shortly and then have to be deallocated, after doing a some pointless counter management.

The garbage collector solves this problem nicely. Both garbage collected strings and static data strings can be used identically in the code and the garbage collector is smart enough to free unused gc’d strings and leave static strings alone. In fact, if these were the only two kinds of strings I used, I wouldn’t even need any real class or code to manage it - I could just use char* data types.

However, I *do* use buffers just often enough for this to not be good enough. If I have a local buffer with character data, and I want to pass it to a function that expects a GC’d string, I have to be careful. If the function stores a copy of the string, the program will end up crashing, because the stored pointer will end up pointing to invalid memory down the line. So the string class/code has to know that if it wants to store a copy of a string somewhere, it only needs to copy the pointer for a gc’d or static string, but that it must make a full copy of the data for string buffers.

To make this fool-proof (or close to it), my new String class can’t be assigned a char pointer. It wouldn’t know whether that pointer is a buffer, a static string, or what. So, it is only capable of being assigned from one of three classes: other Strings (copies the pointer), StaticStrings (also copies the pointer), and StringBuffers (copies the data). So if I want to assign a String the static data “foo”, I’d have to write: String var = StaticString(”foo”);. With proper optimization, the StaticString() construct is essentially a no-op - it only serves to provide type checking for the assignment operator. Likewise, for a buffer, I’d use the StringBuffer() pseudo-class to also perform a no-op that just provides type-checking. Yes, it’s possible to accidently use StaticString when one should’ve used a StringBuffer, but it’s easy to write a sed script to just for such miss uses. (If the argument to StaticString isn’t a string constant, then it’s wrong.)

One problem is left, however. Whenever we use a String for a function argument, any data from a buffer will be copied, even if the function argument is only used for reading/comparing. (i.e., it isn’t stored anywhere else.) That results in a useless copy, which we were trying to avoid. We could make it so that function arguments which are never stored, but only examined, could be a different type that is essentially just a char*, but then that makes it very easy to introduce bugs when function implementations change. We don’t want to go about changing the API because a function decides to keep copies of strings around or anything.

One solution would be to make a custom StringArg type which won’t copy the string, but forces a copy of the string data to be made if you try to copy the string. i.e., String str = StringArg(”foo”) would result in a data copy. This results in wasteful copies being made, however, when a String object or a constant string is passed through a StringArg and then stored in a String. (Like the example I just gave.)

The fix is to have the StringArg type know a little about its payload. Basically, does it need to be copied or not? Of the ways to carry this payload, one struck me as rather simple and clever, if a little less than elegant.

Since allocated or static pointers are always aligned on word boundaries (on my target platforms, anyhow) you end up with at least two bits that are always 0 in the pointer. I already use this fact in Scriptix, to boot (as do quite a few other language VMs), so it’s not like I’m introducing any new portability problems to my code. Those two bits can be used as flags. Just make sure you unset (or mask out) those bits when you try to dereference the pointer. We basically have two states (one bit flag) we are worried about - must we copy the string data to copy the string or not? The only problem is, the Boehm GC (being a conservative GC) can’t work if you munge/mask the pointers. The solution? Set the bit on the non-GCd string buffer pointer, and leave it unset on GCd string pointer or static string pointers. The bit will then only be set on pointers the GC doesn’t need to see or follow then.

Now, if you modify strings a lot, this solution won’t work. Why? The lack of reference counting on the strings means that if you want to modify a string, you *always* must do copy-on-write, because the String class can’t tell if its the only owner of the pointer or not, or if the pointer is to the static data segment. Given that I don’t do much string manipulation tho, and given that the manipulation I do end up doing is usually on data in character buffers, this isn’t a real problem for me.

The only problem I do have at all is that now Scriptix and AweMUD are going to end up depending on yet another library. Drat. ;-)

Zeroconf

Some discussion in GNOME-land has come up regarding service discovery, and the usage of Zeroconf, SLP, and/or UPnP.

UPnP is kind of separate from the other two, because its a hell of a lot more than just service discovery. It’s service discovery plus IPC/communication with services. Because of all the extra bloat and unneeded features it brings in (do we need yet another print protocol? IPP with Zeroconf/SLP for discovery works just fine, thank you.) I’m going to pretend that it is dead in the water so far as GNOME adoption goes. At least for now.

So it comes down to Zeroconf vs SLP. In my mind, Zeroconf is the clear winner so far as the protocol goes. Why? First, it leverages existing code and infrastructure - DNS. SLP requires all new code and expertise/training on the part of programmers and admins. Second, Zeroconf is very widely used in consumer and network devices already thanks to Apple’s Rendezvous marketing. Hard to find a new network printer without Zeroconf support, for example. Finally, it’s capable of doing everything SLP can and then some.

Zeroconf works fine both for home users (zero configuration, after all), ad-hoc networks (did we mention zero configuration?), and corporate networks (the service discovery protocol, dns-sd, allows you to do configuration and management if you really want/need to). In the case of the former two situation, applications and systems can easily start using Zeroconf with no need for new system components are architecture, so long as link-local connections are on. (Which SLP would need in order to work in a similar fashion anyhow.) In a corporate network, DNS-SD could be configured to not publish services over link-local addressing (which would probably be disabled) and service browsers can look at corporate DNS servers. You could even still allow service publishing from client machines using one of the dns-update protocol extensions, with all sorts of policy control on the server as well. Just like SLP. Really, in terms of protocol, Zeroconf is pretty clearly a solid winner.

The problem is, however, that SLP has a fully working implementation in OpenSLP that does everything a corporate environment needs now, while no complete Zeroconf implementation I’m aware of works on non-link-local address. Including Apple’s own Rendezvous implementation. This is changing; I’m told Apple is working on this, and I believe another Open Source implementation is working on it as well. (Was it tmdns? I don’t recall.) Howl only support link-local service discovery, and it doesn’t look like that’ll change soon.

What we end up with is a rather classic situation. We have a better technology, Zeroconf, that just isn’t ready implementation-wise yet, so GNOME may move forward only supporting SLP.

Of course, to be honest, it isn’t that big of a deal. Why? The GNOME developers are uber-cool and intelligent, and instead of relying just on SLP, a generic service discovery layer is being written. So SLP can be used now, and Zeroconf (or even UPnP, or another protocol even) can be plugged in later, and it’ll all work seamlessly to the apps. Rock on GNOME. ^_^

AweMUD Status Page

I wrote a status page for AweMUD.net. Hopefully it looks as nice in everyone else’s browser as it does mine. And hopefully it’ll serve its purpose well, and I’ll get less questions along the lines of, “how done is XYZ in AweMUD?” or “what feature do you need the most help with?”

AweMUD

The AweMUD message boards are getting a little active lately. That’s nice, because it means people are using AweMUD. That’s bad, because now I need to keep watching them and answering questions. ;-)

Bass

Very little progress lately. Not much time to work at it. Hopefully I can rectify that this week, tho.

LARP

Went out to KANAR this weekend. Rather boring weekend, unfortunately. Friday was filled mostly with certifications (i.e., we go to a class to make sure we understand all the rules, then get signed off on having attended the class, so that if we break a rule they can hold us responsible for it.), Saturday was filled with certifications and much standing around.

Sunday was a bit more fun in the evening, since we actually did something. Namely, several high-powered clerics and wizards, several formiddable warriors, a couple low-powered healers and myself all headed off to a nearby town which had been over-run by undead (apparantly the current guess as to what happened is a powerful Lich, or undead wizard, invaded the town, killed everyone, and raised them as ghosts and wraiths and such). I went simply because, as an undead hunter, that’s just what I do. My friend went because that town is his hometown.

Of course, we all got our asses handed to us on silver platters. 300+ ghosts, which are rather powerful undead by KANAR’s rules, most of us barely able to harm them (if at all) while they having several pikemen all guarding the gates (for those who’ve never picked up a weapon, polearm users are nigh-untouchable when they form a wall) and the summoned Lesser Shadow deciding to be useless (the wizard’s enforced agreement was that he’d “see what he can do.” so he walked in, saw that he was immune to the ghosts’ attacks and could kill them quite easily, and said his end of the bargain was fulfilled after having killed at most three ghosts. which was pretty much what I thought would happen after the wizard made his bargain. too bad I wasn’t allowed to distract said wizard and point out the logic loophole…)

The seige was still rather fun. Lots of creative taunting on both sides of the walls. When you have 40+ people standing out in the freezing cold (literally), you’re pretty much forced to come up with active ways of amusing yourselves. ~,^

One notably humorous event (warning: this is probably one of those “had to be there” sorts of things) is when the magic circle for the summoning was being made. Several of us were tasked with guarding the wizard, as neither he nor the circle could be disturbed in any way. One of the fellows playing a ghost walks out of the town. For whatever reason, not one of us thought he was a ghost. I thought he was going to play the summoned creature personally. Anyways, he walks up and is standing on the very very edge of the sommoning circle, a sword in each hand, just staring down towards his feet at the circle. After about half a minute of him standing there, someone finally asks, “what do I see?” (You ask that so the people playing monsters/ghosts/etc can describe what they are.) “I’m a ghost.” So then of course all of us close in, feeling incredibly stupid. We can’t just attack, because that would most certainly result in the magic circle getting trampled over. One of the more powerful warrior-priests steps up and smacks the ghost with an undead-destruction spell. The guy playing the ghost just turns around, says “damn,” and walks back into town. We were all laughing (nervously) for another 10 minutes or so on how we just let him walk up and stand there for so long. :)

Another funny moment (probably, again, “you had to be there”) is as we were running out of the town at full speed, after the last one of us got out, we all heard a big slam as the large doors slammed shut behind us. Funny ghosts.

In any event, we’re all going back with some more powerful weaponry, probably sometime in June. Hopefully by then my character’ll be level 3 or 4, so I can do more than run in, get hit a couple times, run back out yelling for healing, and repeat. I’m actually rather surprised I lived. (hey, it’s the first battle I’ve been in that I *have* lived - woohoO!) I got pressed into a corner and had two spearmen repeatedly trying to stab me. I was down to but a single health point before I got out. One more stab, I’d've been down. I think on the whole I caused a whopping 6 poins of damage, to creatures which need 45 to drop. I pretty much served as a lite damage absorber for the more powerful warriors. Yay uselessness!

Thanks

Thanks for the encouragement from you fellow musicians. :)

And gilbou, I’m not sure I deserve a Journeyer rating for my musical aspirations. ~,^ The Advogato system rather intends the rating to mean my Open Source contribution achievement, of which mine really isn’t worth even an Apprentice rating. Thanks anyhow. :)

OS X Window Grouping Rant

OK, this is going to be a re-hashed version of my gnomesupport.org post on this subject. I feel this worth repeating to a wider audience, however.

The problem is, summed up, that grouping windows based on which application they belong to is useless and even counter-productive for a user.

For example, take a web browser. In OS X, the cmd-tab key would switch between showing all open Safari windows and (for example) all open terminal windows, as they are grouped by application. To switch between windows of the currently selected app, the cmd-~ key must be used. This requires the user to remember which application (or process, basically) is responsible for which window.

Additionally, the grouping is often very different from what the user needs or wants. I don’t think to myself, “these are my web views and these are my terminal views,” I think, “these are the web views and terminal views I need for task A, and these are the views I need for task B.” For example, I might have several Safari windows open; some are in relation to a school project I’m working with, some in relation to one of my Open Source projects. The OS X window management model requires me to group the completely unrelated Safari windows together simply because they are served by the same application/process. There’s no quick/easy way to switch between just the windows related to my school project (say, several Safari windows, an AppleWorks window, and QuickTime), or just the windows related to my programming work.

From a user’s perspective, does it at all, in any way, matter which application is responsible for managing a window? Should I really need to think about and manage the fact that one window full of text is AppleWorks and the other is a text editor? Are all my windows served by a single application supposed to always be related to each other? Am I supposed to be thinking in terms of which tool (application) I’m working with rather than the document/task (set of windows) I’m working on? The answer to all of these should be No.

The Linux desktops are way ahead of OS X in many areas, including this one. GNOME lets me concentrate on my task, by not forcing me to worry about which application serves which window (altho there are cases where this fails - see below) and, if any grouping is to be done, lets me do it based on my critera, by placing windows in separate workspaces. I can have windows from a single Epiphany instance in several workspaces, and they operate completely independently from each other, making it easy to concentrate on and manage the windows related to each individual task separately.

We’re Not That Great Either

The Linux desktops (including GNOME, which you’ve probably noticed I’m going to concentrate on, it being my weapon of choice) don’t avoid the above problem 100%, however. Several situations arise that break the document/task centric work model.

One example is how certain apps, such as gedit, remains open when you close the document you’re working on. The idea is, I imagine, that the user may want the editor open to perform some further work, and launching a new copy is inefficient. Additionally, there is no good application agnostic UI for creating new documents.

The first problem is a UI misdesign caused by technical problems. Breaking the document centric user model simply because closing a window can end the process and restarting it is slow is bad. Instead, work should be done to fix the problem with opening a new document: make the application launch faster, have the application remain running for a short amount of time in case further requests are made on it (from a New Document menu, for example) or so on. Solve the actual problem, don’t just break the UI to avoid it.

For the second issue, the lack of a good New Document UI, I believe work is being done on this already, based on my readings of the GNOME mailing lists.

There are other little niggling issues here and there. For example, I believe Epiphany uses only a single download manager window, even for unrelated downloads initiated by different windows on different workspaces. This reinforces to the user the technical details that there is a single central application (be it the whole browser or just a dedicated download manager) that manages all tasks/documents.

Branding

Another issue in this task vs app model is branding. Corporations, or even Open Source groups, want users to know whose application they’re working with. Corporations want the users to have brand awareness so they’re more likely to think, “hey, that application is great, it’s Foobar by Bazwhiz, I’ll buy their Whammy product and recommend Foobar to my colleagues!” Open Source groups just want their recognition.

The problem is, this almost invariably is done in a way that shatters the task centric user model. The UI is in fact designed to make the users aware of the application as a whole, and how it’s so wonderfully helping them with all their different tasks, and how it’s so wonderfully integrated with itself. This is useless to the user, of course, but it’s great marketing!

There are solutions, however. Spatial Nautilus, for example, can work just as well in its document centric fashion and could easily include branding; just having a big Nautilus and/or GNOME logo in every window (as a background, say). Most developers simply don’t do this, however, which isn’t overly surprising given how few developers bother with good usability at all.

Hypocrisy (Almost)

I find it interesting how people and organizations who understand and even advocate these ideas and principles behind good usability can be such hypocrites. (My apologies; that word is much harsher than I want. I can’t think of a better word, however. Please don’t take this as an insult, it isn’t my intention in the least.)

Take Evolution, for example. Many of the Ximian folk are definitely proponents of a task-centric user model, including many Evolution hackers, yet even the new and sexy Evolution 2.0 completely fails in this regard (and many others). The UI changes are large, but they don’t at all break the unified application system. All my mail folders are still lumped together; my Open Source mail, personal mail, work mail, and school mail. Same for contacts and such. Things like contacts can’t even be viewed without the mail component being shown.

Sure, the shell can (supposedly) be launched with only minimal components (no need to load tasks if you’re just handling mail) but the UI I’ve seen so far still heavily reinforces the fact that all those components are heavily tied together.

In a purer task-centric model, things like contacts would be visible in a window that has no big buttons or other ties to other Evolution components. When viewing individual contacts, a “Send E-Mail” button could launch the mail composer, sure, but that would be about the extend of the UI tie of the mail component to the contacts component. Additionally, one would have better control of the data being browsed. I’d love to have one view showing just selected folders (possibly from multiple accounts) relating to school stuff, for example. This is possible now, but it requires some work setting up on your own, doesn’t get rid of the overly-integrated UI, and isn’t all that ideal of an end result (I’d still see other folders in the folder tree, for example, they’d just be unexpanded).

Another fun example of “hypocrisy” is how Red Hat refuses to remove the RHN icon from the notification area if the up2date monitor is running. The GNOME hackers all pretty much (again, based on my readings of the mailing lists) agree that this sort of thing is an abuse of the spirit of the notification area. Yet some of those same hackers go on to commit such abuse for (as mentioned above) Corporate Branding. I’ve filed a bug against this just to have it closed as WONTFIX. (With an unfortuantely rude response, I might note. Given my lack of social skills, I shouldn’t complain about that though, should I? ~,^ )

Again, I didn’t mean this last section as an insult to either the Evolution hackers or the Red Hat folks. This was just a pondering on how people can play both sides of the game sometimes. (Altho, both of the above issues are things I do believe need fixing, for the benefit of the users.)

Anti-Virus

Well, since Microsoft bought the technology for the RAV engine (a very good, and very cheap, anti-virus solution for Linux, Groupwise, and Windows), we’ve had to find a new virus solution for our mail server.

Granted, once we move to WinXP SP2 on the clients, we may not even need AV on the server (because RAV is being integrated into SP2, apparently), but then doubling up couldn’t hurt.

We’re currently evaluating BitDefender. The installation on the beta had some problems (though the technicians don’t seem to believe; worse is that the problem doesn’t reproduce on a second install attempt, even after manually cleansing the install - weird). Otherwise, tho, it seems to be working.

I had some trouble getting Exim running with BitDefender, mostly because I’m doing things bass-ackwards; I have the mail coming into Exim, then being forwarded to BitDefender which then delivers straight to the Groupwise server. This is a temporary setup until we’re done with the evaluation.

The problem with Exim was that it seems to be nitpicky about redelivering to the localhost, even when I specify a different port. Took a while to find the configuration options needed in several different places to get it to place nice.

In any event, at the moment, we have mail running through two AV solutions (RAV and BitDefender) and two anti-spam solutions (SpamAssassin and BitDefender), so anything that gets into the network deserves to get into the network. We should fall to our knees and pay homage to anything that gets in. ;-)

Bass

Getting better, slowly. Still have a hell of time trying to learn songs by listening. Combination of not being able to pick out the actual bass bits (I get easily confused by all the other instruments playing, especially lead guitar and vocals) and just plain not knowing what to do on a the bass to reproduce the sounds.

Having my father around to show me stuff helps, although he isn’t the world’s greatest teacher. Most of the time he just plays off on a song at a speed I can’t hope to follow, much less play. My lack of short-term memory doesn’t help much, either.

Tablature has been the best help so far. Of course, the songs I’d really like to play either aren’t for bass. Battle of Evermore (Led Zeppelin) for example is played on a Mandolin. A Mandolin! I found a Mandolin at Music 1-2-3 for ~$80US, which I’m very tempted to buy. After I buy a new coat (mine is dead) and some dress shoes.

AweMUD

Little work on AweMUD of late. Contract job and school stuff taking up all my time. I have two different trees, one at home and one at work, with various changes. Stuff at work includes comabt pulled into C++, stuff at home includes character creation pulled into C++.

They worked fine in scripts. They were excellent examples of how cool the script language is. But, they are both big and complex and rather integral to the game engine, and scripts simply don’t offer the level of integration I’d like. Additionally, Scriptix still has some serious flaws that make the work hard, including lack of things like enumerations, easy integration with the network layer, or the ability to use custom database lookups (weapon types and such in combat).

A fully generic engine would of course have everything in scripts, but then AweMUD isn’t going to be fully generic. Flexibility was one of the primary design goals, and still is to a degree, but correctness and power is more important to me now. The system still fully supports putting either of these systems in scripts if a user wants. I should add some compile-time flags for compiling out combat, character creation, and so on if a user wants to code it in scripts and doesn’t need the bloat in C++.

Digital Flat Planel

Finally got my flat panel (Samsung SyncMaster 172t) working in digital mode. The system always locked up before when attempting this in any graphics mode. (Text mode console worked fine.)

I had always assumed this was yet another bug in the binary NVIDIA driver. I was wrong. The bug was a video BIOS bug in VisionTek’s board. After finding out about this (from a user on the NVIDIA forums), I got the BIOS update and applied it, and am now happily enjoying my digital mode.

The NVIDIA driver developers deserve a little more credit than I’ve been giving them; this was almost the only problem I’ve ever had with their driver, and it wasn’t even a problem in the driver. ;-)

The only problem now is that the latest Fedora Devel kernels don’t work with the driver. Bummer. Hopefully it’ll be fixed soonish.