I must have come across and handled this issue before, but at least on my Ubuntu installation, PDF documents generated by sequential latex, dvips, and ps2pdf commands default to A4 size paper. In order to get the correct US style letter paper, I have to explicitly use the “-t letter” option in dvips. After rooting around a bit, the command “texconfig paper letter” has changed my default system paper size for dvips, dvipdf, xdvi, and hopefully I’ll never make this mistake again!

P.S. Metric system? One day we will give in to your logical structure and simple unit conversion. Clearly, though, that day is not today.

Anyhow, one other bit of ultimate news is that Walter Fredrick Morrison has died at the age of 90. Morrison is regarded as the “father of the frisbee”. And to reinforce the notion that the sport is dominated by geeks, the story was picked up by Slashdot and hit the front page a couple days ago.

I always thought that Marty McFly invented the frisbee in 1885 though…

I have established what I hope will become an official community fork of Project Darkstar. This fork will go by the name RedDwarf and is hosted as a sourceforge project. I cannot take credit for the new name, as it was dreamt up by the original instigator of Project Darkstar at Sun Labs (Jeff Kesselman), but I do hope that it will become an even more well known name than Project Darkstar ever was in the games industry. So without further adieu, here are the new community guidelines:

• The sourceforge project is used to host what were once three separate projects on java.net. The reddwarf-server, reddwarf-shared, and reddwarf-java-client all live in the same subversion repository, but under completely separate sub-trees, each with their own trunk, branches, and tags.
• The development process should remain largely unchanged from Project Darkstar. All commits to any trunk repository must undergo a thorough review by at least one other committer, and commit privileges are earned. Review requests should be sent to the mailing list: reddwarf-develop on sourceforge.
• All design, support, and informational documents should go in the Trac instance hosted at sourceforge.
• All issues and bugs should be filed as a ticket in Trac.
• Forum discussions should continue as usual on the sourceforge hosted forums.
• Releases will be done periodically as appropriate. All releases will be published to the central Maven repository rather than the java.net Maven repositories. (This is my first task and may take some time). They will also be published as files for download on the sourceforge site.

Progress will be much slower than before, but I think it’s very important to maintain the structure and code quality standards that we had established as a fully funded project. Will anything come of this effort? I’m not sure. But I think it’s the best chance for coordinated progress to continue with Project Darkstar.

I must say that the shock and disbelief of learning that you’ve been laid off is a predictably emotional time. Of course I didn’t think it would happen to me, but it did: proof that job security is all but an illusion. They say losing your job is like dealing with any other type of loss, which is absolutely true. Knowing this doesn’t make it any easier though. Despite this difficult situation, however, I have received nothing but support from my family, friends, colleagues and even people who were previously just casual acquaintances. Thank you to everyone who has been there so far; I know for a fact I’ll come out the other side of this a strong person.

As for my future, and the future of Project Darkstar? Well both are uncertain. I have already started ramping up and preparing for a full scale job search in the hope that I will find something even better than the best job I’ve ever had. In terms of Project Darkstar, a core group of former members of the team have already started exploring alternative ways to keep the project going. This includes both potential for-profit and volunteer efforts to carry out the original mission objectives laid out years ago. At the very least, after getting myself organized, I personally intend to continue working on the project on a part-time basis during my job search and hopefully beyond. More details to come…

Owen and Katy atop Cannon Mountain

The quality of the weather for this trip was in doubt right up until we took our first step on the trail. A significant rain event was forecast for all day Saturday and into Sunday morning, but it was unclear when things would clear out on Sunday. We were feeling optimistic, though, and were up before sunrise on Sunday and began the two hour ride to Franconia Notch in the dark and in the rain. By the time we reached the trailhead, the rain had stopped and patches of blue sky were trying to break through the low clouds and the fog. Things were looking up and we were on the trail at 8:40AM.

Katy navigating through some serious boulders

Katy doing a Matrix style pose at the summit

Owen overlooking Lonesome Lake on Hi-Cannon Trail

For those keeping track, that’s 3 down with 45 to go!

With this said, though, we can still strive to identify upper bounds on Project Darkstar’s performance at a more fine-grained level. Project Darkstar is an event driven transactional system, so all operations are not without cost. With these micro benchmarks, I’m hoping that I can establish a relative cost to each of the operations using the DataManager, the ChannelManager, and the TaskManager. For example, how expensive is it to retrieve an object using DataManager.getBinding() vs ManagedReference.get(). How much overhead is involved with each transaction? How expensive is it to create a Channel or send a message on a Channel? With more or less users? While the cost of retrieving data from Darkstar’s data store should be an order of magnitude faster than using J2EE and a RDBMS, it is also likely an order of magnitude slower than retrieving data from a data structure that is already in memory and using no synchronization. This is information that users really need to be aware of and be able to take into perspective when designing their game, structuring their tasks, and establishing their own expectations of what the performance should be like.

So over the past couple of days, I’ve been debugging a problem in these benchmarks. In one particular test, I was attempting to measure the raw execution time per call to DataManager.getBinding() from the Project Darkstar API. The test was pretty simple, I just set a large number of bindings in a single set of setup transactions. Then I would time the execution of another set of transactions that would make some subset of calls to getBinding() on the names that I had just setup. Taking into account previously measured transaction overhead I could then come up with a reasonable estimate of the cost per operation. Seems easy right? Well it turns out that I hit a snag. In running this test, I was repeatedly getting a situation where a seemingly random name binding was not being set properly during setup. Most of the calls to getBinding() would work fine, but a couple were throwing NameNotBoundException. What? This didn’t make much sense. I went back and looked over my code many times, I tried a myriad of variations, logging output, and print lines, but still no luck. I was still getting NameNotBoundException for what seemed like a random name in the sequence. Hmmph.

At this point, I went through a whole series of exercises, all centered around one assumption, that my code was right. I tested the native edition vs. the Java edition of BDB, suspecting maybe there was a weird bug in one of them: same result. I tried longer transactions, more operations, larger serialized data objects: same result. I tried running my benchmarks in different orders: same result. I even started writing test cases for DataManager.setBinding() that simulated transaction rollback and retry, large numbers of consecutive calls to setBinding() and binding and rebinding of the same name. I thought I was going to uncover some weird corner case bug. But those tests were passing! I was at a loss. After probably two days of sporadic attempts at debugging this, I finally went back and looked really hard at my own test code. And… I found a bug (doh!). It turns out that I was being too cute with my setup transactions, and was modifying a non-local counter variable inside of my anonymous nested transaction class. In random situations, this class would abort and retry (a normal darkstar operation), but since it was modifying a variable that lived outside of the task itself, this value was not being rolled back. The result was that a name binding would be skipped periodically (exactly the behavior that I was seeing).

So here’s my question. Why did I assume that code that I wrote in less than a day was more likely to be correct than Berkeley DB itself, a project that’s been developed and tested for a couple of decades? Why did I assume that code that I wrote in less than a day was more likely to be correct than Project Darkstar’s Data Service code which has been developed and tested for years? I mean, I knew better than to think that Tim’s code is the likely culprit, but I still started writing test cases thinking I was going to heroically find some obscure bug. This, my friends, is a violation of the number 1 rule of programming: If you’re having problems, It’s Always Your Fault (almost). I mean, don’t get me wrong, I’ve found (and reported) bugs in well established open source projects before, but those situations are actually few and far between. I also don’t mean to suggest that Project Darkstar is bug free. I do think, though, that sometimes it’s too tempting to conclude that there’s a bug in that library you’re using, or there’s a performance limitation in that technology that is fundamentally impossible to overcome. Maybe that’s true, but 99% of the time, it’s your fault.

And with regard to those micro benchmarks, I’m hoping to publish some results soon (assuming I don’t get hung up with any more boneheaded mistakes!)

Ultimate Statistics (since January 2008):
Total Games Played: 252
Total Hours Played: 296

The good news is that I feel like it’s (very slowly) healing. I’ve been doing some no impact strengthening exercises over the past few days and the swelling seems to be ever so slowly going down. I’ve picked Wednesday in my mind as my return date to the field, but may push it back if it doesn’t feel ready. It’s very frustrating to have an injury like this. No matter how many times I deal with them, it always takes a physical and mental toll to not have the ability to be active and out on the field. Regardless, what I always tell others and myself; injuries are a part of every competitive sport. Having the ability to respond and deal with them in a positive way is just as important as your skills in the activity itself. Hoping to be back out there soon…

Ultimate Statistics (since January 2008):
Total Games Played: 240
Total Hours Played: 284

Owen and Katy on top of Mt. Lafayette

I mentioned this idea to Katy, and not ever being one to back down from a challenge, she was in. Now, technically I’ve already done probably about 10 or so peaks on this list (in fact, if I remember correctly, I’ve already hiked 6 of the top 7 peaks, Adams being the lone 5000+ footer that I’ve never scaled). However, I figured we can do it as a team, and restart the list from scratch. So we planned a trip for this past weekend, and stayed at the Lafayette Place Campground in Franconia Notch where several of the 4000 footers flank on both sides…

Owen and Katy on top of Mount Lincoln

Fortunately, Sunday was a completely different story. We woke up to clearing skies and quickly grabbed some breakfast and began getting organized for our hike. Our plan was to attack the classic day hike in the Franconia Range, the Franconia Ridge Loop. This loop actually combines three trails into a 9 mile trek that brings you over the top of Mount Lafayette and Mount Lincoln from the 4000 footers list (technically it also brings you over Little Haystack Mountain but that peak does not satisfy the criteria to be on the 4000 footers list). After collecting some warmer gear, food, water, and gatorade in our packs, we were on the trail at about 8:40AM.

We started towards the summit of Mount Lafayette via the Old Bridle Path. This trail is fairly steep pretty much the whole way, but is very well maintained and easy to navigate. There were a few other groups on the trail, but it was fairly quiet for a near peak foliage weekend. Saturday’s weather likely had something to do with that. By 11:00AM we had made it to the Greenleaf Hut, which sits right at the treeline about 3 miles from the trailhead and 1 mile from the summit. After a quick snack break and topping off our water bottles, we continued on to the summit and were there at about noon. Wahoo! One peak down! We had lunch at the top and about 3 or 4 other groups were up there doing the same.

Traveling across the ridge along the Franconia Ridge Trail was pretty awesome. Weather above the tree line was cool, but mostly clear and calm, a generally rare event. We had nice views of the Pemigewasset Wilderness to our east, and Cannon and the Kinsman Mountains to our west. Foliage was getting close to peak, and the trail was busy but did not have a train of people like I’ve seen in the past. By about 12:30PM we had bagged the Mount Lincoln summit, and by 1:00ish we had reached the trail junction for Falling Waters Trail at Little Haystack Mountain to head back down.

Katy not arguing with the sign after the hike

When we got back to the campground, we had just one passing shower disturb our dinner but otherwise a dry night under an almost full moon in front of the fire. It being a Sunday night, almost the entire campground was empty so we had the place almost completely to ourselves. On Monday, we originally contemplated zipping up and tagging the summit of Cannon Mountain but decided not to overreach on our first trip and get back early enough to unpack and get organized. Overall, it was a fun trip. So 2 summits down, and it may take us a few years, but 46 to go!

P.S. Additional select photos of the trip are available on facebook.

If you’ve been following along with Project Darkstar’s progress over the years, you know that transparent multi-node scaling capabilities are one of its main attractions as a platform. On the other hand, you should also know that with Project Darkstar being a still maturing research project in Sun Labs, these features are not yet done. In fact, we have still not proven if what we are attempting can even be done. Our fearless lead architect, Jim Waldo, has put together an excellent series of posts on his blog outlining why and how we hope to achieve this multi-node scaling. He calls it the “Four Miracles.”

Our goal for Austin was to put together some measure of a compelling visual demonstration of our current progress towards achieving these “Four Miracles.” Specifically, we hoped to show how Ann’s miracle of transparently moving clients from one node to another in a cluster worked in tandem with Keith’s miracle of monitoring a node’s health and intelligently making decisions about which clients to relocate and when. Also, while Jane’s miracle of detecting and organizing clients into groups of affinity groups based on social networking algorithms is not complete, we also hoped to portray how it should work by rigging up an app that formed affinity groups based on a players location in the game (specifically which chat room it was in).

In the weeks leading up to the conference, Keith and I came up with two demos that for the most part seemed to hit the mark. Using the JMX facilities already built into Darkstar, I hacked up a monitoring GUI that tapped into a running Darkstar cluster and displayed each of the nodes as a vertical bar. The height of the bar represented the total number of clients connected to that node, and the color optionally represented either the node health, or fragments of colors represented the affinity group of each connected client. For the first demo, we had Project Snowman running in a multi-node cluster and showed how the new node health monitoring features of darkstar caused client traffic to spill over and be intelligently distributed between the nodes. Here’s a quick video of Keith talking through it on expo floor in Austin:

The second demo had Darkchat running on a multi-node cluster and was designed to show how clients would relocate between nodes depending on which rooms they were connected to in the application. Here’s another quick video given by yours truly on the floor at Austin:

Both of these demos we had running throughout the week on the floor and the response from people who came by was generally positive. I think one of the main differences that I noticed between this year and last year was the amount of quality traffic that we had come through the booth. I talked to a lot of people at Austin last year, but a large percentage of them had never heard of Project Darkstar or were just vaguely familiar with it. This year many people came by who were already committed to a project using Darkstar, or were very interested in our progress, or were familiar with the technology and had a strong desire to learn more. I think the best piece of anecdotal evidence was one person from the Intel booth who came by and said “Oh, so this is for real now?” Referring, of course, to the significant headway that we are finally making and showing on the multi-node scaling capabilities of the Project Darkstar platform.

In a tough year for everyone in just about every industry, I think going to this conference and putting together these demos have injected some energy into both the Project Darkstar community and the team. A few more personal observations:

• The expo floor seemed smaller this year and overall attendance appeared to be down. Not too surprising but hopefully a sign of things past and not things to come.

• A little tidbit on some of the pains of getting the demos setup. Leading up to the conference, we had everything running just fine in Burlington and the demos packaged up and ready to go. After a few hours of pulling cables, moving pods around, and booting up systems, we fired up the node health demo on Tuesday, the day before the expo floor opened. Except, it didn’t work. At least not like it did in Burlington. When simulating an overloaded node, instead of offloading clients onto the other node right away, there was some seemingly random and arbitrary delay of 20 – 30 seconds before any clients would be moved. Huh?

  After some hours of debugging, and pulling Seth into the mix for help, we finally tracked it down. The (still unfinished) node health code offloads identities from a node when it gets overloaded. However, it doesn’t just move client identities, it also moves identities of robots and other NPC’s in the system. Since each snowman game has a number of robots, it was choosing to move the robot identities before moving the client identities. The question, of course, is why didn’t we ever see this behavior in Burlington? Well it turns out that the order in which identities are chosen to be moved is deterministic and seemingly alphabetical. While in Burlington, our client simulated players were generating identity names according to the hostname of the client machine (dstar1, dstar2, dstarX, …). The hostnames of the machines used in Austin? x2250-1, x2250-2, etc. So in our Burlington deployment, the client identities were always getting chosen before the robot identities since they started with a d; in our Austin deployment, the client identities were always getting chosen after the robot identities since they started with an x. Unbelievable.

• Keith gave a talk during a one hour session which was awesome. He went through a number of obstacles and challenges he faced when building Darkchat for JavaOne and I think it came across as very real and genuine.

• One final note, it appears as though Chuck Norris still doesn’t need scalable server technology. All of his CPU’s run faster to get away from him. Also, any code that he writes cannot be optimized. For anyone else, though, Project Darkstar could be a solution.