I have established what I hope will become an official community fork of Project Darkstar. This fork will go by the name RedDwarf and is hosted as a sourceforge project. I cannot take credit for the new name, as it was dreamt up by the original instigator of Project Darkstar at Sun Labs (Jeff Kesselman), but I do hope that it will become an even more well known name than Project Darkstar ever was in the games industry. So without further adieu, here are the new community guidelines:

• The sourceforge project is used to host what were once three separate projects on java.net. The reddwarf-server, reddwarf-shared, and reddwarf-java-client all live in the same subversion repository, but under completely separate sub-trees, each with their own trunk, branches, and tags.
• The development process should remain largely unchanged from Project Darkstar. All commits to any trunk repository must undergo a thorough review by at least one other committer, and commit privileges are earned. Review requests should be sent to the mailing list: reddwarf-develop on sourceforge.
• All design, support, and informational documents should go in the Trac instance hosted at sourceforge.
• All issues and bugs should be filed as a ticket in Trac.
• Forum discussions should continue as usual on the sourceforge hosted forums.
• Releases will be done periodically as appropriate. All releases will be published to the central Maven repository rather than the java.net Maven repositories. (This is my first task and may take some time). They will also be published as files for download on the sourceforge site.

Progress will be much slower than before, but I think it’s very important to maintain the structure and code quality standards that we had established as a fully funded project. Will anything come of this effort? I’m not sure. But I think it’s the best chance for coordinated progress to continue with Project Darkstar.

I must say that the shock and disbelief of learning that you’ve been laid off is a predictably emotional time. Of course I didn’t think it would happen to me, but it did: proof that job security is all but an illusion. They say losing your job is like dealing with any other type of loss, which is absolutely true. Knowing this doesn’t make it any easier though. Despite this difficult situation, however, I have received nothing but support from my family, friends, colleagues and even people who were previously just casual acquaintances. Thank you to everyone who has been there so far; I know for a fact I’ll come out the other side of this a strong person.

As for my future, and the future of Project Darkstar? Well both are uncertain. I have already started ramping up and preparing for a full scale job search in the hope that I will find something even better than the best job I’ve ever had. In terms of Project Darkstar, a core group of former members of the team have already started exploring alternative ways to keep the project going. This includes both potential for-profit and volunteer efforts to carry out the original mission objectives laid out years ago. At the very least, after getting myself organized, I personally intend to continue working on the project on a part-time basis during my job search and hopefully beyond. More details to come…

With this said, though, we can still strive to identify upper bounds on Project Darkstar’s performance at a more fine-grained level. Project Darkstar is an event driven transactional system, so all operations are not without cost. With these micro benchmarks, I’m hoping that I can establish a relative cost to each of the operations using the DataManager, the ChannelManager, and the TaskManager. For example, how expensive is it to retrieve an object using DataManager.getBinding() vs ManagedReference.get(). How much overhead is involved with each transaction? How expensive is it to create a Channel or send a message on a Channel? With more or less users? While the cost of retrieving data from Darkstar’s data store should be an order of magnitude faster than using J2EE and a RDBMS, it is also likely an order of magnitude slower than retrieving data from a data structure that is already in memory and using no synchronization. This is information that users really need to be aware of and be able to take into perspective when designing their game, structuring their tasks, and establishing their own expectations of what the performance should be like.

So over the past couple of days, I’ve been debugging a problem in these benchmarks. In one particular test, I was attempting to measure the raw execution time per call to DataManager.getBinding() from the Project Darkstar API. The test was pretty simple, I just set a large number of bindings in a single set of setup transactions. Then I would time the execution of another set of transactions that would make some subset of calls to getBinding() on the names that I had just setup. Taking into account previously measured transaction overhead I could then come up with a reasonable estimate of the cost per operation. Seems easy right? Well it turns out that I hit a snag. In running this test, I was repeatedly getting a situation where a seemingly random name binding was not being set properly during setup. Most of the calls to getBinding() would work fine, but a couple were throwing NameNotBoundException. What? This didn’t make much sense. I went back and looked over my code many times, I tried a myriad of variations, logging output, and print lines, but still no luck. I was still getting NameNotBoundException for what seemed like a random name in the sequence. Hmmph.

At this point, I went through a whole series of exercises, all centered around one assumption, that my code was right. I tested the native edition vs. the Java edition of BDB, suspecting maybe there was a weird bug in one of them: same result. I tried longer transactions, more operations, larger serialized data objects: same result. I tried running my benchmarks in different orders: same result. I even started writing test cases for DataManager.setBinding() that simulated transaction rollback and retry, large numbers of consecutive calls to setBinding() and binding and rebinding of the same name. I thought I was going to uncover some weird corner case bug. But those tests were passing! I was at a loss. After probably two days of sporadic attempts at debugging this, I finally went back and looked really hard at my own test code. And… I found a bug (doh!). It turns out that I was being too cute with my setup transactions, and was modifying a non-local counter variable inside of my anonymous nested transaction class. In random situations, this class would abort and retry (a normal darkstar operation), but since it was modifying a variable that lived outside of the task itself, this value was not being rolled back. The result was that a name binding would be skipped periodically (exactly the behavior that I was seeing).

So here’s my question. Why did I assume that code that I wrote in less than a day was more likely to be correct than Berkeley DB itself, a project that’s been developed and tested for a couple of decades? Why did I assume that code that I wrote in less than a day was more likely to be correct than Project Darkstar’s Data Service code which has been developed and tested for years? I mean, I knew better than to think that Tim’s code is the likely culprit, but I still started writing test cases thinking I was going to heroically find some obscure bug. This, my friends, is a violation of the number 1 rule of programming: If you’re having problems, It’s Always Your Fault (almost). I mean, don’t get me wrong, I’ve found (and reported) bugs in well established open source projects before, but those situations are actually few and far between. I also don’t mean to suggest that Project Darkstar is bug free. I do think, though, that sometimes it’s too tempting to conclude that there’s a bug in that library you’re using, or there’s a performance limitation in that technology that is fundamentally impossible to overcome. Maybe that’s true, but 99% of the time, it’s your fault.

And with regard to those micro benchmarks, I’m hoping to publish some results soon (assuming I don’t get hung up with any more boneheaded mistakes!)

If you’ve been following along with Project Darkstar’s progress over the years, you know that transparent multi-node scaling capabilities are one of its main attractions as a platform. On the other hand, you should also know that with Project Darkstar being a still maturing research project in Sun Labs, these features are not yet done. In fact, we have still not proven if what we are attempting can even be done. Our fearless lead architect, Jim Waldo, has put together an excellent series of posts on his blog outlining why and how we hope to achieve this multi-node scaling. He calls it the “Four Miracles.”

Our goal for Austin was to put together some measure of a compelling visual demonstration of our current progress towards achieving these “Four Miracles.” Specifically, we hoped to show how Ann’s miracle of transparently moving clients from one node to another in a cluster worked in tandem with Keith’s miracle of monitoring a node’s health and intelligently making decisions about which clients to relocate and when. Also, while Jane’s miracle of detecting and organizing clients into groups of affinity groups based on social networking algorithms is not complete, we also hoped to portray how it should work by rigging up an app that formed affinity groups based on a players location in the game (specifically which chat room it was in).

In the weeks leading up to the conference, Keith and I came up with two demos that for the most part seemed to hit the mark. Using the JMX facilities already built into Darkstar, I hacked up a monitoring GUI that tapped into a running Darkstar cluster and displayed each of the nodes as a vertical bar. The height of the bar represented the total number of clients connected to that node, and the color optionally represented either the node health, or fragments of colors represented the affinity group of each connected client. For the first demo, we had Project Snowman running in a multi-node cluster and showed how the new node health monitoring features of darkstar caused client traffic to spill over and be intelligently distributed between the nodes. Here’s a quick video of Keith talking through it on expo floor in Austin:

The second demo had Darkchat running on a multi-node cluster and was designed to show how clients would relocate between nodes depending on which rooms they were connected to in the application. Here’s another quick video given by yours truly on the floor at Austin:

Both of these demos we had running throughout the week on the floor and the response from people who came by was generally positive. I think one of the main differences that I noticed between this year and last year was the amount of quality traffic that we had come through the booth. I talked to a lot of people at Austin last year, but a large percentage of them had never heard of Project Darkstar or were just vaguely familiar with it. This year many people came by who were already committed to a project using Darkstar, or were very interested in our progress, or were familiar with the technology and had a strong desire to learn more. I think the best piece of anecdotal evidence was one person from the Intel booth who came by and said “Oh, so this is for real now?” Referring, of course, to the significant headway that we are finally making and showing on the multi-node scaling capabilities of the Project Darkstar platform.

In a tough year for everyone in just about every industry, I think going to this conference and putting together these demos have injected some energy into both the Project Darkstar community and the team. A few more personal observations:

• The expo floor seemed smaller this year and overall attendance appeared to be down. Not too surprising but hopefully a sign of things past and not things to come.

• A little tidbit on some of the pains of getting the demos setup. Leading up to the conference, we had everything running just fine in Burlington and the demos packaged up and ready to go. After a few hours of pulling cables, moving pods around, and booting up systems, we fired up the node health demo on Tuesday, the day before the expo floor opened. Except, it didn’t work. At least not like it did in Burlington. When simulating an overloaded node, instead of offloading clients onto the other node right away, there was some seemingly random and arbitrary delay of 20 – 30 seconds before any clients would be moved. Huh?

  After some hours of debugging, and pulling Seth into the mix for help, we finally tracked it down. The (still unfinished) node health code offloads identities from a node when it gets overloaded. However, it doesn’t just move client identities, it also moves identities of robots and other NPC’s in the system. Since each snowman game has a number of robots, it was choosing to move the robot identities before moving the client identities. The question, of course, is why didn’t we ever see this behavior in Burlington? Well it turns out that the order in which identities are chosen to be moved is deterministic and seemingly alphabetical. While in Burlington, our client simulated players were generating identity names according to the hostname of the client machine (dstar1, dstar2, dstarX, …). The hostnames of the machines used in Austin? x2250-1, x2250-2, etc. So in our Burlington deployment, the client identities were always getting chosen before the robot identities since they started with a d; in our Austin deployment, the client identities were always getting chosen after the robot identities since they started with an x. Unbelievable.

• Keith gave a talk during a one hour session which was awesome. He went through a number of obstacles and challenges he faced when building Darkchat for JavaOne and I think it came across as very real and genuine.

• One final note, it appears as though Chuck Norris still doesn’t need scalable server technology. All of his CPU’s run faster to get away from him. Also, any code that he writes cannot be optimized. For anyone else, though, Project Darkstar could be a solution.

First, here is a quick description of the project and the tools that I will be using:

• This will be a purely server side project that will print “Hello Project Darkstar!” when the server is booted.
• I will be using Maven as the build tool for the project.
• The project will leverage the Project Darkstar Maven Plugin to aid in deploying, booting, and shutting down the server according to the new distribution format for the official Project Darkstar server package.
• It should be possible to build, run, and test the raw project outside of an IDE, within Netbeans, and also within Eclipse (with a few tweaks for each respective IDE).

There are three source files that are of interest in this project. The first is the actual Java source file that represents the main class for the server. This class should look something like this:

package my.pkg.hello;
 
import java.io.Serializable;
import java.util.Properties;
import java.util.logging.Level;
import java.util.logging.Logger;
import com.sun.sgs.app.AppListener;
import com.sun.sgs.app.ClientSession;
import com.sun.sgs.app.ClientSessionListener;
 
public class HelloProjectDarkstar implements AppListener, Serializable {
  private static final long serialVersionUID = 1L;
  private static final Logger logger = 
      Logger.getLogger(HelloProjectDarkstar.class.getName());
 
  public void initialize(Properties props) {
    logger.info("Hello Project Darkstar!");
  }
 
  public ClientSessionListener loggedIn(ClientSession session) {
    return null;
  }
}

Second, it is strongly recommended for a Project Darkstar application that you include an app.properties file in the META-INF directory of your resulting JAR file. This file will be used as the base set of configuration properties for the server and is a good place to include default values for application specific configuration properties. In our simple example, we’ll include just two required properties, the name of the server and the listener class:

com.sun.sgs.app.name=HelloProjectDarkstar
com.sun.sgs.app.listener=my.pkg.hello.HelloProjectDarkstar

Third, you need a Maven POM file. If you’re unaware of how Maven works, you can read up on it at the Maven website. Essentially, the POM file is used to declaratively describe your project’s resources and dependencies. Maven then uses this information to correctly build it. Our project has only one dependency (the sgs-server-api) so our POM file should look something like this:

<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
 
  <groupId>my.package.hello</groupId>
  <artifactId>hello-projectdarkstar</artifactId>
  <version>0.1-SNAPSHOT</version>
  <name>Hello Project Darkstar</name>
  <packaging>jar</packaging>
 
  <dependencies>
    <dependency>
      <groupId>com.projectdarkstar.server</groupId>
      <artifactId>sgs-server-api</artifactId>
      <version>${sgs-server.version}</version>
    </dependency>
  </dependencies>
 
  <build>
    <plugins>
      <!-- Use 1.6 for source and target during compile -->
      <plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-compiler-plugin</artifactId>
	<configuration>
	  <source>1.6</source>
	  <target>1.6</target>
	</configuration>
      </plugin>
    </plugins>
  </build>
 
  <properties>
    <sgs-server.version>0.9.9</sgs-server.version>
  </properties>
 
  <repositories>
    <repository>
      <id>java.net</id>
      <name>java.net Maven2 Repository</name>
      <url>http://download.java.net/maven/2/</url>
      <layout>default</layout>
    </repository>
  </repositories>
</project>

These three files should be organized in a directory structure that looks something like this:

- hello-projectdarkstar
  - pom.xml
  - src
    - main
      - java
        - my
          - pkg
            - hello
              - HelloProjectDarkstar.java
    - resources
      - META-INF
        - app.properties

At this point you should be able to build a JAR file that is deployable in a Project Darkstar server. Since this is a Maven project, this should be possible:

• From outside an IDE using the command mvn package
• From within Netbeans (ensure that you have the Maven plugin installed, then try “Open Project” and Netbeans should automatically detect that the root folder is a Maven project)
• Or from within Eclipse (ensure that you have the Maven plugin installed, then choose “Import” from the File menu and import as a “Maven project”).

This is all a good first step, but in order to boot up our server, we still need to manually take the JAR file that is built by Maven, deploy it into a Project Darkstar server, and then fire up the server. This is tedious, and we’d rather be able to do this using one command (or one click from the IDE). Enter the Project Darkstar Maven plugin. The plugin is pretty well documented on its website, but essentially what it does is automate the process of taking your built JAR file, deploying it into a Project Darkstar container, configuring the container with the appropriate configuration files, and then booting up the server. It also includes the capability to shutdown a running server. Using this, therefore, we can automate the process of running our Project Darkstar application by simply adding a configuration entry for this plugin into our POM in a separate Maven profile:

  <profiles>
    <profile>
      <id>run-server</id>
      <build>
	<plugins>
          <plugin>
            <groupId>com.projectdarkstar.maven.plugin</groupId>
            <artifactId>sgs-maven-plugin</artifactId>
            <version>1.0-alpha-3</version>
            <executions>
              <execution>
                <id>sgs-run</id>
                <phase>pre-integration-test</phase>
                <goals>
                  <goal>install</goal>
                  <goal>configure</goal>
                  <goal>deploy</goal>
                  <goal>boot</goal>
                </goals>
              </execution>
            </executions>
            <configuration>
              <version>${sgs-server.version}</version>
              <sgsHome>
                ${project.build.directory}/sgs-server-dist-${sgs-server.version}
              </sgsHome>
              <sgsBoot>${basedir}/conf/hello.boot</sgsBoot>
              <sgsServer>${basedir}/conf/hello.properties</sgsServer>
              <files>
                <file>${project.artifact.file}</file>
              </files>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>

This configuration combines the install, configure, deploy, and boot goals of the sgs-maven-plugin into one. Also notice that there are two additional configuration files that are included in your source tree under the conf directory, hello.boot and hello.properties. These files are respectively used to replace the sgs-boot.properties and sgs-server.properties configuration files in the main Project Darkstar distribution. We can keep these files simple for now, but any set of valid configuration properties can be included:

# hello.boot
# =======
JAVA_OPTS = -server -Xmx768M
 
# hello.properties
# =======
com.sun.sgs.app.root=data

So we now have a complete project that looks something like this:

- hello-projectdarkstar
  - conf
    - hello.boot
    - hello.properties
  - pom.xml
  - src
    - main
      - java
        - my
          - pkg
            - hello
              - HelloProjectDarkstar.java
    - resources
      - META-INF
        - app.properties

The project can be built inside or outside of Netbeans or Eclipse using the same mechanisms as described above. However, with this additional configuration, we can also automatically build, deploy, and boot the application in a Project Darkstar container by simply activating the run-server Maven profile. When this is done, Maven will build the project JAR file, download and install the Project Darkstar distribution into the project’s target directory, deploy the built JAR file into the Project Darkstar installation, copy the hello.boot and hello.properties configuration files into the Project Darkstar installation conf directory, and then boot up the server using the sgs-boot.jar from the Project Darkstar installation. This is completely automated and requires doing just the following for each of our respective environments:

• From outside any IDE: just run the command mvn verify -Prun-server. Shutting down the server requires simply a Ctrl-C. Also, in order to clean out the Project Darkstar datastore in between executions, it is necessary to run mvn clean.
• From Netbeans: You should configure the properties of the project by right clicking the project and selecting the “Properties” option. Configure a new Action (or one of the existing actions like the “Run” action) to run the verify goal with the run-server profile activated. Then you should be able to boot up the server by executing that specific action for your project (Right click the project and choose “Run” or whatever action you created under the “Custom” menu). In order to shutdown the server, you should also configure an additional action (i.e. Stop) that runs the sgs:stop goal with the run-server profile activated. Prefer using this action to shutdown the server rather than just killing the process (since multiple JVMs are started when Project Darkstar boots).
• From Eclipse: Similar to Netbeans, you need to configure two “Run Configurations”, one that runs the verify goal with the run-server profile activated, and one that runs the sgs:stop goal with the run-server profile activated. These two configurations should be respectively used to bootup and shutdown the server.

There are a lot more sophisticated ways that you can setup your project with the Project Darkstar Maven plugin that allows you to customize the runtime environment from the command line. You can also tweak it to support booting multi-node configurations by modifying the boot configuration file, or use filterable properties to customize these values at runtime. I won’t get into these here, but I would suggest looking at Project Snowman’s server POM for some ideas. Hopefully, though, this post should give you enough to get the ball rolling in setting up your development environment for a Project Darkstar application.

I have one more session on my schedule for today but this will likely be my final blog post on this year’s JavaOne conference. A few closing thoughts:

• On the subject of Project Darkstar, I would say it was a generally positive event. Darkstar is a technology that clearly many have heard of, are interested in, and have a sincere desire to apply it for their own needs. With DarkChat, the CommunityOne and JavaOne sessions, and the hands-on lab, this conference seemed to do a decent job of generating some additional buzz.
• On a more general note, there’s obviously a bit of uncertainty surrounding the future of Java, JavaOne, and how Oracle will handle its future direction. I think that showed a bit at this conference, as there was no clear air of excitement and energy around anything in particular. There weren’t really any groundbreaking announcements, no killer apps, no prototypes that can truly be considered game changers. I did see many examples of fine engineering and innovation but nothing worthy of eliciting that “Wow” factor from a general audience.
• Finally, on a personal note, I enjoyed coming to this conference and felt like I was able to connect a little more to the community than at the two GDC conferences I’ve been to this year. While Project Darkstar is a technology targeted specifically at games, I wouldn’t say I consider myself a “game developer” at all. I’m a Java developer, and more generally a software developer, and while it was a long and exhausting week, I was glad I got the opportunity to be here and expand my horizons a bit.

So that’s it for my week at JavaOne 2009. I hope you enjoyed following along. Next week I’ll be back to my regularly scheduled Ultimate-playing ways.

In any case, yesterday was a light day for me at the conference. I attended a session which was essentially a survey of a grab-bag of small test tools and gizmos. An interesting takeaway from that session was that testing Java code is not always best done using Java. Often times certain necessary and useful features of the language (like private methods and security features) make things difficult to test. Technologies like Groovy and JRuby, though, contain hooks that allow you to circumvent some of these features for the purposes of testing, without making your test code ugly, brittle, and difficult to maintain. I think the case that was made was good in the case of some tools, but others just traded one piece of complicated (but recognizable) syntax, to another piece of complicated (and unrecognizable) syntax instead. There were some things worth investigating though.

After that session, as mentioned I also recorded a quick podcast on Project Darkstar at the java.net CommunityCorner. The podcast was in the form of a conversational type interview with one of the editors at java.net and he told me that he’ll be posting it and a blog of the event somewhere on java.net within the next couple of weeks. Stay tuned for that.

I didn’t do a lot Thursday afternoon. I wandered the pavilion floor, talked with a few people, and handed out the rest of the hundred or so Project Darkstar pens that I brought with me to the conference. I then met up with Keith, Karl, and Mike. Mike and his wife offered their generous hospitality with a low-key, very nice get together at their house in Oakland Hills.

I attended James Gosling’s general session “Toy Show” this morning and there were some neat things showcased. One of the cool innovations that was demoed was the mashup of a Wii remote, a JavaFX application being projected onto a piece of transparent, frosted glass, and a glove with infrared sensors on the finger tips. The result was a makeshift touch screen that gave a user interface similar to what they compared to as something out of the movie Minority Report. Pretty slick. There was also a more sobering presentation on how one company is using Java to do image recognition and analysis to identify images, including sharpening the ability to more quickly, easily, and accurately diagnose and identify cancer from biopsy images. Other highlights included a Java powered, energy efficient, hybrid Lincoln Continental, and an Audi capable of driving itself.

I’m getting ready to attend a few more sessions before the conference draws to a close, and I’ll have some closing thoughts a little later.

Today at 11:30AM I’m going to be at the java.net CommunityCorner located at booth 101 in the Pavilion. A few other members of the Project Darkstar team should be joining me as well, and I’ll be doing a brief podcast on Project Darkstar. If you’re here at the conference, please stop by to chat (and take a free pen!).

After the keynote, I sat in on Kohsuke Kawaguchi’s session on bringing continuous integration in Hudson to the cloud. This seemed intriguing and relevant since I maintain a Hudson instance internally for Project Darkstar builds and am looking for ways for us to expand its usefulness. I was also interested to see what exactly he meant by using “the cloud”. The term “cloud computing” is such an overloaded term these days and in fact I would suggest that a very small percentage of things that are named “cloud computing” actually qualify as such. What Kohsuke has done for this session though, is one of those things. Essentially, he’s built a Hudson plugin that will automatically provision, configure and use an Amazon EC2 instance as a Hudson build slave dynamically and on demand based on load. So, if you have a Hudson build cluster, and your build machines are getting swamped with work, Hudson will spin up a new virtual build slave automatically and use it to help handle the additional demand. When the build queue dies down, after a period of time the EC2 instances will shut themselves down and disappear. Really slick.

After taking a break for lunch, as I mentioned yesterday I wanted to make sure I went to Josh Bloch’s session on Effective Java. He talked about several topics including Java generics, enums, varargs, and serialization tricks. It was all really useful stuff and it turns out that I clearly have never really learned how to properly use wildcards with Java generics. In other words when to use List<? extends MyType>, List<? super MyType>, or just List<MyType>. This “PECS” rule, as Josh called it, is so simple that it’s seems almost embarrassing that I didn’t know it. Further incentive for me to read the rest of his book and overall a very worthwhile session.

In the late afternoon, I gave my newly tweaked Project Darkstar talk. I was actually really encouraged by this talk as we had I would say around 100 attendees in the room. Additionally, after the talk (which seemed to go pretty well), there were quite a few good honest questions and several people came by afterwards who were genuinely interested in learning more and doing more with the technology. In fact, there were more questions asked at the end of this talk then any other talk that I’ve attended so far at both CommunityOne and JavaOne this week. It’s interesting too, that even though Project Darkstar is a technology specifically designed for games and game developers, I feel like it has had a much greater presence and interest at JavaOne, a Java developers conference, then at GDC, a game developers conference. That could also simply be because of the quantity and prominence of the sessions and events going on that are related to it though (including DarkChat).

Overall, it was a good day but also pretty busy and exhausting. I met up with Karl, Keith, Mike, and John for a good dinner and am now pretty much ready to crash.