Yesterday, though, I discovered a subtle distinction about the semantic behavior of the Future interface that slightly diminishes its power. I found myself in a situation where I wanted to cancel a Future, but then still wait until its associated task has completed. Initially I thought this might be accomplished with something like this:

    // send an interrupt to the task's thread
    future.cancel(true);
 
    // block until the task is complete
    boolean finished = false;
    boolean interrupted = false;
    while (!finished) {
        try {
            future.get();
            finished = true;
        } catch (CancellationException ce) {
            // task has been cancelled, handle appropriately
            finished = true;
        } catch (ExecutionException ee) {
            // task completed with error, handle appropriately
            finished = true;
        } catch (InterruptedException ie) {
            // current thread has been interrupted, record and continue
            interrupted = true;
        }
    }
    // re-interrupt this thread if we've been interrupted
    if (interrupted) {
        Thread.currentThread().interrupt();
    }

Essentially, call cancel(true) on the Future in order to send an interrupt to the associated task’s thread. Then call get() on the Future in a loop to ensure that the task has properly handled the interruption and completed execution before proceeding. However, the above code does not work as expected. The problem is that once you call cancel() on a Future, the task’s thread will be sent an interrupt if it is running, but then the state associated with the task is completely cast aside. Subsequent calls to get() on the same Future will always immediately throw a CancellationException and there is no available mechanism to block until the task’s thread has completed handling the interruption and finished its execution.

The worst part about this is that it is not immediately clear from the javadoc for the Future interface that this is the actual behavior. The description of the cancel() method is very clear that if the task has not started, it will prevent it from starting, but if it has already started, it can make an attempt to “cancel” it via interruption of the task’s thread. Of course, since in Java interrupting a thread is only a suggestion for it to actually stop what it’s doing, there’s no guarantee about when or if the task will actually stop unless you handle the interruption properly. Therefore, I hoped that calling get() would wait for the computation to complete, and then throw a CancellationException if the thread was interrupted. After testing it and also digging through the Java source code for the Future implementation, I discovered that that is definitely not the case.

I struggled for a bit to come up with a reasonable alternative mechanism for accomplishing this behavior short of re-implementing my own thread pool and managing the threads myself. Fortunately, after speaking with a colleague, I was pointed to the beforeExecute and afterExecute hooks on the ThreadPoolExecutor. It takes a little more work then I had hoped, but using those hooks allows you to augment the functionality of ThreadPoolExecutor to achieve this type of behavior. I’ll leave the details of it up to you.

Anyhow, that’s my tidbit for the day. Be careful when playing with the Future!

With this said, though, we can still strive to identify upper bounds on Project Darkstar’s performance at a more fine-grained level. Project Darkstar is an event driven transactional system, so all operations are not without cost. With these micro benchmarks, I’m hoping that I can establish a relative cost to each of the operations using the DataManager, the ChannelManager, and the TaskManager. For example, how expensive is it to retrieve an object using DataManager.getBinding() vs ManagedReference.get(). How much overhead is involved with each transaction? How expensive is it to create a Channel or send a message on a Channel? With more or less users? While the cost of retrieving data from Darkstar’s data store should be an order of magnitude faster than using J2EE and a RDBMS, it is also likely an order of magnitude slower than retrieving data from a data structure that is already in memory and using no synchronization. This is information that users really need to be aware of and be able to take into perspective when designing their game, structuring their tasks, and establishing their own expectations of what the performance should be like.

So over the past couple of days, I’ve been debugging a problem in these benchmarks. In one particular test, I was attempting to measure the raw execution time per call to DataManager.getBinding() from the Project Darkstar API. The test was pretty simple, I just set a large number of bindings in a single set of setup transactions. Then I would time the execution of another set of transactions that would make some subset of calls to getBinding() on the names that I had just setup. Taking into account previously measured transaction overhead I could then come up with a reasonable estimate of the cost per operation. Seems easy right? Well it turns out that I hit a snag. In running this test, I was repeatedly getting a situation where a seemingly random name binding was not being set properly during setup. Most of the calls to getBinding() would work fine, but a couple were throwing NameNotBoundException. What? This didn’t make much sense. I went back and looked over my code many times, I tried a myriad of variations, logging output, and print lines, but still no luck. I was still getting NameNotBoundException for what seemed like a random name in the sequence. Hmmph.

At this point, I went through a whole series of exercises, all centered around one assumption, that my code was right. I tested the native edition vs. the Java edition of BDB, suspecting maybe there was a weird bug in one of them: same result. I tried longer transactions, more operations, larger serialized data objects: same result. I tried running my benchmarks in different orders: same result. I even started writing test cases for DataManager.setBinding() that simulated transaction rollback and retry, large numbers of consecutive calls to setBinding() and binding and rebinding of the same name. I thought I was going to uncover some weird corner case bug. But those tests were passing! I was at a loss. After probably two days of sporadic attempts at debugging this, I finally went back and looked really hard at my own test code. And… I found a bug (doh!). It turns out that I was being too cute with my setup transactions, and was modifying a non-local counter variable inside of my anonymous nested transaction class. In random situations, this class would abort and retry (a normal darkstar operation), but since it was modifying a variable that lived outside of the task itself, this value was not being rolled back. The result was that a name binding would be skipped periodically (exactly the behavior that I was seeing).

So here’s my question. Why did I assume that code that I wrote in less than a day was more likely to be correct than Berkeley DB itself, a project that’s been developed and tested for a couple of decades? Why did I assume that code that I wrote in less than a day was more likely to be correct than Project Darkstar’s Data Service code which has been developed and tested for years? I mean, I knew better than to think that Tim’s code is the likely culprit, but I still started writing test cases thinking I was going to heroically find some obscure bug. This, my friends, is a violation of the number 1 rule of programming: If you’re having problems, It’s Always Your Fault (almost). I mean, don’t get me wrong, I’ve found (and reported) bugs in well established open source projects before, but those situations are actually few and far between. I also don’t mean to suggest that Project Darkstar is bug free. I do think, though, that sometimes it’s too tempting to conclude that there’s a bug in that library you’re using, or there’s a performance limitation in that technology that is fundamentally impossible to overcome. Maybe that’s true, but 99% of the time, it’s your fault.

And with regard to those micro benchmarks, I’m hoping to publish some results soon (assuming I don’t get hung up with any more boneheaded mistakes!)

I have one more session on my schedule for today but this will likely be my final blog post on this year’s JavaOne conference. A few closing thoughts:

• On the subject of Project Darkstar, I would say it was a generally positive event. Darkstar is a technology that clearly many have heard of, are interested in, and have a sincere desire to apply it for their own needs. With DarkChat, the CommunityOne and JavaOne sessions, and the hands-on lab, this conference seemed to do a decent job of generating some additional buzz.
• On a more general note, there’s obviously a bit of uncertainty surrounding the future of Java, JavaOne, and how Oracle will handle its future direction. I think that showed a bit at this conference, as there was no clear air of excitement and energy around anything in particular. There weren’t really any groundbreaking announcements, no killer apps, no prototypes that can truly be considered game changers. I did see many examples of fine engineering and innovation but nothing worthy of eliciting that “Wow” factor from a general audience.
• Finally, on a personal note, I enjoyed coming to this conference and felt like I was able to connect a little more to the community than at the two GDC conferences I’ve been to this year. While Project Darkstar is a technology targeted specifically at games, I wouldn’t say I consider myself a “game developer” at all. I’m a Java developer, and more generally a software developer, and while it was a long and exhausting week, I was glad I got the opportunity to be here and expand my horizons a bit.

So that’s it for my week at JavaOne 2009. I hope you enjoyed following along. Next week I’ll be back to my regularly scheduled Ultimate-playing ways.

In any case, yesterday was a light day for me at the conference. I attended a session which was essentially a survey of a grab-bag of small test tools and gizmos. An interesting takeaway from that session was that testing Java code is not always best done using Java. Often times certain necessary and useful features of the language (like private methods and security features) make things difficult to test. Technologies like Groovy and JRuby, though, contain hooks that allow you to circumvent some of these features for the purposes of testing, without making your test code ugly, brittle, and difficult to maintain. I think the case that was made was good in the case of some tools, but others just traded one piece of complicated (but recognizable) syntax, to another piece of complicated (and unrecognizable) syntax instead. There were some things worth investigating though.

After that session, as mentioned I also recorded a quick podcast on Project Darkstar at the java.net CommunityCorner. The podcast was in the form of a conversational type interview with one of the editors at java.net and he told me that he’ll be posting it and a blog of the event somewhere on java.net within the next couple of weeks. Stay tuned for that.

I didn’t do a lot Thursday afternoon. I wandered the pavilion floor, talked with a few people, and handed out the rest of the hundred or so Project Darkstar pens that I brought with me to the conference. I then met up with Keith, Karl, and Mike. Mike and his wife offered their generous hospitality with a low-key, very nice get together at their house in Oakland Hills.

I attended James Gosling’s general session “Toy Show” this morning and there were some neat things showcased. One of the cool innovations that was demoed was the mashup of a Wii remote, a JavaFX application being projected onto a piece of transparent, frosted glass, and a glove with infrared sensors on the finger tips. The result was a makeshift touch screen that gave a user interface similar to what they compared to as something out of the movie Minority Report. Pretty slick. There was also a more sobering presentation on how one company is using Java to do image recognition and analysis to identify images, including sharpening the ability to more quickly, easily, and accurately diagnose and identify cancer from biopsy images. Other highlights included a Java powered, energy efficient, hybrid Lincoln Continental, and an Audi capable of driving itself.

I’m getting ready to attend a few more sessions before the conference draws to a close, and I’ll have some closing thoughts a little later.

Today at 11:30AM I’m going to be at the java.net CommunityCorner located at booth 101 in the Pavilion. A few other members of the Project Darkstar team should be joining me as well, and I’ll be doing a brief podcast on Project Darkstar. If you’re here at the conference, please stop by to chat (and take a free pen!).

After the keynote, I sat in on Kohsuke Kawaguchi’s session on bringing continuous integration in Hudson to the cloud. This seemed intriguing and relevant since I maintain a Hudson instance internally for Project Darkstar builds and am looking for ways for us to expand its usefulness. I was also interested to see what exactly he meant by using “the cloud”. The term “cloud computing” is such an overloaded term these days and in fact I would suggest that a very small percentage of things that are named “cloud computing” actually qualify as such. What Kohsuke has done for this session though, is one of those things. Essentially, he’s built a Hudson plugin that will automatically provision, configure and use an Amazon EC2 instance as a Hudson build slave dynamically and on demand based on load. So, if you have a Hudson build cluster, and your build machines are getting swamped with work, Hudson will spin up a new virtual build slave automatically and use it to help handle the additional demand. When the build queue dies down, after a period of time the EC2 instances will shut themselves down and disappear. Really slick.

After taking a break for lunch, as I mentioned yesterday I wanted to make sure I went to Josh Bloch’s session on Effective Java. He talked about several topics including Java generics, enums, varargs, and serialization tricks. It was all really useful stuff and it turns out that I clearly have never really learned how to properly use wildcards with Java generics. In other words when to use List<? extends MyType>, List<? super MyType>, or just List<MyType>. This “PECS” rule, as Josh called it, is so simple that it’s seems almost embarrassing that I didn’t know it. Further incentive for me to read the rest of his book and overall a very worthwhile session.

In the late afternoon, I gave my newly tweaked Project Darkstar talk. I was actually really encouraged by this talk as we had I would say around 100 attendees in the room. Additionally, after the talk (which seemed to go pretty well), there were quite a few good honest questions and several people came by afterwards who were genuinely interested in learning more and doing more with the technology. In fact, there were more questions asked at the end of this talk then any other talk that I’ve attended so far at both CommunityOne and JavaOne this week. It’s interesting too, that even though Project Darkstar is a technology specifically designed for games and game developers, I feel like it has had a much greater presence and interest at JavaOne, a Java developers conference, then at GDC, a game developers conference. That could also simply be because of the quantity and prominence of the sessions and events going on that are related to it though (including DarkChat).

Overall, it was a good day but also pretty busy and exhausting. I met up with Karl, Keith, Mike, and John for a good dinner and am now pretty much ready to crash.

The keynote was great. Chris, of course, kicked off the event and introduced the DarkChat application that will be running throughout the conference. It’s basically a simple social networking tool that shows off JavaFX. The other thing about it, though, is that the backend is built on Project Darkstar, and Keith is the one who had been spending a lot of his time recently getting that put together. Chris also snuck in a plug for the Project Darkstar sessions during that intro (the Project Snowman lab was completely full this afternoon).

After that, obviously, he yielded to CEO Jonathan Schwartz who ran most of the show. There were several announcements, including preview releases of JDK 7 and J2EE 6 and updates to JavaFX. Perhaps the most interesting, though, is the launch of the Java Store beta program. The Java Store is essentially a distribution mechanism that taps into the huge install base of Java to deliver Java applications to consumers in a consistent manner. Think iPhone’s app store except replace iPhone users with Java users. Now, the concept behind the Java Store is not a new idea, but packaging and distributing Java applications has always been sort of a funny animal. I actually think this is a great initiative and hope it takes off.

After the keynote, I attended two sessions. The first was on using Java on game handhelds. This session caught my eye because the description claims that they were able to connect a Sony PSP to a Nintendo DSi using a Project Darkstar server as the game backend. Wow that sounds cool right? Well, it is cool, but not quite as cool as you think. What they actually did is hack the PSP and DSi and put their own firmware on the devices which supports JavaME. They then wrote a game client using the JavaME darkstar client and hooked them both up to the same darkstar server. After some finagling they did eventually get a PSP and two DSi clients logged into the same game, but they were not using anything native to the two devices. A ways off from what we’d like to see but still pretty neat.

The second session that I went to caught my attention because Josh Bloch was one of the presenters. For a while now, I’ve been meaning to get my hands on and read all the way through his book Effective Java. This session further solidified that desire. During the session, he and Neal Gafter analyzed five or six Java code snippets that look correct but actually have tricky and subtle bugs. Not only was the content interesting, but the back and forth banter style presentation format was entertaining and engaging. I plan to attend another of Josh’s talks scheduled for tomorrow.

After a break, as I mentioned above, I proctored and ran the Project Snowman hands-on lab in the late afternoon (with the help of Dan, Keith, and a few other proctors as well). Overall, I would say it ran ok. There was one large and unfortunate problem at the beginning of the lab that reeked havoc throughout the remainder of the session. Essentially, one of the first steps in the lab is to install and configure the Maven plugin for Netbeans. Unfortunately, when all 100 people in the room tried to do that, we overwhelmed something on the network (whether it was the outgoing pipe in the room, the Netbeans server, or what who knows). Getting Maven up and configured for everyone took a while, and once it was up, network connectivity issues continued to plague the attendees as Maven would occasionally search for artifacts on the network causing it to pause or hang during builds. The result is that running the session was not very smooth as I and the other proctors (special thanks to them btw) spent most of the time running around the room debugging these annoying Maven issues rather than Project Darkstar issues. Regardless, though, I think many people did enjoy it and they were given all of the materials that they need to try to complete it on their own.

So that was my day. There’s another session tonight on the Java Collections Framework that I’m currently signed up to attend. But first, dinner. ]]> http://owenkellett.com/2009/06/02/javaone-day-1/feed/ 0 http://owenkellett.com/2009/06/01/communityone/ http://owenkellett.com/2009/06/01/communityone/#comments Tue, 02 Jun 2009 04:52:19 +0000 Owen http://owenkellett.com/?p=386 Getting to JavaOne early to test out the snowman lab actually gave me a chance to attend some of the sessions and events going on today at CommunityOne. This is a free conference that is geared towards users and developers of open source technologies and software. Here’s a rundown of what I did/saw:

• The keynote this morning had two main focuses: cloud computing and OpenSolaris. There was some good stuff in there; Solaris has long had a bad reputation for being very unfriendly, making many oblivious to its core quality. Solaris is consistently the best performer in many of the Project Darkstar benchmarks that we’ve run, for example. I think OpenSolaris is slowly, but surely, making Solaris more accessible to the masses.
• I attended two morning talks, one on the features in the latest Subversion release and one on a Hudson use case. Most of the material I was already aware of but looking at some of the things they were doing with Hudson, I think we should really investigate expanding our Hudson deployment. Currently we have a Hudson server doing continuous integration builds on each commit to the Project Darkstar trunk. However, it’s only deployed on a single Solaris machine so our continuous test platform is limited to that. In the presentation, these guys were using Hudson to fire up multiple virtual machines representing various platforms on demand to do cross-platform testing of Netbeans. This could also be an option for us.
• In the afternoon, I went to Chris Melissinos’s talk on Project Darkstar. Chris is actually the Master of Ceremonies for the entire JavaOne conference and is one of the original promoters of the Project Darkstar project. If you’ve ever met him, you know that he’s one of the most dynamic and outspoken people there is. Which makes this next fact even more surprising in that I actually ended up giving part of Chris’s talk. About five minutes before the session was about to start, I bumped into Chris and chatted with him for a few minutes. Then he said something like “Oh good I’m glad you’re here, do you mind coming up and sitting on stage to answer any technical questions people might have?” Um, sure? The next thing I know some guy is putting a microphone on me and I’m sitting in front of these blinding lights. Then, about halfway through the talk, Chris stumbles across some technical slides and says “Owen do you want to give this part of the presentation? Better you then me butchering it.” Ummmm, ok? I spent the next fifteen minutes or so ad-libbing a sequence of presentation slides, having no idea what was coming next. Thanks Chris! The truth is, the presentation actually did turn out pretty well. Chris was his usual dynamic and entertaining self and I managed to bring just enough technical depth to the show. Karl suggested I stay away from Chris before he gives the JavaOne keynote tomorrow morning though!
• During the keynote this morning I briefly heard them mention something about the OpenSolaris Juicer. The Juicer is a service which allows anyone to submit a piece of software for OpenSolaris for test and review to be included in the official “contrib” repository of OpenSolaris packages. Software packaging and distribution is interesting to me so I took a peak at their late afternoon session on the Juicer. It’s good to see that OpenSolaris is finally approaching something close to the apt system that has been around for years on Debian and now Ubuntu.

That’s about it for my day. I met up with a few others for dinner and am now getting ready to go to sleep. Big day tomorrow with the Project Snowman lab scheduled in the afternoon.

If you haven’t already seen it, John already gave an excellent run down of the Project Darkstar related activities in the Project Darkstar team blog. At the risk of sounding repetitive, I’ll be participating in three of those events this week:

• On Tuesday, I’ll be running a hands on lab on Project Darkstar. The lab will essentially be a step-by-step tutorial on coding up the Project Snowman game. I just zipped through the actual lab on the actual lab machines earlier this afternoon and I think it’s going to turn out to be a pretty fun lab.
• On Wednesday, I’ll be giving a newly tweaked version of my Project Darkstar technical talk. Taking a queue from Brian (award winning high school history teacher), I’ve worked in some verbal sci-fi references to go along with the visuals. This should make it awesome.
• On Thursday, I’ve been tagged to do a quickie podcast in the JavaOne Community Corner. I think it will be a pretty informal chat about Project Darkstar.

So those are my responsibilities this week. Other than that, I’ve scoped out and signed up for some other sessions that peaked my interest, including among others one on Hudson, a few on unit testing, and a couple being given by the Java veteran Josh Bloch. Then there are the keynotes, general sessions, Java Pavilion, and probably too many other events going on for me to keep track of. And of course I’ll try (and most certainly fail) at keeping up with the host of parties and bashes going on in the evenings.

It will be a busy week ahead but I’m pretty excited about it. To be honest, a part of me was dreading this event. Mostly because I’m not a very outspoken person, and often like to keep to the shadows. Being put front and center for these sessions is essentially way outside of my comfort zone. I was also talking to Katy last night about how travelling can be draining for me. Between Austin, GDC, and now JavaOne, this is the third time I’ve travelled in a year’s time. Small potatoes for some, but the most ever for me. Now that I’m here, though, I’m ready for the challenge and am looking forward to a successful (and fun!) week at JavaOne. Maybe I’ll see you there…

• It was really great to finally meet Esko, Jonathan, and Andres – three of the four contest winners from the Project Darkstar Developer Challenge.
• The aforementioned contest winners were interviewed on Chris Melissinos’s radio show SHIFT Radio on Friday morning. I found some time to listen to it after I returned, and honestly, it’s a great display of the internal gears of the Project Darkstar open source community at work. Here we have three different people, from three different countries, all actively contributing to the Project Darkstar effort for three different reasons. Esko, a student, is coming at it with academic curiosity and has independently built a feature into the core that many in the community have been clamoring for. Jonathon, a game developer, has taken an active interest in Project Darkstar from a practical standpoint and put together an ActionScript client that inter-operates with a Project Darkstar server. While Andres, an independent game developer, has built a real game with Project Darkstar. If you’re interested in Project Darkstar at all, have a listen. On a side note, see if you can count how many times Chris says some form of the words ridiculous and awesome.
• Probably the biggest splash that was made by a new player on the expo floor was the completely over-the-top OnLive booth. In twenty five words or less, OnLive is trying to make a grab for some of the console games market with a subscription based, games on demand model. Rather than buying a console and then paying for shrink-wrapped games, consumers get a small, network enabled device that gives them access to the complete library of games that you play over the network for a monthly fee. In terms of the business model, I think they have an approach that could take off (others seemed to disagree with me though). However, I have my doubts that they’ll be able to iron out all of the technical issues with such a setup. Essentially, the only thing happening on the client side is that it is sending controller actions in real time back to the server, the server is processing those actions, and then sending just the video feed back to the client. The guys at the booth were talking mostly about their awesome video compression technology (and in fact I have no doubts that it is, in fact, awesome). However, I don’t see that being the real problem. Because the client and server are in complete, synchronous lock-step, it doesn’t matter how fast you can compress the video, the long tail by far is the network latency. You can improve your chances of low latency by sprinkling OnLive servers all over the globe, but the internet is still prone to blips, spikes, and just general unpredictable latencies. Add to that the fact that during peak demand, the OnLive servers could potentially have millions of users asking for not just processing power, but high performance graphics accelerator type processing power, and I just don’t see how it’s going to scale.
• What’s the deal with so many countries and provinces having extravagant booths? I saw booths for Canada, many of its provinces (Alberta, Newfoundland, etc), Korea, Germany, Scotland, Argentina… I suppose many places see a real value in reaching out to this industry as it can serve as a boon for the economy, it can generate jobs, and it can just generally stir up interest in ways that maybe other industries just can’t do.
• I did get a chance to make it to several sessions and lectures during the conference. Most of the talks that I went to were mostly related to game design and less so on the technical programming problems that we focus on for Project Darkstar. I still found them valuable, though, to get a general sense of the issues that game developers tend to think about.
• Despite its simplicity, I found Hideo Kojima’s Thursday Keynote both really intriguing and really entertaining. He basically chronicled the development lifecycle of the entire Metal Gear series of games, from the original Metal Gear developed for the MSX2 platform, to Metal Gear Solid 4 developed for the Playstation 3. I think what’s most interesting is that many of the game design choices and game play choices were heavily influenced by the capabilities (or lack thereof) of the hardware and software being used.

Overall, GDC 2009 was a great experience. I hope that we get some positive fallout from it.