A couple of postings, ago, I talked about how Project Darkstar is a research effort, and so attempts to do the lowest-risk approach to the problems we are trying to solve aren't really appropriate. After all, if the current high-risk approaches don't work out, we can always move back to the more conservative approaches. These lower-risk approaches, for the most part, move the risk away from us an to the game server programmer.
This is an example of the principle of the conservation of hair, which I first heard from John Rose. John pointed out that the amount of hairy code in a system is always conserved; the best we can do is to move that hairy code around. If the Project Darkstar approach to dealing with the hairy problems of distributed load balancing don't work, we will need to move the hair to the game server, and make it someone else's problem. We might have to do that, but we shouldn't start with that as our goal.
But calling the project research causes some people to hear not product quality. Research code, to this way of thinking, means that the authors of the code are trying to build a quick and dirty prototype to see if their ideas hold water. The code is meant to be stable enough to generate some numbers so that a paper can be written and published. There is no need to worry about whether or not the code really works, or can be maintained, or can be extended. Since it is just research, the code will be discarded after the paper has been written, so who cares if the code actually works?
I'll admit that there is a lot of research code out there for which this characterization is not far off the mark. This is especially true when the ideas being tested are those that can be directly observed. There is even a tradition in user interface studies where there is no code at all-- instead, you get someone to impersonate the computer with which the subjects are interacting to see what the interaction would be like. After all, if the purpose of research is to test ideas with a small group of researchers so that the overall cost of the test is minimized, cutting corners in the implementation makes sense.
But it doesn't make sense for the kind of software that is being produced in Project Darkstar. This is code that is very low level-- we are working on task scheduling, and networking, and distributed load balancing. There aren't a whole lot of obvious ways of observing the effects of our experiments. This is one reason that system demos in general, and distributed system demos in particular, tend to be pretty unexciting. What we can show are packets flowing across the network, or CPUs being utilized. But you have to know what is happening to be impressed, or to even understand what is going on.
For a research effort at this system level, the software is not so much a demonstration of the ideas as it is the instrument that we use to test the ideas. Our software is how we observe the results of the tests of our ideas. It is more like an astronomer's telescope or spectrometer. It is only through our software that we can see the results of our tests.
For that reason, we have to be very sure that our software is correct, just as an astronomer needs to know that the optics of his or her telescope are correct. A bug in the software can mislead us as to whether our ideas are working or not. Our confidence in our research results is only as good as our confidence in our software.
For this reason, the code that we write needs to be better that product quality. Before any API is accepted, it gets reviewed by the whole group, as well as by the Darkstar community. Any code that is written also gets reviewed. Our code reviews are not the classic (and sometimes useless) reviews where the author of the code gets up and explains to a group what the code does. We have found it far more effective to send the code to another engineer, who then gets to figure out what it does by reading the source. And everything has to be tested, at multiple levels.
Nearly all of the engineers on the team have worked in product groups in the past, and have shipped products ranging from software libraries to the Java Virtual Machine itself. In none of those groups was the review process more rigorous than the one we have for code that goes into the Project Darkstar trunk. By the way, for those of you in the community, if you want to have code accepted into the trunk, you should know that it will need to go through the same process. But we aren't picking on community members. We go through the same process ourselves.
The result of this process is, I believe, some of the best code you will find anywhere. It is at least product quality. Maybe better. And it is because we are willing to take great risks in our designs that we have to minimize the possibility of error in the code. This is the balance between research and development. If we are going to be adventurous in our research, we have to take even more care in our development.

More...