I'm working on a web application (JSP), that is running under Tomcat. It also
uses Java3D for off-screen graphics rendering (OpenGL version) to generate
..jpg files.

I have successfully deployed the application on RedHat 9. Unfortunately
"headless" mode does not work for me, since Java3D goes directly to OpenGL
native code. I ended up running Xvfb (X server emulator).

Application itself is working just fine, except that Xvfb over some time
consumes all available memory, and system slowly dies. Restarting JVM does not
help either, although killing and restarting Xvfb does.

Apparently, "real" X server does the same, as application processes requests,
over time it slowly adds memory. Sometimes chunks of memory get released,
memory usage slightly drops, but overall trend is upcoming.

So, I'm trying to figure out why Xvfb fails to release resources.

Similar configuration under Win2K (of course, there is no X there) works with
no problems, and excessive testing indicated no "accumulating" issues.

Here is what I think may happen:

1. The application.
Application itself is seems to be clean, Java runtime shows no signs
of "fatigue" over time, no matter what platform it is running on. Shutting
down Tomcat does not relieve Xvfb (or X for that matter). I could think
about application improperly releasing (or not releasing at all) some kind
of external resources, but what it does, is simply creates an instance of
java.awt.image.BufferedImage class, which J3D runtime somehow connects to
framebuffer.

2. Java3D
This is very suspicious place, we have a little or no control at all on what
is happening between Java world and native code. On other hand, I tried
OpenGL and DirectX versions of Java3D on Win2K platform witn no problems
at all. The only J3D implementation I know for Linux is by BlackDown, which
I think is almost identicall to Sun's reference implementation for OpenGL.
Version 1.3.1 has main memory problems fixed (all of them?).

3. Mesa (OpenGL)
Unfortunately my knowledge is not enough to make assumptions, but this one
seems to be a good candidate.

4. XFree
This is where actual symptoms show themselves. Is this the source of the
problem? It is quite mature product by now (I'm using 4.3), I would not
expect nasty memory leaks not being fixed by now. Some people were saying
about memory fragmentation problems as something non-curable, are they still
there (problems, and people ;-)?

5. Video card driver.
Since OpenGL uses a lot of hardware acceleration, it could be possible, that
a lousy driver screws X server?

6. Hardware
Very unlikely, but who knows?

I would really appreciate any knowledgeable comments / suggestions.