Supermicro X7DBE+ and 32 GB memory - Hardware

This is a discussion on Supermicro X7DBE+ and 32 GB memory - Hardware ; We recently purchased a Supermicro X7DBE+ based system with 32 gigabytes of memory, however it won't boot FC 6 (or any other 64 bit distribution we have at hand) if more than 24 GB is installed. The vendor (mbx) confirms ...

+ Reply to Thread
Results 1 to 18 of 18

Thread: Supermicro X7DBE+ and 32 GB memory

  1. Supermicro X7DBE+ and 32 GB memory


    We recently purchased a Supermicro X7DBE+ based system with 32
    gigabytes of memory, however it won't boot FC 6 (or any other 64 bit
    distribution we have at hand) if more than 24 GB is installed. The
    vendor (mbx) confirms that they can reproduce the problem, and have
    asked Supermicro for fix/work-around/comment. Does anyone here have
    suggestions?

    Details:

    The boot loader loads, and starts the kernel, which dies with the
    following message:

    Kernel Panic - not syncing: Netlabel failed to initialize properly
    (-17).

    The kernel version is 2.6.18-1.2869. We have also tried 32-bit Ubuntu
    6.10 and Suse 10.1 (kernel 2.6.16). In all cases the boot loader
    functions as expected, but the kernel dies with a mysterious message.
    In the case of Suse the message is:

    kernel Panic: Unable to mount rootfs on known-block(1,0)
    [17179571.332000]

    Booting a slightly older FC6 the message is:

    Dwarf2 unwinder stuck at error_exit +0x0/0x84
    leftover inexact backtrace

    followed by a long trace.


    Daniel Feenberg
    617-588-0343
    feenberg isat nber dotte org


  2. Re: Supermicro X7DBE+ and 32 GB memory

    feenberg@gmail.com writes:
    >
    >We recently purchased a Supermicro X7DBE+ based system with 32
    >gigabytes of memory, however it won't boot FC 6 (or any other 64 bit
    >distribution we have at hand) if more than 24 GB is installed. The
    >vendor (mbx) confirms that they can reproduce the problem, and have
    >asked Supermicro for fix/work-around/comment. Does anyone here have
    >suggestions?


    We have this board with 24GB (12x2GB), so I probably can't help you.
    However, the variety of errors you see makes me think of a hardware
    failure (especially if you also see different errors for the same
    kernel). Our vendor had to work a bit to get the system to work; in
    particular, he had to put in a bigger power supply than he originally
    tried, and he had to use a strong and loud fan (80mm 5300rpm) to cool
    the DIMMs enough. With 32GB these problems (especially the DIMM
    cooling) would be even harder.

    - anton
    --
    M. Anton Ertl Some things have to be seen to be believed
    anton@mips.complang.tuwien.ac.at Most things have to be believed to be seen
    http://www.complang.tuwien.ac.at/anton/home.html

  3. Re: Supermicro X7DBE+ and 32 GB memory

    On Thu, 01 Mar 2007 16:43:10 -0800, feenberg wrote:

    >
    > We recently purchased a Supermicro X7DBE+ based system with 32
    > gigabytes of memory, however it won't boot FC 6 (or any other 64 bit
    > distribution we have at hand) if more than 24 GB is installed. The
    > vendor (mbx) confirms that they can reproduce the problem, and have
    > asked Supermicro for fix/work-around/comment. Does anyone here have
    > suggestions?
    >

    I'm 101% sure it's a hardware problem.
    Have you tried to run memtest86+ for several hours?

  4. Re: Supermicro X7DBE+ and 32 GB memory

    On Mar 2, 9:18 am, alfa wrote:
    > On Thu, 01 Mar 2007 16:43:10 -0800,feenbergwrote:
    >
    > > We recently purchased aSupermicro X7DBE+ based system with 32
    > > gigabytes of memory, however it won't boot FC 6 (or any other 64 bit
    > > distribution we have at hand) if more than 24 GB is installed. The
    > > vendor (mbx) confirms that they can reproduce the problem, and have
    > > askedSupermicrofor fix/work-around/comment. Does anyone here have
    > > suggestions?

    >
    > I'm 101% sure it's a hardware problem.
    > Have you tried to run memtest86+ for several hours?


    Yes. It runs memtest86+ overnight without detecting errors. I don't
    think memtest86+ is multi-threaded, so I there may be fault paths
    associated with multiple cpus that are not tested. With Linux, the
    failure always occurs within seconds (even after a cold start) and is
    independent of the arrangement of memory in the system. Also, the
    vendor is testing a different motherboard at his shop, which militates
    against hardware failure.

    We also thought about heat - the memory sticks quickly get too hot to
    touch, however given that memtest can run for hours without recording
    an error makes that unlikely. The system is a Supermicro case-
    motherboard combination so it should have good cooling and power,
    especially since we haven't added any hardware inside the box (which
    supports 8 drives). There are lots of fans, but no baffles to make
    sure the memory gets circulation. If the failures occurred other than
    at the start of operation, I would definitely suspect overheating.

    I still vote for BIOS problems or kernel bugs as most likely.

    Daniel Feenberg
    feenberg isat nber dotte org



  5. Re: Supermicro X7DBE+ and 32 GB memory

    On Thu, 01 Mar 2007 16:43:10 -0800, feenberg wrote:

    > We recently purchased a Supermicro X7DBE+ based system with 32
    > gigabytes of memory, however it won't boot FC 6 (or any other 64 bit
    > distribution we have at hand) if more than 24 GB is installed. The
    > vendor (mbx) confirms that they can reproduce the problem, and have
    > asked Supermicro for fix/work-around/comment. Does anyone here have
    > suggestions?
    >
    > Details:
    >
    > The boot loader loads, and starts the kernel, which dies with the
    > following message:
    >
    > Kernel Panic - not syncing: Netlabel failed to initialize properly
    > (-17).
    >
    > The kernel version is 2.6.18-1.2869. We have also tried 32-bit Ubuntu
    > 6.10 and Suse 10.1 (kernel 2.6.16). In all cases the boot loader
    > functions as expected, but the kernel dies with a mysterious message. In
    > the case of Suse the message is:
    >
    > kernel Panic: Unable to mount rootfs on known-block(1,0)
    > [17179571.332000]
    >
    > Booting a slightly older FC6 the message is:
    >
    > Dwarf2 unwinder stuck at error_exit +0x0/0x84 leftover inexact
    > backtrace
    >
    > followed by a long trace.
    >
    >
    > Daniel Feenberg
    > 617-588-0343
    > feenberg isat nber dotte org


    Have you tried a 2.6.20 kernel? 2.6.18 came out just as the new
    generation of Intel chips was arriving, it might not completely support
    the chipset on that board.

  6. Re: Supermicro X7DBE+ and 32 GB memory

    On Fri, 02 Mar 2007 09:26:35 -0800, feenberg wrote:


    >> I'm 101% sure it's a hardware problem.
    >> Have you tried to run memtest86+ for several hours?

    >
    > Yes. It runs memtest86+ overnight without detecting errors. I don't
    > think memtest86+ is multi-threaded,

    No in fact, afaik.
    Other tests you could run: pick
    http://www.ultimatebootcd.com/

    32G means 32 memory modules. I doubt much there is enough current
    for driving them all, as such configurations are seldom tested.
    Other signs point towards chipset. Some AMD boards won't take more than
    3.5G with an nforce chipset.

    > so I there may be fault paths
    > associated with multiple cpus that are not tested. With Linux, the
    > failure always occurs within seconds (even after a cold start) and is
    > independent of the arrangement of memory in the system. Also, the
    > vendor is testing a different motherboard at his shop, which militates
    > against hardware failure.
    >


    Have you tried a non-smp kernel first?
    Fedora uses an smp one for all x86_64 configurations.

    > We also thought about heat - the memory sticks quickly get too hot to
    > touch,


    No, no no. That's very bad, it should not happen even under heavy
    overclocking.
    Very often a broken module heats like crazy.
    I had problems with one 1G module, for a total of 2G, not 32 like yours.
    Yet I was able to recompile a kernel over and over again without
    triggering a memory problem (make, make -j2, etc without a sig11).
    Pull out, say, half of modules and try again.


    > however given that memtest can run for hours without recording
    > an error makes that unlikely. The system is a Supermicro case-
    > motherboard combination so it should have good cooling and power,
    > especially since we haven't added any hardware inside the box (which
    > supports 8 drives). There are lots of fans, but no baffles to make
    > sure the memory gets circulation. If the failures occurred other than
    > at the start of operation, I would definitely suspect overheating.
    >
    > I still vote for BIOS problems or kernel bugs as most likely.
    >


    Then another kernel should just work.
    I still believe a ram problem. Keep us informed.

  7. Re: Supermicro X7DBE+ and 32 GB memory

    >
    > 32G means 32 memory modules. I doubt much there is enough current for
    > driving them all, as such configurations are seldom tested. Other signs
    > point towards chipset. Some AMD boards won't take more than 3.5G with an
    > nforce chipset.


    This is a little off topic. However your statement about AMD boards with
    NForce chipsets is wrong. There is a BIOS option called Memory Hole
    Remapping which fixes the problem. If Memory Hole Remapping is off you
    can only see 3.5G of RAM because the IOMMU was mapped into the upper part
    of the 4G memory space. However with Memory Hole Remapping On all 4G of
    RAM are visible. You need a current kernel to be able to handle memory
    hole remapping, there was a problem with either 2.6.13 or 2.6.14 (I don't
    remember which it was) but as long as you are using 2.6.15 or later you
    can handle the Memory Hole Remapping. I don't know if this is something
    that's automatically handled now. My Nforce4 board is a couple of years
    old and it required that you enable Memory Hole Remapping in the BIOS. My
    most recent system is Core2 based and it has no such option in the BIOS,
    it just works, so it's possible that the BIOS writers are doing the
    remapping automatically now. The memory hole is an artifact going back to
    the days when 32 bit addressing was introduced. The IO space was mapped
    to the top of the 32 bit address space. Back then you were lucky to have
    8M of RAM so who cared if the upper 512M of the 4G space was used for IO.
    It became a problem when 1G DIMMs were introduced and 4G of RAM became
    possible. BTW WinXP Pro can't handle more than 3.5G because MS use the 32
    bit address mapping mode in Pro and reserves the 36 bit map for their
    Server products (I assume that this is fixed in Vista which uses 64 bit
    addressing). Every major 32 bit Linux distro has always included
    Enterprise kernels which use the extended address map so this was never a
    Linux problem.

  8. Re: Supermicro X7DBE+ and 32 GB memory

    On Fri, 02 Mar 2007 19:32:17 +0000, General Schvantzkoph wrote:

    >>
    >> 32G means 32 memory modules. I doubt much there is enough current for
    >> driving them all, as such configurations are seldom tested. Other signs
    >> point towards chipset. Some AMD boards won't take more than 3.5G with an
    >> nforce chipset.

    >
    > This is a little off topic. However your statement about AMD boards with
    > NForce chipsets is wrong.

    I've just cited some manuals, for which no more than 3.5G was allowed.
    Alas I don't recall which one, (not that it matters at this point).
    The problem was independent from the os and the bios.

  9. Re: Supermicro X7DBE+ and 32 GB memory

    On Fri, 02 Mar 2007 19:50:05 +0000, alfa wrote:

    > On Fri, 02 Mar 2007 19:32:17 +0000, General Schvantzkoph wrote:
    >
    >
    >>> 32G means 32 memory modules. I doubt much there is enough current for
    >>> driving them all, as such configurations are seldom tested. Other
    >>> signs point towards chipset. Some AMD boards won't take more than 3.5G
    >>> with an nforce chipset.

    >>
    >> This is a little off topic. However your statement about AMD boards
    >> with NForce chipsets is wrong.

    > I've just cited some manuals, for which no more than 3.5G was allowed.
    > Alas I don't recall which one, (not that it matters at this point). The
    > problem was independent from the os and the bios.


    I'm guessing those manuals were written before the fix was put into the
    BIOS. Since most motherboard makers have never heard of Linux (SuperMicro
    is an exception, but they are in the server space where Linux has a very
    large market share), they wouldn't have needed to update the manuals
    because their XP users can't take advantage of more that 3.5G anyway.
    Everybody buys their BIOS from the same two sources so I'm sure that
    every NForce motherboard that has an updateable BIOS can handle 4G now.

  10. Re: Supermicro X7DBE+ and 32 GB memory

    On Fri, 02 Mar 2007 20:23:02 +0000, General Schvantzkoph wrote:


    >>>> 32G means 32 memory modules. I doubt much there is enough current for
    >>>> driving them all, as such configurations are seldom tested. Other
    >>>> signs point towards chipset. Some AMD boards won't take more than 3.5G
    >>>> with an nforce chipset.
    >>>
    >>> This is a little off topic. However your statement about AMD boards
    >>> with NForce chipsets is wrong.

    >> I've just cited some manuals, for which no more than 3.5G was allowed.
    >> Alas I don't recall which one, (not that it matters at this point). The
    >> problem was independent from the os and the bios.

    >
    > I'm guessing those manuals were written before the fix was put into the
    > BIOS.

    As far as I could remember it was an electrical problem.

  11. Re: Supermicro X7DBE+ and 32 GB memory

    On Fri, 02 Mar 2007 22:47:34 +0000, alfa wrote:

    > On Fri, 02 Mar 2007 20:23:02 +0000, General Schvantzkoph wrote:
    >
    >
    >>>>> 32G means 32 memory modules. I doubt much there is enough current
    >>>>> for driving them all, as such configurations are seldom tested.
    >>>>> Other signs point towards chipset. Some AMD boards won't take more
    >>>>> than 3.5G with an nforce chipset.
    >>>>
    >>>> This is a little off topic. However your statement about AMD boards
    >>>> with NForce chipsets is wrong.
    >>> I've just cited some manuals, for which no more than 3.5G was allowed.
    >>> Alas I don't recall which one, (not that it matters at this point).
    >>> The problem was independent from the os and the bios.

    >>
    >> I'm guessing those manuals were written before the fix was put into the
    >> BIOS.

    > As far as I could remember it was an electrical problem.


    There could have been an issue with supporting four double sided DIMMs,
    which might have limited you to 3G, but there isn't anything that could
    limit you to 3.5G. The memory controller is built into the CPU so there
    is nothing the board manufacturer could do to screw it up. You do have to
    set the maps correctly but that's a software issue not hardware.

  12. Re: Supermicro X7DBE+ and 32 GB memory

    alfa writes:
    >On Fri, 02 Mar 2007 09:26:35 -0800, feenberg wrote:

    ....
    >32G means 32 memory modules.


    No, 16x2GB (or 8x4GB, but that is rather theoretical); the board only
    has 16 DIMM slots.

    > I doubt much there is enough current
    >for driving them all, as such configurations are seldom tested.


    If you mean signal drive power, that is not a problem with the
    FB-DIMMs used in this board, because the chipset only talks to the
    first DIMM in the channel (and the first DIMM talks to the second one
    etc.)

    >> We also thought about heat - the memory sticks quickly get too hot to
    >> touch,

    >
    >No, no no. That's very bad, it should not happen even under heavy
    >overclocking.


    These are FB-DIMMs. They do get very hot even if not broken. Most of
    the heat comes from the interface chip that propagates the signals
    between the DIMMs and to and from the RAM chips.

    - anton
    --
    M. Anton Ertl Some things have to be seen to be believed
    anton@mips.complang.tuwien.ac.at Most things have to be believed to be seen
    http://www.complang.tuwien.ac.at/anton/home.html

  13. Re: Supermicro X7DBE+ and 32 GB memory

    General Schvantzkoph writes:
    >On Fri, 02 Mar 2007 19:50:05 +0000, alfa wrote:
    >
    >> On Fri, 02 Mar 2007 19:32:17 +0000, General Schvantzkoph wrote:
    >>>> Some AMD boards won't take more than 3.5G
    >>>> with an nforce chipset.

    ....
    >I'm sure that
    >every NForce motherboard that has an updateable BIOS can handle 4G now.


    Just to support this with some data, here's what free outputs on two
    of our machines:

    Tyan S2865AG2NRF Tomcat K8E, nForce 4:
    total used free shared buffers cached
    Mem: 4045276 3817508 227768 0 514688 2079620
    ....

    Tyan S2892 Thunder K8SE, nForce 4 professional 2200:
    total used free shared buffers cached
    Mem: 8176628 1473744 6702884 0 250256 820040
    ....

    - anton
    --
    M. Anton Ertl Some things have to be seen to be believed
    anton@mips.complang.tuwien.ac.at Most things have to be believed to be seen
    http://www.complang.tuwien.ac.at/anton/home.html

  14. Re: Supermicro X7DBE+ and 32 GB memory

    On Fri, 02 Mar 2007 18:28:45 +0000, Anton Ertl wrote:

    > alfa writes:
    >>On Fri, 02 Mar 2007 09:26:35 -0800, feenberg wrote:

    > ...
    >>32G means 32 memory modules.

    >
    > No, 16x2GB (or 8x4GB, but that is rather theoretical); the board only
    > has 16 DIMM slots.
    >
    >> I doubt much there is enough current
    >>for driving them all, as such configurations are seldom tested.

    >
    > If you mean signal drive power, that is not a problem with the
    > FB-DIMMs used in this board, because the chipset only talks to the
    > first DIMM in the channel (and the first DIMM talks to the second one
    > etc.)
    >

    I've just seen the webpage of the mb. It needs 2G buffered
    memories, which allow work temperatures up to 95 celsius. So the crazy
    temperature memory has seems not to be a problem. I'd still pull out half
    of it and test the box. And pass the parameter mem= at boot time.

    Now, the board has also an intel 5000 chipset, which is not very diffuse.
    In fact google returns the same error on this chipset with less memory
    onboard. Perhaps the two facts are correlated.





  15. Re: Supermicro X7DBE+ and 32 GB memory

    alfa writes:
    >I've just seen the webpage of the mb. It needs 2G buffered
    >memories, which allow work temperatures up to 95 celsius. So the crazy
    >temperature memory has seems not to be a problem. I'd still pull out half
    >of it and test the box.


    The OP obviously has, as he reports it working with 24GB.

    >Now, the board has also an intel 5000 chipset, which is not very diffuse.
    >In fact google returns the same error on this chipset with less memory
    >onboard. Perhaps the two facts are correlated.


    You lost me here. Anyway, both the OP and we have that board (and
    thus the chipset) working with 24GB. The current uptime of our system
    is 51 days, and we have no problem with it.

    - anton
    --
    M. Anton Ertl Some things have to be seen to be believed
    anton@mips.complang.tuwien.ac.at Most things have to be believed to be seen
    http://www.complang.tuwien.ac.at/anton/home.html

  16. Re: Supermicro X7DBE+ and 32 GB memory

    On Sat, 03 Mar 2007 17:07:33 +0000, Anton Ertl wrote:

    > alfa writes:
    >>I've just seen the webpage of the mb. It needs 2G buffered
    >>memories, which allow work temperatures up to 95 celsius. So the crazy
    >>temperature memory has seems not to be a problem. I'd still pull out half
    >>of it and test the box.

    >
    > The OP obviously has, as he reports it working with 24GB.

    Which one, how are they populated? First slots, last slots, the ones in
    the middle? DOes it depend upon the slot number or the memory module
    itself?
    >
    >>Now, the board has also an intel 5000 chipset, which is not very
    >>diffuse. In fact google returns the same error on this chipset with less
    >>memory onboard. Perhaps the two facts are correlated.

    >
    > You lost me here. Anyway, both the OP and we have that board (and thus
    > the chipset) working with 24GB. The current uptime of our system is 51
    > days, and we have no problem with it.
    >
    > - anton

    This board has been built with 32 bit addressing in mind. Perhaps it has
    some strange issues because of that. The Dwarf2 unwinder message points in
    this direction.

  17. Re: Supermicro X7DBE+ and 32 GB memory

    On Mar 1, 7:43 pm, feenb...@gmail.com wrote:
    > We recently purchased aSupermicro X7DBE+ based system with 32
    > gigabytes of memory, however it won't boot FC 6 (or any other 64 bit
    > distribution we have at hand) if more than 24 GB is installed. The


    Thanks for all your comments. The vendor came back with this
    explanation:

    >We did a FC5 install from a DVD with kernel 2.6.15 and it
    >works fine with all 32GB. My Linux guys are telling me the reasoning for
    >this is the 2.6.18 kernel must be compiled differently or with different
    >options than the 2.6.15 kernel and is causing this error.


    I have no idea if that is likely, but I can't get FC5 to work either.
    Even if I could it doesn't have suitable drivers for the onboard
    controllers so I would have to patch together some hybrid of FC6 and
    FC5 which I don't the kernel knowledge to do. We will try 2.6.20 but
    if that fails I think we will just give up and restrict ourselves to
    24 GB. It is disappointing when such an expensive server MB fails to
    live up to its promises.

    Daniel Feenberg
    feenberg isat nber dotte org


  18. Re: Supermicro X7DBE+ and 32 GB memory

    feenberg@gmail.com writes:
    >The vendor came back with this
    >explanation:
    >
    >>We did a FC5 install from a DVD with kernel 2.6.15 and it
    >>works fine with all 32GB. My Linux guys are telling me the reasoning for
    >>this is the 2.6.18 kernel must be compiled differently or with different
    >>options than the 2.6.15 kernel and is causing this error.

    >
    >I have no idea if that is likely, but I can't get FC5 to work either.
    >Even if I could it doesn't have suitable drivers for the onboard
    >controllers


    We originally used the Debian Etch kernel 2.6.17-2 kernel, and it
    worked with that board. So if the change is really in 2.6.18, this
    might be an option for you. I think this kernel is no longer current
    in Etch, so I have put our copy of it on
    .

    - anton
    --
    M. Anton Ertl Some things have to be seen to be believed
    anton@mips.complang.tuwien.ac.at Most things have to be believed to be seen
    http://www.complang.tuwien.ac.at/anton/home.html

+ Reply to Thread