Installing onto a >2TB drive - Hardware

This is a discussion on Installing onto a >2TB drive - Hardware ; A HP DL320S Prioliant server with P400 raid controller has recently arrived on my doormat, and I'm attempting to install on it. It has 5 (for now) 750GB drives in a hardware RAID 5 set up, giving a single logical ...

+ Reply to Thread
Results 1 to 13 of 13

Thread: Installing onto a >2TB drive

  1. Installing onto a >2TB drive

    A HP DL320S Prioliant server with P400 raid controller has recently
    arrived on my doormat, and I'm attempting to install on it. It has 5
    (for now) 750GB drives in a hardware RAID 5 set up, giving a single
    logical drive of 3TB (plan is to move to 12*750 RAID6 for 9TB of
    storage once things are worked out)

    I've tried a variety of ditrobutions, however grml 0.9 (based on
    debian, but with a modern kernel) appears the most promising. It boots
    from my pendrive with no problems, and can access and partition the
    entire drive via "parted /dev/cciss/c0d0". It even installs to hard
    drive.

    I was estatic. I partitioned up as following initially
    label: gpt
    50MB fat32 partition -- bootable
    9950MB swap
    40GB /
    25GB /var
    15GB /tmp
    (Rest to be LVMed later)

    Installed to the root partition, everything appeared fine.

    I then went arorund working with elilo, however attempts to install
    to /dev/cciss/c0d0p1, while appearing to succed, were fruitless. The
    boot fell past the HDD and onto network boot.

    I'm convinced that if I can get it booting, it will work swimmingly,
    as it does work when I boot from USB.

    Booting from USB with a root on the disk isn't an ideal solution,
    however it could be made to work.

    Has anyone attempted anything similar? Most elilo docs I find are to
    do with Macs. Am I barking up the wrong tree? Is the answer staring me
    in the face?


  2. Re: Installing onto a >2TB drive

    Paul Weaver wrote:
    > A HP DL320S Prioliant server with P400 raid controller has recently
    > arrived on my doormat, and I'm attempting to install on it. It has 5
    > (for now) 750GB drives in a hardware RAID 5 set up, giving a single
    > logical drive of 3TB (plan is to move to 12*750 RAID6 for 9TB of
    > storage once things are worked out)
    >


    Standard partition tables cannot handle greater than 2TB.

    > I've tried a variety of ditrobutions, however grml 0.9 (based on
    > debian, but with a modern kernel) appears the most promising. It boots
    > from my pendrive with no problems, and can access and partition the
    > entire drive via "parted /dev/cciss/c0d0". It even installs to hard
    > drive.


    You can break the array into multiple disks and then use LVM to
    place the physical volumes into a volume group and in turn into
    a 3TB logical volume.

    Alternatively, you can try to uses a different partition table
    format (???)... but I haven't tried this under Linux. Probably
    will cause more problems than you want.

    Ever run fsck on a multi-TB filesystem?? You may find yourself
    wishing your filesystems weren't so big someday.

  3. Re: Installing onto a >2TB drive

    On 2007-03-16, Paul Weaver wrote:
    > A HP DL320S Prioliant server with P400 raid controller has recently
    > arrived on my doormat, and I'm attempting to install on it. It has 5
    > (for now) 750GB drives in a hardware RAID 5 set up, giving a single
    > logical drive of 3TB (plan is to move to 12*750 RAID6 for 9TB of
    > storage once things are worked out)
    >
    > I've tried a variety of ditrobutions, however grml 0.9 (based on
    > debian, but with a modern kernel) appears the most promising. It boots
    > from my pendrive with no problems, and can access and partition the
    > entire drive via "parted /dev/cciss/c0d0". It even installs to hard
    > drive.


    You can't boot from a >2TB device. Period. On such a device you must use a
    gpt partition table (as you did), and no boot loader currently supports
    booting off of a gpt labeled device.

    --
    Joshua Baker-LePain
    Department of Biomedical Engineering
    Duke University

  4. Re: Installing onto a >2TB drive

    On Mar 17, 4:01 am, Chris Cox wrote:
    > Paul Weaver wrote:
    > > A HP DL320S Prioliant server with P400 raid controller has recently
    > > arrived on my doormat, and I'm attempting to install on it. It has 5
    > > (for now) 750GB drives in a hardware RAID 5 set up, giving a single
    > > logical drive of 3TB (plan is to move to 12*750 RAID6 for 9TB of
    > > storage once things are worked out)

    >
    > Standard partition tables cannot handle greater than 2TB.
    >
    > > I've tried a variety of ditrobutions, however grml 0.9 (based on
    > > debian, but with a modern kernel) appears the most promising. It boots
    > > from my pendrive with no problems, and can access and partition the
    > > entire drive via "parted /dev/cciss/c0d0". It even installs to hard
    > > drive.

    >
    > You can break the array into multiple disks and then use LVM to
    > place the physical volumes into a volume group and in turn into
    > a 3TB logical volume.


    Unfortunatly, as I understand it, that would reduce the beenfit of
    being able to use hardware raid 5 (or 6)

    > Alternatively, you can try to uses a different partition table
    > format (???)... but I haven't tried this under Linux. Probably
    > will cause more problems than you want.


    GPT, we'll all be using it in a couple of years -- 750GB drives are
    pretty old hat now, 2TB should be arround by Summer 08

    > Ever run fsck on a multi-TB filesystem?? You may find yourself
    > wishing your filesystems weren't so big someday.


    Plan was to partition the OS part into 40GB partitions or less, and
    the remaining tbytes into 200-500GB partitions, depending on
    performance and file systems.


  5. Re: Installing onto a >2TB drive

    On Mar 17, 11:59 am, Joshua Baker-LePain
    wrote:
    > On 2007-03-16, Paul Weaver wrote:
    > > > I've tried a variety of ditrobutions, however grml 0.9 (based on

    > > debian, but with a modern kernel) appears the most promising. It boots
    > > from my pendrive with no problems, and can access and partition the
    > > entire drive via "parted /dev/cciss/c0d0". It even installs to hard
    > > drive.

    >
    > You can't boot from a >2TB device. Period. On such a device you must use a
    > gpt partition table (as you did), and no boot loader currently supports
    > booting off of a gpt labeled device.


    That's what I was beginning to fear when 10PM on Friday night came and
    went

    In that case it looks like a USB key for booting. The question is, can
    I boot from a kernel loaded on the USB key and pass control to a root
    partition on the large volume? It's been a long time since tom's root
    and boot. Would I need to create an initrd on the key as well? I'm
    afraid that I have always avoided the need by compiling ext3 into the
    kernel and booting from that!

    I think that's preferable to the other options I can think of:
    1) A seperate system drive, however that would mean losing another
    drive for mirroring.
    2) 4096 byte disk sectors, which I'm even sure is possible
    3) Boot and root on the USB key loading the entire file system into a
    ramdisk on boot for performance (the machine has 8GB of memory, and is
    only going to be serving files via nfs/smb/http)


  6. Re: Installing onto a >2TB drive

    Paul Weaver staggered into the Black Sun and said:
    > Joshua Baker-LePain wrote:
    >> You can't boot from a >2TB device. Period. On such a device you
    >> must use a gpt partition table (as you did), and no boot loader
    >> currently supports booting off of a gpt labeled device.


    Hm. Seems like that'd require the GRUB and LILO people to get a move
    on, then. Real-mode assembly is a PITA, for sure, but gpt tables are
    only going to get more common.

    > In that case it looks like a USB key for booting. The question is, can
    > I boot from a kernel loaded on the USB key and pass control to a root
    > partition on the large volume?


    Of course. All you need on the boot disk is what's in /boot , after
    all. You could have / on a USB key as well, since if you have /var and
    /usr and /tmp partitions, / is <= 200M.

    > Would I need to create an initrd on the key as well?


    Nope. I'd say to avoid initrd entirely if the hardware involved is
    going to stay the same for > 6 months. Just build SCSI support, SCSI
    disk support, support for your SCSI card, and ext[23] filesystem support
    into the kernel.

    > 4096-byte disk sectors, which I'm [not] even sure is possible


    I doubt it. Too much stuff assumes disk sectors are 512 bytes. This is
    annoying because disk sizes and file sizes keep going up.

    > Boot and root on the USB key loading the entire file system into a
    > ramdisk on boot for performance (the machine has 8GB of memory, and is
    > only going to be serving files via nfs/smb/http)


    ....possible. That seems like a whole lot of RAM+disk; hope the machine
    has enough bandwidth available.

    --
    Love is a laser.
    DO NOT LOOK INTO LASER WITH REMAINING HEART.
    --Tori and Greg Andrews
    Matt G|There is no Darkness in Eternity/But only Light too dim for us to see

  7. Re: Installing onto a >2TB drive

    "Paul Weaver" writes:
    >In that case it looks like a USB key for booting. The question is, can
    >I boot from a kernel loaded on the USB key and pass control to a root
    >partition on the large volume?


    Sure, if the kernel knows about the disk controller, the partition
    table format, and the file system.

    > Would I need to create an initrd on the key as well?


    If you are using an initrd, I think that yes.

    Or you can compile all the parts needed for accessing the root
    partition into the kernel, then you don't need an initrd.

    >3) Boot and root on the USB key loading the entire file system into a
    >ramdisk on boot for performance (the machine has 8GB of memory, and is
    >only going to be serving files via nfs/smb/http)


    Why load it into a RAM disk? Boot and root for a system that only
    does these few things should easily fit into a 1GB stick (and if not,
    larger sticks are not that expensive, either). The system will cache
    as much as needed in RAM anyway.

    - anton
    --
    M. Anton Ertl Some things have to be seen to be believed
    anton@mips.complang.tuwien.ac.at Most things have to be believed to be seen
    http://www.complang.tuwien.ac.at/anton/home.html

  8. Re: Installing onto a >2TB drive

    Paul Weaver wrote:
    > On Mar 17, 4:01 am, Chris Cox wrote:
    >> Paul Weaver wrote:
    >>> A HP DL320S Prioliant server with P400 raid controller has recently
    >>> arrived on my doormat, and I'm attempting to install on it. It has 5
    >>> (for now) 750GB drives in a hardware RAID 5 set up, giving a single
    >>> logical drive of 3TB (plan is to move to 12*750 RAID6 for 9TB of
    >>> storage once things are worked out)

    >> Standard partition tables cannot handle greater than 2TB.
    >>
    >>> I've tried a variety of ditrobutions, however grml 0.9 (based on
    >>> debian, but with a modern kernel) appears the most promising. It boots
    >>> from my pendrive with no problems, and can access and partition the
    >>> entire drive via "parted /dev/cciss/c0d0". It even installs to hard
    >>> drive.

    >> You can break the array into multiple disks and then use LVM to
    >> place the physical volumes into a volume group and in turn into
    >> a 3TB logical volume.

    >
    > Unfortunatly, as I understand it, that would reduce the beenfit of
    > being able to use hardware raid 5 (or 6)


    Why? It does not defeat this in any way.

    >
    >> Alternatively, you can try to uses a different partition table
    >> format (???)... but I haven't tried this under Linux. Probably
    >> will cause more problems than you want.

    >
    > GPT, we'll all be using it in a couple of years -- 750GB drives are
    > pretty old hat now, 2TB should be arround by Summer 08
    >
    >> Ever run fsck on a multi-TB filesystem?? You may find yourself
    >> wishing your filesystems weren't so big someday.

    >
    > Plan was to partition the OS part into 40GB partitions or less, and
    > the remaining tbytes into 200-500GB partitions, depending on
    > performance and file systems.
    >


    There is NO problem with a 9TB drive... only a problem when
    creating a 9TB partition. So perhaps I misled you a bit
    (unintentionally). I meant to say that the problem is creating
    a PARTITION that is >2TB. Large disk... not a problem.

    I don't recommend partition (filesystem) sizes above 800G.... just
    takes WAY too long to fsck if there every is a serious problem.


  9. Re: Installing onto a >2TB drive

    On Mar 18, 7:06 pm, Chris Cox wrote:
    > Paul Weaver wrote:
    > > On Mar 17, 4:01 am, Chris Cox wrote:
    > >> Paul Weaver wrote:
    > >>> A HP DL320S Prioliant server with P400 raid controller has recently
    > >>> arrived on my doormat, and I'm attempting to install on it. It has 5
    > >>> (for now) 750GB drives in a hardware RAID 5 set up, giving a single
    > >>> logical drive of 3TB (plan is to move to 12*750 RAID6 for 9TB of
    > >>> storage once things are worked out)
    > >> Standard partition tables cannot handle greater than 2TB.

    >
    > >>> I've tried a variety of ditrobutions, however grml 0.9 (based on
    > >>> debian, but with a modern kernel) appears the most promising. It boots
    > >>> from my pendrive with no problems, and can access and partition the
    > >>> entire drive via "parted /dev/cciss/c0d0". It even installs to hard
    > >>> drive.
    > >> You can break the array into multiple disks and then use LVM to
    > >> place the physical volumes into a volume group and in turn into
    > >> a 3TB logical volume.

    >
    > > Unfortunatly, as I understand it, that would reduce the beenfit of
    > > being able to use hardware raid 5 (or 6)

    >
    > Why? It does not defeat this in any way.


    The controller creates a virtual volume of xTB, which is presented to
    linux as /dev/cciss/c0d0. You assign the physical disks installed, and
    select raid 0, raid 1, raid 5 with or without a spare, or raid 6
    (that's 2 parity drives). Linux is unaware of this.

    Now you could present as 12 750GB drivers to linux, and have it do
    raid in software, but that seems a waste of a perfectly good hardware
    raid controller.

    > >> Alternatively, you can try to uses a different partition table
    > >> format (???)... but I haven't tried this under Linux. Probably
    > >> will cause more problems than you want.

    >
    > > GPT, we'll all be using it in a couple of years -- 750GB drives are
    > > pretty old hat now, 2TB should be arround by Summer 08

    >
    > >> Ever run fsck on a multi-TB filesystem?? You may find yourself
    > >> wishing your filesystems weren't so big someday.

    >
    > > Plan was to partition the OS part into 40GB partitions or less, and
    > > the remaining tbytes into 200-500GB partitions, depending on
    > > performance and file systems.

    >
    > There is NO problem with a 9TB drive... only a problem when
    > creating a 9TB partition. So perhaps I misled you a bit
    > (unintentionally). I meant to say that the problem is creating
    > a PARTITION that is >2TB. Large disk... not a problem.
    >
    > I don't recommend partition (filesystem) sizes above 800G.... just
    > takes WAY too long to fsck if there every is a serious problem.



    I was going for arround 6 490GB partitions (jfs perhaps) (on a 3TB
    drive), and 60GB for root/var/tmp etc (ext3)

    However to do this with a >2TB drive you need to use GPT, not an msdos
    partition table. Booting from GPT appears unsupported, as for starters
    it doesn't seem to have a master boot record.


  10. Re: Installing onto a >2TB drive

    Dances With Crows wrote:
    > Paul Weaver staggered into the Black Sun and said:
    >> Joshua Baker-LePain wrote:
    >>> You can't boot from a >2TB device. Period. On such a device you
    >>> must use a gpt partition table (as you did), and no boot loader
    >>> currently supports booting off of a gpt labeled device.

    >
    > Hm. Seems like that'd require the GRUB and LILO people to get a move
    > on, then. Real-mode assembly is a PITA, for sure, but gpt tables are
    > only going to get more common.


    AFAIK, it's not a GRUB/LILO problem but a BIOS problem. Machines with
    EFI firmware (mainly Itanium-based systems) can boot from a GPT parition
    table using ELILO. (It was the EFI firmware specification that first
    introduced the GPT partition table structure, AIUI.) But if your BIOS
    doesn't understand the GPT layout, I don't see how GRUB or LILO could
    overcome that to boot from such a device.

  11. Re: Installing onto a >2TB drive

    Paul Weaver wrote:
    > On Mar 18, 7:06 pm, Chris Cox wrote:
    >> Paul Weaver wrote:
    >>> On Mar 17, 4:01 am, Chris Cox wrote:
    >>>> Paul Weaver wrote:
    >>>>> A HP DL320S Prioliant server with P400 raid controller has recently
    >>>>> arrived on my doormat, and I'm attempting to install on it. It has 5
    >>>>> (for now) 750GB drives in a hardware RAID 5 set up, giving a single
    >>>>> logical drive of 3TB (plan is to move to 12*750 RAID6 for 9TB of
    >>>>> storage once things are worked out)
    >>>> Standard partition tables cannot handle greater than 2TB.
    >>>>> I've tried a variety of ditrobutions, however grml 0.9 (based on
    >>>>> debian, but with a modern kernel) appears the most promising. It boots
    >>>>> from my pendrive with no problems, and can access and partition the
    >>>>> entire drive via "parted /dev/cciss/c0d0". It even installs to hard
    >>>>> drive.
    >>>> You can break the array into multiple disks and then use LVM to
    >>>> place the physical volumes into a volume group and in turn into
    >>>> a 3TB logical volume.
    >>> Unfortunatly, as I understand it, that would reduce the beenfit of
    >>> being able to use hardware raid 5 (or 6)

    >> Why? It does not defeat this in any way.

    >
    > The controller creates a virtual volume of xTB, which is presented to
    > linux as /dev/cciss/c0d0. You assign the physical disks installed, and
    > select raid 0, raid 1, raid 5 with or without a spare, or raid 6
    > (that's 2 parity drives). Linux is unaware of this.
    >
    > Now you could present as 12 750GB drivers to linux, and have it do
    > raid in software, but that seems a waste of a perfectly good hardware
    > raid controller.


    True... I guess we're just not understanding each other.

    You can expose multiple fully HW RAID'd drives via
    the SmartArray controller and use them as physical
    volumes in an LVM config and STILL have ONLY hw RAID.

    >
    >>>> Alternatively, you can try to uses a different partition table
    >>>> format (???)... but I haven't tried this under Linux. Probably
    >>>> will cause more problems than you want.
    >>> GPT, we'll all be using it in a couple of years -- 750GB drives are
    >>> pretty old hat now, 2TB should be arround by Summer 08
    >>>> Ever run fsck on a multi-TB filesystem?? You may find yourself
    >>>> wishing your filesystems weren't so big someday.
    >>> Plan was to partition the OS part into 40GB partitions or less, and
    >>> the remaining tbytes into 200-500GB partitions, depending on
    >>> performance and file systems.

    >> There is NO problem with a 9TB drive... only a problem when
    >> creating a 9TB partition. So perhaps I misled you a bit
    >> (unintentionally). I meant to say that the problem is creating
    >> a PARTITION that is >2TB. Large disk... not a problem.
    >>
    >> I don't recommend partition (filesystem) sizes above 800G.... just
    >> takes WAY too long to fsck if there every is a serious problem.

    >
    >
    > I was going for arround 6 490GB partitions (jfs perhaps) (on a 3TB
    > drive), and 60GB for root/var/tmp etc (ext3)
    >
    > However to do this with a >2TB drive you need to use GPT, not an msdos
    > partition table. Booting from GPT appears unsupported, as for starters
    > it doesn't seem to have a master boot record.
    >


    I believe there are other choices as well... but all with some
    issues.

    But with the partition sizes you're suggesting.. you should
    have no problem.

  12. Re: Installing onto a >2TB drive

    John-Paul Stewart staggered into the Black Sun and said:
    > Dances With Crows wrote:
    >> Joshua Baker-LePain wrote:
    >>> You can't boot from a >2TB device. Period. On such a device you
    >>> must use a gpt partition table (as you did), and no boot loader
    >>> currently supports booting off of a gpt labeled device.

    >> Hm. Seems like that'd require the GRUB and LILO people to get a move
    >> on, then. Real-mode assembly is a PITA, for sure, but gpt tables are
    >> only going to get more common.

    > AFAIK, it's not a GRUB/LILO problem but a BIOS problem. Machines with
    > EFI firmware (mainly Itanium-based systems) can boot from a GPT
    > parition table using ELILO. (It was the EFI firmware specification
    > that first introduced the GPT partition table structure, AIUI.)


    Interesting. I wonder whether this is insanely relevant for most users,
    since the number of amd64/Core Duo machines >> the number of Itanic
    machines. Or did they kludge up some sort of bug-for-bug compatible EFI
    workalike for amd64/Core Duo? (I haven't been paying much attention to
    all this; the largest RAID I have access to is only 600G and that
    machine boots from a 20G RAID-1.)

    > if your BIOS doesn't understand the GPT layout, I don't see how GRUB
    > or LILO could overcome that to boot from such a device.


    True dat. I didn't know that it was a BIOS thing, thanks.

    --
    There is a budding morrow in midnight
    There is a triple sight in blindness keen...
    --John Keats, "To Homer"
    Matt G|There is no Darkness in Eternity/But only Light too dim for us to see

  13. Re: Installing onto a >2TB drive

    Dances With Crows wrote:
    > John-Paul Stewart staggered into the Black Sun and said:
    >> Dances With Crows wrote:
    >>> Joshua Baker-LePain wrote:
    >>>> You can't boot from a >2TB device. Period. On such a device you
    >>>> must use a gpt partition table (as you did), and no boot loader
    >>>> currently supports booting off of a gpt labeled device.
    >>> Hm. Seems like that'd require the GRUB and LILO people to get a move
    >>> on, then. Real-mode assembly is a PITA, for sure, but gpt tables are
    >>> only going to get more common.

    >> AFAIK, it's not a GRUB/LILO problem but a BIOS problem. Machines with
    >> EFI firmware (mainly Itanium-based systems) can boot from a GPT
    >> parition table using ELILO. (It was the EFI firmware specification
    >> that first introduced the GPT partition table structure, AIUI.)

    >
    > Interesting. I wonder whether this is insanely relevant for most users,
    > since the number of amd64/Core Duo machines >> the number of Itanic
    > machines. Or did they kludge up some sort of bug-for-bug compatible EFI
    > workalike for amd64/Core Duo? (I haven't been paying much attention to
    > all this; the largest RAID I have access to is only 600G and that
    > machine boots from a 20G RAID-1.)


    It's not a huge problem. The problem exists for commercial Unix's
    as well... though I think Solaris switched to EFI GPT in version 10.

    Which is to say that 99% (or more) of all Solaris installations
    can't deal (well) with partitions over 2TB. In fact, it wasn't
    all that long ago when Solaris 8 couldn't see DISKS above 1TB.

    >
    >> if your BIOS doesn't understand the GPT layout, I don't see how GRUB
    >> or LILO could overcome that to boot from such a device.

    >
    > True dat. I didn't know that it was a BIOS thing, thanks.
    >


+ Reply to Thread