12 Veliciraptors again w/x4 card (~1gbyte/sec aggregate read)! - Kernel

This is a discussion on 12 Veliciraptors again w/x4 card (~1gbyte/sec aggregate read)! - Kernel ; Each PCI-e x1 card has 1 veliciraptor on it now. Got an x4 card wit 4 sata ports: Not quite the > 1 gbyte/sec I was hoping for in regards to the reads but pretty close! (For my RAID5) Previously ...

+ Reply to Thread
Results 1 to 8 of 8

Thread: 12 Veliciraptors again w/x4 card (~1gbyte/sec aggregate read)!

  1. 12 Veliciraptors again w/x4 card (~1gbyte/sec aggregate read)!

    Each PCI-e x1 card has 1 veliciraptor on it now.
    Got an x4 card wit 4 sata ports:

    Not quite the > 1 gbyte/sec I was hoping for in regards to the reads
    but pretty close!

    (For my RAID5)
    Previously my write was limited to 400-420MiB/s now I see an additional
    120-125 MiB/s increase!

    jpiszcz@p34:/x/f$ dd if=/dev/zero of=bigfile bs=1M count=10240
    10240+0 records in
    10240+0 records out
    10737418240 bytes (11 GB) copied, 20.7054 s, 519 MB/s
    jpiszcz@p34:/x/f$ sync
    jpiszcz@p34:/x/f$ dd if=/dev/zero of=bigfile.1 bs=1M count=10240
    10240+0 records in
    10240+0 records out
    10737418240 bytes (11 GB) copied, 20.4973 s, 524 MB/s
    jpiszcz@p34:/x/f$ sync
    jpiszcz@p34:/x/f$ dd if=bigfile of=/dev/null bs=1M count=10240
    10240+0 records in
    10240+0 records out
    10737418240 bytes (11 GB) copied, 11.3529 s, 946 MB/s
    jpiszcz@p34:/x/f$ sync
    jpiszcz@p34:/x/f$ dd if=bigfile.1 of=/dev/null bs=1M count=10240
    10240+0 records in
    10240+0 records out
    10737418240 bytes (11 GB) copied, 11.2635 s, 953 MB/s
    jpiszcz@p34:/x/f$

    --

    For all disks:

    Something I noticed is the x1 PCI-e cards are doing around 68MiB/s each
    for 3 of them where the x4 has no issue pumping out 100MiB/s+ without a
    problem, however keep in mind the bus is probably already taxed from the
    6 sata drives on the southbridge.

    vmstat output:

    procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

    1 VR
    0 1 160 45220 341772 6480468 0 0 122112 0 584 2082 1 7 73 20
    0 1 160 46592 455436 6362088 0 0 113664 0 495 1968 0 4 74 21
    2 VR
    1 1 160 45540 3027724 3720340 0 0 243216 0 1006 4030 0 9 74 17
    0 2 160 44988 3262220 3480648 0 0 234480 0 1008 4134 0 8 73 19
    3 VR
    1 2 160 44816 6600068 50476 0 0 330248 16 1342 4126 0 12 70 18
    0 3 160 45440 6599812 50264 0 0 316032 8 1296 3878 0 12 72 17
    4 VR
    0 4 160 44504 6602488 47644 0 0 495232 0 1992 6081 0 20 57 23
    1 3 160 45500 6602796 45980 0 0 483968 0 1915 6207 0 20 54 26
    5 VR
    1 5 160 43932 6602972 45304 0 0 606080 0 2375 6622 0 25 56 19
    1 4 160 45412 6601852 45160 0 0 618756 0 2431 6791 0 25 53 21
    6 VR
    0 6 160 45000 6602348 44512 0 0 683904 8 2746 7880 0 31 42 27
    0 6 160 45248 6602028 44460 0 0 705792 0 2754 7564 0 31 45 24
    7 VR
    2 6 160 46744 6599020 44688 0 0 748204 17 3042 9084 0 34 40 26
    3 6 160 46592 6598824 44372 0 0 747520 8 2975 9047 1 33 31 36
    8 VR
    2 7 160 46512 6598612 44580 0 0 761184 16 3089 9937 0 36 40 24
    2 7 160 44528 6600392 44360 0 0 759720 8 2993 9522 0 36 36 28
    9 VR
    2 8 160 47152 6596824 44572 0 0 767016 0 3075 9730 1 37 39 24
    2 7 160 46576 6597728 44688 0 0 771200 0 3032 9568 0 37 40 23
    10 VR
    0 10 160 45048 6598240 44428 0 0 889072 8 3599 11561 0 47 20 33
    2 10 160 45232 6598116 44772 0 0 890112 0 3495 11547 0 46 23 31
    11 VR
    4 8 160 45536 6594716 44600 0 0 996352 0 3947 12134 1 62 13 25
    2 9 160 45348 6594912 44096 0 0 1009152 0 3949 11949 0 63 10 28
    12 VR
    6 8 160 45092 6583136 47016 0 0 1063200 0 4187 12394 1 71 9 21
    3 11 160 47080 6578492 47588 0 0 1058412 0 4224 12547 1 72 8 20

    Just about 1 gigabyte per second total aggregate read for all drives on a
    965 chipset!

    Justin.
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  2. Re: 12 Veliciraptors again w/x4 card (1.1gbytes/sec aggregate read)!



    On Mon, 7 Jul 2008, Justin Piszcz wrote:

    >
    >
    > On Mon, 7 Jul 2008, Justin Piszcz wrote:
    >
    >> Each PCI-e x1 card has 1 veliciraptor on it now.
    >> Got an x4 card wit 4 sata ports:
    >>
    >> Not quite the > 1 gbyte/sec I was hoping for in regards to the reads
    >> but pretty close!

    >
    > Going to remove one of the drives from the x1 card and put it on the x4
    > card instead, then I will use all 4 SATA ports on the x4 and hopefully get
    > better bw.
    >


    Four drives on the x4 card, MAX bandwidth for every disk.

    p34:~# dd if=/dev/sdi of=/dev/null bs=1M &
    [1] 4720
    p34:~# dd if=/dev/sdj of=/dev/null bs=1M &
    [2] 4721
    p34:~# dd if=/dev/sdk of=/dev/null bs=1M &
    [3] 4722
    p34:~# dd if=/dev/sdl of=/dev/null bs=1M &
    [4] 4723
    p34:~#

    120MiB/s per each one!

    Re-running dd test with all 12 disks:

    1.1 gigabytes per second read!

    r b swpd free buff cache si so bi bo in cs us sy id wa
    1 0 120 59104 6632220 52228 0 0 0 40 168 517 0 0 100 0
    0 0 120 59104 6632220 52228 0 0 0 0 20 291 0 0 100 0
    3 10 120 43516 6635576 51924 0 0 1051776 62 4221 12301 1 70 11 19
    6 9 160 44420 6634720 51788 0 0 1117284 0 4435 12308 1 75 5 19
    6 9 160 47436 6631100 51676 0 0 1110300 0 4449 11438 1 76 3 20
    2 10 160 46740 6632048 51948 0 0 1137920 0 4447 12251 1 75 8 17
    9 7 160 45248 6632056 52004 0 8 1127940 45 4559 13259 1 74 9 17
    3 9 160 44152 6634780 49960 0 0 1132032 12 4471 12962 0 75 8 16
    4 9 160 44160 6634960 49380 0 0 1129216 8 4430 12545 0 76 7 16

    After:

    About the same for write:
    $ dd if=/dev/zero of=bigfile.1 bs=1M count=10240
    10240+0 records in
    10240+0 records out
    10737418240 bytes (11 GB) copied, 20.4056 s, 526 MB/s

    'nuff said for read
    $ dd if=bigfile.1 of=/dev/null bs=1M count=10240
    10240+0 records in
    10240+0 records out
    10737418240 bytes (11 GB) copied, 10.2841 s, 1.0 GB/s

    Justin.
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  3. Re: 12 Veliciraptors again w/x4 card (1.1gbytes/sec aggregate read)!



    On Mon, 7 Jul 2008, Justin Piszcz wrote:

    >
    >
    > On Mon, 7 Jul 2008, Justin Piszcz wrote:
    >
    >>
    >>
    >> On Mon, 7 Jul 2008, Justin Piszcz wrote:
    >>
    >>> Each PCI-e x1 card has 1 veliciraptor on it now.
    >>> Got an x4 card wit 4 sata ports:
    >>>
    >>> Not quite the > 1 gbyte/sec I was hoping for in regards to the reads
    >>> but pretty close!

    >>
    >> Going to remove one of the drives from the x1 card and put it on the x4
    >> card instead, then I will use all 4 SATA ports on the x4 and hopefully get
    >> better bw.
    >>

    >
    > Four drives on the x4 card, MAX bandwidth for every disk.
    >
    > p34:~# dd if=/dev/sdi of=/dev/null bs=1M &
    > [1] 4720
    > p34:~# dd if=/dev/sdj of=/dev/null bs=1M &
    > [2] 4721
    > p34:~# dd if=/dev/sdk of=/dev/null bs=1M &
    > [3] 4722
    > p34:~# dd if=/dev/sdl of=/dev/null bs=1M &
    > [4] 4723
    > p34:~#
    >
    > 120MiB/s per each one!
    >
    > Re-running dd test with all 12 disks:
    >
    > 1.1 gigabytes per second read!
    >


    bonnie++ benchmarks:

    http://home.comcast.net/~jpiszcz/200...s_with_x4.html

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  4. Re: 12 Veliciraptors again w/x4 card (~1gbyte/sec aggregate read)!

    Justin Piszcz wrote:
    > Each PCI-e x1 card has 1 veliciraptor on it now.
    > Got an x4 card wit 4 sata ports:


    Useful - which card?

    David
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  5. Re: 12 Veliciraptors again w/x4 card (~1gbyte/sec aggregate read)!



    On Tue, 8 Jul 2008, David Greaves wrote:

    > Justin Piszcz wrote:
    >> Each PCI-e x1 card has 1 veliciraptor on it now.
    >> Got an x4 card wit 4 sata ports:

    >
    > Useful - which card?
    >
    > David
    >
    >


    StarTech 4 Port PCI Express x4 SATA II Card Model PEXSATA24E

    Justin.


    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  6. Re: 12 Veliciraptors again w/x4 card (~1gbyte/sec aggregate read)!



    On Tue, 8 Jul 2008, Justin Piszcz wrote:

    >
    >
    > On Tue, 8 Jul 2008, David Greaves wrote:
    >
    >> Justin Piszcz wrote:
    >>> Each PCI-e x1 card has 1 veliciraptor on it now.
    >>> Got an x4 card wit 4 sata ports:

    >>
    >> Useful - which card?
    >>
    >> David
    >>
    >>

    >
    > StarTech 4 Port PCI Express x4 SATA II Card Model PEXSATA24E
    >
    > Justin.
    >
    >
    >

    Chipset: Marvell 88SX7042
    SATA Connectivity: Use four internal ports at the same time or two
    internal and two external ports

    Which is fully supported in the latest kernels (didn't try an old kernel):

    linux-2.6.25.10/drivers/ata/sata_mv.c

    /* Marvell 7042 support */
    { PCI_VDEVICE(MARVELL, 0x7042), chip_7042 },

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  7. Re: 12 Veliciraptors again w/x4 card (~1gbyte/sec aggregate read)!

    On Tue, Jul 08, 2008 at 04:39:34AM -0400, Justin Piszcz wrote:
    > On Tue, 8 Jul 2008, Justin Piszcz wrote:
    >> On Tue, 8 Jul 2008, David Greaves wrote:
    >>> Justin Piszcz wrote:
    >>>> Each PCI-e x1 card has 1 veliciraptor on it now.
    >>>> Got an x4 card wit 4 sata ports:
    >>> Useful - which card?

    >>
    >> StarTech 4 Port PCI Express x4 SATA II Card Model PEXSATA24E
    >>
    >>

    > Chipset: Marvell 88SX7042
    > SATA Connectivity: Use four internal ports at the same time or two internal
    > and two external ports
    >
    > Which is fully supported in the latest kernels (didn't try an old kernel):
    >
    > linux-2.6.25.10/drivers/ata/sata_mv.c
    >
    > /* Marvell 7042 support */
    > { PCI_VDEVICE(MARVELL, 0x7042), chip_7042 },


    How have that card and the array attached to it been doing since you
    originally posted this?

    Doesn't that HighPoint 23xx card also use the Marvell 88SX7042 chip?
    I remember seeing the threads about the HighPoint card silently
    corrupting data by silently writing over parts of the disk(s) with
    its own info. I'm guessing that's not a "feature" of the Marvell
    chip, but still... admittedly irrational fear here

    It would be nice if that Supermicro 8-port SATA card (AOC-SAT2-MV8)
    was available in PCIe... I wonder if it's possible to just graft
    multiple SiI 3132 controllers (2 SATA each) on a single PCIe card?

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  8. Re: 12 Veliciraptors again w/x4 card (~1gbyte/sec aggregate read)!



    On Thu, 31 Jul 2008, Matt Garman wrote:

    > On Tue, Jul 08, 2008 at 04:39:34AM -0400, Justin Piszcz wrote:
    >> On Tue, 8 Jul 2008, Justin Piszcz wrote:
    >>> On Tue, 8 Jul 2008, David Greaves wrote:
    >>>> Justin Piszcz wrote:
    >>>>> Each PCI-e x1 card has 1 veliciraptor on it now.
    >>>>> Got an x4 card wit 4 sata ports:
    >>>> Useful - which card?
    >>>
    >>> StarTech 4 Port PCI Express x4 SATA II Card Model PEXSATA24E
    >>>
    >>>

    >> Chipset: Marvell 88SX7042
    >> SATA Connectivity: Use four internal ports at the same time or two internal
    >> and two external ports
    >>
    >> Which is fully supported in the latest kernels (didn't try an old kernel):
    >>
    >> linux-2.6.25.10/drivers/ata/sata_mv.c
    >>
    >> /* Marvell 7042 support */
    >> { PCI_VDEVICE(MARVELL, 0x7042), chip_7042 },

    >
    > How have that card and the array attached to it been doing since you
    > originally posted this?

    No problems to report yet, working well, I wish I had multiple x4 or x16
    slots and use those instead of x1 cards (the mobo would need to support
    the I/O as well)..

    >
    > Doesn't that HighPoint 23xx card also use the Marvell 88SX7042 chip?

    I read remeber 'locked' 64kb read problem but not any corruption issue.

    > I remember seeing the threads about the HighPoint card silently
    > corrupting data by silently writing over parts of the disk(s) with

    Also I MD5SUM -c (verify) 100-150GiB of data daily or so to make sure it
    is OK, so far, no problems.

    > its own info. I'm guessing that's not a "feature" of the Marvell
    > chip, but still... admittedly irrational fear here

    Have not seen it yet.

    >
    > It would be nice if that Supermicro 8-port SATA card (AOC-SAT2-MV8)
    > was available in PCIe... I wonder if it's possible to just graft
    > multiple SiI 3132 controllers (2 SATA each) on a single PCIe card?

    Yes, I asked Supermicro about it and they never mailed me back about a
    PCIe version..

    That's a good question, I suppose if it was easy it would have already
    been done?

    Justin.
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

+ Reply to Thread