IOPS benchmarks - looking for opinions - Storage

This is a discussion on IOPS benchmarks - looking for opinions - Storage ; Hi everyone, I recently read about the heated contest between two Storage vendors regarding some IOPS tests one did with the other's gear. The latter's claim is that the IOPS benchmark is biased and irrelevant, and therefore they don't run ...

+ Reply to Thread
Results 1 to 7 of 7

Thread: IOPS benchmarks - looking for opinions

  1. IOPS benchmarks - looking for opinions

    Hi everyone,

    I recently read about the heated contest between two Storage vendors
    regarding some IOPS tests one did with the other's gear. The latter's claim
    is that the IOPS benchmark is biased and irrelevant, and therefore they
    don't run it with their own products. Not being neither a HPC or a high IO
    expert, I can't really decide whether or how relevant the claims are -
    whilst I found some arguments compelling.

    Being curious, I'll be interested in reading what some of the knowledgeable
    minds haunting this group would have to say regarding the IOPS benchmark
    validity and how relevant they are to the "real world" - whether or not they
    are a good proxy to assess how a given piece of hardware is able to sustain
    an heavy IO load.

    I don't want to discuss the validation process used in by one of the vendor,
    I have my own opinion on this matter, and I'd rather prefer to avoid
    replicating the heated exchanges I've read in the respective blogs of those
    two vendors :-)

    Thanks in advance.

    Fx

    --
    "Ah, the beauty of OSS. Hundreds of volunteers worldwide volunteering
    their time inventing and implementing new, exciting ways for software
    to suck." -- Toni L. in alt.sysadmin.recovery

  2. Re: IOPS benchmarks - looking for opinions

    fx [François-Xavier Peretmere] wrote:
    > Hi everyone,
    >
    > I recently read about the heated contest between two Storage vendors
    > regarding some IOPS tests one did with the other's gear. The latter's
    > claim is that the IOPS benchmark is biased and irrelevant, and therefore
    > they don't run it with their own products. Not being neither a HPC or a
    > high IO expert, I can't really decide whether or how relevant the claims
    > are - whilst I found some arguments compelling.
    >
    > Being curious, I'll be interested in reading what some of the
    > knowledgeable minds haunting this group would have to say regarding the
    > IOPS benchmark validity and how relevant they are to the "real world" -
    > whether or not they are a good proxy to assess how a given piece of
    > hardware is able to sustain an heavy IO load.
    >
    > I don't want to discuss the validation process used in by one of the
    > vendor, I have my own opinion on this matter, and I'd rather prefer to
    > avoid replicating the heated exchanges I've read in the respective blogs
    > of those two vendors :-)


    IOPs are much like having a speedometer that goes to 200 miles per hour,
    but an engine that may only stay put together at speeds up to 65 miles
    per hour.

    Put a set of file systems or luns on half of the storage pool on a set
    of arrays. Then benchmark. Then put two file systems or luns on the
    same set of arrays such that you fill the arrays. Crank up benchmarks
    on both. Some arrays will do very well, some won't. What have you
    demonstrated as far as your actual application performance? Very little
    if nothing at all.

    There are better tools, most are application specific and some of those
    can be coaxed into producing results predictive of performance on that
    application.

  3. Re: IOPS benchmarks - looking for opinions


    ""fx [François-Xavier Peretmere]"" wrote in message
    news:47ec3902@ac-versailles.fr...
    > Hi everyone,
    >
    > I recently read about the heated contest between two Storage vendors
    > regarding some IOPS tests one did with the other's gear. The latter's
    > claim is that the IOPS benchmark is biased and irrelevant, and therefore
    > they don't run it with their own products. Not being neither a HPC or a
    > high IO expert, I can't really decide whether or how relevant the claims
    > are - whilst I found some arguments compelling.
    >
    > Being curious, I'll be interested in reading what some of the
    > knowledgeable minds haunting this group would have to say regarding the
    > IOPS benchmark validity and how relevant they are to the "real world" -
    > whether or not they are a good proxy to assess how a given piece of
    > hardware is able to sustain an heavy IO load.


    IOPS are a valid test. All IOPS calcuations (atleast the ones advertised by
    a vendor) are performed using 512B transfer sizes which is
    the typical sector size used by most block SCSI devices. It gives you an
    indication of how well the vendors HBA driver and firmware perform.
    IOPS are also dependent on the setup because this value is not dependent on
    just the HBA driver and firmware, but is dependent on the system itself.

    Da Mooj
    >
    > I don't want to discuss the validation process used in by one of the
    > vendor, I have my own opinion on this matter, and I'd rather prefer to
    > avoid replicating the heated exchanges I've read in the respective blogs
    > of those two vendors :-)
    >
    > Thanks in advance.
    >
    > Fx
    >
    > --
    > "Ah, the beauty of OSS. Hundreds of volunteers worldwide volunteering
    > their time inventing and implementing new, exciting ways for software
    > to suck." -- Toni L. in alt.sysadmin.recovery




  4. Re: IOPS benchmarks - looking for opinions

    moojit wrote:
    > ""fx [François-Xavier Peretmere]"" wrote in message
    > news:47ec3902@ac-versailles.fr...
    >> Hi everyone,
    >>
    >> I recently read about the heated contest between two Storage vendors
    >> regarding some IOPS tests one did with the other's gear. The latter's
    >> claim is that the IOPS benchmark is biased and irrelevant, and therefore
    >> they don't run it with their own products. Not being neither a HPC or a
    >> high IO expert, I can't really decide whether or how relevant the claims
    >> are - whilst I found some arguments compelling.
    >>
    >> Being curious, I'll be interested in reading what some of the
    >> knowledgeable minds haunting this group would have to say regarding the
    >> IOPS benchmark validity and how relevant they are to the "real world" -
    >> whether or not they are a good proxy to assess how a given piece of
    >> hardware is able to sustain an heavy IO load.

    >
    > IOPS are a valid test. All IOPS calcuations (atleast the ones advertised by
    > a vendor) are performed using 512B transfer sizes which is
    > the typical sector size used by most block SCSI devices. It gives you an
    > indication of how well the vendors HBA driver and firmware perform.
    > IOPS are also dependent on the setup because this value is not dependent on
    > just the HBA driver and firmware, but is dependent on the system itself.
    >


    Although 512 is the scsi size, it isnt the size used for IOPS
    calculations. Nor frankly is it even relevant for arrays that use a
    much larger transfer size and still have the same IOP. Some arrays may
    have a lower IOP count for 512 byte transfers, or not.

    Nor is 512 byte transfers overly useful for predicting application
    performance with apps that regularly use multi-megabyte transfers. e.g.
    databases.


  5. Re: IOPS benchmarks - looking for opinions

    Lon wrote:
    > moojit wrote:
    >> ""fx [Fran?ois-Xavier Peretmere]"" wrote in message
    >> news:47ec3902@ac-versailles.fr...
    >>> Hi everyone,
    >>>
    >>> I recently read about the heated contest between two Storage vendors
    >>> regarding some IOPS tests one did with the other's gear. The latter's
    >>> claim is that the IOPS benchmark is biased and irrelevant, and therefore
    >>> they don't run it with their own products. Not being neither a HPC or a
    >>> high IO expert, I can't really decide whether or how relevant the claims
    >>> are - whilst I found some arguments compelling.
    >>>
    >>> Being curious, I'll be interested in reading what some of the
    >>> knowledgeable minds haunting this group would have to say regarding the
    >>> IOPS benchmark validity and how relevant they are to the "real world" -
    >>> whether or not they are a good proxy to assess how a given piece of
    >>> hardware is able to sustain an heavy IO load.

    >>
    >> IOPS are a valid test. All IOPS calcuations (atleast the ones advertised by
    >> a vendor) are performed using 512B transfer sizes which is
    >> the typical sector size used by most block SCSI devices. It gives you an
    >> indication of how well the vendors HBA driver and firmware perform.
    >> IOPS are also dependent on the setup because this value is not dependent on
    >> just the HBA driver and firmware, but is dependent on the system itself.
    >>

    >
    > Although 512 is the scsi size, it isnt the size used for IOPS
    > calculations. Nor frankly is it even relevant for arrays that use a
    > much larger transfer size and still have the same IOP. Some arrays may
    > have a lower IOP count for 512 byte transfers, or not.


    small tranfers are pretty taxing and a good way to see how a system
    performs. Reading and writing large files and getting decent speeds isn't
    hard.

    > Nor is 512 byte transfers overly useful for predicting application
    > performance with apps that regularly use multi-megabyte transfers. e.g.
    > databases.


    Not everything is a database with tranfer to and from the storage unit
    being nearly ideal in size.

    weak performance for small tranfers can mean horrible backup and restore
    times as well.

    even untarring a bunch of small files is tedious on a crappy array.


  6. Re: IOPS benchmarks - looking for opinions

    fx [François-Xavier Peretmere] wrote:
    > Hi everyone,
    >
    > I recently read about the heated contest between two Storage vendors
    > regarding some IOPS tests one did with the other's gear. The latter's
    > claim is that the IOPS benchmark is biased and irrelevant, and therefore
    > they don't run it with their own products. Not being neither a HPC or a
    > high IO expert, I can't really decide whether or how relevant the claims
    > are - whilst I found some arguments compelling.
    >
    > Being curious, I'll be interested in reading what some of the
    > knowledgeable minds haunting this group would have to say regarding the
    > IOPS benchmark validity and how relevant they are to the "real world" -
    > whether or not they are a good proxy to assess how a given piece of
    > hardware is able to sustain an heavy IO load.
    >
    > I don't want to discuss the validation process used in by one of the
    > vendor, I have my own opinion on this matter, and I'd rather prefer to
    > avoid replicating the heated exchanges I've read in the respective blogs
    > of those two vendors :-)
    >


    For the record, I assume you are talking about the Netapp/EMC SPC-1
    results that Netapp "kindly" published on EMC's behalf. This is much
    more complex than simply running 512KB IoMeter type loads.

    --
    Nik Simpson

  7. Re: IOPS benchmarks - looking for opinions

    On Fri, 28 Mar 2008 12:58:16 -0500, nik Simpson
    wrote:

    >fx [François-Xavier Peretmere] wrote:
    >> Hi everyone,
    >>
    >> I recently read about the heated contest between two Storage vendors
    >> regarding some IOPS tests one did with the other's gear. The latter's
    >> claim is that the IOPS benchmark is biased and irrelevant, and therefore
    >> they don't run it with their own products. Not being neither a HPC or a
    >> high IO expert, I can't really decide whether or how relevant the claims
    >> are - whilst I found some arguments compelling.
    >>
    >> Being curious, I'll be interested in reading what some of the
    >> knowledgeable minds haunting this group would have to say regarding the
    >> IOPS benchmark validity and how relevant they are to the "real world" -
    >> whether or not they are a good proxy to assess how a given piece of
    >> hardware is able to sustain an heavy IO load.
    >>
    >> I don't want to discuss the validation process used in by one of the
    >> vendor, I have my own opinion on this matter, and I'd rather prefer to
    >> avoid replicating the heated exchanges I've read in the respective blogs
    >> of those two vendors :-)
    >>

    >
    >For the record, I assume you are talking about the Netapp/EMC SPC-1
    >results that Netapp "kindly" published on EMC's behalf. This is much
    >more complex than simply running 512KB IoMeter type loads.


    I thought Dave Hitz explained the situation very well and had good
    points. But I hate EMC so my bias is unfettered.

    But, to the point. Synthetic tests are just that, synthetic. They
    can give you great insight into worst case or specific case IO
    scenario's. Some of those may or may not apply to your workload.

    Real World is so diverse that there's no Real World validation other
    than each users own application.

    Oracle data warehousing v. OLTP v. EDA grid apps v. web server v.
    .......

    You get the picture. The truth of it all is that you HAVE to know
    what your target is, or at least what your current IO is. You can't
    just say "build me a house" and expect to be happy with the outcome.
    There are details involved.

    ~F

+ Reply to Thread