Re: Parallel sort -m ? - Unix

This is a discussion on Re: Parallel sort -m ? - Unix ; Matt C wrote: > I'm not sure that merging files A & B into X separately from C & D into Y > then going onto merging X and Y would be very efficient in disk > reads/writes. This will ...

+ Reply to Thread
Results 1 to 3 of 3

Thread: Re: Parallel sort -m ?

  1. Re: Parallel sort -m ?

    Matt C wrote:
    > I'm not sure that merging files A & B into X separately from C & D into Y
    > then going onto merging X and Y would be very efficient in disk
    > reads/writes. This will probably outway *any* amount of CPU usuage as you
    > are effectively reading the same files twice instead of once.


    Aye, but the subsequent reads could be cached. Placing the files in
    some tree-like structure and having parent sort -m's actually does speed
    things along considerably as long as you have an SMP box. A sort -m on
    named pipes (from `child' sort -m's) is CPU-bound, not IO, vs. a lot of
    IO on the data file sort -m's.

    It may not be efficient IO wise, but total wall clock time is less.

    --
    JQ
    http://xepoch.com

  2. Re: Parallel sort -m ?

    "JQ" wrote in message
    news:10hgijkq8ih420a@corp.supernews.com...
    > Matt C wrote:
    > > I'm not sure that merging files A & B into X separately from C & D into

    Y
    > > then going onto merging X and Y would be very efficient in disk
    > > reads/writes. This will probably outway *any* amount of CPU usuage as

    you
    > > are effectively reading the same files twice instead of once.

    >
    > Aye, but the subsequent reads could be cached. Placing the files in
    > some tree-like structure and having parent sort -m's actually does speed
    > things along considerably as long as you have an SMP box. A sort -m on
    > named pipes (from `child' sort -m's) is CPU-bound, not IO, vs. a lot of
    > IO on the data file sort -m's.
    >
    > It may not be efficient IO wise, but total wall clock time is less.
    >
    > --
    > JQ
    > http://xepoch.com


    I'm really surprised that a merge takes up much CPU!?



  3. Re: Parallel sort -m ?

    Matt C wrote:
    > I'm really surprised that a merge takes up much CPU!?


    Me too, but yep it does. Again, though, only on those being fed from
    named pipes, not those reading IO from auxillary storage.

    --
    JQ
    http://xepoch.com

+ Reply to Thread