47,900 MP3's - How To Organize Hard Discs? - Storage

This is a discussion on 47,900 MP3's - How To Organize Hard Discs? - Storage ; I have a little over 310 GB of music in MP3/WMA and am wondering the best way to organize my hard discs to maximize computer speed. I have two new Maxtor 250 GB hard discs (7200 rpm), 4GB RAM, and ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 22

Thread: 47,900 MP3's - How To Organize Hard Discs?

  1. 47,900 MP3's - How To Organize Hard Discs?


    I have a little over 310 GB of music in MP3/WMA and am wondering the
    best way to organize my hard discs to maximize computer speed. I have
    two new Maxtor 250 GB hard discs (7200 rpm), 4GB RAM, and 3.2GHz dual
    processors.

    The two discs are currently divided into about ten 50GB partitions and
    the MP3's are distributed somewhat evenly between the partitions.

    Is this most efficient way to store these files or should I have less
    partitions with bigger capacity?

    At times my computer runs slow when accessing the whole list and I
    just want to make sure its not because of my file organization.

    Thanks!
    Matthew


  2. Re: 47,900 MP3's - How To Organize Hard Discs?

    Previously matthewhargrove@msn.com wrote:

    > I have a little over 310 GB of music in MP3/WMA and am wondering the
    > best way to organize my hard discs to maximize computer speed.


    What do you mean by "speed"?

    > I have
    > two new Maxtor 250 GB hard discs (7200 rpm), 4GB RAM, and 3.2GHz dual
    > processors.


    > The two discs are currently divided into about ten 50GB partitions and
    > the MP3's are distributed somewhat evenly between the partitions.


    > Is this most efficient way to store these files or should I have less
    > partitions with bigger capacity?


    It depends on what you want to optimize.

    Search speed would need something like separate subdirectories for
    each starting letter or use of a database system in the first place.

    Space would mean to store them in archives, e.g. .zip.

    > At times my computer runs slow when accessing the whole list and I
    > just want to make sure its not because of my file organization.


    That may be due to directory fragmentation. A simpe disk defrag
    may cure it.

    Arno

  3. Re: 47,900 MP3's - How To Organize Hard Discs?

    On 2007-09-10, matthewhargrove@msn.com wrote:

    > The two discs are currently divided into about ten 50GB partitions and
    > the MP3's are distributed somewhat evenly between the partitions.
    >
    > Is this most efficient way to store these files or should I have less
    > partitions with bigger capacity?
    >
    > At times my computer runs slow when accessing the whole list and I
    > just want to make sure its not because of my file organization.


    On what OS ? For Unix filesystems, the bigger the filesystem the
    farther away a directory entry will be from the inode on
    average. That's no good.

    Directory fragmentation will slow things down, as Arno pointed
    out. On the other hand, if you create subdirectories to fight
    that, fewer directories will be fragmented but there will be
    more of them so the total seek time may actually increase.

    There are good reasons for avoiding directories with large
    number of entries, though (CPU).

    --
    André Majorel
    (Counterfeit: wog@skiff.com aqif@baldwin.com)
    There is always someone somewhere who needs a good laugh.

  4. Re: 47,900 MP3's - How To Organize Hard Discs?

    matthewhargrove@msn.com wrote:

    >I have a little over 310 GB of music in MP3/WMA


    Steal much?


  5. Re: 47,900 MP3's - How To Organize Hard Discs?

    On Sep 10, 8:43 am, chrisv wrote:
    > matthewhargr...@msn.com wrote:
    > >I have a little over 310 GB of music in MP3/WMA

    >
    > Steal much?



    No Chris, I don't steal at all, but thanks for the implication. I
    have been collecting music for more than 30 years and have digitized
    most of my albums and tapes, the more than 3,000 CD's I have purchased
    since my first one in 1985, and have been a subscriber to eMusic since
    its launch. Digitizing all the music of has taken me years of
    meticulous cataloguing, expense, and tons and tons of time. Since I
    am not a computer guy I came here for assistance on how to best
    organize it. Your post was not very helpful.


  6. Re: 47,900 MP3's - How To Organize Hard Discs?

    Thanks to Arno and Andre for the posts.

    I have defragged and have an organized directory tree set-up (each
    letter of alphabet --> band name --> album --> song files), and am
    running Windows XP.

    I am trying to decide if I should have just a few large partitions or
    a bunch or smaller ones to organize the hard disks for speed of
    accessing the data. Thanks again.


  7. Re: 47,900 MP3's - How To Organize Hard Discs?

    On 10 Sep 2007 05:34:59 GMT, Arno Wagner put finger
    to keyboard and composed:

    >Previously matthewhargrove@msn.com wrote:
    >
    >> I have a little over 310 GB of music in MP3/WMA and am wondering the
    >> best way to organize my hard discs to maximize computer speed.

    >
    >What do you mean by "speed"?
    >
    >> I have
    >> two new Maxtor 250 GB hard discs (7200 rpm), 4GB RAM, and 3.2GHz dual
    >> processors.

    >
    >> The two discs are currently divided into about ten 50GB partitions and
    >> the MP3's are distributed somewhat evenly between the partitions.

    >
    >> Is this most efficient way to store these files or should I have less
    >> partitions with bigger capacity?

    >
    >It depends on what you want to optimize.
    >
    >Search speed would need something like separate subdirectories for
    >each starting letter or use of a database system in the first place.
    >
    >Space would mean to store them in archives, e.g. .zip.


    MP3s are already compressed. Winzip doesn't appear to shrink them by
    very much.

    - Franc Zabkar
    --
    Please remove one 'i' from my address when replying by email.

  8. Re: 47,900 MP3's - How To Organize Hard Discs?

    Previously Franc Zabkar wrote:
    > On 10 Sep 2007 05:34:59 GMT, Arno Wagner put finger
    > to keyboard and composed:


    >>Previously matthewhargrove@msn.com wrote:
    >>
    >>> I have a little over 310 GB of music in MP3/WMA and am wondering the
    >>> best way to organize my hard discs to maximize computer speed.

    >>
    >>What do you mean by "speed"?
    >>
    >>> I have
    >>> two new Maxtor 250 GB hard discs (7200 rpm), 4GB RAM, and 3.2GHz dual
    >>> processors.

    >>
    >>> The two discs are currently divided into about ten 50GB partitions and
    >>> the MP3's are distributed somewhat evenly between the partitions.

    >>
    >>> Is this most efficient way to store these files or should I have less
    >>> partitions with bigger capacity?

    >>
    >>It depends on what you want to optimize.
    >>
    >>Search speed would need something like separate subdirectories for
    >>each starting letter or use of a database system in the first place.
    >>
    >>Space would mean to store them in archives, e.g. .zip.


    > MP3s are already compressed. Winzip doesn't appear to shrink them by
    > very much.


    Indeed. But if you have small mp3s and a large cluster size, you get
    less filesystem loss. Does not matter that much, admittedly.

    Arno

  9. Re: 47,900 MP3's - How To Organize Hard Discs?


    wrote in message
    news:1189393219.094091.285450@50g2000hsm.googlegro ups.com...
    >
    > I have a little over 310 GB of music in MP3/WMA and am wondering the
    > best way to organize my hard discs to maximize computer speed. I have
    > two new Maxtor 250 GB hard discs (7200 rpm), 4GB RAM, and 3.2GHz dual
    > processors.
    >
    > The two discs are currently divided into about ten 50GB partitions and
    > the MP3's are distributed somewhat evenly between the partitions.
    >
    > Is this most efficient way to store these files or should I have less
    > partitions with bigger capacity?
    >
    > At times my computer runs slow when accessing the whole list and I
    > just want to make sure its not because of my file organization.
    >
    > Thanks!
    > Matthew
    >


    Hi Matthew

    I see youve already had good advice re managing the file distribution.

    FWIW I did wonder if you have any backup regime for all your music data
    files?

    Given the amount of time and effort it has taken it would be more than a bit
    upsetting if one of you drives expired (sorry to say that IMHO Maxtors dont
    have the best reputation for reliabilty).

    Given the amont of data I'd suggest using disk imaging software and another
    drive to save to - perhaps in a removeable caddy.

    Food for thought....

    Hope this is helpful.

    Ian







  10. Re: 47,900 MP3's - How To Organize Hard Discs?

    > FWIW I did wonder if you have any backup regime for all your music data
    > files?


    My backup regime is not very good... I have everything burned to DVD's
    using Nero's Back It Up. My initial backup took over 50 disks, and
    since then I just periodically burn a new disc with what I have
    transferred recently. Would love advice on how to back it up if an
    easier way exists. I have never tested the backup, so I have
    nightmares that I lose the data and it doesn't work, then I am back to
    transferring everything again.

    However, I am still unclear if a few larger partitions are better than
    a bunch of smaller ones.


  11. Re: 47,900 MP3's - How To Organize Hard Discs?

    On 11 Sep 2007 02:55:10 GMT, Arno Wagner put finger
    to keyboard and composed:

    >Previously Franc Zabkar wrote:
    >> On 10 Sep 2007 05:34:59 GMT, Arno Wagner put finger
    >> to keyboard and composed:

    >
    >>>Previously matthewhargrove@msn.com wrote:
    >>>
    >>>> I have a little over 310 GB of music in MP3/WMA and am wondering the
    >>>> best way to organize my hard discs to maximize computer speed.
    >>>
    >>>What do you mean by "speed"?
    >>>
    >>>> I have
    >>>> two new Maxtor 250 GB hard discs (7200 rpm), 4GB RAM, and 3.2GHz dual
    >>>> processors.
    >>>
    >>>> The two discs are currently divided into about ten 50GB partitions and
    >>>> the MP3's are distributed somewhat evenly between the partitions.
    >>>
    >>>> Is this most efficient way to store these files or should I have less
    >>>> partitions with bigger capacity?
    >>>
    >>>It depends on what you want to optimize.
    >>>
    >>>Search speed would need something like separate subdirectories for
    >>>each starting letter or use of a database system in the first place.
    >>>
    >>>Space would mean to store them in archives, e.g. .zip.

    >
    >> MP3s are already compressed. Winzip doesn't appear to shrink them by
    >> very much.

    >
    >Indeed. But if you have small mp3s and a large cluster size, you get
    >less filesystem loss. Does not matter that much, admittedly.
    >
    >Arno


    IME MP3 files seem to be about 3MB in size. Assuming a max cluster
    size of 32KB, the average slack space per file would be about
    32K/3MB/2 = 0.5%. As you said, that isn't much of a saving.

    - Franc Zabkar
    --
    Please remove one 'i' from my address when replying by email.

  12. Re: 47,900 MP3's - How To Organize Hard Discs?

    Andre Majorel wrote:

    > There are good reasons for avoiding directories with large
    > number of entries, though (CPU).
    >


    Depends on the filesystem. ReiserFS handles them very well, but I guess
    its future is in doubt.


  13. Re: 47,900 MP3's - How To Organize Hard Discs?

    matthewhargrove@msn.com wrote:
    > I have a little over 310 GB of music in MP3/WMA and am wondering the
    > best way to organize my hard discs to maximize computer speed. I have
    > two new Maxtor 250 GB hard discs (7200 rpm), 4GB RAM, and 3.2GHz dual
    > processors.
    >
    > The two discs are currently divided into about ten 50GB partitions and
    > the MP3's are distributed somewhat evenly between the partitions.
    >
    > Is this most efficient way to store these files or should I have less
    > partitions with bigger capacity?
    >
    > At times my computer runs slow when accessing the whole list and I
    > just want to make sure its not because of my file organization.
    >
    > Thanks!
    > Matthew
    >


    Over-partitioning. Why all the partitions? They don't do anything but
    add complication.

    What you should do is get two discs, each capable of holding the entire
    collection plus some room to grow. Use one for every day use, and
    another for backup. On each, put a single partition with the entire
    collection on it.

    A 500GB drive is only a few bucks more than a 250. For the time and
    expense in your collection, it would be a real shame not to have a
    backup. (And RAID is not a backup - you need protection against
    accidental deletion or corruption, too).

    If searching through the music is taking too long, just dump out a text
    file with the filename of every track, and search that instead. It's as
    easy as

    dir /s > all_music.txt

    then look through it with a text editor

  14. Re: 47,900 MP3's - How To Organize Hard Discs?

    On Tue, 11 Sep 2007 11:48:12 -0600, timeOday
    put finger to keyboard and composed:

    >If searching through the music is taking too long, just dump out a text
    >file with the filename of every track, and search that instead. It's as
    >easy as
    >
    >dir /s > all_music.txt
    >
    >then look through it with a text editor


    Here's another way:

    dir drive:\ /b /s /on | find /i "name_of_artist_or_album_or track"

    - Franc Zabkar
    --
    Please remove one 'i' from my address when replying by email.

  15. Re: 47,900 MP3's - How To Organize Hard Discs?

    Franc Zabkar wrote:
    > On Tue, 11 Sep 2007 11:48:12 -0600, timeOday
    > put finger to keyboard and composed:
    >
    >> If searching through the music is taking too long, just dump out a text
    >> file with the filename of every track, and search that instead. It's as
    >> easy as
    >>
    >> dir /s > all_music.txt
    >>
    >> then look through it with a text editor

    >
    > Here's another way:
    >
    > dir drive:\ /b /s /on | find /i "name_of_artist_or_album_or track"
    >
    > - Franc Zabkar


    Since it will take a while to traverse the whole directory tree, I would
    still recommend dumping the listing to a file only once (each time you
    add a new album), then searching that file with "find," or a text
    editor, etc.


  16. Re: 47,900 MP3's - How To Organize Hard Discs?

    On 2007-09-10, matthewhargrove@msn.com wrote:

    > I have defragged and have an organized directory tree set-up (each
    > letter of alphabet --> band name --> album --> song files),


    So you have about 4000 directories with a dozen of entries each.
    Directories with so few entries are unlikely to become
    fragmented. On the other hand, that makes a lot of them.

    Scanning that tree means opening 4000 directories. That's
    expensive. Unless NTFS wildly differs from the file systems I
    know, opening a directory has a significant fixed cost. Assuming
    no fragmentation, listing 10 directories with 10 entries each
    takes more time than listing 1 directory with 100 entries.

    You could make fewer directories with more entries. For example,
    one directory per artist. But that won't solve everything
    because scanning the tree still means stating nearly 50,000
    files.

    Alternatively, you could have a cron job (or whatever periodic
    tasks are called on Windows) slowly scan the tree every few
    minutes. A simple "DIR D:\ E:\ F:\ ... /-O /-P /S >NUL" would be
    a good start. That would help the directories and file metadata
    remain in the cache. Not exactly elegant, but IMO more
    convenient and effective than one directory per artist.

    > I am trying to decide if I should have just a few large partitions or
    > a bunch or smaller ones to organize the hard disks for speed of
    > accessing the data. Thanks again.


    Your many-small-partitions scheme makes day-to-day management
    more difficult but I don't think it makes things slower. In
    fact, it probably makes them a bit faster by improving locality.

    --
    André Majorel
    (Counterfeit: urojowoh@carry.com exaf@exoskeleton.com)
    There is always someone somewhere who needs a good laugh.

  17. Re: 47,900 MP3's - How To Organize Hard Discs?

    On 2007-09-11, matthewhargrove@msn.com wrote:
    >> FWIW I did wonder if you have any backup regime for all your music data
    >> files?

    >
    > My backup regime is not very good... I have everything burned to DVD's
    > using Nero's Back It Up. My initial backup took over 50 disks, and
    > since then I just periodically burn a new disc with what I have
    > transferred recently. Would love advice on how to back it up if an
    > easier way exists. I have never tested the backup, so I have
    > nightmares that I lose the data and it doesn't work, then I am back to
    > transferring everything again.


    Buy a 500 GB disk (A). Copy everything you have on it. Give it
    to a friend or relative.

    On the next visit, buy a second 500 GB disk (B). Copy everything
    you have on it, give it to him/her and take disk (A) back with
    you. Repeat, alternating between (A) and (B).

    At any time, you have a backup at hand and another one that's
    not as fresh but will survive a flood/fire/burglary of your
    residence.

    --
    André Majorel
    (Counterfeit: higid@amethystine.com okegobed@buckthorn.com)
    There is always someone somewhere who needs a good laugh.

  18. Re: 47,900 MP3's - How To Organize Hard Discs?

    Andre Majorel wrote
    > matthewhargrove@msn.com wrote


    >> I have defragged and have an organized directory tree set-up
    >> (each letter of alphabet --> band name --> album --> song files),

    >
    > So you have about 4000 directories with a dozen of entries each.
    > Directories with so few entries are unlikely to become fragmented.


    It aint the directory that gets fragmented, its the files.

    > On the other hand, that makes a lot of them.


    > Scanning that tree means opening 4000 directories. That's
    > expensive. Unless NTFS wildly differs from the file systems I
    > know, opening a directory has a significant fixed cost. Assuming
    > no fragmentation, listing 10 directories with 10 entries each
    > takes more time than listing 1 directory with 100 entries.


    Fragmentation has no effect on listing directorys.

    > You could make fewer directories with more entries. For example,
    > one directory per artist. But that won't solve everything because
    > scanning the tree still means stating nearly 50,000 files.


    But you only need to scan directory of the artist you are looking at.

    > Alternatively, you could have a cron job (or whatever periodic
    > tasks are called on Windows) slowly scan the tree every few
    > minutes. A simple "DIR D:\ E:\ F:\ ... /-O /-P /S >NUL" would be
    > a good start. That would help the directories and file metadata
    > remain in the cache.


    No it wouldnt with that many files.

    > Not exactly elegant, but IMO more convenient
    > and effective than one directory per artist.


    Wrong again.

    >> I am trying to decide if I should have just a few large
    >> partitions or a bunch or smaller ones to organize the
    >> hard disks for speed of accessing the data.


    > Your many-small-partitions scheme makes day-to-day management
    > more difficult but I don't think it makes things slower.


    It can do.

    > In fact, it probably makes them a bit faster by improving locality.


    Pity about farting around changing the current partition.



  19. Re: 47,900 MP3's - How To Organize Hard Discs?

    Rod Speed wrote in news:5kumtaF5gl22U1@mid.individual.net
    > Andre Majorel wrote
    > > matthewhargrove@msn.com wrote

    >
    > > > I have defragged and have an organized directory tree set-up
    > > > (each letter of alphabet --> band name --> album --> song files),

    > >
    > > So you have about 4000 directories with a dozen of entries each.
    > > Directories with so few entries are unlikely to become fragmented.


    Provided they all fit in the initially allocated directory space.
    If there is even such a thing in NTFS.

    > It aint the directory that gets fragmented, its the files.


    Nope. He only adds to his collection.
    So it's the directories that can get fragmented, not the files.

    >
    > > On the other hand, that makes a lot of them.

    >
    > > Scanning that tree means opening 4000 directories. That's
    > > expensive. Unless NTFS wildly differs from the file systems I
    > > know, opening a directory has a significant fixed cost. Assuming
    > > no fragmentation, listing 10 directories with 10 entries each
    > > takes more time than listing 1 directory with 100 entries.


    > Fragmentation has no effect on listing directorys.


    Of course it does, if they're scattered all over the drive.

    [snip]

  20. Re: 47,900 MP3's - How To Organize Hard Discs?

    On 2007-09-14, Rod Speed wrote:
    > Andre Majorel wrote
    >> matthewhargrove@msn.com wrote

    >
    >>> I have defragged and have an organized directory tree set-up
    >>> (each letter of alphabet --> band name --> album --> song files),

    >>
    >> So you have about 4000 directories with a dozen of entries each.
    >> Directories with so few entries are unlikely to become fragmented.

    >
    > It aint the directory that gets fragmented, its the files.


    If I'm not mistaken the OP has not been complaining about file I/O.

    >> On the other hand, that makes a lot of them.

    >
    >> Scanning that tree means opening 4000 directories. That's
    >> expensive. Unless NTFS wildly differs from the file systems I
    >> know, opening a directory has a significant fixed cost. Assuming
    >> no fragmentation, listing 10 directories with 10 entries each
    >> takes more time than listing 1 directory with 100 entries.

    >
    > Fragmentation has no effect on listing directorys.


    Unless your disk drive has zero seek time, it has.

    >> You could make fewer directories with more entries. For example,
    >> one directory per artist. But that won't solve everything because
    >> scanning the tree still means stating nearly 50,000 files.

    >
    > But you only need to scan directory of the artist you are looking at.


    I assume the OP is scanning all (or a significant portion of)
    the tree. Otherwise it wouldn't be slow enough to notice.

    >> Alternatively, you could have a cron job (or whatever periodic
    >> tasks are called on Windows) slowly scan the tree every few
    >> minutes. A simple "DIR D:\ E:\ F:\ ... /-O /-P /S >NUL" would be
    >> a good start. That would help the directories and file metadata
    >> remain in the cache.

    >
    > No it wouldnt with that many files.


    $ time find / | head -n 50000 | wc -l
    50000

    real 0m7.794s
    user 0m0.169s
    sys 0m0.291s
    $ time find / | head -n 50000 | wc -l
    50000

    real 0m0.301s
    user 0m0.134s
    sys 0m0.166s

    Works for me, on Ext3 at least.

    >>> I am trying to decide if I should have just a few large
    >>> partitions or a bunch or smaller ones to organize the
    >>> hard disks for speed of accessing the data.

    >
    >> Your many-small-partitions scheme makes day-to-day management
    >> more difficult but I don't think it makes things slower.

    >
    > It can do.


    In the OP's situation ? How ?

    >> In fact, it probably makes them a bit faster by improving locality.

    >
    > Pity about farting around changing the current partition.


    That's Windows for you. :->

    --
    André Majorel
    (Counterfeit: keqehefeb@sinuous.com elerys@rxmedszone.info)
    There is always someone somewhere who needs a good laugh.

+ Reply to Thread
Page 1 of 2 1 2 LastLast