Fast Bulk Ingestion of dicom files into Conquest/MySQL
I'm using MySQL 5.0 (win32 with 1gb ram) as the database engine for a
conquest dicom server. There are millions of dicom medical image files
to be imported. I started with a batch of about 300,000 files and the
indexing went pretty fast at first but has slowed to a crawl. I'm
hoping you might advise me on how to optimize MySQL's performance for
this process. (and Conquest too if possible.)
My plan/hope is to:
(a) load 300gb or more worth of dicom files into a folder (with many
(b) define that as MAGx (where x is initially 0)
(c) tell Conquest to index that data (first one on installation page,
later ones on Maintenance page, one mag device at a time.)
(d) migrate the files to an HSM/tape library after they are indexed by
[the HSM software makes them appear to still be disk files, but we
certainly don't want to have Conquest try to index them *after* they've
gone to tape.]
(e) close Conquest and define another root folder as MAGx+1
Then repeat the steps till all is done. It may be 10 cycles. I think
you can't go past "mag9" with conquest, so 10 may be the maximum number
of chunks I can use.
Also, is there any way to optimize the servers (Conquest and MySQL)
for ingesting many dicoms quickly? Since this is old data, and its on
an HSM anyway, the eventual retrieval speed only needs to be
"adequate". But there's so much of it that I need to make the
ingestion of the files as speedy as possible.
Innodb is the default. Using MySQL Administrator, I've increased the
Key Buffer allocation to 128mb, and haven't noticed any difference.
Someone suggested to set "innodb_flush_log_at_trx_commit = 0. Can't
see how to do that in the MySQL admin GUI. guess it's a command-line
Any help will be greatly appreciated.