Help needed with JPEG tests - DICOM

This is a discussion on Help needed with JPEG tests - DICOM ; Bonjour to you all As some of you know, one of the popular library used to encode and decode the JPEG lossless images in DICOM streams is the PVRG library. You may also know that this library has a bug ...

+ Reply to Thread
Results 1 to 5 of 5

Thread: Help needed with JPEG tests

  1. Help needed with JPEG tests

    Bonjour to you all

    As some of you know, one of the popular library used to encode and
    decode the JPEG lossless images in DICOM streams is the PVRG library.
    You may also know that this library has a bug for 16 bits images. At
    first the bug was not too noticeable since most images were only 12
    bits anyway. But since a couple of years, manufacturers started using
    16 bits of resolution in their images.

    This means that the images encoded with the PVRG library could only be
    displayed with viewers using the same library. Using a different
    library would either give erroneous results or simply crash the
    program altogether. And for the same reason, viewers using PVRG could
    not display images that were encoded with other libraries. (Sample of
    images having the PVRG bug can be found in David's sample files. In
    the JPLL section the MR1, MR3 and NM1 images have the PVRG bug).

    I don't want to criticize the PVRG library, apart from that detail,
    it's a good library and I was using it myself (I still use it for
    lossy images). The problem is that you need to know in advance if the
    images you are reading have the bug or not. If you don't, the library
    will probably crash while reading the data.

    In my programs I've tried to detect and correct this problem. I
    thought I had a good heuristic to detect the problem from the first
    pixel in the image, but I just received a few images from a Hitachi
    scanner that my algorithm failed to detect has having the bug

    So I finally decided to byte the bullet and last week I did my own
    decoder for the lossless JPEG. In keeping with the philosophy of my
    applications, I aim for robustness instead of speed. So now, even if
    I don't predict the bug at the first try, my application will not
    crash and I can then attempt a second reading with different
    settings...

    This brings me to the "help needed" part:

    I need to get the last few bugs out of my new library.

    I've tried it with all the Lossless JPEG images I have, but I'm sure
    that there are some images out there that will crash it! Among the
    possible problems, I'm not sure I treat the "DRI" (Define Restart
    Interval) or the "DNL" (Define Number of Lines) correctly, but none of
    the images I have use these marker

    If you could try it out with different images having JPEG lossless
    data in an attempt to make it crash, I would appreciate any feedback
    and sample of problem images

    I placed the latest version of TomoVision (my viewer) on my web site
    (it's the 1.4 rev-0 beta version). You can downloaf it at:
    http://www.tomovision.com/download/tomovision.htm

    Merci,

    Yves


  2. Re: Help needed with JPEG tests

    Hi Yves

    Do you have a suggested patch for PVRG code that would stop it
    generating this bug ?

    Do you have a description of your new detection algorithm ?

    I guess I should make some new test images, with and without the bug.

    David

    Yves Martel wrote:

    > Bonjour to you all
    >
    > As some of you know, one of the popular library used to encode and
    > decode the JPEG lossless images in DICOM streams is the PVRG library.
    > You may also know that this library has a bug for 16 bits images. At
    > first the bug was not too noticeable since most images were only 12
    > bits anyway. But since a couple of years, manufacturers started using
    > 16 bits of resolution in their images.
    >
    > This means that the images encoded with the PVRG library could only be
    > displayed with viewers using the same library. Using a different
    > library would either give erroneous results or simply crash the
    > program altogether. And for the same reason, viewers using PVRG could
    > not display images that were encoded with other libraries. (Sample of
    > images having the PVRG bug can be found in David's sample files. In
    > the JPLL section the MR1, MR3 and NM1 images have the PVRG bug).
    >
    > I don't want to criticize the PVRG library, apart from that detail,
    > it's a good library and I was using it myself (I still use it for
    > lossy images). The problem is that you need to know in advance if the
    > images you are reading have the bug or not. If you don't, the library
    > will probably crash while reading the data.
    >
    > In my programs I've tried to detect and correct this problem. I
    > thought I had a good heuristic to detect the problem from the first
    > pixel in the image, but I just received a few images from a Hitachi
    > scanner that my algorithm failed to detect has having the bug
    >
    > So I finally decided to byte the bullet and last week I did my own
    > decoder for the lossless JPEG. In keeping with the philosophy of my
    > applications, I aim for robustness instead of speed. So now, even if
    > I don't predict the bug at the first try, my application will not
    > crash and I can then attempt a second reading with different
    > settings...
    >
    > This brings me to the "help needed" part:
    >
    > I need to get the last few bugs out of my new library.
    >
    > I've tried it with all the Lossless JPEG images I have, but I'm sure
    > that there are some images out there that will crash it! Among the
    > possible problems, I'm not sure I treat the "DRI" (Define Restart
    > Interval) or the "DNL" (Define Number of Lines) correctly, but none of
    > the images I have use these marker
    >
    > If you could try it out with different images having JPEG lossless
    > data in an attempt to make it crash, I would appreciate any feedback
    > and sample of problem images
    >
    > I placed the latest version of TomoVision (my viewer) on my web site
    > (it's the 1.4 rev-0 beta version). You can downloaf it at:
    > http://www.tomovision.com/download/tomovision.htm
    >
    > Merci,
    >
    > Yves
    >



  3. Re: Help needed with JPEG tests

    Salut Yves,

    I tried version from zip file: tomo_14_r0.zip
    It seg fault on the very first image I tried (to be honest I knew there
    would be few chance it will). The DICOM image is

    gdcm-JPEG-LossLessThoravision.dcm

    You can get it using cvs, follow instruction from:

    http://www.creatis.insa-lyon.fr/Public/Gdcm/Cvs.html

    On linux I do:

    cvs -d server:anonymous@cvs.creatis.insa-lyon.fr:/cvs/public co gdcmData


    Good luck, this image is tricky
    Mathieu



    Yves Martel wrote:
    > Bonjour to you all
    >
    > As some of you know, one of the popular library used to encode and
    > decode the JPEG lossless images in DICOM streams is the PVRG library.
    > You may also know that this library has a bug for 16 bits images. At
    > first the bug was not too noticeable since most images were only 12
    > bits anyway. But since a couple of years, manufacturers started using
    > 16 bits of resolution in their images.
    >
    > This means that the images encoded with the PVRG library could only be
    > displayed with viewers using the same library. Using a different
    > library would either give erroneous results or simply crash the
    > program altogether. And for the same reason, viewers using PVRG could
    > not display images that were encoded with other libraries. (Sample of
    > images having the PVRG bug can be found in David's sample files. In
    > the JPLL section the MR1, MR3 and NM1 images have the PVRG bug).
    >
    > I don't want to criticize the PVRG library, apart from that detail,
    > it's a good library and I was using it myself (I still use it for
    > lossy images). The problem is that you need to know in advance if the
    > images you are reading have the bug or not. If you don't, the library
    > will probably crash while reading the data.
    >
    > In my programs I've tried to detect and correct this problem. I
    > thought I had a good heuristic to detect the problem from the first
    > pixel in the image, but I just received a few images from a Hitachi
    > scanner that my algorithm failed to detect has having the bug
    >
    > So I finally decided to byte the bullet and last week I did my own
    > decoder for the lossless JPEG. In keeping with the philosophy of my
    > applications, I aim for robustness instead of speed. So now, even if
    > I don't predict the bug at the first try, my application will not
    > crash and I can then attempt a second reading with different
    > settings...
    >
    > This brings me to the "help needed" part:
    >
    > I need to get the last few bugs out of my new library.
    >
    > I've tried it with all the Lossless JPEG images I have, but I'm sure
    > that there are some images out there that will crash it! Among the
    > possible problems, I'm not sure I treat the "DRI" (Define Restart
    > Interval) or the "DNL" (Define Number of Lines) correctly, but none of
    > the images I have use these marker
    >
    > If you could try it out with different images having JPEG lossless
    > data in an attempt to make it crash, I would appreciate any feedback
    > and sample of problem images
    >
    > I placed the latest version of TomoVision (my viewer) on my web site
    > (it's the 1.4 rev-0 beta version). You can downloaf it at:
    > http://www.tomovision.com/download/tomovision.htm
    >
    > Merci,
    >
    > Yves
    >



  4. Re: Help needed with JPEG tests

    David Clunie wrote:
    > Hi Yves
    >
    > Do you have a suggested patch for PVRG code that would stop it
    > generating this bug ?
    >
    > Do you have a description of your new detection algorithm ?
    >
    > I guess I should make some new test images, with and without the bug.


    So you acknowledge the DICOM image is buggy but those images should
    remain 'DICOM' images ? As long as they will be around people will have
    to deal/worry with those.

    I would really appreciate if -at least- your baseline was exempt from
    images that do not respect properly the JPEG compression. Those image
    should clearly be mark as either not DICOM, sort of compatible or close
    to DICOM specifications.

    Or else we need a Transfer Syntax for the buggy PVRG jpeg.

    Regards,
    Mathieu

  5. Re: Help needed with JPEG tests


    >Hi Yves
    >
    >Do you have a suggested patch for PVRG code that would stop it
    >generating this bug ?
    >


    Bonjour David,

    I must admit I didn't look to closely at the "encoding" side of the
    problem in PVRG, I only use the decoder...

    In the losslessDecodeDC() function of decode.c, you have the lines:

    if ( s== 16 )
    return( 32768 )

    So if you have a 16 bit encoded word, it will automatically return the
    value 32768 instead of computing the actual value from the next 16
    bits of data.

    Just commenting these lines solve the problem on the decoder side. I
    suspect it might be more complex to fix on the encoding side. You do
    have a matching line in EncodeDC():

    if (S!=16) fputv(s,coef) ;

    But I'm not sure if just removing this test will not have
    repercussions in the rest of the code...

    >Do you have a description of your new detection algorithm ?
    >



    Basically, PVRG does not seem to be able to encode actual 16 bit
    values. As soon as the difference between a value and it's predictor
    is greater than 32768, it is encode as 32768 with just the index in
    the DC table (=16) without the actual value.

    This means that if you read a stream from PVRG with a normal decoder
    an you hit on a value encode as 32768 by PVRG, the normal decoder will
    read a supplementary 16 bits from the stream. After that, your
    lost...

    On the other hand, if you read a non-PVRG stream with the PVRG
    library, the library will hit the code for 16 bits and return 32768
    without reading the next 16 bits. These 16 bits will be read as a new
    pixels and again, your lost.

    How do you know if the image is PVRG or not?

    We do know that the first predictor for a 16 bit image is:
    1 << (Frame_P - Scan_Al - 1)
    with:
    Frame_P = Frame precision = 16
    Sacn_Al = point transform = 0 to 15, but usually 0
    So the predictor is usually = 32768

    So if we want the first pixel to be between 0 to 1000, we will need
    the first value to be between -32768 to -31768. Both will be encoded
    as 16 bits values, so the very first pixel in a typical 16 bit image
    is a 16 bit value!

    I read and decode this first pixel using a non-PVRG algo. I then look
    at the resulting value. Is it something that make sense in a medical
    image context (Is its value between +- 6000 of the predictor)?

    But of course, this does not work all the time

    In the an Hitachi image I have, the values are signed, and the first
    pixel is -2048. This encode to 0xF800. To get this value, you only
    need to add 7800 to the predictor, and that's a 15 bit value...

    So if after my prediction, if the decoder get lost during reading, I
    assume I predicted incorrectly and re-try to decode the stream with
    the other algorithm.

    And even if the decoder did not get lost, but the dynamic range of the
    image look suspicious, I try with the different algorithm to make sure
    I do not get better results.

    >I guess I should make some new test images, with and without the bug.
    >


    I have a few MR images of the head of my girl friend from a Siemens
    scanner. I do beleive they are 16 bits and they do not use PVRG. Do
    you want them? (I've ask and she doesn't mind, I'll anonymize them
    anyway...)


    Does this help?

    Merci,

    Yves

    >David
    >
    >Yves Martel wrote:
    >
    >> Bonjour to you all
    >>
    >> As some of you know, one of the popular library used to encode and
    >> decode the JPEG lossless images in DICOM streams is the PVRG library.
    >> You may also know that this library has a bug for 16 bits images. At
    >> first the bug was not too noticeable since most images were only 12
    >> bits anyway. But since a couple of years, manufacturers started using
    >> 16 bits of resolution in their images.
    >>
    >> This means that the images encoded with the PVRG library could only be
    >> displayed with viewers using the same library. Using a different
    >> library would either give erroneous results or simply crash the
    >> program altogether. And for the same reason, viewers using PVRG could
    >> not display images that were encoded with other libraries. (Sample of
    >> images having the PVRG bug can be found in David's sample files. In
    >> the JPLL section the MR1, MR3 and NM1 images have the PVRG bug).
    >>
    >> I don't want to criticize the PVRG library, apart from that detail,
    >> it's a good library and I was using it myself (I still use it for
    >> lossy images). The problem is that you need to know in advance if the
    >> images you are reading have the bug or not. If you don't, the library
    >> will probably crash while reading the data.
    >>
    >> In my programs I've tried to detect and correct this problem. I
    >> thought I had a good heuristic to detect the problem from the first
    >> pixel in the image, but I just received a few images from a Hitachi
    >> scanner that my algorithm failed to detect has having the bug
    >>
    >> So I finally decided to byte the bullet and last week I did my own
    >> decoder for the lossless JPEG. In keeping with the philosophy of my
    >> applications, I aim for robustness instead of speed. So now, even if
    >> I don't predict the bug at the first try, my application will not
    >> crash and I can then attempt a second reading with different
    >> settings...
    >>
    >> This brings me to the "help needed" part:
    >>
    >> I need to get the last few bugs out of my new library.
    >>
    >> I've tried it with all the Lossless JPEG images I have, but I'm sure
    >> that there are some images out there that will crash it! Among the
    >> possible problems, I'm not sure I treat the "DRI" (Define Restart
    >> Interval) or the "DNL" (Define Number of Lines) correctly, but none of
    >> the images I have use these marker
    >>
    >> If you could try it out with different images having JPEG lossless
    >> data in an attempt to make it crash, I would appreciate any feedback
    >> and sample of problem images
    >>
    >> I placed the latest version of TomoVision (my viewer) on my web site
    >> (it's the 1.4 rev-0 beta version). You can downloaf it at:
    >> http://www.tomovision.com/download/tomovision.htm
    >>
    >> Merci,
    >>
    >> Yves
    >>



+ Reply to Thread