> After reading PS 3.10 - 2008, 7.1 DICOM FILE META INFORMATION, I
> believe that a DICOM file is valid even if the elements in the File
> Meta Information are declared as UN instead of there respective VR.
> I know the case is hairy, but when converting a Little Endian
> Implicit file meta information header (typically from GE), I do not
> wish to load the data dictionary in which case I default to using
> VR=UN.

Certainly a gray area, but I believe that encoding meta-headers as UN
will break many toolkits (certainly ours, i.e. DCMTK, in the default
configuration) and is not a good idea.
The only reason to ever create UN elements is when converting from
Implicit VR to Explicit VR and reading an attribute that is unknown
to the application. However, no metadata attributes have been defined
since part 10 was written, so there is no reason why an application
writing a part 10 metaheader should ever encounter an unknown attribute.
Also, "conversion from Implicit VR metaheader" is a use-case that does
not exist in DICOM since a metaheader cannot ever exist in Implicit VR
format (of course we all know that not all developers read the Final Text
version of the specs, but correct behaviour in face of faulty data is
something that is an area where DICOM does not define any rules).

In summary, not wishing to load a data dictionary (which in this case
would just need to contain about 10 attributes) is a rather lame excuse
and the converted files will probably cause more problems than the faulty
orginal ones.

Regards,
Marco Eichelberg

On Mar 4, 9:23 am, Marco Eichelberg
wrote:
> > After reading PS 3.10 - 2008, 7.1 DICOM FILE META INFORMATION, I
> > believe that a DICOM file is valid even if the elements in the File
> > Meta Information are declared as UN instead of there respective VR.
> > I know the case is hairy, but when converting a Little Endian
> > Implicit file meta information header (typically from GE), I do not
> > wish to load the data dictionary in which case I default to using
> > VR=UN.
>
> Certainly a gray area, but I believe that encoding meta-headers as UN
> will break many toolkits (certainly ours, i.e. DCMTK, in the default
> configuration) and is not a good idea.
> The only reason to ever create UN elements is when converting from
> Implicit VR to Explicit VR and reading an attribute that is unknown
> to the application. However, no metadata attributes have been defined
> since part 10 was written, so there is no reason why an application
> writing a part 10 metaheader should ever encounter an unknown attribute.
> Also, "conversion from Implicit VR metaheader" is a use-case that does
> not exist in DICOM since a metaheader cannot ever exist in Implicit VR
> format (of course we all know that not all developers read the Final Text
> version of the specs, but correct behaviour in face of faulty data is
> something that is an area where DICOM does not define any rules).
>
> In summary, not wishing to load a data dictionary (which in this case
> would just need to contain about 10 attributes) is a rather lame excuse
> and the converted files will probably cause more problems than the faulty
> orginal ones.

Depends what you call 'lame'. I call it a lame excuse not to go and
fix a toolkit for 'gray area' of the standard :-P
Anyway -at least for me- it completely breaks the separation in
between Part 3.5 and Part 3.6/Part 3.3. No data dictionary should be
needed to read a DICOM file (parsing is a different operation than
interpreting).
Call it a design issue if you want, but having the data dictionary
pulls in for example dependency to an XML library. And even in C++
constructing a hash table (std::map) is insanely slow.

And for reference, the following do not seems to have any particular
issues with such files:
- dicom3tools
- TomoVision
- irfanview
- dicomworks
- Efilm 153

-Mathieu
Ref:
http://gdcm.sf.net/PICKER-16-MONO2-N...3_Preamble.dcm

Hi Mathieu

This is a really bad idea.

Just in case it is not obvious, I will put in a CP to explicitly
state that these attributes shall never use the UN VR.

David

Mathieu Malaterre wrote:
> Hi there,
>
> After reading PS 3.10 - 2008, 7.1 DICOM FILE META INFORMATION, I
> believe that a DICOM file is valid even if the elements in the File
> Meta Information are declared as UN instead of there respective VR.
> I know the case is hairy, but when converting a Little Endian
> Implicit file meta information header (typically from GE), I do not
> wish to load the data dictionary in which case I default to using
> VR=UN.
>
> Thanks
> -Mathieu

Mathieu Malaterre wrote:
> Depends what you call 'lame'. I call it a lame excuse not to go and
> fix a toolkit for 'gray area' of the standard :-P

Point taken :-)

That said, most of the "gray areas" in DICOM only turned out to be
gray areas when somebody came up with an interpretation I had never
thought to be ambiguous - like this example.

With DCMTK 3.5.4, dcmdump will refuse to even read the file,
while dcmdump --convert-un will work correctly.

The problem is that by default DCMTK makes use of the meta-header
group length attribute, and the parser refuses to interpret that as UL
if encoded in UN in the dataset. One could certainly implement a workaround
(such as enabling --convert-un by default when reading a meta-header),
but that is not present in any DCMTK release up to now.

> Anyway -at least for me- it completely breaks the separation in
> between Part 3.5 and Part 3.6/Part 3.3. No data dictionary should be
> needed to read a DICOM file (parsing is a different operation than
> interpreting).

That is a good question. Without a dictionary, you cannot identify
sequences (with explicit length), so your parser cannot create an
in-memory tree representing all elements. You need to treat sequences
as "black boxes" and parse them at a later point when their VR is known,
which again breaks the separation. In summary, I don't think that the
DICOM encoding really permits a "clean" parsing without a data dictionary.
If Implicit VR had gone away in 1993, this might be easier, but with
Implicit VR you simply need a-priori knowledge if you want your parser
to produce anything useful.

> Call it a design issue if you want, but having the data dictionary
> pulls in for example dependency to an XML library. And even in C++
> constructing a hash table (std::map) is insanely slow.

That might depend on the hash table implementation - std::map might
not be the optimal choice here. I agree that creating the dictionary
is relatively expensive (in the order of a few milliseconds on modern hardware),
but not to the degree that it really hurts.

One compromise would be to create a "skeleton dictionary" that is always
available and built in. DCMTK actually does this, although the skeleton
only contains (xxxx,0000), (fffe,e000), (fffe,e00d) and (fffe,e0dd),
i.e. group length and the special-purpose tags for items and delimiters.
One could imagine extending that with the few group 0000 and 0002 tags
which would permit a reliable reading at least of metaheaders and network
messages. Such a dictionary with, say, 20 tags should not be expensive
to construct.

Regards,
Marco Eichelberg

On Mar 4, 1:25 pm, Marco Eichelberg
wrote:
> Mathieu Malaterre wrote:
> > Depends what you call 'lame'. I call it a lame excuse not to go and
> > fix a toolkit for 'gray area' of the standard :-P
>
> Point taken :-)
>
> That said, most of the "gray areas" in DICOM only turned out to be
> gray areas when somebody came up with an interpretation I had never
> thought to be ambiguous - like this example.
>
> With DCMTK 3.5.4, dcmdump will refuse to even read the file,
> while dcmdump --convert-un will work correctly.

Cool did not realize that ! My biggest issue was that I am using
dcmdump to verify the output of gdcm is always valid. So at least I
have a temporary solution.

> The problem is that by default DCMTK makes use of the meta-header
> group length attribute, and the parser refuses to interpret that as UL
> if encoded in UN in the dataset. One could certainly implement a workaround
> (such as enabling --convert-un by default when reading a meta-header),
> but that is not present in any DCMTK release up to now.
>
> > Anyway -at least for me- it completely breaks the separation in
> > between Part 3.5 and Part 3.6/Part 3.3. No data dictionary should be
> > needed to read a DICOM file (parsing is a different operation than
> > interpreting).
>
> That is a good question. Without a dictionary, you cannot identify
> sequences (with explicit length), so your parser cannot create an
> in-memory tree representing all elements. You need to treat sequences
> as "black boxes" and parse them at a later point when their VR is known,
> which again breaks the separation. In summary, I don't think that the
> DICOM encoding really permits a "clean" parsing without a data dictionary.
> If Implicit VR had gone away in 1993, this might be easier, but with
> Implicit VR you simply need a-priori knowledge if you want your parser
> to produce anything useful.

If you get a chance, please try gdcm2 (from svn on sf.net). The Parser
is not (physically) link to the dictionary and so far I did not find a
case where I could not recognize a SQ in implicit dataset. The whole
idea is to look for item start/end in the binary blob. Of course one
could imagine forging an image that exactly start with an item start
end finish with an item end... But I like it this way, it allowed me
to find private SQ without prior knowledge of private vendor data
elements dictionary.

> > Call it a design issue if you want, but having the data dictionary
> > pulls in for example dependency to an XML library. And even in C++
> > constructing a hash table (std::map) is insanely slow.
>
> That might depend on the hash table implementation - std::map might
> not be the optimal choice here. I agree that creating the dictionary
> is relatively expensive (in the order of a few milliseconds on modern hardware),
> but not to the degree that it really hurts.

People south of France likes to exagerate a little...just a little

> One compromise would be to create a "skeleton dictionary" that is always
> available and built in. DCMTK actually does this, although the skeleton
> only contains (xxxx,0000), (fffe,e000), (fffe,e00d) and (fffe,e0dd),
> i.e. group length and the special-purpose tags for items and delimiters.
> One could imagine extending that with the few group 0000 and 0002 tags
> which would permit a reliable reading at least of metaheaders and network
> messages. Such a dictionary with, say, 20 tags should not be expensive
> to construct.

Correct. I was more concerned with the maintaining, but since meta
header has not seen any new elements since its creation I believe this
is safe.

In summary, because of the coming CP (thanks David), I'll go ahead and
fix GDCM 2.x

Sorry for the noise,
-Mathieu

This is not really an 'idea' but mainly because my implementation of
File Meta Header derive from a DataSet class, which -by definition-
uses VR=UN when the VR is unknown. I'll make sure to have a separate
implementation for Meta Header.

thx
-Mathieu

On Mar 4, 12:50 pm, David Clunie wrote:
> Hi Mathieu
>
> This is a really bad idea.
>
> Just in case it is not obvious, I will put in a CP to explicitly
> state that these attributes shall never use the UN VR.
>
> David
>
> Mathieu Malaterre wrote:
> > Hi there,
>
> > After reading PS 3.10 - 2008, 7.1 DICOM FILE META INFORMATION, I
> > believe that a DICOM file is valid even if the elements in the File
> > Meta Information are declared as UN instead of there respective VR.
> > I know the case is hairy, but when converting a Little Endian
> > Implicit file meta information header (typically from GE), I do not
> > wish to load the data dictionary in which case I default to using
> > VR=UN.
>
> > Thanks
> > -Mathieu