IP Identification field is zero - why? - TCP-IP

This is a discussion on IP Identification field is zero - why? - TCP-IP ; Why is the DF (don't fragment) flag set in all these packets, and why is the IPv4 "identification" word set to zero in two of them? IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: UDP (17), ...

+ Reply to Thread
Results 1 to 16 of 16

Thread: IP Identification field is zero - why?

  1. IP Identification field is zero - why?

    Why is the DF (don't fragment) flag set in all these packets, and why
    is the IPv4 "identification" word set to zero in two of them?

    IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: UDP (17), length: 84) aaaa.3300 > bbbb.53: 8299+ [1au] PTR? ... (56)
    IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto: UDP (17), length: 140) bbbb.53 > aaaaa.3300: 8299 q: PTR? ...

    IP (tos 0x0, ttl 64, id 23635, offset 0, flags [DF], proto: UDP (17), length: 32) aaaa.2458 > cccc.666: [udp sum ok] UDP, length 4

    These packets were captured on a Linux 2.6 host with tcpdump. The
    local machine is "aaaa" in the slightly anonymized dump above.

    The two first packets are two bind name servers talking to each other.
    The third packet, with non-zero id 23635, is me sending a single UDP
    datagram using netcat.

    Is the DF bit on UDP packets part of some path MTU discovery scheme?
    And if so, does it explain the "id 0" on only some of the packets?

    As I understand it, Identification is mostly used for fragment
    reassembly, so it is probably not that useful when DF is set.

    Or is tcpdump on Linux capturing the packets too early, before lower
    layers of the network stack fix them up?

    /Jorgen

    --
    // Jorgen Grahn \X/ snipabacken.se> R'lyeh wgah'nagl fhtagn!

  2. Re: IP Identification field is zero - why?

    In article ,
    Jorgen Grahn wrote:

    > Why is the DF (don't fragment) flag set in all these packets, and why
    > is the IPv4 "identification" word set to zero in two of them?
    >
    > IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: UDP (17), length:
    > 84) aaaa.3300 > bbbb.53: 8299+ [1au] PTR? ... (56)
    > IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto: UDP (17), length:
    > 140) bbbb.53 > aaaaa.3300: 8299 q: PTR? ...
    >
    > IP (tos 0x0, ttl 64, id 23635, offset 0, flags [DF], proto: UDP (17),
    > length: 32) aaaa.2458 > cccc.666: [udp sum ok] UDP, length 4
    >
    > These packets were captured on a Linux 2.6 host with tcpdump. The
    > local machine is "aaaa" in the slightly anonymized dump above.
    >
    > The two first packets are two bind name servers talking to each other.
    > The third packet, with non-zero id 23635, is me sending a single UDP
    > datagram using netcat.
    >
    > Is the DF bit on UDP packets part of some path MTU discovery scheme?


    Most likely. There's not much other use for it.

    > And if so, does it explain the "id 0" on only some of the packets?


    That I'm not sure of.

    >
    > As I understand it, Identification is mostly used for fragment
    > reassembly, so it is probably not that useful when DF is set.


    That's the ONLY thing that Identification is used for, so this probably
    explains it.

    But then it's strange that packet 3 has an ID even though DF is set.
    I'm not sure why it would make a difference what application sent it, as
    I don't think there's an API that allows application to control this.
    But I'm probably out of touch with recent additions to the sockets API.

    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE post questions in newsgroups, not directly to me ***
    *** PLEASE don't copy me on replies, I'll read them in the group ***

  3. Re: IP Identification field is zero - why?

    On Wed, 09 Jan 2008 05:34:22 -0500, Barry Margolin wrote:
    > In article ,
    > Jorgen Grahn wrote:
    >
    >> Why is the DF (don't fragment) flag set in all these packets, and why
    >> is the IPv4 "identification" word set to zero in two of them?

    ....

    >> Is the DF bit on UDP packets part of some path MTU discovery scheme?

    >
    > Most likely. There's not much other use for it.
    >
    >> And if so, does it explain the "id 0" on only some of the packets?

    >
    > That I'm not sure of.
    >
    >> As I understand it, Identification is mostly used for fragment
    >> reassembly, so it is probably not that useful when DF is set.

    >
    > That's the ONLY thing that Identification is used for, so this probably
    > explains it.


    Yes, but at least the older RFCs say the host should increase this field
    for each packet it sends. I think Stevens discusses this somewhere --
    how BSD chose to keep one counter per interface instead, or something.

    > But then it's strange that packet 3 has an ID even though DF is set.
    > I'm not sure why it would make a difference what application sent it, as
    > I don't think there's an API that allows application to control this.
    > But I'm probably out of touch with recent additions to the sockets API.


    And like me, you are probably not familiar with the odd corners of the
    Linux IP stack. I would rather suspect the Linux kernel than the name
    server application -- that some condition makes it choose to skip the
    ID.

    I am interested in more suggestions. Partly because I am curious, but
    mostly because my team has a performance problem on the receiving side
    (which is an embedded system) of an UDP-based protocol. We suspect
    that its IP stack gets confused by the zero Identification field.

    /Jorgen

    --
    // Jorgen Grahn \X/ snipabacken.se> R'lyeh wgah'nagl fhtagn!

  4. Re: IP Identification field is zero - why?

    Jorgen Grahn writes:

    >I am interested in more suggestions. Partly because I am curious, but
    >mostly because my team has a performance problem on the receiving side
    >(which is an embedded system) of an UDP-based protocol. We suspect
    >that its IP stack gets confused by the zero Identification field.


    If you can disable path mtu discovery on the sending side, the IP ID fields
    will not be 0 any longer. (Not that this helps if you have no control over
    your clients in production)

    Casper
    --
    Expressed in this posting are my opinions. They are in no way related
    to opinions held by my employer, Sun Microsystems.
    Statements on Sun products included here are not gospel and may
    be fiction rather than truth.

  5. Re: IP Identification field is zero - why?

    In article ,
    Jorgen Grahn wrote:
    >> But then it's strange that packet 3 has an ID even though DF is set.
    >> I'm not sure why it would make a difference what application sent it, as
    >> I don't think there's an API that allows application to control this.
    >> But I'm probably out of touch with recent additions to the sockets API.

    >
    >And like me, you are probably not familiar with the odd corners of the
    >Linux IP stack. I would rather suspect the Linux kernel than the name
    >server application -- that some condition makes it choose to skip the
    >ID.


    I agree, it's likely the IP stack. The name server application doesn't
    build the IP header, so it has little influence over what the kernel
    puts in the IP header. It should be easy enough to find the code that
    builds the IP header and see how they decide what to put in the ID field
    of the IP header.

    >I am interested in more suggestions. Partly because I am curious, but
    >mostly because my team has a performance problem on the receiving side
    >(which is an embedded system) of an UDP-based protocol. We suspect
    >that its IP stack gets confused by the zero Identification field.


    What stack is the embedded system using? Don't you have sources for
    that stack? Only the IP stack should be looking at the ID field, and
    then really only when reassembling fragmented packets. And certainly,
    UDP-based protocols have nothing to do with the IP ID field. Do you see
    similar behavior with TCP? Is it only DNS that you're seeing this issue
    with?

    Patrick
    ========= For LAN/WAN Protocol Analysis, check out PacketView Pro! =========
    Patrick Klos Email: patrick@klos.com
    Klos Technologies, Inc. Web: http://www.klos.com/
    ================================================== ==========================

  6. Re: IP Identification field is zero - why?

    On Wed, 9 Jan 2008 14:22:22 +0000 (UTC), Patrick Klos wrote:
    > In article ,
    > Jorgen Grahn wrote:
    >>> But then it's strange that packet 3 has an ID even though DF is set.
    >>> I'm not sure why it would make a difference what application sent it, as
    >>> I don't think there's an API that allows application to control this.
    >>> But I'm probably out of touch with recent additions to the sockets API.

    >>
    >>And like me, you are probably not familiar with the odd corners of the
    >>Linux IP stack. I would rather suspect the Linux kernel than the name
    >>server application -- that some condition makes it choose to skip the
    >>ID.

    >
    > I agree, it's likely the IP stack. The name server application doesn't
    > build the IP header, so it has little influence over what the kernel
    > puts in the IP header. It should be easy enough to find the code that
    > builds the IP header and see how they decide what to put in the ID field
    > of the IP header.


    Yes, shouldn't be too hard. Easier to ask here though -- and maybe
    get an answer which covers other stacks (And other versions of the
    Linux stack, for that matter. RedHat has probably patched our kernel
    heavily).

    >>I am interested in more suggestions. Partly because I am curious, but
    >>mostly because my team has a performance problem on the receiving side
    >>(which is an embedded system) of an UDP-based protocol. We suspect
    >>that its IP stack gets confused by the zero Identification field.

    >
    > What stack is the embedded system using? Don't you have sources for
    > that stack?


    It's BSD-derived, and owned by a partner.

    > Only the IP stack should be looking at the ID field, and
    > then really only when reassembling fragmented packets. And certainly,
    > UDP-based protocols have nothing to do with the IP ID field. Do you see
    > similar behavior with TCP? Is it only DNS that you're seeing this issue
    > with?


    DNS was just an example; the real problem (which I hesitate to go into
    details with) concerns another UDP-based protocol, and two load
    generators. One of them assembles its own IP packets (and clears DF
    and sets id), but the other one uses sockets and displays the
    behaviour described above.

    /Jorgen

    --
    // Jorgen Grahn \X/ snipabacken.se> R'lyeh wgah'nagl fhtagn!

  7. Re: IP Identification field is zero - why?

    In article ,
    Jorgen Grahn wrote:

    > DNS was just an example; the real problem (which I hesitate to go into
    > details with) concerns another UDP-based protocol, and two load
    > generators. One of them assembles its own IP packets (and clears DF
    > and sets id), but the other one uses sockets and displays the
    > behaviour described above.


    So when you said the other packets were from netcat, you really meant
    they were from this application that uses raw sockets? It would have
    been much clearer if you'd said this the first time.

    What's apparently going on is that the OS defaults to using PMTU
    Discovery, so it sets DF and doesn't fill in the ID. Any apps that use
    normal TCP or UDP sockets are affected by this. Apps that use raw
    sockets and fill in the header themselves are not, and can do whatever
    they want.

    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE post questions in newsgroups, not directly to me ***
    *** PLEASE don't copy me on replies, I'll read them in the group ***

  8. Re: IP Identification field is zero - why?

    On Thu, 10 Jan 2008 21:51:19 -0500, Barry Margolin wrote:
    > In article ,
    > Jorgen Grahn wrote:
    >
    >> DNS was just an example; the real problem (which I hesitate to go into
    >> details with) concerns another UDP-based protocol, and two load
    >> generators. One of them assembles its own IP packets (and clears DF
    >> and sets id), but the other one uses sockets and displays the
    >> behaviour described above.

    >
    > So when you said the other packets were from netcat, you really meant
    > they were from this application that uses raw sockets? It would have
    > been much clearer if you'd said this the first time.


    No, sorry for being unclear. The DNS-and-netcat example was real,
    and taken during an attempt to investigate the issue in a cleaner
    network. I was hoping to see netcat's packets with ID clear, but
    instead I stumbled on these DNS packets.

    The best thing for me to do is what someone suggested: read the
    Linux source. When/if I can get the time to do that, I will
    report what I find with respect to the Identification field.

    BR,
    /Jörgen

    --
    // Jorgen Grahn \X/ snipabacken.se> R'lyeh wgah'nagl fhtagn!

  9. Re: IP Identification field is zero - why?

    Barry Margolin wrote:
    > What's apparently going on is that the OS defaults to using PMTU
    > Discovery, so it sets DF and doesn't fill in the ID. Any apps that
    > use normal TCP or UDP sockets are affected by this. Apps that use
    > raw sockets and fill in the header themselves are not, and can do
    > whatever they want.


    From the manpage for "udp" on one of my "linux" systems:

    By default Linux UDP does path MTU (Maximum Transmission Unit)
    discov- ery. This means the kernel will keep track of the MTU
    to a specific target IP address and return EMSGSIZE when a UDP
    packet write exceeds it. When this happens the application
    should decrease the packet size. Path MTU discovery can be
    also turned off using the IP_MTU_DISCOVER socket option or the
    ip_no_pmtu_disc sysctl, see ip(7) for details. When turned off
    UDP will fragment outgoing UDP packets that exceed the
    interface MTU. However disabling it is not recommended for
    performance and reliability reasons.

    Manpages in linux (networking) can sometimes be rather out of date,
    but the above seems consistent with what has been said thusfar.

    rick jones

    setting IP ID to zero when DF is set has never sat well with me

    --
    Wisdom Teeth are impacted, people are affected by the effects of events.
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  10. Re: IP Identification field is zero - why?

    On Jan 11, 10:18 am, Rick Jones wrote:

    > By default Linux UDP does path MTU (Maximum Transmission Unit)
    > discov- ery. This means the kernel will keep track of the MTU
    > to a specific target IP address and return EMSGSIZE when a UDP
    > packet write exceeds it. When this happens the application
    > should decrease the packet size. Path MTU discovery can be
    > also turned off using the IP_MTU_DISCOVER socket option or the
    > ip_no_pmtu_disc sysctl, see ip(7) for details. When turned off
    > UDP will fragment outgoing UDP packets that exceed the
    > interface MTU. However disabling it is not recommended for
    > performance and reliability reasons.


    Wow, you learn something new every day. This seems like horribly
    broken behavior to me.

    The kernel is welcome to use path MTU discovery. But it should not, by
    default, assume that every program that uses UDP can deal with a
    reduction in the effective MTU. I was always under the impression that
    it would fragment by default.

    DS


  11. Re: IP Identification field is zero - why?

    David Schwartz writes:

    >On Jan 11, 10:18 am, Rick Jones wrote:


    >> By default Linux UDP does path MTU (Maximum Transmission Unit)
    >> discov- ery. This means the kernel will keep track of the MTU
    >> to a specific target IP address and return EMSGSIZE when a UDP
    >> packet write exceeds it. When this happens the application
    >> should decrease the packet size. Path MTU discovery can be
    >> also turned off using the IP_MTU_DISCOVER socket option or the
    >> ip_no_pmtu_disc sysctl, see ip(7) for details. When turned off
    >> UDP will fragment outgoing UDP packets that exceed the
    >> interface MTU. However disabling it is not recommended for
    >> performance and reliability reasons.


    >Wow, you learn something new every day. This seems like horribly
    >broken behavior to me.


    >The kernel is welcome to use path MTU discovery. But it should not, by
    >default, assume that every program that uses UDP can deal with a
    >reduction in the effective MTU. I was always under the impression that
    >it would fragment by default.


    And it should; I find it very strange that path MTU discover is made
    visible at the application layer. That must break an unmentionable
    number of applications.

    The application should be aware that UDP packets can be dropped for
    reasons of MTU discovery and so it should be prepared to retransmit
    the same packet (which the kernel would fragment properly the
    second time round)

    Casper
    --
    Expressed in this posting are my opinions. They are in no way related
    to opinions held by my employer, Sun Microsystems.
    Statements on Sun products included here are not gospel and may
    be fiction rather than truth.

  12. Re: IP Identification field is zero - why?

    On Tue, 15 Jan 2008 04:02:51 -0800, David Schwartz wrote:

    > On Jan 11, 10:18 am, Rick Jones wrote:
    >
    >> By default Linux UDP does path MTU (Maximum Transmission Unit)
    >> discov- ery. This means the kernel will keep track of the MTU
    >> to a specific target IP address and return EMSGSIZE when a UDP
    >> packet write exceeds it. When this happens the application
    >> should decrease the packet size. Path MTU discovery can be also
    >> turned off using the IP_MTU_DISCOVER socket option or the
    >> ip_no_pmtu_disc sysctl, see ip(7) for details. When turned off
    >> UDP will fragment outgoing UDP packets that exceed the interface
    >> MTU. However disabling it is not recommended for performance
    >> and reliability reasons.

    >
    > Wow, you learn something new every day. This seems like horribly broken
    > behavior to me.


    It's not, according to that man page:
    ] ip_no_pmtu_disc (Boolean; default: disabled)
    ] If enabled, don’t do Path MTU Discovery for TCP sockets by
    ^^^^^^^^^^^^^^^
    ] default. Path MTU discovery may fail if misconfigured firewalls
    ] (that drop all ICMP packets) or misconfigured interfaces (e.g., a
    ] point-to-point link where the both ends don’t agree on the
    ] MTU) are on the path. It is better to fix the broken routers on the
    ] path than to turn off Path MTU Discovery globally, because
    ] not doing it incurs a high cost to the network.


    > The kernel is welcome to use path MTU discovery. But it should not, by
    > default, assume that every program that uses UDP can deal with a
    > reduction in the effective MTU. I was always under the impression that
    > it would fragment by default.


    Which, I think, is exactly what happens under Linux.

    M4

  13. Re: IP Identification field is zero - why?

    It could be that the manpage is incomplete. One possible test I
    suppose would be to run a netperf UDP_STREAM test across a dumbell
    network and see if netperf starts spitting-out errors.

    rick jones
    --
    portable adj, code that compiles under more than one compiler
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  14. Re: IP Identification field is zero - why?

    On Jan 15, 5:56 am, Martijn Lievaart wrote:

    > Which, I think, is exactly what happens under Linux.


    Yep, turns out I'm not crazy. Or, at least, not about this.

    DS

  15. Re: IP Identification field is zero - why?

    On Fri, 11 Jan 2008 18:18:58 +0000 (UTC), Rick Jones wrote:
    > Barry Margolin wrote:
    >> What's apparently going on is that the OS defaults to using PMTU
    >> Discovery, so it sets DF and doesn't fill in the ID. Any apps that
    >> use normal TCP or UDP sockets are affected by this. Apps that use
    >> raw sockets and fill in the header themselves are not, and can do
    >> whatever they want.

    >
    > From the manpage for "udp" on one of my "linux" systems:
    >
    > By default Linux UDP does path MTU (Maximum Transmission Unit)
    > discov- ery. This means the kernel will keep track of the MTU
    > to a specific target IP address and return EMSGSIZE when a UDP
    > packet write exceeds it. When this happens the application
    > should decrease the packet size. [---]


    > Manpages in linux (networking) can sometimes be rather out of date,
    > but the above seems consistent with what has been said thusfar.


    Indeed -- on my system udp(7), dated 1998, simply suggests that you
    should use UDP Path MTU discovery, and ip(7) from 2001 seems to
    indicate that the application, not the administrator, needs to enable
    it:

    The system-wide default is controlled by the ip_no_pmtu_disc
    sysctl for SOCK_STREAM sockets, and disabled on all others.

    I'm reading the networking sources for Linux 2.6.18.8, and it's
    unclear at a glance if this is true or not ... but I can't see
    anything in net/ipv4/af_inet.c that suggests that STREAM/DGRAM makes
    any difference at all.

    [Side note: it was a surprise to me, and I bet many others, that Path
    MTU discovery may prevent you from sending long UDP datagrams. It is
    not in Stevens' books as far as I recall. I always thought of it as
    just another TCP tweak ...]

    > setting IP ID to zero when DF is set has never sat well with me


    And the Linux people seem to do that.

    There's ip_select_ident() in include/net/ip.h. If DF is clear, it
    uses some route-specific and semi-cryptographic ways to calculate the
    IP ID.

    For packets with DF *set*, however, it appears to me that if the socket
    is connect()ed, a per-socket counter is used as IP ID, otherwise it is
    set to zero.

    And that might be what makes a difference in my case: one application
    connect()s its UDP socket, the other doesn't.

    Tracing "netcat -u localhost discard" reveals that it does call
    connect(), and I bet named doesn't. Hell, it was probably its
    listening socket on port 53 I saw in action.

    I think this answers my original question. Thanks!

    /Jorgen

    --
    // Jorgen Grahn \X/ snipabacken.se> R'lyeh wgah'nagl fhtagn!

  16. Re: IP Identification field is zero - why?

    On Tue, 15 Jan 2008 14:56:27 +0100, Martijn Lievaart wrote:
    > On Tue, 15 Jan 2008 04:02:51 -0800, David Schwartz wrote:
    >
    >> On Jan 11, 10:18 am, Rick Jones wrote:
    >>
    >>> By default Linux UDP does path MTU (Maximum Transmission Unit)
    >>> discov- ery. This means the kernel will keep track of the MTU
    >>> to a specific target IP address and return EMSGSIZE when a UDP
    >>> packet write exceeds it. When this happens the application
    >>> should decrease the packet size. Path MTU discovery can be also
    >>> turned off using the IP_MTU_DISCOVER socket option or the
    >>> ip_no_pmtu_disc sysctl, see ip(7) for details. When turned off
    >>> UDP will fragment outgoing UDP packets that exceed the interface
    >>> MTU. However disabling it is not recommended for performance
    >>> and reliability reasons.

    >>
    >> Wow, you learn something new every day. This seems like horribly broken
    >> behavior to me.

    >
    > It's not, according to that man page:
    > ] ip_no_pmtu_disc (Boolean; default: disabled)
    > ] If enabled, don?t do Path MTU Discovery for TCP sockets by
    > ^^^^^^^^^^^^^^^
    > ] default. Path MTU discovery may fail if misconfigured firewalls
    > ] (that drop all ICMP packets) or misconfigured interfaces (e.g., a
    > ] point-to-point link where the both ends don?t agree on the
    > ] MTU) are on the path. It is better to fix the broken routers on the
    > ] path than to turn off Path MTU Discovery globally, because
    > ] not doing it incurs a high cost to the network.


    You underlined the phrase "for TCP sockets", but I'm sitting with a
    Linux 2.6.18 kernel right now, and I see it directly affecting UDP
    sockets, too -- if PMTU is enabled (the disabling is disabled, to use
    the man page's terms), DF gets set in the UDP datagram's IP header.

    >> The kernel is welcome to use path MTU discovery. But it should not, by
    >> default, assume that every program that uses UDP can deal with a
    >> reduction in the effective MTU. I was always under the impression that
    >> it would fragment by default.

    >
    > Which, I think, is exactly what happens under Linux.


    Fragment by default, you mean. No -- to me it seem like you're likely
    to get DF set, and lose the datagram rather than have it fragmented
    for you by the stack. Depending on if the socket is connected or not --
    I think we discussed that part elsewhere in the thread, back in January.

    (I'm not trying to resurrect this old thread, but I felt I had to
    point out what seems like an error in the documentation quoted above.)

    /Jorgen

    --
    // Jorgen Grahn \X/ snipabacken.se> R'lyeh wgah'nagl fhtagn!

+ Reply to Thread