Slow response with multiple KDCs - Kerberos

This is a discussion on Slow response with multiple KDCs - Kerberos ; My Kerberos admins recently changed all the KDCs in our realm and started distributing a new standard krb5.conf file. Now... instead of taking sec to get a password prompt from "kinit", it takes 40-50 secs. The old file lists 6 ...

+ Reply to Thread
Results 1 to 3 of 3

Thread: Slow response with multiple KDCs

  1. Slow response with multiple KDCs

    My Kerberos admins recently changed all the KDCs in our realm and started
    distributing a new standard krb5.conf file. Now... instead of taking < 1
    sec to get a password prompt from "kinit", it takes 40-50 secs.

    The old file lists 6 KDCs using IP addresses instead of hostnames. The
    new file lists 10 KDCs using hostnames... so obviously it has something to
    do with DNS.

    Using krb5-1.4.4, and running strace on kinit, it appears to be doing
    multiple DNS requests for EVERY KDC listed in the krb5.conf file. This
    seems to be why it takes so long. In fact... it looks like for 10 KDCs,
    "kinit" ends up making 316 DNS requests.

    Why does it make so many requests? Why does it make DNS requests for ALL
    the KDCs even if the first one returned results. Is this a function of
    the kerberos library itself or something else?

    I've tried setting the following in krb5.conf, but they don't seem to make
    a difference:

    dns_lookup_realm = false
    dns_lookup_kdc = false
    dns_fallback = false

    I've also tried 1.4.3 compiled WITHOUT --disable-dns-for-realm and 1.4.4
    compiled WITH --disable-dns-for-realm, but that didn't make a difference
    either.

    PS. The reason I'm concerned about this is because I need to build a new
    krb5-1.4.4 package to be distributed to all our developers that contains
    the new krb5.conf file. I don't want to get a bunch of users telling me
    how slow kinit has become.
    ________________________________________________
    Kerberos mailing list Kerberos@mit.edu
    https://mailman.mit.edu/mailman/listinfo/kerberos


  2. Re: Slow response with multiple KDCs

    On Sep 18, 2006, at 17:15, petesea@bigfoot.com wrote:
    > My Kerberos admins recently changed all the KDCs in our realm and
    > started
    > distributing a new standard krb5.conf file. Now... instead of
    > taking < 1
    > sec to get a password prompt from "kinit", it takes 40-50 secs.
    >
    > The old file lists 6 KDCs using IP addresses instead of hostnames.
    > The
    > new file lists 10 KDCs using hostnames... so obviously it has
    > something to
    > do with DNS.
    >
    > Using krb5-1.4.4, and running strace on kinit, it appears to be doing
    > multiple DNS requests for EVERY KDC listed in the krb5.conf file.
    > This
    > seems to be why it takes so long. In fact... it looks like for 10
    > KDCs,
    > "kinit" ends up making 316 DNS requests.


    That's... fascinating....

    > Why does it make so many requests? Why does it make DNS requests
    > for ALL
    > the KDCs even if the first one returned results. Is this a
    > function of
    > the kerberos library itself or something else?


    You (understandably) didn't include the 316 requests in your message,
    but I'll take a shot:

    It's probably doubling queries, once in the "look for UDP-accessible
    KDCs" path and once in the "look for TCP-accessible KDCs" path (which
    don't have to return the same results if the data is coming from DNS
    SRV records, but SRV-vs-krb5.conf isn't known at that level).

    It may be doubling the address queries per name, one A for IPv4
    addresses and one AAAA for IPv6 addresses. (An interesting, um,
    quirk, that I've seen in one implementation: If you don't have an
    AAAA record for foo.example.com, and you don't put a trailing dot on
    the end of the name you look up, it may then look up
    foo.example.com.example.com, even though you have other data -- like
    an A record -- for foo.example.com. So we could be looking at a
    multiplier of three here, or more if you use a domain search list.)

    At one point, the library may try to look up the "master KDC" (so if
    you get an "incorrect password" type result but were talking to a
    slave KDC that may not have your password change from 30 seconds ago,
    it then tries a KDC that would have it); offhand, I'm not sure how
    many DNS queries that's likely to generate. Here at MIT, we've got a
    SRV record for _kerberos_master._udp.athena.mit.edu listing one host,
    so we do get one additional lookup for that name. (Oddly, we don't
    get two, for A and AAAA; I should look at why that is.)


    Try adding "." to the end of your hostnames in krb5.conf, assuming
    they're all FQDNs, and see if that helps.


    Our library is currently set up to collect all the addresses, and
    then start trying to contact servers. Integrating these loops better
    has been on the to-do list for a while, but will be kind of ugly.
    (We don't want to try one, time out, give up, look up the next
    address, try it, etc. We want to fail over to the next KDC fairly
    quickly, but we also want to keep listening for responses to earlier
    attempts, in case it's just a slow KDC or slow network.)

    We do generally assume that local caching will make the occasional
    repeated DNS query fast. But 316? Ow.

    If you want to send me your config file and strace output (off-list,
    I suggest, and run with -s99 or something like that to record the
    full query packet), I can take a closer look.

    > I've tried setting the following in krb5.conf, but they don't seem
    > to make
    > a difference:
    >
    > dns_lookup_realm = false


    That controls the use of TXT records for figuring out the realm of a
    host; normally disabled.

    > dns_lookup_kdc = false


    That controls the use of SRV records for finding the KDCs given a
    realm name; normally enabled, but having the data in krb5.conf should
    prevent it from getting used for the regular KDC lookup. Lookup of
    _kerberos-master._udp.REALM.NAME (and likewise for _tcp) would
    probably still happen unless you use this option, or specify
    "master_kdc=..." in krb5.conf.

    > dns_fallback = false


    This (poorly-named?) option controls both of the above explicit uses
    of DNS, letting you switch both on or off with one option. It
    doesn't control the implicit uses of DNS via getaddrinfo or
    gethostbyname.

    > I've also tried 1.4.3 compiled WITHOUT --disable-dns-for-realm and
    > 1.4.4
    > compiled WITH --disable-dns-for-realm, but that didn't make a
    > difference
    > either.


    Configuration-time options corresponding to the above control for
    TXT queries.

    > PS. The reason I'm concerned about this is because I need to build
    > a new
    > krb5-1.4.4 package to be distributed to all our developers that
    > contains
    > the new krb5.conf file. I don't want to get a bunch of users
    > telling me
    > how slow kinit has become.


    Perfectly understandable....

    Ken
    ________________________________________________
    Kerberos mailing list Kerberos@mit.edu
    https://mailman.mit.edu/mailman/listinfo/kerberos


  3. Re: Slow response with multiple KDCs

    Ken Raeburn wrote:
    > At one point, the library may try to look up the "master KDC" (so if
    > you get an "incorrect password" type result but were talking to a
    > slave KDC that may not have your password change from 30 seconds ago,
    > it then tries a KDC that would have it); offhand, I'm not sure how
    > many DNS queries that's likely to generate. Here at MIT, we've got a
    > SRV record for _kerberos_master._udp.athena.mit.edu listing one host,
    > so we do get one additional lookup for that name. (Oddly, we don't
    > get two, for A and AAAA; I should look at why that is.)
    >


    The DNS will always return all matches to the query including queries
    for SRV requests. When you do the additional lookup for the name,
    getaddrinfo() I assume, the lookup returns all AAAA and A addresses
    unless you have configured the call to only look up one or the other.
    There is no need for a separate lookup. getaddrinfo() returns ALL
    addresses that matches the query.

    Danny
    ________________________________________________
    Kerberos mailing list Kerberos@mit.edu
    https://mailman.mit.edu/mailman/listinfo/kerberos


+ Reply to Thread