htdig on mandriva 2008.0 "host not found" - Networking

This is a discussion on htdig on mandriva 2008.0 "host not found" - Networking ; This is driving me crazy: I've installed htdig, htdig-web and apache on 2008.0 [Mandriva seperates the cgi stuff into a seperate -web rpm]. I've configured apache to allow indexing of directories and run htdig with the following result: # su ...

+ Reply to Thread
Results 1 to 11 of 11

Thread: htdig on mandriva 2008.0 "host not found"

  1. htdig on mandriva 2008.0 "host not found"

    This is driving me crazy:

    I've installed htdig, htdig-web and apache on 2008.0 [Mandriva seperates
    the cgi stuff into a seperate -web rpm].

    I've configured apache to allow indexing of directories and run htdig
    with the following result:

    # su - htdig -c "/usr/bin/rundig -a -vvv"
    ht://dig Start Time: Sun Feb 24 13:55:34 2008
    1:1:http://myserver/
    New server: myserver, 80
    - Persistent connections: enabled
    - HEAD before GET: enabled
    - Timeout: 30
    - Connection space: 0
    - Max Documents: -1
    - TCP retries: 1
    - TCP wait time: 5
    - Accept-Language:
    Trying to retrieve robots.txt file
    Trying local files
    tried local file /var/www/html/robots.txt
    pushed
    pick: myserver, # servers = 1
    > myserver supports HTTP persistent connections (infinite)

    0:2:0:http://myserver/: Trying local files
    tried local file /var/www/html/index.html
    Local retrieval failed, trying HTTP
    host not found
    pick: myserver, # servers = 1
    > myserver supports HTTP persistent connections (infinite)

    ht://dig End Time: Sun Feb 24 13:55:34 2008
    Deleted, not found: ID: 2 URL: http://myserver/
    #

    The first part of this I take to mean that the server has actually been
    found, but later I get "host not found". I've tried:

    # su - htdig -c "host myserver"
    myserver.mydomain has address 192.168.1.28

    and have also changed start_url in /etc/htdig/htdig.conf to localhost,
    127.0.0.1 and 192.168.1.28 and still get the above result. I've also
    tried commenting out local_urls and local_user_urls, to force htdig to
    use http - no change, except for missing out the attempt at using the
    local filesystem.

    Apache's logs show no attempts at connection.

    Any ideas?

    Sarah

  2. Re: htdig on mandriva 2008.0 "host not found"

    On Sun, 24 Feb 2008 09:11:00 -0500, Sarah wrote:

    > # su - htdig -c "host myserver"
    > myserver.mydomain has address 192.168.1.28
    >
    > and have also changed start_url in /etc/htdig/htdig.conf to localhost,
    > 127.0.0.1 and 192.168.1.28 and still get the above result. I've also


    Have you tried using myserver.mydomain in htdig.conf? I've just done
    some limited testing, and it appears you cannot specify an ip address
    in htdig.conf, and must specify a fully qualified name, that can be
    resolved by dns lookup.

    Regards, Dave Hodgins

    --
    Change nomail.afraid.org to ody.ca to reply by email.
    (nomail.afraid.org has been set up specifically for
    use in usenet. Feel free to use it yourself.)

  3. Re: htdig on mandriva 2008.0 "host not found"

    David W. Hodgins wrote:
    > On Sun, 24 Feb 2008 09:11:00 -0500, Sarah wrote:
    >
    >> # su - htdig -c "host myserver"
    >> myserver.mydomain has address 192.168.1.28
    >>
    >> and have also changed start_url in /etc/htdig/htdig.conf to localhost,
    >> 127.0.0.1 and 192.168.1.28 and still get the above result. I've also

    >
    > Have you tried using myserver.mydomain in htdig.conf? I've just done
    > some limited testing, and it appears you cannot specify an ip address
    > in htdig.conf, and must specify a fully qualified name, that can be
    > resolved by dns lookup.
    >
    > Regards, Dave Hodgins


    I've tried it now - no better :-(

    I really don't understand it - how can it have problems finding the host
    even when I've given it the IP address!!! And if "host myserver" works
    in the same environment then I can only assume that htdig isn't using
    the standard libraries for resolving the hostname.

    Thanks for the suggestion.

    Sarah

  4. Re: htdig on mandriva 2008.0 "host not found"

    Sarah wrote:
    > This is driving me crazy:


    [Snip htdig failing utterly to index my site!]

    > Any ideas?
    >
    > Sarah


    Looking through Freshmeat I have found two other apps that have stock
    Mandriva rpms. Both have been updated within the last year or so.

    Hyperestraier:

    Swish-e:

    Anyone with experience of these?

    Sarah

  5. Re: htdig on mandriva 2008.0 "host not found"

    On Sun, 24 Feb 2008 18:08:43 -0500, Sarah wrote:

    > I really don't understand it - how can it have problems finding the host
    > even when I've given it the IP address!!! And if "host myserver" works
    > in the same environment then I can only assume that htdig isn't using
    > the standard libraries for resolving the hostname.


    htdig and htsearch are old, unmaintained packages. I looked through them
    quite a bit, some time ago, when debugging the creation of kde helpcenter
    indexes, which also uses htdig and htsearch.

    Not working with an ip address makes sense, if it's doing a lookup of
    get ip from hostname. I don't remember exactly what it does, and no
    longer have the source installed, to look at.

    Why it's failing when the host command is working, I don't know. I do
    know there are several methods for resolving the ip of a hostname.

    For example, I'm running a local nameserver, with hodgins.homeip.net
    defined as 127.0.0.1. The nameserver on 127.0.0.1 is specified as
    the first server to search, by listing it in /etc/resovconf/resolv.conf/head.
    The dynamically assigned ip is also updated at dyndns.org.

    How I lookup the address will determine the result I get.

    # dig +trace hodgins.homeip.net
    hodgins.homeip.net. 60 IN A 216.240.2.97
    ;; Received 232 bytes from 203.62.195.75#53(ns5.dyndns.org) in 428 ms
    # host hodgins.homeip.net
    hodgins.homeip.net has address 127.0.0.1

    I also have an ad sever, "mserv.zap2it.com", only in my /etc/hosts file,
    directed to 127.0.0.1

    The host command ignores the /etc/hosts file.
    # host mserv.zap2it.com
    mserv.zap2it.com has address 206.18.98.140
    yet traceroute, and my web browser use it.
    # traceroute mserv.zap2it.com
    traceroute to mserv.zap2it.com (127.0.0.1), 30 hops max, 38 byte packets
    1 hodgins (127.0.0.1) 0.116 ms 0.088 ms 0.051 ms

    How are you getting myserver.mydomain resolved to an ip address?

    Regards, Dave Hodgins

    --
    Change nomail.afraid.org to ody.ca to reply by email.
    (nomail.afraid.org has been set up specifically for
    use in usenet. Feel free to use it yourself.)

  6. Re: htdig on mandriva 2008.0 "host not found"

    On Mon, 25 Feb 2008 13:21:22 -0500, Sarah wrote:

    > Looking through Freshmeat I have found two other apps that have stock
    > Mandriva rpms. Both have been updated within the last year or so.
    > Hyperestraier:
    > Swish-e:
    > Anyone with experience of these?


    I've installed both. They both require some work to get the indexing
    configured correctly.

    In both cases, I have some problems with the search results, that I
    haven't sorted out yet.

    With Hyperestraier, using http://hodgins.homeip.net/cgi-bin/estseek.cgi
    to search for the word "Works", the resulting file names, linked to by
    the search results are in the form file:///var/www/html/mimetex.html,
    which of course only works if you are searching, from the localhost, and
    most browsers will give you security warnings, when you try to access
    the local file.

    With swish, using http://hodgins.homeip.net/cgi-bin/swish.cgi to search
    for the word "Works", the resulting file names, linked to by the search
    results are in the form http://hodgins.homeip.net/var/www/html/mimetex.html,
    which doesn't work, as the /var/www/html should not be included.

    Both packages are designed to build indexes that can be searched from
    the command line, as well as via cgi scripts. As above, this does cause
    some problems, since the proper paths to the files, varies depending on
    how you are accessing them. The correct link in the above examples
    shoule be http://hodgins.homeip.net/mimetex.html.

    I expect, If I dug through the documentation, these can both be fixed,
    but it isn't obvious how, from a quick skim of the documentation.

    I'd be inclined to sort out the dns problems with htdig/search, as once
    that's solved, the both "just work", and have for a long time.

    Regards, Dave Hodgins

    --
    Change nomail.afraid.org to ody.ca to reply by email.
    (nomail.afraid.org has been set up specifically for
    use in usenet. Feel free to use it yourself.)

  7. Re: htdig on mandriva 2008.0 "host not found"

    David W. Hodgins wrote:

    [some excellent stuff on his experiences with swish-e and hyperestraier]

    > I'd be inclined to sort out the dns problems with htdig/search, as once
    > that's solved, the both "just work", and have for a long time.
    >
    > Regards, Dave Hodgins


    I'm going to press on with swish-e for a while - I get an impression
    that it might be a bit easier than hyperestraier. In the long run if
    htdig isn't maintained it will just get more troublesome. I'll report
    back if I get any of them working.

    Thanks again,

    Sarah

  8. Re: htdig on mandriva 2008.0 "host not found"

    David W. Hodgins wrote:
    > On Sun, 24 Feb 2008 18:08:43 -0500, Sarah wrote:
    >
    >> I really don't understand it - how can it have problems finding the host
    >> even when I've given it the IP address!!! And if "host myserver" works
    >> in the same environment then I can only assume that htdig isn't using
    >> the standard libraries for resolving the hostname.


    > htdig and htsearch are old, unmaintained packages.


    I've noticed. It even appears to need a patch for glibc >= 2.2.2, which
    came out on 2001-02-16!

    > How I lookup the address will determine the result I get.


    Now that is useful info.

    > How are you getting myserver.mydomain resolved to an ip address?


    I'm using dnsmasq (thanks Simon Kelley, great package!) on another PC.
    It crosses my mind that this might be the problem, since this is
    presumably an unusual setup and Google has found no other similar reports.

    > Regards, Dave Hodgins


    Thanks very much for your input Dave. To be honest, the best option
    seems to be to find a better (or at least better maintained) indexing
    package. htdig seems to be abandonware. Odd, really, that it seems to be
    better known than the alternatives.

    Sarah

  9. Re: htdig on mandriva 2008.0 "host not found"

    Sarah wrote:
    > David W. Hodgins wrote:
    >
    > [some excellent stuff on his experiences with swish-e and hyperestraier]
    >
    >> I'd be inclined to sort out the dns problems with htdig/search, as once
    >> that's solved, the both "just work", and have for a long time.
    >>
    >> Regards, Dave Hodgins

    >
    > I'm going to press on with swish-e for a while - I get an impression
    > that it might be a bit easier than hyperestraier. In the long run if
    > htdig isn't maintained it will just get more troublesome. I'll report
    > back if I get any of them working.
    >
    > Thanks again,
    >
    > Sarah


    I've managed to get swish-e to work. The key to generating the correct
    urls in the search results is

    (1) Building the index with the following in the configuration file:

    ReplaceRules regex "|^/var/www/html/||"

    (2) Insert in the swish.cgi configuration file:

    prepend_path => 'http://mywebserver/'

    Alternatively you can do the same with just:

    ReplaceRules regex "|^/var/www/html/|http://mywebserver/|"


    Sarah

  10. mdns4_minimal added to /etc/nsswitch.conf (was Re: htdig on mandriva2008.0 "host not found"

    On Tue, 26 Feb 2008 17:50:31 -0500, Sarah wrote:

    > David W. Hodgins wrote:
    >> How are you getting myserver.mydomain resolved to an ip address?

    > I'm using dnsmasq (thanks Simon Kelley, great package!) on another PC.
    > It crosses my mind that this might be the problem, since this is
    > presumably an unusual setup and Google has found no other similar reports.


    I've just noticed that mdns4_minimal has been added to hosts line in
    /etc/nsswitch.conf and it appears to be intended to lookup names on
    windows domains, or via the avahi daemon.

    For anyone else, like me, who doesn't have a network, or doesn't use
    windows for domain resolution, remove it.

    Given the age of the htdig software, that is probably the problem with
    it failing to resolv the hostname.

    I'd set up a nameserver (aka bind), to resolve the hostname, rather
    then use anything using m$ methods. I expect that would solve the
    problem with htdig, if you choose to try it again.

    Regards, Dave Hodgins

    --
    Change nomail.afraid.org to ody.ca to reply by email.
    (nomail.afraid.org has been set up specifically for
    use in usenet. Feel free to use it yourself.)

  11. Re: htdig on mandriva 2008.0 "host not found"

    On Wed, 27 Feb 2008 18:24:25 -0500, Sarah wrote:

    > I've managed to get swish-e to work. The key to generating the correct
    > urls in the search results is
    > (1) Building the index with the following in the configuration file:
    > ReplaceRules regex "|^/var/www/html/||"
    > (2) Insert in the swish.cgi configuration file:
    > prepend_path => 'http://mywebserver/'
    > Alternatively you can do the same with just:
    > ReplaceRules regex "|^/var/www/html/|http://mywebserver/|"


    Thanks for the update.

    Regards, Dave Hodgins

    --
    Change nomail.afraid.org to ody.ca to reply by email.
    (nomail.afraid.org has been set up specifically for
    use in usenet. Feel free to use it yourself.)

+ Reply to Thread