A script for scanning the internet for one particular index.html? - Networking

This is a discussion on A script for scanning the internet for one particular index.html? - Networking ; Hello, I want to know how I make a script for scanning the entire internet (127.0.0.1 - 255.255.255.255) for one unique index.html file (I got the index.html file I'm looking for) and then whenever it finds a match to the ...

+ Reply to Thread
Results 1 to 9 of 9

Thread: A script for scanning the internet for one particular index.html?

  1. A script for scanning the internet for one particular index.html?

    Hello,

    I want to know how I make a script for scanning the entire internet
    (127.0.0.1 - 255.255.255.255)
    for one unique index.html file (I got the index.html file I'm looking for)
    and then whenever it finds a
    match to the index.html file saving the results into a textfile, all
    scanning done using standar http port 80.

    So the end result will be a text file contaning the ip addresses found
    matching the index.html I got.
    I'm sure it can be done, I'm just not that good at linux scripting.


    --
    Tee Jay/Sapphire
    (support@telenor.com)




  2. Re: A script for scanning the internet for one particular index.html?

    > I want to know how I make a script for scanning the entire internet
    > (127.0.0.1 - 255.255.255.255)
    > for one unique index.html file (I got the index.html file I'm looking for)


    use wget to fetch the index.html into a file then grep -i its content.

    --
    @~@ Might, Courage, Vision, SINCERITY.
    / v \ Simplicity is Beauty! May the Force and Farce be with you!
    /( _ )\ (Xubuntu 7.04) Linux 2.6.22.6
    ^ ^ 22:58:01 up 1 day 7:14 1 user load average: 0.00 0.00 0.00
    news://news.3home.net news://news.hkpcug.org news://news.newsgroup.com.hk

  3. Re: A script for scanning the internet for one particular index.html?

    In comp.os.linux.networking Man-wai Chang ToDie :
    >> I want to know how I make a script for scanning the entire internet
    >> (127.0.0.1 - 255.255.255.255)
    >> for one unique index.html file (I got the index.html file I'm looking for)


    > use wget to fetch the index.html into a file then grep -i its content.


    I'd suggest to learn how to use a search engine and stop
    multi-posting...

    --
    Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
    mail: echo zvpunry@urvzvat.qr | perl -pe 'y/a-z/n-za-m/'
    #bofh excuse 157: Incorrect time synchronization

  4. Re: A script for scanning the internet for one particularindex.html?

    Man-wai Chang ToDie wrote:
    >> I want to know how I make a script for scanning the entire internet
    >> (127.0.0.1 - 255.255.255.255)
    >> for one unique index.html file (I got the index.html file I'm looking for)

    >
    >use wget to fetch the index.html into a file then grep -i its content.


    How many decades will it take a PC to search half the Internet that way?

    --
    Floyd L. Davidson
    Ukpeagvik (Barrow, Alaska) floyd@apaflo.com

  5. Re: A script for scanning the internet for one particular index.html?

    Floyd L. Davidson wrote:
    > Man-wai Chang ToDie wrote:
    >>> I want to know how I make a script for scanning the entire internet
    >>> (127.0.0.1 - 255.255.255.255)
    >>> for one unique index.html file (I got the index.html file I'm looking for)

    >>
    >> use wget to fetch the index.html into a file then grep -i its content.

    >
    > How many decades will it take a PC to search half the Internet that way?


    Not quite half.

    Routable addresses are (theoretically):
    the range 1.0.0.0/8 - 223.0.0.0/8
    minus:
    10.0.0.0/8
    127.0.0.0/8
    169.254.0.0/16
    172.16.0.0/12
    192.0.2.0/24
    192.168.0.0/16

    In the meantime, however, this request seems an awful lot like what bots
    do to look for new servers they can infect.

  6. Re: A script for scanning the internet for one particular index.html?

    Hi Allen and the group -

    On Sat, 22 Sep 2007 12:11:23 -0500, Allen Kistler
    wrote:

    >Floyd L. Davidson wrote:
    >> Man-wai Chang ToDie wrote:
    >>>> I want to know how I make a script for scanning the entire internet
    >>>> (127.0.0.1 - 255.255.255.255)
    >>>> for one unique index.html file (I got the index.html file I'm looking for)
    >>>
    >>> use wget to fetch the index.html into a file then grep -i its content.

    >>
    >> How many decades will it take a PC to search half the Internet that way?

    >
    >Not quite half.
    >
    >Routable addresses are (theoretically):
    > the range 1.0.0.0/8 - 223.0.0.0/8
    >minus:
    > 10.0.0.0/8
    > 127.0.0.0/8
    > 169.254.0.0/16
    > 172.16.0.0/12
    > 192.0.2.0/24
    > 192.168.0.0/16
    >
    >In the meantime, however, this request seems an awful lot like what bots
    >do to look for new servers they can infect.


    That comment brings up another point. With the proliferation of
    virtual named hosts, going by IP address is going to miss a lot. I
    have multiple domains on my server, and in some cases multiple hosts
    per domain. And the default container (which is what access by IP
    address without a host name or with the IP address as host name would
    get) does not contain any of those sites.

    --
    Ken
    http://www.kensims.net/

  7. Re: A script for scanning the internet for one particular index.html?


    "Allen Kistler" wrote in message
    news:%0cJi.10362$924.3565@newssvr23.news.prodigy.n et...
    > Floyd L. Davidson wrote:
    > > Man-wai Chang ToDie wrote:
    > >>> I want to know how I make a script for scanning the entire internet
    > >>> (127.0.0.1 - 255.255.255.255)
    > >>> for one unique index.html file (I got the index.html file I'm looking

    for)
    > >>
    > >> use wget to fetch the index.html into a file then grep -i its content.

    > >
    > > How many decades will it take a PC to search half the Internet that way?

    >
    > Not quite half.
    >
    > Routable addresses are (theoretically):
    > the range 1.0.0.0/8 - 223.0.0.0/8
    > minus:
    > 10.0.0.0/8
    > 127.0.0.0/8
    > 169.254.0.0/16
    > 172.16.0.0/12
    > 192.0.2.0/24
    > 192.168.0.0/16
    >
    > In the meantime, however, this request seems an awful lot like what bots
    > do to look for new servers they can infect.


    Ok, that I can ensure it's not for. It's actually more a hypothetic question
    rather than a request. I just wanna know stuff like that


    --
    Tee Jay




  8. Re: A script for scanning the internet for one particularindex.html?

    On Sat, 22 Sep 2007 20:32:59 +0200, Tee Jay rearranged some electrons to
    say:

    > "Allen Kistler" wrote in message
    > news:%0cJi.10362$924.3565@newssvr23.news.prodigy.n et...
    >> Floyd L. Davidson wrote:
    >> > Man-wai Chang ToDie wrote:
    >> >>> I want to know how I make a script for scanning the entire internet
    >> >>> (127.0.0.1 - 255.255.255.255)
    >> >>> for one unique index.html file (I got the index.html file I'm
    >> >>> looking

    > for)
    >> >>
    >> >> use wget to fetch the index.html into a file then grep -i its
    >> >> content.
    >> >
    >> > How many decades will it take a PC to search half the Internet that
    >> > way?

    >>
    >> Not quite half.
    >>
    >> Routable addresses are (theoretically):
    >> the range 1.0.0.0/8 - 223.0.0.0/8
    >> minus:
    >> 10.0.0.0/8
    >> 127.0.0.0/8
    >> 169.254.0.0/16
    >> 172.16.0.0/12
    >> 192.0.2.0/24
    >> 192.168.0.0/16
    >>
    >> In the meantime, however, this request seems an awful lot like what
    >> bots do to look for new servers they can infect.

    >
    > Ok, that I can ensure it's not for. It's actually more a hypothetic
    > question rather than a request. I just wanna know stuff like that



    "I want to know how I make a script for scanning the entire internet
    for one unique index.html file (I got the index.html file I'm looking for)
    and then whenever it finds a match to the index.html file saving the
    results into a textfile, all scanning done using standar http port 80.
    So the end result will be a text file contaning the ip addresses found
    matching the index.html I got"

    Doesn't sound like a hypothetical to me.. sounds like you are doing
    something specific (and, suspicious).





  9. Re: A script for scanning the internet for one particular index.html?

    Tee Jay wrote:
    > I want to know how I make a script for scanning the entire internet
    > (127.0.0.1 - 255.255.255.255)


    That is not the entire internet, as already mentioned.

    An alternative approach might be to find some sort of fingerprint (unique
    segment) in the index file, and utilize a search engine (such as google) to try
    and match the fingerprint. (The search engine may provide an API that can be
    utilized.)

    The search will of course be limited to pages indexed by the search engine,
    but you will need a lot of computing power and sufficient bandwidth if you
    want to outperform google. (There are billions of pages to process.)

    Regards,

    Mark.

    --
    Mark Hobley
    393 Quinton Road West
    QUINTON
    Birmingham
    B32 1QE

    Email: markhobley at hotpop dot donottypethisbit com

    http://markhobley.yi.org/


+ Reply to Thread