Suggestions for my Linux app...? - Linux

This is a discussion on Suggestions for my Linux app...? - Linux ; I am trying to develop a (very simple!) application in C++ that will spider my site's web pages at pre-arranged times (the reasons for this scheduling is very politically driven, so please don't ask! My aim is a simple linked ...

+ Reply to Thread
Results 1 to 8 of 8

Thread: Suggestions for my Linux app...?

  1. Suggestions for my Linux app...?

    I am trying to develop a (very simple!) application in C++ that will
    spider my site's web pages at pre-arranged times (the reasons for this
    scheduling is very politically driven, so please don't ask!

    My aim is a simple linked list, as follows, using two web pages: (in
    pseudo code)

    spider web page 1
    work out when next look is scheduled
    - put a wait signal in the scheduling queue
    -when wait expires, spider web page 2
    work out when next look is scheduled after next re-look at page 1
    wait
    spider web page 1
    etc. etc.

    essentially this means that the queue will be
    webpage request
    wait
    webpage request
    wait etc.
    with the queue being populated with new commands to re-look at certain
    webpages after so many hours or days.

    Personally, I don't think that threading is needed in such a simple
    regime, but its simplicity is warning me that this is the wrong way to
    do this! Other people say that a spin-wait routine like that above is
    wasteful of CPU cycles.

    Does anyone have any comments or suggestions?

    TIA

    Trev


  2. Re: Suggestions for my Linux app...?

    Trev wrote:
    > I am trying to develop a (very simple!) application in C++ that will
    > spider my site's web pages at pre-arranged times (the reasons for this
    > scheduling is very politically driven, so please don't ask!
    >
    > My aim is a simple linked list, as follows, using two web pages: (in
    > pseudo code)
    >
    > spider web page 1
    > work out when next look is scheduled
    > - put a wait signal in the scheduling queue
    > -when wait expires, spider web page 2
    > work out when next look is scheduled after next re-look at page 1
    > wait
    > spider web page 1
    > etc. etc.
    >
    > essentially this means that the queue will be
    > webpage request
    > wait
    > webpage request
    > wait etc.
    > with the queue being populated with new commands to re-look at certain
    > webpages after so many hours or days.
    >
    > Personally, I don't think that threading is needed in such a simple
    > regime, but its simplicity is warning me that this is the wrong way to
    > do this! Other people say that a spin-wait routine like that above is
    > wasteful of CPU cycles.
    >
    > Does anyone have any comments or suggestions?


    Nothing you said above implies a spin-wait. But spin-wait would be bad.
    Use sleep() or select() to wait until the earliest expiration.

  3. Re: Suggestions for my Linux app...?


    Trev wrote:
    > I am trying to develop a (very simple!) application in C++ that will
    > spider my site's web pages at pre-arranged times (the reasons for this
    > scheduling is very politically driven, so please don't ask!
    >
    > My aim is a simple linked list, as follows, using two web pages: (in
    > pseudo code)
    >
    > spider web page 1
    > work out when next look is scheduled
    > - put a wait signal in the scheduling queue
    > -when wait expires, spider web page 2
    > work out when next look is scheduled after next re-look at page 1
    > wait
    > spider web page 1
    > etc. etc.
    >
    > essentially this means that the queue will be
    > webpage request
    > wait
    > webpage request
    > wait etc.
    > with the queue being populated with new commands to re-look at certain
    > webpages after so many hours or days.
    >
    > Personally, I don't think that threading is needed in such a simple
    > regime, but its simplicity is warning me that this is the wrong way to
    > do this! Other people say that a spin-wait routine like that above is
    > wasteful of CPU cycles.
    >
    > Does anyone have any comments or suggestions?
    >
    > TIA
    >
    > Trev


    To do something like that in C++ would be utterly idiotic.
    Try Python.


  4. Re: Suggestions for my Linux app...?

    sndive@gmail.com wrote:

    > To do something like that in C++ would be utterly idiotic.


    No. It's only cumbersome, especially if you are no C++ crack.

    > Try Python.


    Agreed.

    Regards,


    Björn

    --
    BOFH excuse #388:

    Bad user karma.


  5. Re: Suggestions for my Linux app...?

    Joe Beanfish wrote:

    > Trev wrote:
    >> I am trying to develop a (very simple!) application in C++ that will
    >> spider my site's web pages at pre-arranged times (the reasons for this
    >> scheduling is very politically driven, so please don't ask!
    >>
    >> My aim is a simple linked list, as follows, using two web pages: (in
    >> pseudo code)
    >>
    >> spider web page 1
    >> work out when next look is scheduled
    >> - put a wait signal in the scheduling queue
    >> -when wait expires, spider web page 2
    >> work out when next look is scheduled after next re-look at page 1
    >> wait
    >> spider web page 1
    >> etc. etc.
    >>
    >> essentially this means that the queue will be
    >> webpage request
    >> wait
    >> webpage request
    >> wait etc.
    >> with the queue being populated with new commands to re-look at certain
    >> webpages after so many hours or days.
    >>
    >> Personally, I don't think that threading is needed in such a simple
    >> regime, but its simplicity is warning me that this is the wrong way to
    >> do this! Other people say that a spin-wait routine like that above is
    >> wasteful of CPU cycles.
    >>
    >> Does anyone have any comments or suggestions?

    >
    > Nothing you said above implies a spin-wait. But spin-wait would be bad.
    > Use sleep() or select() to wait until the earliest expiration.



    Wouldn't this hog most of the CPU though? What if he wanted to allow
    user interaction, such as webforms etc.?

    --
    ----
    Home: http://www.paullee.com
    Woes: http://www.dr_paul_lee.btinternet.co.uk/zzq.shtml

  6. Re: Suggestions for my Linux app...?

    Kwebway Konongo wrote:
    > Joe Beanfish wrote:
    >
    >> Trev wrote:
    >>> I am trying to develop a (very simple!) application in C++ that will
    >>> spider my site's web pages at pre-arranged times (the reasons for this
    >>> scheduling is very politically driven, so please don't ask!
    >>>
    >>> My aim is a simple linked list, as follows, using two web pages: (in
    >>> pseudo code)
    >>>
    >>> spider web page 1
    >>> work out when next look is scheduled
    >>> - put a wait signal in the scheduling queue
    >>> -when wait expires, spider web page 2
    >>> work out when next look is scheduled after next re-look at page 1
    >>> wait
    >>> spider web page 1
    >>> etc. etc.
    >>>
    >>> essentially this means that the queue will be
    >>> webpage request
    >>> wait
    >>> webpage request
    >>> wait etc.
    >>> with the queue being populated with new commands to re-look at certain
    >>> webpages after so many hours or days.
    >>>
    >>> Personally, I don't think that threading is needed in such a simple
    >>> regime, but its simplicity is warning me that this is the wrong way to
    >>> do this! Other people say that a spin-wait routine like that above is
    >>> wasteful of CPU cycles.
    >>>
    >>> Does anyone have any comments or suggestions?

    >> Nothing you said above implies a spin-wait. But spin-wait would be bad.
    >> Use sleep() or select() to wait until the earliest expiration.

    >
    >
    > Wouldn't this hog most of the CPU though? What if he wanted to allow
    > user interaction, such as webforms etc.?


    Spin-wait would hog the cpu. That's why sleep() or select() would be better.
    Use select() for better than second resolution or to allow interaction
    with the user and/or other resources.

  7. Re: Suggestions for my Linux app...?

    Like this?

    struct timeval tv;
    tv.tv_sec = 0;
    tv.tv_usec = 200000; /* 0.2 seconds */
    select (0, NULL, NULL, NULL, &tv);


  8. Re: Suggestions for my Linux app...?

    On 2007-01-17, trashman.horlicks@btinternet.com wrote:
    > Like this?
    >
    > struct timeval tv;
    > tv.tv_sec = 0;
    > tv.tv_usec = 200000; /* 0.2 seconds */
    > select (0, NULL, NULL, NULL, &tv);


    that'd work.

    usleep(200000) would have the same effect.

    nanosleep would be another option. (usleep is easier to use)


    If it's for a server typically you use select and list put the FD of all the
    sockets you're listening on in the readfds... and any with writes pending
    in the writefds

    Bye.
    Jasen

+ Reply to Thread