Fun with Slackware: bandwidth monitor - Slackware

This is a discussion on Fun with Slackware: bandwidth monitor - Slackware ; Here's some scripting i've found to be useful. I began wondering one day, just how much of my bandwidth does my little server actually use? The web logs are intersting but don't answr the question. After a bit of poking ...

+ Reply to Thread
Results 1 to 16 of 16

Thread: Fun with Slackware: bandwidth monitor

  1. Fun with Slackware: bandwidth monitor


    Here's some scripting i've found to be useful. I began
    wondering one day, just how much of my bandwidth does my little
    server actually use? The web logs are intersting but don't
    answr the question. After a bit of poking around, i decided
    that ifconfig is the easiest tool to get the raw information.
    I could periodically read data from ifconfig and log it to
    a file for later analysis.

    So I wrote this little script here to create my log file, and
    named it 'logifx', and stuffed it into /root/cron/ (where i keep
    root cron stuff):


    #!/bin/bash
    # A script to periodically calculate the total number of bytes
    # sent and received on the given interface;
    # useful for running under cron, for example.
    # raw data is read from ifconfig(1); the script calculates its
    # output from the current reading and from data stored in a state
    # file by the previous run; the current values are then stored in
    # the state file for the next run, overwriting the previous state.
    # if the state file does not exist then no output is generated, but
    # the new STATEF file is still initialized with current data.
    #--------------------------------------------------------------
    # this is a useful escape in case of trouble:
    #--------------------------------------------------------------
    function PANIC () { echo PANIC: $* ; exit ; }
    #--------------------------------------------------------------
    # syntax: logifx
    #--------------------------------------------------------------
    [ "$1" == "" ] && PANIC interface required.
    IF=$1
    #--------------------------------------------------------------
    # FILES & DIRECTORIES:
    #--------------------------------------------------------------
    STATED=/var/run/logifx
    [ -d "${STATED}" ] || mkdir -p ${STATED} || PANC bad ${STATED} ?
    STATEF=${STATED}/ifxlog-${IF}.dat
    OUTD=/var/log
    OUTF=${OUTD}/ifx_${IF}
    #--------------------------------------------------------------
    # read ifconfig and return RX and TX values:
    #--------------------------------------------------------------
    function readif() {
    ifconfig ${IF} | while read LINE ; do
    DATA=($(echo ${LINE} | grep 'RX bytes:'))
    [ $? = 0 ] && echo ${DATA[1]/bytes:} ${DATA[5]/bytes:}
    done
    }
    #--------------------------------------------------------------
    # read the clock:
    #--------------------------------------------------------------
    STAMP=$( date '+%Y%m%d%H%M' ) # timestamp for logfile line
    SEC=$( date '+%s' ) # time used for calculations
    #--------------------------------------------------------------
    # read the data - note how an array is used to receive
    # two return values from the function call
    #--------------------------------------------------------------
    CURRENT=($( readif )) || PANIC reading ifconfig
    CRX=${CURRENT[0]}
    CTX=${CURRENT[1]}
    #--------------------------------------------------------------
    # if state file exists, read prior values and calculate output.
    # negative results indicate a stale state file, so reset it.
    # (not absolutely correct, but best guess and close enough);
    # then append the output to the log file:
    #--------------------------------------------------------------
    if [ -e ${STATEF} ] ; then
    read PSEC PRX PTX < ${STATEF}
    SPAN=$(( ${SEC} - ${PSEC} ))
    [ $SPAN -gt 0 ] || PANIC WTF? SPAN=${SPAN}
    RX=$(( ${CRX} - ${PRX} ))
    TX=$(( ${CTX} - ${PTX} ))
    [ $RX -lt 0 ] && RX=${CRX}
    [ $TX -lt 0 ] && TX=${CTX}
    echo ${STAMP} ${SPAN} ${RX} ${TX} >> ${OUTF} || PANIC ${OUTF}
    fi
    #--------------------------------------------------------------
    # update state file (overwrite)
    #--------------------------------------------------------------
    echo ${SEC} ${CRX} ${CTX} > ${STATEF} || PANIC updating ${STATEF}
    #--------------------------------------------------------------
    # the end.


    Okay? Okay! With that in /root/cron/logifx, now, the next step is
    to stuff it into cron, using crontab -l of course.

    * * * * * /root/cron/logifx eth0
    * * * * * /root/cron/logifx eth1
    * * * * * /root/cron/logifx eth2

    Note that any errors running the script are written to stdout,
    which cron will mail to the owner (root). So check for that.

    Note that the output record is formatted:
    1: timestamp YYYYMMDDHHMM
    2: interval, in seconds
    3: total bytes received during interval preceeding timestamp;
    4: total bytes sent during interval preceding timestamp;

    Okay? Okay! Now comes the fun part - analyzing the data. One
    could simply use a pager (ex: less) to view the accumulated
    readings, but a graphical interpretation makes it so much easier
    to visualize the data and observe patterns. Since I work mainly
    at text consoles, I use a histogram for this.

    Awk is convenient for this, but the same thing could be whacked
    out in most any language. I use two scripts for this, the first
    one i call view_RX, as follows:


    #!/bin/awk -f
    function histo( X ) {
    B=""
    N=X^0.5 # NOTE scaling ...
    for (a=0;a return B
    }
    {
    Data = ($3)
    Size = Data / 1024 # kilobytes
    Rate = Data / 128 / ($2) # kilobits per second
    Graph = histo( Rate )
    mask = "%s %5.0f KB %8.3f Kb/s %s\n"
    printf( mask, $1, Size, Rate, Graph )
    }
    # the end.


    The second script i call view_TX and it is exactly the same
    as view_RX except for one line:
    Data = ($3)
    is replaced with:
    Data = ($4)

    I know i should rewrite these into one script, in such a way
    that RX vs TX would be indicated by the script name, or an
    argument, or whatever. But for my purpose this is good enough,
    and maybe some day I'll rewrite it (again :*)

    These scripts get called something like:

    view_TX /var/log/ifx_eth0 | less

    and generate a report something like this:

    [...]
    200801222211 852 KB 113.586 Kb/s ***********
    200801222212 4445 KB 592.651 Kb/s *************************
    200801222213 10 KB 1.337 Kb/s **
    200801222214 505 KB 66.245 Kb/s *********
    200801222215 1232 KB 164.312 Kb/s *************
    200801222216 403 KB 54.646 Kb/s ********
    200801222217 3379 KB 450.586 Kb/s **********************
    200801222218 172 KB 22.913 Kb/s *****
    200801222219 19 KB 2.580 Kb/s **
    200801222220 705 KB 94.007 Kb/s **********
    [...]


    Okay? Okay!

    The same ifx_eth0 file can also be used to calculate totals
    for longer intervals, like monthly, for example. I rotate
    my logs into a file tree like /var/log/YYYY/MM/DD/, so:

    cat /var/log/2007/12/*/ifx_eth0 | while read A B C D ; do
    TOTRX=$(( $TOTRX + $C ))
    done
    echo "TOTAL RX for December, $(( $TOTRX / 1024 / 1024 )) MB."

    Okay? Okay!

    Have fun,

    --
    William Hunt, Portland Oregon USA

  2. Re: Fun with Slackware: bandwidth monitor

    On Wed, 23 Jan 2008 02:04:05 -0800, William Hunt wrote:

    >Here's some scripting i've found to be useful. I began
    >wondering one day, just how much of my bandwidth does my little
    >server actually use?


    I made one recently too With awk and bash and a web page for the
    results, in pictures... http://bugsplatter.mine.nu/netdraw/ -- source
    is there too.

    Grant.
    --
    http://bugsplatter.mine.nu/

  3. Re: Fun with Slackware: bandwidth monitor

    On Wed, 23 Jan 2008 02:04:05 -0800, William Hunt wrote:

    >
    > Here's some scripting i've found to be useful. I began wondering one day,
    > just how much of my bandwidth does my little server actually use?
    > BIG SNIP


    So what was the fun part, William? Writing the scripts, or did you learn
    something interesting about your server's use of bandwidth?

    --
    Chick Tower

    For e-mail: aols2 DOT sent DOT towerboy AT xoxy DOT net


  4. Re: Fun with Slackware: bandwidth monitor

    On Thu, 24 Jan 2008, Chick Tower wrote:
    > On Wed, 23 Jan 2008 02:04:05 -0800, William Hunt wrote:
    >> Here's some scripting i've found to be useful. I began wondering one day,
    >> just how much of my bandwidth does my little server actually use?
    >> BIG SNIP

    >
    > So what was the fun part, William? Writing the scripts, or did you learn
    > something interesting about your server's use of bandwidth?


    Yes.

    I apologize if it did not rise to meet your standard for 'fun'.
    Perhaps it is a mental defect which leads me to be amused
    by scripting a trivial ascii histogram. Or perhaps my life
    is so shallow and joyless that I mistook my noodling for
    entertainment. In either case, I should have recognized
    that the article's lack of troll-baiting and bickering was
    boring and unacceptably off-topic for this newsgroup. I will
    endeavour to corrct these failures of mine in any future
    posting. Thank you for your interest.
    --
    William Hunt, Portland Oregon USA

  5. Re: Fun with Slackware: bandwidth monitor

    William Hunt typed on 2008-01-24:



    > I apologize if it did not rise to meet your standard for 'fun'.
    > Perhaps it is a mental defect which leads me to be amused
    > by scripting a trivial ascii histogram. Or perhaps my life
    > is so shallow and joyless that I mistook my noodling for
    > entertainment. In either case, I should have recognized
    > that the article's lack of troll-baiting and bickering was
    > boring and unacceptably off-topic for this newsgroup. I will
    > endeavour to corrct these failures of mine in any future
    > posting. Thank you for your interest.




    --
    Bartosz Oudekerk
    I think a better name for PAM might be SCAM, for Swiss Cheese Authentication
    Modules, and have never felt that the small amount of convenience it provides
    is worth the great loss of system security. -- Patrick Volkerding

  6. Re: Fun with Slackware: bandwidth monitor

    On Thu, 24 Jan 2008 12:35:28 -0800, William Hunt wrote:

    > On Thu, 24 Jan 2008, Chick Tower wrote:
    >> So what was the fun part, William? Writing the scripts, or did you
    >> learn something interesting about your server's use of bandwidth?

    >
    > Yes.
    >
    > I apologize if it did not rise to meet your standard for 'fun'.


    I wasn't making fun of you, or trying to be sly, William. The subject led
    me to believe that you found out something interesting or you had fun
    monitoring your bandwidth, but all you discussed was the scripts. I can
    understand the fun in scripting. I just wondered if you discovered
    anything useful about your bandwidth usage with your scripts.

    --
    Chick Tower

    For e-mail: aols2 DOT sent DOT towerboy AT xoxy DOT net


  7. Re: Fun with Slackware: bandwidth monitor

    On Fri, 25 Jan 2008, Chick Tower wrote:
    > On Thu, 24 Jan 2008 12:35:28 -0800, William Hunt wrote:
    >> On Thu, 24 Jan 2008, Chick Tower wrote:
    >>> So what was the fun part, William? Writing the scripts, or did you
    >>> learn something interesting about your server's use of bandwidth?

    >>
    >> Yes.
    >> I apologize if it did not rise to meet your standard for 'fun'.

    >
    > I wasn't making fun of you, or trying to be sly, William. The subject led
    > me to believe that you found out something interesting or you had fun
    > monitoring your bandwidth, but all you discussed was the scripts. I can
    > understand the fun in scripting. I just wondered if you discovered
    > anything useful about your bandwidth usage with your scripts.


    :*)

    Thanks, Chick, and I hope you understand my reply was meant in jest.
    And my hide is made of leather, i don't mind being made fun with.
    And IMHO any article with 'fun' in the subject is, well, open for fun.

    Seriously now, the 'fun' part was simply writing the crude scripts
    and seeing it all work as intended. I was pleased with myself for
    concocting the various components - periodically extracting raw
    data from the system, and the use of an ascii logarithmic histogram
    to view the accumulated data. Certainly not groundbreaking work.

    Personally, i am amused by almost any crude technology that performs
    within acceptable parameters. (The ancient rust-bucket truck i use
    is a good example :*) and an ongoing joke amongst my friends but it
    gets the job done reliably and efficiently).

    I believe that, to improve one's coding skills, it is -essential-
    to read other's code, and conversely, to be able to 'walk through'
    one's own code for others. Giving a code walk-through can be a
    fun but humbling experience :*)

    So what did i learn about my bandwidth ? ummmmm, wellllll, a couple
    of things -

    1), that my inbound bandwidth max's out at about 80% of the nominal
    rate, and outbound about 50% of nominal. ie, 1.5Mb/s ADSL inbound
    rarely exceeds 1.2Mb/s and never exceeds 1.4Mb/s. These figures
    suggest that there may be some issues with outbound service.

    2), that my traffic averages 3:1 inboundutbound, so the ADSL nominal
    ratio of 2:1 inboundutbound is acceptable, but it also suggests
    that there may be some issues with outbound service.

    3), that my little server has gobs of bandwidth available, averaging
    only about 2% of the theortical max, maybe 10% of a practical max.
    This tells me I have plenty of room for more webhosting clients
    without degrading service, plenty of room for my LAN users to play
    in.

    4), many small blocks of sustained 30Kb/s outbound suggest that
    many of my visitors are stilll using dialup connections.

    The main benefits i get from bandwidth monitoring stem from the
    confidence i have in my numbers. In the past, when I was getting
    unacceptable service, I found that useful in harranging my DSL
    provider to correct the situation. It has also been very useful
    in discussions with webhost clients, to assure them that their
    sites are being adequately served. It has been useful to me,
    during a period when I was leasing a remote host, to discover
    that the bandwidth provided was in that case totally unacceptable.

    I also run similar monitors to log and review ping times with my
    upstream gateway, useful again in discovering and handling problems
    with my service providers.

    I also run similar monitors to log and review system load averages.
    This has been extremely useful in making me aware of some bizarre
    and as yet undiagnosed problem with one server which from time to
    time, causes the load average to skyrocket on precise 7 minute
    intervals ... for which the only solution i've found so far is
    (ugh!) a reboot.

    Okay, now, wasn't that fun ? :*)

    (I know i'm a little whacko, but i still think an ascii histogram
    is somehow incongrous and funny, like maybe a dog wearing a straw
    hat and sunglasses :*)

    PS, i still haven't gotten around to examing Grant's solution,
    except to note that it certainly looks more proffesional than
    mine. I may have to throw away years of hard noodling now and
    adopt/adapt his code yet :*)


    --
    William Hunt, Portland Oregon USA

  8. Re: Fun with Slackware: bandwidth monitor

    Grant wrote:

    > I made one recently too With awk and bash and a web page for the
    > results, in pictures... http://bugsplatter.mine.nu/netdraw/ --
    > source is there too.


    Thanks, I like it.

    -- Simon


  9. Re: Fun with Slackware: bandwidth monitor

    On Fri, 25 Jan 2008 13:41:10 -0800, William Hunt wrote:
    ....
    >PS, i still haven't gotten around to examing Grant's solution,
    >except to note that it certainly looks more proffesional than
    >mine. I may have to throw away years of hard noodling now and
    >adopt/adapt his code yet :*)


    Thanks, I thought your data collection more accurate than what I'm
    doing (one minute sampling with delta secs vs five minute sampling)
    but I'm not about to change mine now

    Also, I've done ascii art chart for a different project monitoring
    the junk being dropped by iptables, mostly windows leakage from my
    ISP's CIDR block:

    22/tcp 2 |( . . . . 0.8
    80/tcp 30 |((((((((((((( . . . 12.6
    135/tcp 75 |(((((((((((((((((((((((((((((((( . 31.5
    139/tcp 5 |(( . . . . 2.1
    445/tcp 81 |(((((((((((((((((((((((((((((((((( . 34.0
    1433/tcp 11 |((((( . . . . 4.6
    1434/udp 3 |( . . . . 1.3
    1504/tcp 1 | . . . . 0.4
    1505/tcp 1 | . . . . 0.4
    2967/tcp 1 | . . . . 0.4
    3067/tcp 11 |((((( . . . . 4.6
    8080/tcp 3 |( . . . . 1.3
    10622/tcp 1 | . . . . 0.4
    16756/tcp 1 | . . . . 0.4
    43486/tcp 3 |( . . . . 1.3
    54356/tcp 5 |(( . . . . 2.1
    others 4 |(( . . . . 1.7
    total 238 + - - - - + - - - - + - - - - + - - - - + - - -
    0 10.0% 20.0% 30.0% 40.0%

    http://bugsplatter.mine.nu/junkshow/

    Grant.
    --
    http://bugsplatter.mine.nu/

  10. Re: Fun with Slackware: bandwidth monitor

    On Sat, 26 Jan 2008 01:53:51 +0100, Simon Sibbez wrote:

    >Grant wrote:
    >
    >> I made one recently too With awk and bash and a web page for the
    >> results, in pictures... http://bugsplatter.mine.nu/netdraw/ --
    >> source is there too.

    >
    >Thanks, I like it.


    Feedback! Thank you

    Grant.
    --
    http://bugsplatter.mine.nu/

  11. Re: Fun with Slackware: bandwidth monitor

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    Grant wrote:
    | On Sat, 26 Jan 2008 01:53:51 +0100, Simon Sibbez
    wrote:
    |
    |> Grant wrote:
    |>
    |>> I made one recently too With awk and bash and a web page for the
    |>> results, in pictures... http://bugsplatter.mine.nu/netdraw/ --
    |>> source is there too.
    |> Thanks, I like it.
    |
    | Feedback! Thank you

    Bug report I'm afraid...

    netdrawhtml fails with the message:

    *** glibc detected *** free(): invalid pointer: 0x08093340 ***
    Aborted

    It's a slack 10.2 install with gawk 3.1.5

    Looks like a bug in the build of gawk itself, is the slack 10.2 build
    known to be faulty?

    Blumf
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.7 (GNU/Linux)
    Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

    iD8DBQFHnLOkMid3IcxolsoRAnhmAJ9wkSGUix/KJr2wo74SBJO3hZMiWgCfc0/C
    33Ef1iigwekXCoHGwl4FusA=
    =/oM7
    -----END PGP SIGNATURE-----

  12. Re: Fun with Slackware: bandwidth monitor

    On Sun, 27 Jan 2008 16:39:01 GMT, Blumf wrote:

    >Bug report I'm afraid...
    >
    >netdrawhtml fails with the message:
    >
    >*** glibc detected *** free(): invalid pointer: 0x08093340 ***
    >Aborted
    >
    >It's a slack 10.2 install with gawk 3.1.5
    >
    >Looks like a bug in the build of gawk itself, is the slack 10.2 build
    >known to be faulty?


    Yep, bug in gawk, install gawk-3.1.5 from slackware-11.0 should fix it.

    Alternately a copy of the official patch is here:

    http://bugsplatter.mine.nu/bash/patc...nvalid-pointer

    if you want to recompile your source, I also patched:

    http://bugsplatter.mine.nu/bash/patc...awk.SlackBuild

    --- gawk.SlackBuild.old 2005-09-07 13:54:44.000000000 +1000
    +++ gawk.SlackBuild 2006-03-11 15:24:57.000000000 +1100
    @@ -9,7 +9,7 @@

    VERSION=3.1.5
    ARCH=${ARCH:-i486}
    -BUILD=${BUILD:-1}
    +BUILD=${BUILD:-1a}

    if [ "$ARCH" = "i386" ]; then
    SLKCFLAGS="-O2 -march=i386 -mcpu=i686"

    Cheers,
    Grant.
    --
    http://bugsplatter.mine.nu/

  13. Re: Fun with Slackware: bandwidth monitor

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    Grant wrote:
    | On Sun, 27 Jan 2008 16:39:01 GMT, Blumf wrote:
    |
    |> Bug report I'm afraid...
    |>
    |> netdrawhtml fails with the message:
    |>
    |> *** glibc detected *** free(): invalid pointer: 0x08093340 ***
    |> Aborted
    |>
    |> It's a slack 10.2 install with gawk 3.1.5
    |>
    |> Looks like a bug in the build of gawk itself, is the slack 10.2 build
    |> known to be faulty?
    |
    | Yep, bug in gawk, install gawk-3.1.5 from slackware-11.0 should fix it.

    Thought so, a quick google showed other people reporting it. I notice
    there's a 3.1.6 out, would that include the fixes? (can't see a
    changelog anywhere)

    Blumf
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.7 (GNU/Linux)
    Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

    iD8DBQFHnR8LMid3IcxolsoRAlcDAKCHKikQ4k5M8pLfxSFtty AtBumNUwCcDfyZ
    Q92Xka5st206Lc7Yi1LVek8=
    =++wO
    -----END PGP SIGNATURE-----

  14. Re: Fun with Slackware: bandwidth monitor

    On Mon, 28 Jan 2008 00:17:16 GMT, Blumf wrote:

    >Grant wrote:

    ....
    >|> Looks like a bug in the build of gawk itself, is the slack 10.2 build
    >|> known to be faulty?
    >|
    >| Yep, bug in gawk, install gawk-3.1.5 from slackware-11.0 should fix it.
    >
    >Thought so, a quick google showed other people reporting it. I notice
    >there's a 3.1.6 out, would that include the fixes? (can't see a
    >changelog anywhere)


    Haven't tried 3.1.6, expect it would have the fix. gawk-3.1.5 with the
    patch is what comes with slack-11 and slack-12.

    Grant.
    --
    http://bugsplatter.mine.nu/

  15. Re: Fun with Slackware: bandwidth monitor

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    Grant wrote:
    | Haven't tried 3.1.6, expect it would have the fix. gawk-3.1.5 with the
    | patch is what comes with slack-11 and slack-12.

    Tried 3.1.6 and it works. Thanks.

    Blumf
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.7 (GNU/Linux)
    Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

    iD8DBQFHna9YMid3IcxolsoRAnxTAKCDLBIHsWBZdSaKnQA26U K5alwBgACfbF1W
    0NrpVQWsQvF9J2yQ/zn7HOA=
    =Q1q1
    -----END PGP SIGNATURE-----

  16. Re: Fun with Slackware: bandwidth monitor

    On Sat, 26 Jan 2008, Grant wrote:
    > On Fri, 25 Jan 2008 13:41:10 -0800, William Hunt wrote:

    [...]
    > Thanks, I thought your data collection more accurate than what I'm
    > doing (one minute sampling with delta secs vs five minute sampling)
    > but I'm not about to change mine now


    Thanks :*) - i put the delta secs in there so that the data point
    would be useful regardless of sampling intervals. it can cron at
    5min or hourly just as well, or even irregular intervals. or,
    ( while usleep $X ; do logifx $Y ; done )&

    i ended up using 1 min intervals, and haven't put any thought on
    handling oddities like reboot or integer rollover. It's all
    just QND.

    I thought about using iptables for raw data, ( using counters,
    not -j LOG ), but i went with ifconfig because i'm often mucking
    with my filter tables, so that would be less stable and not so
    portable.

    The 'view_[RT]' script is just a quick hack. I've also been messing
    with wrapping it in a /bin/dialog interface at the console, and planning
    on a cgi wrapper using "pic|groff>*.png" to draw a pretty picture with
    RX and TX together. The awk histogram is a quick idiom i use in a few
    different things and one-off analysis.

    But now:

    > Also, I've done ascii art chart for a different project monitoring
    > the junk being dropped by iptables, mostly windows leakage from my
    > ISP's CIDR block:
    >

    [... snip, nice ascii art chart ... ]
    >
    > http://bugsplatter.mine.nu/junkshow/



    That's really really nice. I want one. :*)


    The ascii art is cute but i am sooo most impressed by your
    tables. 'networkmonitor' and 'netdraw' both look extremely
    well done, the code all looks excellent. I'm quite impressed
    and looking forward to going through it all - I will no doubt
    be using most all of both of those in my scheme of things, ASAP!

    That's some really good work you've doen there, Grant.
    Thanks !


    --
    William Hunt, Portland Oregon USA

+ Reply to Thread