OpenBSD 3.7, wget, squid, slow download rate - BSD

This is a discussion on OpenBSD 3.7, wget, squid, slow download rate - BSD ; Hi, I have a performance problem using OpenBSD 3.7 on Sparc64. I have the following environment: - DSL 6MBit down, 512k up, no router, the firewall does pppoe to the modem - Firewall: Sun Ultra 1/170MHz, 192MB RAM, (hme1 is ...

+ Reply to Thread
Results 1 to 5 of 5

Thread: OpenBSD 3.7, wget, squid, slow download rate

  1. OpenBSD 3.7, wget, squid, slow download rate

    Hi,

    I have a performance problem using OpenBSD 3.7 on Sparc64.

    I have the following environment:
    - DSL 6MBit down, 512k up, no router, the firewall does pppoe to the
    modem
    - Firewall: Sun Ultra 1/170MHz, 192MB RAM, (hme1 is external interface,
    hme0 is dmz and hme4 is lan), OpenBSD 3.7 sparc64 with pf,
    Generic-Kernel
    - Proxy in the dmz: Sun Netra T1 AC200 500MHz, 256MB RAM,
    gem0-Interface, sparc64, OpenBSD 3.7 fully patched, Generic-Kernel

    Used Software:
    - Squid 2.5stable9 (standard-package) and tried also 2.5stable10 from
    ports
    - wget 1.8.2 (standard-package) and tried also wget 1.10 from ports

    I tested against a fast ftp-server which delivers about 640kBytes/s.
    OpenBSD only gives me 320kBytes/s. I tested with and without proxy,
    with and without pf (the last thing naturally from the firewall). The
    result is the same: between 320kBytes/s and 330kBytes/s. The load on
    the system doesn't matter: at the moment I'm using the system alone !!!
    The external
    NIC was at first the le0 (10MBit), but is now hme1 (100MBit), no
    difference.

    Now the funny things:
    - a Windows machine in the LAN using wget 1.8.2 gives the expected
    640kBytes (going directly without proxy)
    - a Linux-machine in the LAN using wget 1.9.1 gives the expected
    640kBytes (going directly without proxy)
    - an OpenBSD/i386 3.7 machine in the LAN with wget 1.8.2 gives me also
    only 320kBytes (going directly without proxy)
    - using ncftpget on the proxy-machine in the dmz gives me the expected
    640kBytes/s (going directly without proxy)
    - using Samba from LAN to the proxy gives me about 4MBytes/s
    - doing 2 simultaneous wgets from 2 separate machines without proxy
    give me for each connection about 320kBytes

    My conclusion:
    - pf isn't the problem (with and without pf I only get 320k on the
    OpenBSDs but 640k with Linux and Windows)
    - OpenBSD in itself isn't the problem since samba or ncftpget give me
    the expected 640k
    - wget in itself isn't the problem (since wget 1.8.2 on Windows does
    ok)
    - load isn't a problem (according to vmstat and top)
    - the sparc64-architecture isn't the problem since i386 has the same
    problem

    That means that squid and wget on OpenBSD are somewhat weird ?!? Where
    is my mistake ??? I'm completely confused !!!

    Any help would be appreciated.
    Regards,
    Karsten

    If it helps, my dmesgs:
    Firewall:
    =========
    console is /sbus@1f,0/zs@f,1100000:a
    Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California. All rights
    reserved.
    Copyright (c) 1995-2005 OpenBSD. All rights reserved.
    http://www.OpenBSD.org

    OpenBSD 3.7 (GENERIC) #431: Sun Mar 20 14:10:02 MST 2005

    dera...@sparc64.openbsd.org:/usr/src/sys/arch/sparc64/compile/GENERIC
    total memory = 201326592
    avail memory = 172679168
    using 1228 buffers containing 10059776 bytes of memory
    bootpath: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0
    mainbus0 (root): Sun Ultra 1 SBus (UltraSPARC 167MHz)
    cpu0 at mainbus0: SUNW,UltraSPARC @ 166.982 MHz, version 0 FPU
    cpu0: physical 32K instruction (32 b/l), 16K data (32 b/l), 512K
    external (64 b/l)
    timer0 at mainbus0 addr 0xfffc7c00 irq vectors 7f0 and 7f1
    sbus0 at mainbus0 addr 0xfffcc000: clock = 25 MHz
    STC0 on /mainbus enabled
    DVMA map: ff800000 to ffffe000
    IOTDB: fb2000 to fb4000
    audiocs0 at sbus0 slot 13 offset 0xc000000 vector 24 ipl 8
    audio0 at audiocs0
    auxio0 at sbus0 slot 15 offset 0x1900000
    flashprom at sbus0 slot 15 offset 0x0 not configured
    fdc0 at sbus0 slot 15 offset 0x1400000 vector 29 ipl 11: no drives
    attached
    clock1 at sbus0 slot 15 offset 0x1200000: mk48t59: hostid 80891a56
    zs0 at sbus0 slot 15 offset 0x1100000 vector 28 ipl 12 softpri 6
    zstty0 at zs0 channel 0 (console i/o)
    zstty1 at zs0 channel 1
    zs1 at sbus0 slot 15 offset 0x1000000 vector 28 ipl 12 softpri 6
    zskbd0 at zs1 channel 0: no keyboard
    zstty2 at zs1 channel 1, mouse
    uperf0 at sbus0 slot 15 offset 0x1300000: model SUNW,sc-up (0/0) ports
    3
    SUNW,pll at sbus0 slot 15 offset 0x1304000 not configured
    dma0 at sbus0 slot 14 offset 0x8400000: dma rev 2
    esp0 at dma0 slot 14 offset 0x8800000 vector 20 ipl 3: ESP200, 40MHz,
    SCSI ID 7
    scsibus0 at esp0: 8 targets
    sd0 at scsibus0 targ 0 lun 0: SCSI2
    0/direct fixed
    sd0: 4345MB, 6144 cyl, 8 head, 181 sec, 512 bytes/sec, 8899737 sec
    total
    cd0 at scsibus0 targ 6 lun 0: SCSI2
    5/cdrom removable
    ledma0 at sbus0 slot 14 offset 0x8400010: dma rev 2
    le0 at ledma0 slot 14 offset 0x8c00000 vector 21 ipl 6: address
    08:00:20:89:1a:56
    le0: 8 receive buffers, 2 transmit buffers
    SUNW,bpp at sbus0 slot 14 offset 0xc800000 vector 22 ipl 2 not
    configured
    hme0 at sbus0 slot 0 offset 0x8c00000 vector 4 ipl 6: address
    08:00:20:9d:c3:d8
    qsphy0 at hme0 phy 1: QS6612 10/100 PHY, rev. 1
    hme1 at sbus0 slot 0 offset 0x8c10000 vector 4 ipl 6: address
    08:00:20:9d:c3:d9
    qsphy1 at hme1 phy 1: QS6612 10/100 PHY, rev. 1
    hme2 at sbus0 slot 0 offset 0x8c20000 vector 4 ipl 6: address
    08:00:20:9d:c3:da
    qsphy2 at hme2 phy 1: QS6612 10/100 PHY, rev. 1
    hme3 at sbus0 slot 0 offset 0x8c30000 vector 4 ipl 6: address
    08:00:20:9d:c3:db
    qsphy3 at hme3 phy 1: QS6612 10/100 PHY, rev. 1
    hme4 at sbus0 slot 2 offset 0x8c00000 vector 4 ipl 6: address
    08:00:20:89:1a:56
    nsphy0 at hme4 phy 1: DP83840 10/100 PHY, rev. 1
    pcons at mainbus0 not configured
    root on sd0a
    rootdev=0x700 rrootdev=0x1100 rawdev=0x1102

    Proxy:
    ======
    Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California. All rights
    reserved.
    Copyright (c) 1995-2005 OpenBSD. All rights reserved.
    http://www.OpenBSD.org

    OpenBSD 3.7 (GENERIC) #0: Thu Aug 11 23:45:13 CEST 2005

    r...@rohrpostix.dorfplatz:/usr/src/kernel/sys/arch/sparc64/compile/GENERIC
    total memory = 268435456
    avail memory = 234061824
    using 1638 buffers containing 13418496 bytes of memory
    bootpath: /pci@1f,0/pci@1,0/scsi@8,0/disk@0,0
    mainbus0 (root): Netra T1 200 (UltraSPARC-IIe 500MHz)
    cpu0 at mainbus0: SUNW,UltraSPARC-IIe @ 500 MHz, version 0 FPU
    cpu0: physical 32K instruction (32 b/l), 16K data (32 b/l), 1024K
    external (64 b/l)
    psycho0 at mainbus0
    SUNW,sabre: impl 0, version 0: ign 7c0 bus range 0 to 2; PCI bus 0
    DVMA map: c0000000 to e0000000
    IOTDB: 1360000 to 13e0000
    pci0 at psycho0
    ppb0 at pci0 dev 1 function 1 "Sun Simba PCI-PCI" rev 0x13
    pci1 at ppb0 bus 1
    ebus0 at pci1 dev 12 function 0 "Sun PCIO Ebus2 (US III)" rev 0x01
    flashprom at ebus0 addr 0-fffff not configured
    clock1 at ebus0 addr 0-1fff: mk48t59: hostid 830bae1f
    ebus_attach: idprom: incomplete
    SUNW,lomh at ebus0 addr 200000-200003 ipl 42 not configured
    gem0 at pci1 dev 12 function 1 "Sun ERI Ether" rev 0x01: ivec 3006,
    address 00:03:ba:0b:ae:1f
    ukphy0 at gem0 phy 1: Generic IEEE 802.3u media interface
    ukphy0: OUI 0x0010dd, model 0x0002, rev. 1
    ohci0 at pci1 dev 12 function 3 "Sun USB" rev 0x01: ivec 24, version
    1.0, legacy support
    usb0 at ohci0: USB revision 1.0
    uhub0 at usb0
    uhub0: Sun OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
    uhub0: 4 ports with 4 removable, self powered
    "Acer Labs M7101 Power Mgmt" rev 0x00 at pci1 dev 3 function 0 not
    configured
    "Acer Labs M7101 Power Mgmt" rev 0x00 at pci1 dev 3 function 0 not
    configured
    ebus1 at pci1 dev 7 function 0 "Acer Labs M1533 ISA" rev 0x00
    power at ebus1 addr 800-82f ipl 37 not configured
    com0 at ebus1 addr 3f8-3ff ipl 43: ns16550a, 16 byte fifo
    com0: console
    com1 at ebus1 addr 2e8-2ef ipl 43: ns16550a, 16 byte fifo
    pciide0 at pci1 dev 13 function 0 "Acer Labs M5229 UDMA IDE" rev 0xc3:
    DMA, channel 0 configured to native-PCI, channel 1 configured to
    native-PCI
    pciide0: using ivec 180c for native-PCI interrupt
    atapiscsi0 at pciide0 channel 0 drive 0
    scsibus0 at atapiscsi0: 2 targets
    cd0 at scsibus0 targ 0 lun 0: SCSI0 5/cdrom
    removable
    cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
    pciide0: channel 1 disabled (no drives)
    gem1 at pci1 dev 5 function 1 "Sun ERI Ether" rev 0x01: ivec 301c,
    address 00:03:ba:0b:ae:20
    ukphy1 at gem1 phy 1: Generic IEEE 802.3u media interface
    ukphy1: OUI 0x0010dd, model 0x0002, rev. 1
    ohci1 at pci1 dev 5 function 3 "Sun USB" rev 0x01: ivec 26, version
    1.0, legacy support
    usb1 at ohci1: USB revision 1.0
    uhub1 at usb1
    uhub1: Sun OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
    uhub1: 4 ports with 4 removable, self powered
    ppb1 at pci0 dev 1 function 0 "Sun Simba PCI-PCI" rev 0x13
    pci2 at ppb1 bus 2
    siop0 at pci2 dev 8 function 0 "Symbios Logic 53c896" rev 0x07: ivec
    20, using 8K of on-board RAM
    scsibus1 at siop0: 16 targets
    sd0 at scsibus1 targ 0 lun 0: SCSI3
    0/direct fixed
    sd0: 17274MB, 7508 cyl, 19 head, 248 sec, 512 bytes/sec, 35378533 sec
    total
    siop1 at pci2 dev 8 function 1 "Symbios Logic 53c896" rev 0x07: ivec
    20, using 8K of on-board RAM
    scsibus2 at siop1: 16 targets
    pcons at mainbus0 not configured
    No counter-timer -- using %tick at 500MHz as system clock.
    root on sd0a
    siop0: target 0 now using tagged 16 bit 40.0 MHz 31 REQ/ACK offset
    xfers
    rootdev=0x700 rrootdev=0x1100 rawdev=0x1102
    ohci1: 1 scheduling overruns
    ohci0: 1 scheduling overruns


  2. Re: OpenBSD 3.7, wget, squid, slow download rate

    > I have a performance problem using OpenBSD 3.7 on Sparc64.

    This story is so strange i even tried to copy your situation.
    I'm running Obsd 3.7 Stable on an amd


    # dmesg | grep GHz
    cpu0: AMD Athlon(TM) XP1700+ ("AuthenticAMD" 686-class) 1.11 GHz
    # wget -V | head -1
    GNU Wget 1.8.2
    # wget ftp://ftp.nl.uu.net/pub/XFree86/4.5.....5.0-src-7.tgz
    20:25:23 (664.07 KB/s) - `XFree86-4.5.0-src-7.tgz' saved [4741560]

    # /usr/wget/bin/wget -V | head -1
    GNU Wget 1.10.1
    # /usr/wget/bin/wget
    ftp://ftp.nl.uu.net/pub/XFree86/4.5.....5.0-src-7.tgz
    20:27:34 (786.15 KB/s) - `XFree86-4.5.0-src-7.tgz.1' saved [4741560]

    Not as much as you have, but i also see a performance change thats
    somewhat big. Note i did flush the squid cache.

    -Bas



  3. Re: OpenBSD 3.7, wget, squid, slow download rate

    Hi Bas,

    I could reproduce the problem with wget 1.8.2 AND wget 1.10. For
    testing I didn't even use squid. I still haven't a solution. The only
    thing I just noticed, is that I get slightly more collisions on my
    external interface when using wget. Using ncftpget I get a little less
    collisions even though I get about double as much transfer rate.
    Using OpenBSD/i386 I could also reproduce the problem.
    I'm still totally without a clue :-(((

    Greets,
    Karsten

    Bas Keur schrieb:

    > > I have a performance problem using OpenBSD 3.7 on Sparc64.

    >
    > This story is so strange i even tried to copy your situation.
    > I'm running Obsd 3.7 Stable on an amd
    >
    >
    > # dmesg | grep GHz
    > cpu0: AMD Athlon(TM) XP1700+ ("AuthenticAMD" 686-class) 1.11 GHz
    > # wget -V | head -1
    > GNU Wget 1.8.2
    > # wget ftp://ftp.nl.uu.net/pub/XFree86/4.5.....5.0-src-7.tgz
    > 20:25:23 (664.07 KB/s) - `XFree86-4.5.0-src-7.tgz' saved [4741560]
    >
    > # /usr/wget/bin/wget -V | head -1
    > GNU Wget 1.10.1
    > # /usr/wget/bin/wget
    > ftp://ftp.nl.uu.net/pub/XFree86/4.5.....5.0-src-7.tgz
    > 20:27:34 (786.15 KB/s) - `XFree86-4.5.0-src-7.tgz.1' saved [4741560]
    >
    > Not as much as you have, but i also see a performance change thats
    > somewhat big. Note i did flush the squid cache.
    >
    > -Bas



  4. Re: OpenBSD 3.7, wget, squid, slow download rate

    Hi all,

    does nobody have an idea ? Or at least some hints where I can track it
    down some more ?
    Help :-((( !

    Greetings,
    Karsten

    grmmpf@yahoo.de schrieb:

    > Hi Bas,
    >
    > I could reproduce the problem with wget 1.8.2 AND wget 1.10. For
    > testing I didn't even use squid. I still haven't a solution. The only
    > thing I just noticed, is that I get slightly more collisions on my
    > external interface when using wget. Using ncftpget I get a little less
    > collisions even though I get about double as much transfer rate.
    > Using OpenBSD/i386 I could also reproduce the problem.
    > I'm still totally without a clue :-(((
    >
    > Greets,
    > Karsten
    >
    > Bas Keur schrieb:
    >
    > > > I have a performance problem using OpenBSD 3.7 on Sparc64.

    > >
    > > This story is so strange i even tried to copy your situation.
    > > I'm running Obsd 3.7 Stable on an amd
    > >
    > >
    > > # dmesg | grep GHz
    > > cpu0: AMD Athlon(TM) XP1700+ ("AuthenticAMD" 686-class) 1.11 GHz
    > > # wget -V | head -1
    > > GNU Wget 1.8.2
    > > # wget ftp://ftp.nl.uu.net/pub/XFree86/4.5.....5.0-src-7.tgz
    > > 20:25:23 (664.07 KB/s) - `XFree86-4.5.0-src-7.tgz' saved [4741560]
    > >
    > > # /usr/wget/bin/wget -V | head -1
    > > GNU Wget 1.10.1
    > > # /usr/wget/bin/wget
    > > ftp://ftp.nl.uu.net/pub/XFree86/4.5.....5.0-src-7.tgz
    > > 20:27:34 (786.15 KB/s) - `XFree86-4.5.0-src-7.tgz.1' saved [4741560]
    > >
    > > Not as much as you have, but i also see a performance change thats
    > > somewhat big. Note i did flush the squid cache.
    > >
    > > -Bas



  5. Re: OpenBSD 3.7, wget, squid, slow download rate

    On 25 Aug 2005 02:09:22 -0700, grmmpf@yahoo.de wrote:

    > does nobody have an idea ? Or at least some hints where I can track it
    > down some more ?


    You seem to have done a number of different tests, but not two that precisely
    show what you seem to conclude: that wget is slower than ncftpget all other
    things being exactly the same.

    Forget about squid completely for now, don't use it.

    Don't trust any external servers or that PPPoE link. Set up a LOCAL web server
    and test everything against that. Use a large enough file, so the entire
    transfer takes at least 20s (dd if=/dev/arandom of=big bs=1024 count=40960).
    Using a 4MB file to test transfer speeds of ~700KB/s, which takes around 5s,
    just doesn't give enough precision.

    Use one OpenBSD/i386 client box. Fetch the file using ncftpget (and ftp(1) in
    base, and lynx -source -o). Write the output to /dev/null, so disk writing
    speed is irrelevant. Use time(1) to manually measure execution time, and
    manually check that the speeds printed by various tools are about right.
    Note the average speed you get and the approximate standard deviation.

    Now, on the SAME client box, without changing ANYTHING else, use wget to
    fetch the same file. Is it really notably slower?

    It isn't, in all my tests. It happily fetches files at >7MB/s speeds on the
    local 100Mbit/s LAN, much like any other tool I tried.

    If you find that wget isn't actually slower, all other things equal, you can
    go find what OTHER thing wasn't equal in the tests that did show a difference.
    Change only one other thing at a time, so you don't jump to conclusions

    Daniel

+ Reply to Thread