Hello,

I updated a system with 12 dc-interfaces to a new hardware
with 14 em-interfaces. Yes, it is a firewall.
New System is 6.2-RELEASE-p8.

What I now experience between two internal networks (100MBit/s each)
is the following:
1318 packets transmitted, 1317 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.387/246.153/2441.392/324.142 ms

tcpdump on the firewall shows similar delays (on the outgoing
interface).

tcpdump on the system I ping however shows very quick responses
for incoming packages (ie usually less than a millisecond).

I therefore assume that the problem is between receiving the
irq from em and getting the data from the interface on the firewall
itself.

My first option would be to activate polling on em-interfaces - but as
I did not experience this sort of notieceable slowdown with the old
dc-based firewall (without polling), maybe someone can shed some light
on this strange behaviour or has other suggestions as well?

More details and dmesg below.

Help and suggestions welcome :-)

Best regards,
Holger Kipp

PS: Regarding the MPSAFE-issue - I need to use ipsec, so don't have a choice
here.




On the firewall I see many irqs (sometimes more than 5000) very often
on one of the em-interfaces using sysctl/vmstat:

--------------------------------------------------------------------------------
2 users Load 0.00 0.00 0.04 Feb 27 12:20

Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 14804 3724 32800 3800 1327728 count
All 722324 5248 76723364 5696 pages
Interrupts
Proc:r p d s w Csw Trp Sys Int Sof Flt cow 7606 total
27 7862 1 462 4105 158840 wire 1: atkb
14988 act 6: fdc0
0.2%Sys 3.0%Intr 0.0%User 0.0%Nice 96.8%Idl 547868 inact 12: psm
| | | | | | | | | | 1244 cache 14: ata
++ 1326484 free 359 16: em4
daefr 2761 17: em5
Namei Name-cache Dir-cache prcfr 28: twe
Calls hits % hits % react 306 52: em0
pdwak 1 53: em1
zfod pdpgs 177 55: em3
Disks twed0 ozfod intrn 2001 cpu0: time
KB/t 21.33 %slo-z 114464 buf 2001 cpu1: time
tps 1 1 tfree 10 dirtybuf
MB/s 0.01 100000 desiredvnodes
% busy 0 66594 numvnodes
19758 freevnodes
--------------------------------------------------------------------------------

dmesg

Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RELEASE-p8 #1: Thu Oct 4 16:07:31 CEST 2007
WARNING: debug.mpsafenet forced to 0 as ipsec requires Giant
WARNING: MPSAFE network stack disabled, expect reduced performance.
MPTable:
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 CPU 6420 @ 2.13GHz (2128.01-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6
Features=0xbfebfbff
Features2=0xe3bd,CX16,,>
AMD Features=0x20100000
AMD Features2=0x1
Cores per package: 2
real memory = 2146349056 (2046 MB)
avail memory = 2095296512 (1998 MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0: Assuming intbase of 0
ioapic1: Assuming intbase of 24
ioapic2: Assuming intbase of 48
ioapic0 irqs 0-23 on motherboard
ioapic1 irqs 24-47 on motherboard
ioapic2 irqs 48-71 on motherboard
kbd1 at kbdmux0
cpu0 on motherboard
cpu1 on motherboard
pcib0: pcibus 0 on motherboard
pci0: on pcib0
pcib0: unable to route slot 28 INTB
pcib1: irq 16 at device 1.0 on pci0
pci1: on pcib1
pcib2: at device 0.0 on pci1
pci2: on pcib2
twe0: <3ware Storage Controller. Driver version 1.50.01.002> port 0x4000-0x400f mem 0xe0100000-0xe010000f,0xe0800000-0xe0ffffff irq 28 at device 2.0 on pci2
twe0: [GIANT-LOCKED]
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048
pci1: at device 0.1 (no driver attached)
pcib3: at device 0.2 on pci1
pci3: on pcib3
pcib4: at device 3.0 on pci3
pci4: on pcib4
em0: port 0x5000-0x503f mem 0xe1000000-0xe101ffff irq 52 at device 4.0 on pci4
em0: Ethernet address: 00:0e:0c:c6:cc:54
em0: [GIANT-LOCKED]
em1: port 0x5040-0x507f mem 0xe1020000-0xe103ffff irq 53 at device 4.1 on pci4
em1: Ethernet address: 00:0e:0c:c6:cc:55
em1: [GIANT-LOCKED]
em2: port 0x5080-0x50bf mem 0xe1040000-0xe105ffff irq 54 at device 6.0 on pci4
em2: Ethernet address: 00:0e:0c:c6:cc:56
em2: [GIANT-LOCKED]
em3: port 0x50c0-0x50ff mem 0xe1060000-0xe107ffff irq 55 at device 6.1 on pci4
em3: Ethernet address: 00:0e:0c:c6:cc:57
em3: [GIANT-LOCKED]
pci1: at device 0.3 (no driver attached)
pcib5: irq 16 at device 3.0 on pci0
pci10: on pcib5
pcib6: at device 0.0 on pci10
pci11: on pcib6
pcib7: at device 0.0 on pci11
pci12: on pcib7
em4: port 0x6000-0x601f mem 0xe1120000-0xe113ffff,0xe1100000-0xe111ffff irq 16 at device 0.0 on pci12
em4: Memory Access and/or Bus Master bits were not set!
em4: Ethernet address: 00:15:17:25:48:88
em4: [GIANT-LOCKED]
em5: port 0x6020-0x603f mem 0xe1160000-0xe117ffff,0xe1140000-0xe115ffff irq 17 at device 0.1 on pci12
em5: Memory Access and/or Bus Master bits were not set!
em5: Ethernet address: 00:15:17:25:48:89
em5: [GIANT-LOCKED]
pcib8: at device 1.0 on pci11
pci13: on pcib8
em6: port 0x7000-0x701f mem 0xe1220000-0xe123ffff,0xe1200000-0xe121ffff irq 16 at device 0.0 on pci13
em6: Memory Access and/or Bus Master bits were not set!
em6: Ethernet address: 00:15:17:25:48:8a
em6: [GIANT-LOCKED]
em7: port 0x7020-0x703f mem 0xe1260000-0xe127ffff,0xe1240000-0xe125ffff irq 17 at device 0.1 on pci13
em7: Memory Access and/or Bus Master bits were not set!
em7: Ethernet address: 00:15:17:25:48:8b
em7: [GIANT-LOCKED]
pcib9: irq 17 at device 28.0 on pci0
pci14: on pcib9
pcib10: at device 0.0 on pci14
pci15: on pcib10
pcib11: at device 0.0 on pci15
pci16: on pcib11
em8: port 0x8000-0x801f mem 0xe1320000-0xe133ffff,0xe1300000-0xe131ffff irq 16 at device 0.0 on pci16
em8: Memory Access and/or Bus Master bits were not set!
em8: Ethernet address: 00:15:17:25:48:64
em8: [GIANT-LOCKED]
em9: port 0x8020-0x803f mem 0xe1360000-0xe137ffff,0xe1340000-0xe135ffff irq 17 at device 0.1 on pci16
em9: Memory Access and/or Bus Master bits were not set!
em9: Ethernet address: 00:15:17:25:48:65
em9: [GIANT-LOCKED]
pcib12: at device 1.0 on pci15
pci17: on pcib12
em10: port 0x9000-0x901f mem 0xe1420000-0xe143ffff,0xe1400000-0xe141ffff irq 16 at device 0.0 on pci17
em10: Memory Access and/or Bus Master bits were not set!
em10: Ethernet address: 00:15:17:25:48:66
em10: [GIANT-LOCKED]
em11: port 0x9020-0x903f mem 0xe1460000-0xe147ffff,0xe1440000-0xe145ffff irq 17 at device 0.1 on pci17
em11: Memory Access and/or Bus Master bits were not set!
em11: Ethernet address: 00:15:17:25:48:67
em11: [GIANT-LOCKED]
pcib13: irq 17 at device 28.4 on pci0
pci18: on pcib13
em12: port 0xa000-0xa01f mem 0xe1500000-0xe151ffff irq 16 at device 0.0 on pci18
em12: Ethernet address: 00:30:48:90:86:7a
em12: [GIANT-LOCKED]
pcib14: irq 7 at device 28.5 on pci0
pci19: on pcib14
em13: port 0xb000-0xb01f mem 0xe1600000-0xe161ffff irq 17 at device 0.0 on pci19
em13: Ethernet address: 00:30:48:90:86:7b
em13: [GIANT-LOCKED]
pcib15: at device 30.0 on pci0
pci20: on pcib15
pci20: at device 4.0 (no driver attached)
isab0: at device 31.0 on pci0
isa0: on isab0
atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x3020-0x302f at device 31.1 on pci0
ata0: on atapci0
ata1: on atapci0
pci0: at device 31.3 (no driver attached)
pmtimer0 on isa0
orm0: at iomem 0xc0000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff on isa0
atkbdc0: at port 0x60,0x64 on isa0
atkbd0: irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model MouseMan+, device ID 0
fdc0: at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: parallel port not found.
sc0: at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
unknown: can't assign resources (port)
unknown: can't assign resources (memory)
unknown: can't assign resources (memory)
unknown: can't assign resources (irq)
unknown: can't assign resources (port)
unknown: can't assign resources (port)
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
IP Filter: v4.1.13 initialized. Default = pass all, Logging = enabled
acd0: DVDROM at ata0-slave UDMA33
twed0: on twe0
twed0: 76318MB (156299440 sectors)
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/twed0s1a
em9: link state changed to UP
em8: link state changed to UP
em10: link state changed to UP
em0: link state changed to UP
em3: link state changed to UP
[...]
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"