Why is this super simple ethernet bridge not working? - BSD

This is a discussion on Why is this super simple ethernet bridge not working? - BSD ; I am having trouble setting up a fresh install of FreeBSD as an ethernet bridge. I have two boxes, call them source, 10.2.0.1, and sink 10.2.0.2. When they are on the same network they can ping each other fine. However, ...

+ Reply to Thread
Results 1 to 6 of 6

Thread: Why is this super simple ethernet bridge not working?

  1. Why is this super simple ethernet bridge not working?

    I am having trouble setting up a fresh install of FreeBSD as an
    ethernet bridge.
    I have two boxes, call them source, 10.2.0.1, and sink 10.2.0.2. When
    they are on the same network they can ping each other fine. However,
    when I insert the FreeBSD box, call it bridge, between source and
    sink, they can no longer reach each other.

    source <--net1-->(le1)bridge0(le2)<--net2--> sink

    In particular, if I ping sink from source, then I see:
    On net1, as seen from both source's nic and le1: arp who-has 10.2.0.2
    (sink)
    On net2, as seen from sink's nic: arp who-has 10.2.0.2 and arp reply
    10.2.0.2 is-at 00:0c:29:f3:64:fb
    On net2, as seen from a fourth machine monitoring traffic for
    debugging this: arp who-has 10.2.0.2 and arp reply 10.2.0.2 is-at
    00:0c:29:f3:64:fb
    On net2, as seen from le2: arp who-has 10.2.0.2

    Why doesn't the arp reply 1) show up on le2 and 2) make it back to
    net1 so source can see it? Is even the problem?

    For some sanity checks:
    If I assign le1 10.2.0.3, then bridge can ping 10.2.0.1 (source).
    If I assign le2 10.2.0.3 then bridge can ping 10.2.0.2 (sink).
    If I assign bridge0 10.2.0.3, then I can not ping either source or
    sink.

    If manually set the MAC address of 10.2.0.2 on 10.2.0.1, then I see:
    On net2, nothing.
    On net1, as seen from le1, nothing.
    On net1, as seen from source's nic and a sniffer on the network: IP
    10.2.0.1 > 10.2.0.2: icmp 64: echo request seq

    I'm new to FreeBSD (previously long time Linux user), so there could
    be something basic I'm missing. If it matters, all three machines are
    VMWare virtual machines and the networks are virtual networks.

    I've been banging my head against a wall all day trying to figure this
    out, please help. Thank you,
    Ryan

    [user@bridge]$ uname -a
    FreeBSD bridge.localdomain 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sun Feb
    24 19:59:52 UTC 2008 root@logan.cse.buffalo.edu:/usr/obj/usr/src/
    sys/GENERIC i386

    [user@bridge]$ ifconfig
    lo0: flags=8049 metric 0 mtu 16384
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    inet6 ::1 prefixlen 128
    inet 127.0.0.1 netmask 0xff000000
    le0: flags=8843 metric 0 mtu
    1500
    options=8
    ether 00:0c:29:1c:c0:a8
    inet 172.16.0.129 netmask 0xffffff00 broadcast 172.16.0.255
    media: Ethernet autoselect
    status: active
    le1: flags=8943 metric
    0 mtu 1500
    options=8
    ether 00:0c:29:1c:c0:b2
    media: Ethernet autoselect
    status: active
    le2: flags=8943 metric
    0 mtu 1500
    options=8
    ether 00:0c:29:1c:c0:bc
    media: Ethernet autoselect
    status: active
    bridge0: flags=8843 metric 0
    mtu 1500
    ether 66:0c:0e:5f:57:56
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: le2 flags=143
    member: le1 flags=143


    [user@bridge]$ sysctl -a | grep bridge
    kern.hostname: voipbridge.localdomain
    net.link.bridge.ipfw: 0
    net.link.bridge.log_stp: 0
    net.link.bridge.pfil_local_phys: 0
    net.link.bridge.pfil_member: 1
    net.link.bridge.pfil_bridge: 1
    net.link.bridge.ipfw_arp: 0
    net.link.bridge.pfil_onlyip: 1
    dev.pcib.0.%desc: ACPI Host-PCI bridge
    dev.pcib.1.%desc: ACPI PCI-PCI bridge
    dev.hostb.0.%desc: Host to PCI bridge
    dev.agp.0.%desc: Intel 82443BX (440 BX) host to PCI bridge
    dev.isab.0.%desc: PCI-ISA bridge

    [user@bridge]$ sysctl -a | grep ether
    kern.random.sys.harvest.ethernet: 1
    net.link.ether.inet.log_arp_permanent_modify: 1
    net.link.ether.inet.log_arp_movements: 1
    net.link.ether.inet.log_arp_wrong_iface: 1
    net.link.ether.inet.proxyall: 0
    net.link.ether.inet.useloopback: 1
    net.link.ether.inet.maxtries: 5
    net.link.ether.inet.max_age: 1200
    net.link.ether.ipfw: 0

    [user@bridge]$ sysctl -a | grep icmp
    net.inet.icmp.maskrepl: 0
    net.inet.icmp.icmplim: 200
    net.inet.icmp.bmcastecho: 0
    net.inet.icmp.quotelen: 8
    net.inet.icmp.reply_from_interface: 0
    net.inet.icmp.reply_src:
    net.inet.icmp.icmplim_output: 1
    net.inet.icmp.log_redirect: 0
    net.inet.icmp.drop_redirect: 0
    net.inet.icmp.maskfake: 0
    net.inet.tcp.icmp_may_rst: 1
    net.inet6.icmp6.rediraccept: 1
    net.inet6.icmp6.redirtimeout: 600
    net.inet6.icmp6.nd6_prune: 1
    net.inet6.icmp6.nd6_delay: 5
    net.inet6.icmp6.nd6_umaxtries: 3
    net.inet6.icmp6.nd6_mmaxtries: 3
    net.inet6.icmp6.nd6_useloopback: 1
    net.inet6.icmp6.nodeinfo: 3
    net.inet6.icmp6.errppslimit: 100
    net.inet6.icmp6.nd6_maxnudhint: 0
    net.inet6.icmp6.nd6_debug: 0
    net.inet6.icmp6.nd6_maxqueuelen: 1

    [user@bridge]$ sysctl -a | grep arp
    net.inet.ip.same_prefix_carp_only: 0
    net.link.ether.inet.log_arp_permanent_modify: 1
    net.link.ether.inet.log_arp_movements: 1
    net.link.ether.inet.log_arp_wrong_iface: 1
    net.link.bridge.ipfw_arp: 0

    [user@bridge]$ cat /boot/loader.conf
    #bridge_load="YES"
    #ipfw_load="YES"
    ipdivert_load="YES"
    if_bridge_load="YES"
    bridgestp_load="YES"

    [user@bridge]$ cat /etc/rc.conf
    # -- sysinstall generated deltas -- # Mon Aug 25 17:23:46 2008
    # Created: Mon Aug 25 17:23:46 2008
    # Enable network daemons for user convenience.
    # Please make all changes to this file, not to /etc/defaults/rc.conf.
    # This file now contains just the overrides from /etc/defaults/
    rc.conf.
    gateway_enable="YES"
    hostname="voipbridge.localdomain"
    ifconfig_le0="DHCP"
    inetd_enable="NO"
    linux_enable="YES"
    sshd_enable="YES"

    gateway_enable="YES"

    firewall_enable="YES"
    firewall_type="open"
    firewall_quiet="YES"
    firewall_logging="YES"

    cloned_interfaces="bridge0"
    ifconfig_bridge0="addm le1 addm le2 up"
    ifconfig_le1="up"
    ifconfig_le2="up"

    [user@bridge]$ ipfw show
    00100 0 0 allow ip from any to any via lo0
    00200 0 0 deny ip from any to 127.0.0.0/8
    00300 0 0 deny ip from 127.0.0.0/8 to any
    65000 0 0 allow ip from any to any
    65535 0 0 deny ip from any to any


  2. Re: Why is this super simple ethernet bridge not working?

    ryanjfarley@gmail.com wrote:

    > I am having trouble setting up a fresh install of FreeBSD as an
    > ethernet bridge.
    > I have two boxes, call them source, 10.2.0.1, and sink 10.2.0.2. When
    > they are on the same network they can ping each other fine. However,
    > when I insert the FreeBSD box, call it bridge, between source and
    > sink, they can no longer reach each other.
    >
    > source <--net1-->(le1)bridge0(le2)<--net2--> sink
    >
    > In particular, if I ping sink from source, then I see:
    > On net1, as seen from both source's nic and le1: arp who-has 10.2.0.2
    > (sink)
    > On net2, as seen from sink's nic: arp who-has 10.2.0.2 and arp reply
    > 10.2.0.2 is-at 00:0c:29:f3:64:fb
    > On net2, as seen from a fourth machine monitoring traffic for
    > debugging this: arp who-has 10.2.0.2 and arp reply 10.2.0.2 is-at
    > 00:0c:29:f3:64:fb
    > On net2, as seen from le2: arp who-has 10.2.0.2
    >
    > Why doesn't the arp reply 1) show up on le2 and 2) make it back to
    > net1 so source can see it? Is even the problem?
    >

    [snippage]

    It has been since a very long time that I've played with a bridge. I don't
    have the magic bullet answer and my memory is very hazy at this point, but
    I believe at first glance what you may be missing is arp-proxy. It's built
    into the arp command, so man arp for more details.

    Example from googling:

    % arp -s 1.2.3.4 AA:BB:CC:11:22:33 pub

    To verify it worked:

    % arp -a

    (1.2.3.100) at aa:bb:cc:11:22:33 on ed0 [ethernet]
    (1.2.3.4.) at aa:bb:cc:11:22:33 on ed0 permanent published [ethernet]

    The pub or "published" switch is what invokes the ARP proxy.

    Your situation is more complicated and will take some figuring and
    configuring, but this may be the missing piece. Try googling for more stuff
    on arp-proxy for a handle. YMMV

    -Mike


  3. Re: Why is this super simple ethernet bridge not working?

    On Aug 27, 8:24 pm, Jason Bourne
    wrote:
    > ryanjfar...@gmail.com wrote:
    > > I am having trouble setting up a fresh install of FreeBSD as an
    > > ethernet bridge.
    > > I have two boxes, call them source, 10.2.0.1, and sink 10.2.0.2. When
    > > they are on the same network they can ping each other fine. However,
    > > when I insert the FreeBSD box, call it bridge, between source and
    > > sink, they can no longer reach each other.

    >
    > > source <--net1-->(le1)bridge0(le2)<--net2--> sink

    >
    > > In particular, if I ping sink from source, then I see:
    > > On net1, as seen from both source's nic and le1: arp who-has 10.2.0.2
    > > (sink)
    > > On net2, as seen from sink's nic: arp who-has 10.2.0.2 and arp reply
    > > 10.2.0.2 is-at 00:0c:29:f3:64:fb
    > > On net2, as seen from a fourth machine monitoring traffic for
    > > debugging this: arp who-has 10.2.0.2 and arp reply 10.2.0.2 is-at
    > > 00:0c:29:f3:64:fb
    > > On net2, as seen from le2: arp who-has 10.2.0.2

    >
    > > Why doesn't the arp reply 1) show up on le2 and 2) make it back to
    > > net1 so source can see it? Is even the problem?

    >
    > [snippage]
    >
    > It has been since a very long time that I've played with a bridge. I don't
    > have the magic bullet answer and my memory is very hazy at this point, but
    > I believe at first glance what you may be missing is arp-proxy. It's built
    > into the arp command, so man arp for more details.
    >
    > Example from googling:
    >
    > % arp -s 1.2.3.4 AA:BB:CC:11:22:33 pub
    >
    > To verify it worked:
    >
    > % arp -a
    >
    > (1.2.3.100) at aa:bb:cc:11:22:33 on ed0 [ethernet]
    > (1.2.3.4.) at aa:bb:cc:11:22:33 on ed0 permanent published [ethernet]
    >
    > The pub or "published" switch is what invokes the ARP proxy.
    >
    > Your situation is more complicated and will take some figuring and
    > configuring, but this may be the missing piece. Try googling for more stuff
    > on arp-proxy for a handle. YMMV
    >
    > -Mike


    I've since looked into arp-proxy and it unfortunately doesn't work. I
    may look into netgraph, but am unsure. I've tried a Linux ethernet
    bridge in place of the FreeBSD bridge and have gotten the same
    results. This seems like the most trivial bridge example, I feel dumb.

    The strange thing is that if I make net1 and net2 different subnets
    and make bridge an IP router between them, then source can ping sink
    so it's not a physical connectivity issue. I'm looking into whether
    this is a problem due to running on vmnets -- I'll use the same config
    for bridge but on a physical machine. I'm wondering if this isn't a
    config or even FreeBSD thing, but possibly bc vmware handles something
    funky.

    -Ryan

    For completeness, here's an email I sent a colleague trying to figure
    this out:
    source:eth0(00:0c:29:be:f3:7f|10.2.0.1) -> le1:bridge:le2 -
    >sink:eth0(00:0c:29:45:d1:77|10.2.0.2)

    bridge0 on host bridge has members le1 and le2.

    [root@bridge /usr/home/gmusip]# ifconfig bridge0 addr
    00:0c:29:be:f3:7f Vlan1 le1 1084 flags=0<>
    00:0c:29:1c:c0:a8 Vlan1 le2 810 flags=0<>

    Here is a sample trace of a frame when source runs 'ping 10.2.0.2'
    As seen from source:eth0
    00:0c:29:be:f3:7f (oui Unknown) > Broadcast, ethertype ARP (0x0806),
    length 60: arp who-has 10.2.0.2 tell 10.2.0.1

    As seen from bridge:le1
    00:0c:29:be:f3:7f (oui Unknown) > Broadcast, ethertype ARP (0x0806),
    length 60: arp who-has 10.2.0.2 tell 10.2.0.1

    As seen from the vmware host on the vmnet that contains source:eth0
    and bridge:le1
    00:0c:29:be:f3:7f > Broadcast, ethertype ARP (0x0806), length 60: arp
    who-has 10.2.0.2 tell 10.2.0.1

    As seen from bridge:bridge0
    00:0c:29:be:f3:7f > Broadcast, ethertype ARP (0x0806), length 60: arp
    who-has 10.2.0.2 tell 10.2.0.1

    As seen from bridge:le2
    00:0c:29:be:f3:7f (oui Unknown) > Broadcast, ethertype ARP (0x0806),
    length 60: arp who-has 10.2.0.2 tell 10.2.0.1

    As seen from sink:eth0
    00:0c:29:be:f3:7f (oui Unknown) > Broadcast, ethertype ARP (0x0806),
    length 60: arp who-has 10.2.0.2 tell 10.2.0.1
    00:0c:29:45:d1:77 (oui Unknown) > 00:0c:29:be:f3:7f (oui Unknown),
    ethertype ARP (0x0806), length 42: arp reply 10.2.0.2 is-at 00:0c:
    29:45:d1:77 (oui Unknown)

    As seen from the vmware host on the vmnet that contains bridge:le2 and
    sink:eth0
    00:0c:29:be:f3:7f > Broadcast, ethertype ARP (0x0806), length 60: arp
    who-has 10.2.0.2 tell 10.2.0.1
    00:0c:29:45:d1:77 > 00:0c:29:be:f3:7f, ethertype ARP (0x0806),
    length42: arp reply 10.2.0.2 is-at 00:0c:29:45:d1:77

    Note that bridge:le2 and sink:eth0 are be on the same network, yet
    they are seeing different things.

    I've tried the setup with NAT'ed vmnets and the results are the same
    as hostonly vmnets. I've tried a Linux bridge and the results are the
    same as a FreeBSD bridge.

    Here is the trace when source runs 'arp -s 10.2.0.2 00:0c:29:45:d1:77
    && ping 10.2.0.2'
    As seen from source:eth0
    00:0c:29:be:f3:7f (oui Unknown) > 00:0c:29:45:d1:77 (oui Unknown),
    ethertype IPv4 (0x0800), length 98: 10.2.0.1 > 10.2.0.2: ICMP echo
    request, id 15624, seq 1603, length 64

    As seen from bridge:le1


    As seen from the vmware host on the vmnet that contains source:eth0
    and bridge:le1
    00:0c:29:be:f3:7f > 00:0c:29:45:d1:77, ethertype IPv4 (0x0800), length
    98: IP 10.2.0.1 > 10.2.0.2: icmp 64: echo request seq 1603

    As seen from bridge:bridge0


    As seen from bridge:le2


    As seen from sink:eth0


    As seen from the vmware host on the vmnet that contains bridge:le2 and
    sink:eth0



  4. Re: Why is this super simple ethernet bridge not working?

    rfarley wrote:
    > I've since looked into arp-proxy and it unfortunately doesn't work.


    I doubted it anyway. You don't need proxy arp, you need a bridge.

    > This seems like the most trivial bridge example, I feel dumb.


    Can you tell us what you actually configured on your BSD box?

    And, just for curiosity's sake - why not just take a small
    switch for 20 bucks and let that one do the bridging? Is this
    a "I want to get this working" thing or a "I need special
    features the vanilla switch doesn't give me" issue?

    If the latter, which exactly?

    Regards

    Christop

  5. Re: Why is this super simple ethernet bridge not working?

    On Wed, 27 Aug 2008 12:18:49 -0700, ryanjfarley@gmail.com wrote:

    > I am having trouble setting up a fresh install of FreeBSD as an ethernet
    > bridge.
    > I have two boxes, call them source, 10.2.0.1, and sink 10.2.0.2. When they
    > are on the same network they can ping each other fine. However, when I
    > insert the FreeBSD box, call it bridge, between source and sink, they can
    > no longer reach each other.
    >
    > source <--net1-->(le1)bridge0(le2)<--net2--> sink
    >
    > In particular, if I ping sink from source, then I see: On net1, as seen
    > from both source's nic and le1: arp who-has 10.2.0.2 (sink)
    > On net2, as seen from sink's nic: arp who-has 10.2.0.2 and arp reply
    > 10.2.0.2 is-at 00:0c:29:f3:64:fb
    > On net2, as seen from a fourth machine monitoring traffic for debugging
    > this: arp who-has 10.2.0.2 and arp reply 10.2.0.2 is-at 00:0c:29:f3:64:fb
    > On net2, as seen from le2: arp who-has 10.2.0.2
    >
    > Why doesn't the arp reply 1) show up on le2 and 2) make it back to net1 so
    > source can see it? Is even the problem?
    >
    > For some sanity checks:
    > If I assign le1 10.2.0.3, then bridge can ping 10.2.0.1 (source). If I
    > assign le2 10.2.0.3 then bridge can ping 10.2.0.2 (sink). If I assign
    > bridge0 10.2.0.3, then I can not ping either source or sink.
    >
    > If manually set the MAC address of 10.2.0.2 on 10.2.0.1, then I see: On
    > net2, nothing.
    > On net1, as seen from le1, nothing.
    > On net1, as seen from source's nic and a sniffer on the network: IP
    > 10.2.0.1 > 10.2.0.2: icmp 64: echo request seq
    >
    > I'm new to FreeBSD (previously long time Linux user), so there could be
    > something basic I'm missing. If it matters, all three machines are VMWare
    > virtual machines and the networks are virtual networks.


    When you configure your virtual machines you have the opportunity to
    configure networking. Perhaps you should look at vmware help regarding
    networking between boxes. Getting vmware to network realboxes is trivial
    but perhaps there is some extra info required when networking virtual
    boxes. I dunno. never done it myself. Just my 2 cents.

    --
    ___ _______ ___ ___ ___ __ ____
    / _ \/ __/ _ | / _ \ / _ \/ _ |/ / / / /
    / // / _// __ |/ // / / ___/ __ / /_/ / /__
    /____/___/_/ |_/____/ /_/ /_/ |_\____/____/


  6. Re: Why is this super simple ethernet bridge not working?

    Thanks for all the follow-ups.

    > why not just take a small switch for 20 bucks and let that one do the bridging?


    Feel free to let me know if there is better output to describe the
    machine's config than what's in the first email.

    This falls into the "I need special features the vanilla switch
    doesn't give me" issue. I'm a grad student and work in a lab with
    mostly VMs. The goal is to build a small plug-and-go device that I can
    use for monitoring and manipulating frames in a test environment.

    > When you configure your virtual machines you have the opportunity to
    > configure networking. Perhaps you should look at vmware help regarding
    > networking between boxes. Getting vmware to network realboxes is trivial
    > but perhaps there is some extra info required when networking virtual
    > boxes. I dunno. never done it myself. Just my 2 cents.


    VMware networking settings "shouldn't" be the issue, unless VMware
    vmnets handle this situation abnormally. I can successfully use the
    same VM (the one named bridge) as a router if the same VMnets are on
    different subnets.

    > You've left filtering enabled on your bridge.

    [root@bridge]# sysctl net.link.bridge.pfil_member=0
    net.link.bridge.pfil_member: 1 -> 0
    [root@bridge]# sysctl net.link.bridge.pfil_bridge=0
    net.link.bridge.pfil_bridge: 1 -> 0
    [root@bridge]# sysctl net.link.bridge.pfil_onlyip=0
    net.link.bridge.pfil_onlyip: 1 -> 0

    No effects. Results same as before.

    It seems like there should be a one-liner I forgot to run or some
    setting that should be disabled. I'm going to keep plugging away at
    this.

    Willingly accepting any ideas, thanks,
    -Ryan

    On Aug 29, 5:20 pm, Christoph Weber-Fahr wrote:
    > rfarley wrote:
    > > I've since looked into arp-proxy and it unfortunately doesn't work.

    >
    > I doubted it anyway. You don't need proxy arp, you need a bridge.
    >
    > > This seems like the most trivial bridge example, I feel dumb.

    >
    > Can you tell us what you actually configured on your BSD box?
    >
    > And, just for curiosity's sake - why not just take a small
    > switch for 20 bucks and let that one do the bridging? Is this
    > a "I want to get this working" thing or a "I need special
    > features the vanilla switch doesn't give me" issue?
    >
    > If the latter, which exactly?
    >
    > Regards
    >
    > Christop



+ Reply to Thread