My thoughts on configuring zones with shared IP instances and the 'defrouter' paramet
An occasional call or email I receive has questions about routing issues when using Solaris Zones in the (default) shared IP Instance configuration. Everything works well when the non-global zones are on the same IP subnet (lets say 172.16.1.0/24) as the global zone. Routing gets a little tricky when the non-global zones are on a different subnet.
My general recommendation is to isolate. This means:[LIST] [*]Separate subnets for the global zone (administration, backup) and the non-global zones (applications, data). [*]Separate data-links for the global and non-global zones.[LIST] [*]The non-global zones can share a data-link [*]Non-global zones on different IP subnets use different data-links [/LIST][/LIST]Using separate data-links is not always possible. I was concerned whether this would actually work.
So I did some testing, and exchanged some emails because of a comment I made regarding [URL="http://arc.opensolaris.org/caselog/PSARC/2008/057/"]PSARC/2008/057[/URL] and the automatic removal of a default route when the zone is halted.
Turns out I have been very restrictive in [I]suggesting[/I] that the global and non-global zones not share a data-link. While I think that is a good administrative policy, to separate administrative and application traffic, it is not a requirement. It is OK to have the global zone and one or more non-global zones share the same data-link. However, if the non-global zones are to have different default routes, they must be on subnets that the global zone is not on.
My test case running Solaris 10 10/09 has the global zone on the 220.127.116.11/24 network and the non-global zone on the 172.16.27.0/24 network. global# ifconfig -a...e1000g0: flags=1000843 mtu 1500 index 2 inet 18.104.22.168 netmask ffffff00 broadcast 22.214.171.124 ether 0:14:4f:ac:57:c4e1000g0:1: flags=1000843 mtu 1500 index 2 zone shared1 inet 172.16.27.27 netmask ffffff00 broadcast 172.16.27.255global# zonecfg -z shared1 info netnet: address: 172.16.27.27/24 physical: e1000g0 defrouter: 172.16.27.16The routing table as seen from both are:global# netstat -rnRouting Table: IPv4 Destination Gateway Flags Ref Use Interface-------------------- -------------------- ----- ----- ---------- ---------default 126.96.36.199 UG 1 123default 172.16.27.16 UG 1 7 e1000g0188.8.131.52 184.108.40.206 U 1 50 e1000g0220.127.116.11 18.104.22.168 U 1 0 e1000g0127.0.0.1 127.0.0.1 UH 3 80 lo0shared1# netstat -rnRouting Table: IPv4 Destination Gateway Flags Ref Use Interface-------------------- -------------------- ----- ----- ---------- ---------default 172.16.27.16 UG 1 7 e1000g0172.16.27.0 172.16.27.27 U 1 3 e1000g0:122.214.171.124 172.16.27.27 U 1 0 e1000g0:1127.0.0.1 127.0.0.1 UH 4 78 lo0:1While the global zone shows both routes, only the default applying to its subnet will be used. And for traffic leaving the non-global zone, only its default will be used.
You may notice that the [I]Interface[/I] for the global zone's default router is blank. That is because I have set the default route via [I]/etc/defaultrouter[/I]. I noticed that if it is determined via the route discovery daemon, it will be listed as being on [I]e1000g0[/I]! This does not affect the behavior, however it may be visually confusing, which is probably why I initially leaned towards saying to not share the data-link.
There are multiple ways to determining which route might be used, including ping(1M) and traceroute(1M). I like the output of the [I]route get[/I] command.global# route get 172.16.29.1 route to: 172.16.29.1destination: default mask: default gateway: 126.96.36.199 interface: e1000g0 flags: recvpipe sendpipe ssthresh rtt,ms rttvar,ms hopcount mtu expire 0 0 0 0 0 0 1500 0shared1# route get 172.16.28.1 route to: 172.16.28.1destination: default mask: default gateway: 172.16.27.16 interface: e1000g0:1 flags: recvpipe sendpipe ssthresh rtt,ms rttvar,ms hopcount mtu expire 0 0 0 0 0 0 1500 0This quickly shows which interfaces and IP addresses are being used. If there are multiple default routes, repeated invocations of this will show a rotation in the selection of the default routes.
Thanks to Erik Nordmark and Penny Cotten for their insights on this topic!