This is a discussion on My thoughts on configuring zones with shared IP instances and the 'defrouter' paramet - Solaris Rss ; An occasional call or email I receive has questions about routing issues when using Solaris Zones in the (default) shared IP Instance configuration. Everything works well when the non-global zones are on the same IP subnet (lets say 172.16.1.0/24) as ...
An occasional call or email I receive has questions about routing issues when using Solaris Zones in the (default) shared IP Instance configuration. Everything works well when the non-global zones are on the same IP subnet (lets say 172.16.1.0/24) as the global zone. Routing gets a little tricky when the non-global zones are on a different subnet.
My general recommendation is to isolate. This means:
Using separate data-links is not always possible. I was concerned whether this would actually work.
- Separate subnets for the global zone (administration, backup) and the non-global zones (applications, data).
- Separate data-links for the global and non-global zones.
- The non-global zones can share a data-link
- Non-global zones on different IP subnets use different data-links
So I did some testing, and exchanged some emails because of a comment I made regarding PSARC/2008/057 and the automatic removal of a default route when the zone is halted.
Turns out I have been very restrictive in suggesting that the global and non-global zones not share a data-link. While I think that is a good administrative policy, to separate administrative and application traffic, it is not a requirement. It is OK to have the global zone and one or more non-global zones share the same data-link. However, if the non-global zones are to have different default routes, they must be on subnets that the global zone is not on.
My test case running Solaris 10 10/09 has the global zone on the 220.127.116.11/24 network and the non-global zone on the 172.16.27.0/24 network. global# ifconfig -a...e1000g0: flags=1000843 mtu 1500 index 2 inet 18.104.22.168 netmask ffffff00 broadcast 22.214.171.124 ether 0:14:4f:ac:57:c4e1000g0:1: flags=1000843 mtu 1500 index 2 zone shared1 inet 172.16.27.27 netmask ffffff00 broadcast 172.16.27.255global# zonecfg -z shared1 info netnet: address: 172.16.27.27/24 physical: e1000g0 defrouter: 172.16.27.16The routing table as seen from both are:global# netstat -rnRouting Table: IPv4 Destination Gateway Flags Ref Use Interface-------------------- -------------------- ----- ----- ---------- ---------default 126.96.36.199 UG 1 123default 172.16.27.16 UG 1 7 e1000g0188.8.131.52 184.108.40.206 U 1 50 e1000g0220.127.116.11 18.104.22.168 U 1 0 e1000g0127.0.0.1 127.0.0.1 UH 3 80 lo0shared1# netstat -rnRouting Table: IPv4 Destination Gateway Flags Ref Use Interface-------------------- -------------------- ----- ----- ---------- ---------default 172.16.27.16 UG 1 7 e1000g0172.16.27.0 172.16.27.27 U 1 3 e1000g0:122.214.171.124 172.16.27.27 U 1 0 e1000g0:1127.0.0.1 127.0.0.1 UH 4 78 lo0:1While the global zone shows both routes, only the default applying to its subnet will be used. And for traffic leaving the non-global zone, only its default will be used.
You may notice that the Interface for the global zone's default router is blank. That is because I have set the default route via /etc/defaultrouter. I noticed that if it is determined via the route discovery daemon, it will be listed as being on e1000g0! This does not affect the behavior, however it may be visually confusing, which is probably why I initially leaned towards saying to not share the data-link.
There are multiple ways to determining which route might be used, including ping(1M) and traceroute(1M). I like the output of the route get command.global# route get 172.16.29.1 route to: 172.16.29.1destination: default mask: default gateway: 126.96.36.199 interface: e1000g0 flags: recvpipe sendpipe ssthresh rtt,ms rttvar,ms hopcount mtu expire 0 0 0 0 0 0 1500 0shared1# route get 172.16.28.1 route to: 172.16.28.1destination: default mask: default gateway: 172.16.27.16 interface: e1000g0:1 flags: recvpipe sendpipe ssthresh rtt,ms rttvar,ms hopcount mtu expire 0 0 0 0 0 0 1500 0This quickly shows which interfaces and IP addresses are being used. If there are multiple default routes, repeated invocations of this will show a rotation in the selection of the default routes.
Thanks to Erik Nordmark and Penny Cotten for their insights on this topic!