Lately, I've been playing around with smaller OpenSolaris guests. Not wanting to muck around with setting up an AI installer, I ended up doing a "fresh install" into a new Boot Environment (BE). Although none of this is overly practical, it is interesting.. I thought I would share what I learned..
This works just as well on metal, in a VirtualBox OpenSolaris guest, and in a xVM OpenSolaris guest. For this example, I'll run through an xVM OpenSolaris guest. You'll need to change your package list slightly for the other ones.
First, a little off subject, lets install a fresh OpenSolaris guest.. This assumes your already running a OpenSolaris dom0. : core2[1]#; virt-install -n opensolaris -r 1024 -p --nographics \ --noautoconsole -l /net/192.168.0.71/tank/isos/solaris/os2009.06.iso \ -f /vdisks/opensolaris -s 20 Starting install... Retrieving file unix... 100% |=========================| 1.4 MB 00:00 Retrieving file x86.micro 100% |=========================| 36 MB 00:01 Creating storage file... 100% |=========================| 20 B 00:00 Creating domain... 0 B 00:05 Domain installation still in progress. You can reconnect to the console to complete the installation process. : core2[1]#; virsh console opensolaris v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b' SunOS Release 5.11 Version snv_111b 32-bit Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: opensolaris Remounting root read/write Probing for device nodes ... Preparing live image for use Done mounting Live image USB keyboard 1. Albanian 23. Lithuanian 2. Belarusian 24. Latvian 3. Belgian 25. Macedonian 4. Brazilian 26. Malta_UK 5. Bulgarian 27. Malta_US 6. Canadian-Bilingual 28. Norwegian 7. Croatian 29. Polish 8. Czech 30. Portuguese 9. Danish 31. Russian 10. Dutch 32. Serbia-And-Monte***** 11. Finnish 33. Slovenian 12. French 34. Slovakian 13. French-Canadian 35. Spanish 14. Hungarian 36. Swedish 15. German 37. Swiss-French 16. Greek 38. Swiss-German 17. Icelandic 39. Traditional-Chinese 18. Italian 40. TurkishQ 19. Japanese-type6 41. TurkishF 20. Japanese 42. UK-English 21. Korean 43. US-English 22. Latin-American To select the keyboard layout, enter a number [default 43]: 1. Arabic 2. Chinese - Simplified 3. Chinese - Traditional 4. Czech 5. Dutch 6. English 7. French 8. German 9. Greek 10. Hebrew 11. Hungarian 12. Indonesian 13. Italian 14. Japanese 15. Korean 16. Polish 17. Portuguese - Brazil 18. Russian 19. Slovak 20. Spanish 21. Swedish To select desktop language, enter a number [default is 6]: User selected: English Configuring devices. Mounting cdroms Reading ZFS config: done. opensolaris console login: jack Password: Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008 jack@opensolaris:~$ jack@opensolaris:~$ Aug 17 02:58:54 opensolaris in.routed[696]: route 0.0.0.0/8 --> 0.0.0.0 nexthop is not directly connected jack@opensolaris:~$ ifconfig xnf0 xnf0: flags=1004843 mtu 1500 index 2 inet 192.168.0.147 netmask ffffff00 broadcast 192.168.0.255 jack@opensolaris:~$ jack@opensolaris:~$ (^] to exit console) : core2[1]#; /usr/lib/xen/bin/xenstore-ls | grep passwd passwd = "EJzbFnyg" : core2[1]#; Connect a vncviewer to the OpenSolaris guest, using the vnc password above. : core2[1]#; vncviewer 192.168.0.147:0 &> /dev/null & Once you've completed the OpenSolaris install, lets disable gdm and intrd (I like to do this for small guests), and apply the workaround for a zfs bug (6840704 osol_0906 PV guests sometimes hang at login prompt). Then lets create a working BE. If we mess anything up, we can always go back to where we started from. : core2[1]#; virsh start opensolaris;virsh console opensolaris Domain opensolaris started v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b' SunOS Release 5.11 Version snv_111b 64-bit Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: opensolaris Configuring devices. Loading smf(5) service descriptions: 150/150 svccfg import warnings. See /var/svc/log/system-manifest-import:default.log . Reading ZFS config: done. Mounting ZFS filesystems: (6/6) Creating new rsa public/private host key pair Creating new dsa public/private host key pair opensolaris console login: myuser Password: Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008 myuser@opensolaris:~$ pfexec su - Aug 17 10:58:54 opensolaris su: 'su root' succeeded for myuser on /dev/console Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008 root@opensolaris:~# svcadm disable gdm root@opensolaris:~# svcadm disable intrd root@opensolaris:~# echo -e "\n"\ "forceload: drv/domcaps\n"\ "forceload: drv/xencons\n"\ "forceload: drv/xenbus\n"\ "forceload: drv/balloon\n"\ "forceload: drv/evtchn\n"\ "forceload: drv/privcmd\n"\ "forceload: drv/xdf\n"\ "forceload: drv/xnf\n\n" >> /etc/system root@opensolaris:~# bootadm update-archive updating //platform/i86pc/boot_archive updating //platform/i86pc/amd64/boot_archive root@opensolaris:~# beadm create snv111b root@opensolaris:~# beadm activate snv111b root@opensolaris:~# reboot Aug 17 11:36:01 opensolaris reboot: initiated by myuser on /dev/console syncing file systems... done rebooting... v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b' SunOS Release 5.11 Version snv_111b 64-bit Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: opensolaris Reading ZFS config: done. Mounting ZFS filesystems: (6/6) opensolaris console login: Now lets get to the point of this blog entry. Lets do a fresh install to a new BE.
When you do a beadm create, you are creating a copy-on-write(COW) based clone of your current root. We don't want that though.. We want an empty root directory to start with.. So we'll create it by hand.. We'll need a uuid. You can write a little program using libuuid(3LIB) or just make one up. Also, we'll put the mountdir to a temporary location during the install. opensolaris console login: myuser Password: Last login: Mon Aug 17 11:33:40 on console Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008 myuser@opensolaris:~$ pfexec su - Aug 17 11:39:04 opensolaris su: 'su root' succeeded for myuser on /dev/console Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008 root@opensolaris:~# zfs create rpool/ROOT/small-be root@opensolaris:~# zfs set canmount=noauto rpool/ROOT/small-be root@opensolaris:~# zfs set mountpoint="/mnt" rpool/ROOT/small-be root@opensolaris:~# zfs set org.opensolaris.libbe:uuid=f0fa607f-7d1c-66ca-caf9-e04cbf rpool/ROOT/small-be root@opensolaris:~# zfs mount rpool/ROOT/small-be Next, we'll setup a new packaging environment in the new "BE", install a custom set of packages, seed SMF, setup vfstab and /dev, cleanup some OpenSolaris cruft, setup the new BE to prompt for configuration on the next boot, and then apply the workaround for the zfs bug mentioned above. root@opensolaris:~# export ROOTDIR=/mnt root@opensolaris:~# pkg image-create -f -F -a opensolaris.org=http://pkg.opensolaris.org/ $ROOTDIR root@opensolaris:~# export PKGS="entire \ SUNWcsd \ SUNWcs \ SUNWcarx \ SUNWcakrx \ SUNWos86r \ SUNWkvm \ SUNWrmodr \ SUNWpsdcr \ SUNWpsdir \ SUNWcnetr \ SUNWesu \ SUNWkey \ SUNWuprl \ SUNWkrb \ SUNWbip \ SUNWzfskr \ SUNWbash \ SUNWipf \ SUNWbash \ SUNWgrub \ SUNWtoo \ SUNWbind \ SUNWrcmdc \ SUNWmkcd \ SUNWPython \ SUNWPython-extra \ SUNWipkg \ SUNWinstall \ SUNWbeadm \ SUNWadmap \ SUNWadmlib-sysid \ SUNWadmr" root@opensolaris:~# pkg -R $ROOTDIR install $PKGS DOWNLOAD PKGS FILES XFER (MB) Completed 67/67 8542/8542 93.32/93.32 PHASE ACTIONS Install Phase 14726/14726 PHASE ITEMS Reading Existing Index 8/8 Indexing Packages 67/67 Optimizing Index... PHASE ITEMS Indexing Packages 67/67 root@opensolaris:~# rm -rf $ROOTDIR/var/pkg/download/* root@opensolaris:~# /usr/bin/cp $ROOTDIR/lib/svc/seed/global.db $ROOTDIR/etc/svc/repository.db root@opensolaris:~# chmod 600 $ROOTDIR/etc/svc/repository.db root@opensolaris:~# cd $ROOTDIR/var/svc/profile/ root@opensolaris:/mnt/var/svc/profile# ln -s generic_limited_net.xml generic.xml root@opensolaris:/mnt/var/svc/profile# ln -s ns_files.xml name_service.xml root@opensolaris:/mnt/var/svc/profile# cd root@opensolaris:~# cp /etc/vfstab $ROOTDIR/etc/vfstab root@opensolaris:~# /usr/sbin/devfsadm -R $ROOTDIR root@opensolaris:~# echo -e "/lib/svc/method/sshd\n\ /usr/sbin/sysidkbd\n\ /usr/sbin/sysidpm\n\ /lib/svc/method/net-nwam\n\ /usr/lib/cc-ccr/bin/eraseCCRRepository" > $ROOTDIR/etc/.sysidconfig.apps root@opensolaris:~# /usr/sbin/sys-unconfig -R $ROOTDIR sys-unconfig started Mon Aug 17 12:32:56 2009 rm: cannot remove `/mnt/etc/vfstab.sys-u': No such file or directory grep: /mnt/etc/dumpadm.conf: No such file or directory sys-unconfig completed Mon Aug 17 12:32:56 2009 root@opensolaris:~# cat $ROOTDIR/etc/passwd | sed '/^jack/d' > $ROOTDIR/etc/passwd.new;mv -f $ROOTDIR/etc/passwd.new $ROOTDIR/etc/passwd root@opensolaris:~# cat $ROOTDIR/etc/shadow | sed '/^jack/d' > $ROOTDIR/etc/shadow.new;mv -f $ROOTDIR/etc/shadow.new $ROOTDIR/etc/shadow root@opensolaris:~# cat $ROOTDIR/etc/user_attr | sed 's/^root::::type=role;/root::::/g' > $ROOTDIR/etc/user_attr.new;mv -f $ROOTDIR/etc/user_attr.new $ROOTDIR/etc/user_attr root@opensolaris:~# echo -e "\n"\ "forceload: drv/domcaps\n"\ "forceload: drv/xencons\n"\ "forceload: drv/xenbus\n"\ "forceload: drv/balloon\n"\ "forceload: drv/evtchn\n"\ "forceload: drv/privcmd\n"\ "forceload: drv/xdf\n"\ "forceload: drv/xnf\n\n" >> $ROOTDIR/etc/system root@opensolaris:~# /usr/sbin/bootadm update-archive -R $ROOTDIR updating /mnt//platform/i86pc/boot_archive updating /mnt//platform/i86pc/amd64/boot_archive Now lets unmount our new BE, and setup the correct mountpoint. root@opensolaris:~# zfs umount rpool/ROOT/small-be root@opensolaris:~# zfs set mountpoint="/" rpool/ROOT/small-be root@opensolaris:~# beadm list BE Active Mountpoint Space Policy Created -- ------ ---------- ----- ------ ------- opensolaris - - 4.01M static 2009-08-17 10:05 small-be - - 357.76M static 2009-08-17 11:49 snv111b NR / 3.16G static 2009-08-17 11:39 root@opensolaris:~# Time to switch to our new BE and run through the configure. root@opensolaris:~# beadm activate small-be root@opensolaris:~# beadm list BE Active Mountpoint Space Policy Created -- ------ ---------- ----- ------ ------- opensolaris - - 4.01M static 2009-08-17 10:05 small-be R - 357.76M static 2009-08-17 11:49 snv111b N / 3.16G static 2009-08-17 11:39 root@opensolaris:~# reboot Aug 17 12:38:24 opensolaris reboot: initiated by myuser on /dev/console syncing file systems... done rebooting... v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b' SunOS Release 5.11 Version snv_111b 64-bit Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: unknown Configuring devices. Loading smf(5) service descriptions: 78/78 Reading ZFS config: done. Mounting ZFS filesystems: (8/8) What type of terminal are you using? 1) ANSI Standard CRT [CUT] Configuring network interface addresses: xnf0. System identification is completed. unknown console login: root Password: Aug 17 09:45:08 unknown login: ROOT LOGIN /dev/console Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008 root@unknown:~# We went from around 3G to around 370M for disk footprint using the custom set of packages above. Not too bad, but we can improve this over time. Filesystem kbytes used avail capacity Mounted on rpool/ROOT/small-be 20514816 373259 15924049 3% / rpool/ROOT/snv111b 20514816 3005343 15920934 16% /mnt Now lets create a COW based clone of out new BE and switch to it. root@unknown:~# beadm list BE Active Mountpoint Space Policy Created -- ------ ---------- ----- ------ ------- opensolaris - - 4.01M static 2009-08-17 07:05 small-be NR / 364.43M static 2009-08-17 08:49 snv111b - - 3.17G static 2009-08-17 08:39 root@unknown:~# bootadm update-archive updating //platform/i86pc/boot_archive updating //platform/i86pc/amd64/boot_archive root@unknown:~# beadm create small-be-clone root@unknown:~# beadm activate small-be-clone root@unknown:~# beadm list BE Active Mountpoint Space Policy Created -- ------ ---------- ----- ------ ------- opensolaris - - 4.01M static 2009-08-17 07:05 small-be N / 19.5K static 2009-08-17 08:49 small-be-clone R - 364.55M static 2009-08-17 09:56 snv111b - - 3.17G static 2009-08-17 08:39 root@unknown:~# root@unknown:~# reboot Aug 17 09:57:20 unknown reboot: initiated by root on /dev/console syncing file systems... done rebooting... v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b' SunOS Release 5.11 Version snv_111b 64-bit Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: unknown Reading ZFS config: done. Mounting ZFS filesystems: (9/9) unknown console login: root Password: Aug 17 13:46:34 unknown login: ROOT LOGIN /dev/console Last login: Mon Aug 17 13:44:19 on console Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008 root@unknown:~# root@unknown:~# beadm list BE Active Mountpoint Space Policy Created -- ------ ---------- ----- ------ ------- opensolaris - - 4.01M static 2009-08-17 10:05 small-be - - 3.78M static 2009-08-17 11:49 small-be-clone NR / 420.45M static 2009-08-17 12:56 snv111b - - 3.17G static 2009-08-17 11:39 From here, we are going to try two different things.. The first thing we want to try is to make sure we can delete the new BE and it's clone. We also want to get a little crazy and see if we can get rid of the original opensolaris and snv111b snapshot. But before we continue on, lets snapshot the vdisk so we can rollback to this point so we don't need to install from scratch again. root@unknown:~# poweroff Aug 17 13:47:20 unknown poweroff: initiated by root on /dev/console syncing file systems... done : core2[1]#; vdiskadm -u xvm snapshot /vdisks/opensolaris@pre-destroy : core2[1]#; virsh start opensolaris;virsh console opensolaris Lets remove the small-be-clone clone and the small-be BE. For small-be, since we created this by hand, we will want to remove it by hand. opensolaris console login: root Password: Last login: Mon Aug 17 11:46:42 on console Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008 root@unknown:~# beadm activate snv111b root@unknown:~# reboot Aug 17 09:01:18 unknown reboot: initiated by root on /dev/console syncing file systems... done rebooting... v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b' SunOS Release 5.11 Version snv_111b 64-bit Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: opensolaris Reading ZFS config: done. Mounting ZFS filesystems: (9/9) opensolaris console login: myuser Password: Last login: Mon Aug 17 11:46:42 on console Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008 myuser@opensolaris:~$ pfexec su - Aug 17 13:10:19 opensolaris su: 'su root' succeeded for myuser on /dev/console Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008 root@opensolaris:~# beadm destroy small-be-clone Are you sure you want to destroy small-be-clone? This action cannot be undone(y/[n]): y root@opensolaris:~# zfs destroy rpool/ROOT/small-be root@opensolaris:~# beadm list BE Active Mountpoint Space Policy Created -- ------ ---------- ----- ------ ------- opensolaris - - 4.01M static 2009-08-17 10:05 snv111b NR / 3.18G static 2009-08-17 11:39 root@opensolaris:~# Now, lets rollback the vdisk and try the second part of our test. root@unknown:~# poweroff Aug 17 15:14:27 unknown poweroff: initiated by root on /dev/console syncing file systems... done : core2[1]#; vdiskadm -u xvm rollback /vdisks/opensolaris@pre-destroy : core2[1]#; virsh start opensolaris;virsh console opensolaris Domain opensolaris started v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b' SunOS Release 5.11 Version snv_111b 64-bit Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: unknown Reading ZFS config: done. Mounting ZFS filesystems: (9/9) unknown console login: root Password: Aug 17 15:22:00 unknown login: ROOT LOGIN /dev/console Last login: Mon Aug 17 13:46:34 on console Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008 root@unknown:~# beadm list BE Active Mountpoint Space Policy Created -- ------ ---------- ----- ------ ------- opensolaris - - 4.62M static 2009-08-17 10:05 small-be - - 1.72M static 2009-08-17 13:23 small-be-clone NR / 377.48M static 2009-08-17 13:45 snv111b - - 3.17G static 2009-08-17 13:20 root@unknown:~# root@unknown:~# beadm destroy snv111b Are you sure you want to destroy snv111b? This action cannot be undone(y/[n]): y root@unknown:~# beadm destroy opensolaris Are you sure you want to destroy opensolaris? This action cannot be undone(y/[n]): y root@unknown:~# beadm list BE Active Mountpoint Space Policy Created -- ------ ---------- ----- ------ ------- small-be - - 1.72M static 2009-08-17 13:23 small-be-clone NR / 377.48M static 2009-08-17 13:45 root@unknown:~# beadm activate small-be root@unknown:~# root@unknown:~# reboot Aug 17 13:52:22 unknown reboot: initiated by root on /dev/console syncing file systems... done rebooting... v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b' SunOS Release 5.11 Version snv_111b 64-bit Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: unknown Reading ZFS config: done. Mounting ZFS filesystems: (7/7) unknown console login: root Password: Aug 17 13:53:22 unknown login: ROOT LOGIN /dev/console Last login: Mon Aug 17 13:44:19 on console Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008 root@unknown:~# beadm destroy small-be-clone Are you sure you want to destroy small-be-clone? This action cannot be undone(y/[n]): y root@unknown:~# Finally, lets create a new cloned BE and upgrade it to snv118. Here we'll run into some minor pkg bugs.. They don't hurt anything though. You see that we grew the root a little after our upgrade. root@unknown:~# beadm create snv118 root@unknown:~# beadm mount snv118 /mnt root@unknown:~# pkg -R /mnt set-publisher -O http://pkg.opensolaris.org/dev opensolaris.org root@unknown:~# pkg -R /mnt install SUNWipkg No updates available for this image. root@unknown:~# pkg -R /mnt install entire@0.5.11-0.118 DOWNLOAD PKGS FILES XFER (MB) Completed 73/73 4311/4311 79.34/79.34 PHASE ACTIONS Removal Phase 1569/1569 Install Phase 2619/2619 Update Phase 5470/5698 driver (softmac) upgrade (removal of policy'read_priv_set=net_rawaccess write_priv_set=net_rawaccess) failed: minor node spec required. driver (vnic) upgrade (removal of policy'read_priv_set=net_rawaccess write_priv_set=net_rawaccess) failed: minor node spec required. driver (aggr) upgrade (removal of policy'read_priv_set=net_rawaccess write_priv_set=net_rawaccess) failed: minor node spec required. Update Phase 5610/5698 driver (dnet) upgrade (removal of policy'read_priv_set=net_rawaccess write_priv_set=net_rawaccess) failed: minor node spec required. driver (elxl) upgrade (removal of policy'read_priv_set=net_rawaccess write_priv_set=net_rawaccess) failed: minor node spec required. driver (iprb) upgrade (removal of policy'read_priv_set=net_rawaccess write_priv_set=net_rawaccess) failed: minor node spec required. Update Phase 5698/5698 PHASE ITEMS Reading Existing Index 8/8 Indexing Packages 73/73 Optimizing Index... PHASE ITEMS Indexing Packages 73/73 root@unknown:~# bootadm update-archive -R /mnt updating /mnt//platform/i86pc/boot_archive updating /mnt//platform/i86pc/amd64/boot_archive root@unknown:~# beadm umount snv118 root@unknown:~# beadm activate snv118 root@unknown:~# reboot Aug 17 14:05:37 unknown reboot: initiated by root on /dev/console syncing file systems... done rebooting... v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b' SunOS Release 5.11 Version snv_118 64-bit Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: unknown Configuring devices. Loading smf(5) service descriptions: 6/6 Reading ZFS config: done. Mounting ZFS filesystems: (7/7) unknown console login: root Password: Aug 17 14:06:39 unknown login: ROOT LOGIN /dev/console Last login: Mon Aug 17 13:53:22 on console Sun Microsystems Inc. SunOS 5.11 snv_118 November 2008 root@unknown:~# rm -rf /var/pkg/download/* root@unknown:~# df -lk Filesystem kbytes used avail capacity Mounted on rpool/ROOT/snv118 20514816 418664 18756788 3% /

More...