OpenSolaris milestone/xvm grub dom0_mem problem
Posted in Solaris, Solution on 02/28/2010 06:53 pm by jmccune
I’ve recently been struggling to track down a problem with my OpenSolaris xVM system. I’m running xvm in OpenSolaris b133. The issue is that my manual configuration of dom0_mem in /rpool/boot/grub/menu.lst seems to constantly be overwritten upon reboot. This is a problem since I need dom0 to be clamped down to prevent Xen’s balloon feature from fighting with the ZFS arc. In addition to this problem, there are bugs in the b132 and b133 of OpenSolaris which require the config/dom0-min-mem SMF property to be set to match dom0_mem.
I’ve also been running into the dom0-min-mem issues documented at My South – Sun xVM 3.4.2 available, dom0_min_mem. Pascal also mentions setting the dom0-min-mem propery, but doesn’t appear to be running into the issue I have with b132 and b133 where the property is consistently changes by the xvm-milestone service method script.
The problem is caused by the SMF xvm milestone ( svc:/milestone/xvm) constantly re-writing these properties and the menu.lst file. The solution is to disable the xvm milestone and re-enable all of the xvm services manually. This will allow you to make manual changes to the menu.lst file without the xvm milestone interfering with you.
OpenSolaris introduced the xvm milestone in b126 around October of 2009. Please see [xen-discuss] FYI: enable/disable the xVM hypervisor.
Here is the recipe to fix the problem. First, make a backup copy of your menu.lst file, then disable the xvm milestone, enable the other xvm SMF services, and finally restore your menu.lst file. We do this because disabling the xvm milestone disables all of xvm, where we really just want to prevent /lib/svc/method/xvm-milestone from executing.
This assumes you already have xVM enabled through the use of svcadn enable milestone/xvm.
cd /rpool/boot/grub
pfexec cp -p menu.lst menu.lst.milestone-xvm.enabled
pfexec svcadm disable milestone/xvm
pfexec svcadm enable -r svc:/system/xvm/domains:default
pfexec cp -p menu.lst menu.lst.milestone-xvm.disabled
pfexec cp -p menu.lst.milestone-xvm.enabled menu.lst
Before rebooting, ensure the dom0_mem setting is something reasonable. I find 1.5GB to be a good balance.
title os-133-xvm1
findroot (pool_rpool,0,a)
bootfs rpool/ROOT/os-133-xvm1
kernel$ /boot/$ISADIR/xen.gz console=vga dom0_mem=1536M dom0_vcpus_pin=false watchdog=false
module$ /platform/i86xpv/kernel/$ISADIR/unix /platform/i86xpv/kernel/$ISADIR/unix -B $ZFS-BOOTFS
module$ /platform/i86pc/$ISADIR/boot_archive
Finally, ensure SMF properties match the dom0_mem value:
svccfg -s svc:/milestone/xvm listprop hypervisor/dom0_mem
svccfg -s xend listprop config/dom0-min-mem
If they don’t match, they may be set using:
pfexec svccfg -s svc:/system/xvm/xend setprop config/dom0-min-mem = 1536
pfexec /usr/sbin/svccfg -s svc:/milestone/xvm setprop hypervisor/dom0_mem = 1536
I plan to diagnose just why the xvm-milestone service method script is misbehaving so much and file the appropriate bug reports. If anyone has any suggestions or ideas, please let me know.
