North Star Labs

I’ve recently been struggling to track down a problem with my OpenSolaris xVM system. I’m running xvm in OpenSolaris b133. The issue is that my manual configuration of dom0_mem in /rpool/boot/grub/menu.lst seems to constantly be overwritten upon reboot. This is a problem since I need dom0 to be clamped down to prevent Xen’s balloon feature from fighting with the ZFS arc. In addition to this problem, there are bugs in the b132 and b133 of OpenSolaris which require the config/dom0-min-mem SMF property to be set to match dom0_mem.

I’ve also been running into the dom0-min-mem issues documented at My South – Sun xVM 3.4.2 available, dom0_min_mem. Pascal also mentions setting the dom0-min-mem propery, but doesn’t appear to be running into the issue I have with b132 and b133 where the property is consistently changes by the xvm-milestone service method script.

The problem is caused by the SMF xvm milestone ( svc:/milestone/xvm) constantly re-writing these properties and the menu.lst file. The solution is to disable the xvm milestone and re-enable all of the xvm services manually. This will allow you to make manual changes to the menu.lst file without the xvm milestone interfering with you.

OpenSolaris introduced the xvm milestone in b126 around October of 2009. Please see [xen-discuss] FYI: enable/disable the xVM hypervisor.

Here is the recipe to fix the problem. First, make a backup copy of your menu.lst file, then disable the xvm milestone, enable the other xvm SMF services, and finally restore your menu.lst file. We do this because disabling the xvm milestone disables all of xvm, where we really just want to prevent /lib/svc/method/xvm-milestone from executing.

This assumes you already have xVM enabled through the use of svcadn enable milestone/xvm.

cd /rpool/boot/grub
pfexec cp -p menu.lst menu.lst.milestone-xvm.enabled
pfexec svcadm disable milestone/xvm
pfexec svcadm enable -r svc:/system/xvm/domains:default
pfexec cp -p menu.lst menu.lst.milestone-xvm.disabled
pfexec cp -p menu.lst.milestone-xvm.enabled menu.lst


Before rebooting, ensure the dom0_mem setting is something reasonable. I find 1.5GB to be a good balance.

title os-133-xvm1
findroot (pool_rpool,0,a)
bootfs rpool/ROOT/os-133-xvm1
kernel$ /boot/$ISADIR/xen.gz console=vga dom0_mem=1536M dom0_vcpus_pin=false watchdog=false
module$ /platform/i86xpv/kernel/$ISADIR/unix /platform/i86xpv/kernel/$ISADIR/unix -B $ZFS-BOOTFS
module$ /platform/i86pc/$ISADIR/boot_archive


Finally, ensure SMF properties match the dom0_mem value:

svccfg -s svc:/milestone/xvm listprop hypervisor/dom0_mem
svccfg -s xend listprop config/dom0-min-mem


If they don’t match, they may be set using:

pfexec svccfg -s svc:/system/xvm/xend setprop config/dom0-min-mem = 1536
pfexec /usr/sbin/svccfg -s svc:/milestone/xvm setprop hypervisor/dom0_mem = 1536


I plan to diagnose just why the xvm-milestone service method script is misbehaving so much and file the appropriate bug reports. If anyone has any suggestions or ideas, please let me know.

Using OpenSolaris 2008.11, it appears the most complete way to obtain a full-featured development tool chain is to install the ss-dev and gcc-dev package clusters.

pfexec pkg install gcc-dev ss-dev

References:

• Sun Studio Express 11/08 Packages

I spent the better part of the evening attempting to get a reliable, responsive and otherwise unobtrusive DLNA media server running on my new OpenSolaris home file server. I finally stumbled upon PMS which “just works” after using X11 forwarding over ssh once in order to get at the GUI configuration screen. I went ahead and tried the Linux tarball. None of the included binaries execute on solaris, but the jar file appears to run great.

Once running, my PS3 sees the media player quickly and easily and streams my MP3’s nicely.

I’m planning on cooking up a SMF profile to keep this running as a service and figure out the mplayer calls in order to stream my favorite web streams directly to the PS3.

Other DLNA media servers I tried were Coherence (no documentation, didn’t work out of box), Mediatomb (needed to hack the source to get it to run on solaris. When running, rarely showed up in the XMB), fuppes (compile issues).

Contrary to recommendations from seasoned Unix admins, it’s perfectly acceptable to change the root shell from the bourne shell to something like bash. The most common reason to leave the root shell alone usually goes something like, “you need a valid and statically linked shell defined in /etc/passwd to boot into single user mode if you need to recover your system.”

There’s a really nice list of Solaris root shell misconceptions published at http://www.roble.com/docs/sol_root_shell.html.

Fortunately for me, this isn’t the case in Solaris 10. While setting up a new Solaris 10 system today, I accidentally set root’s shell to /sbin/bash instead of /usr/bin/bash. /sbin/bash doesn’t exist, so I could no longer log into the system.

Luckily, this is a system with a Dell RAC card setup for remote console access. I logged into the RAC and issued a “graceful shutdown” power off command, which Solaris responded to nicely and brought the system entirely down. Once I powered the system back on, it’s simply a matter of booting into single user mode by passing the -s flag to the kernel.

Solaris 10 is smart enough to fall back to /sbin/sh if it can’t invoke the shell defined in /etc/passwd booted into single user mode. So long as you don’t horribly mangle /sbin/sh and the libraries it’s linked to, you’ll be fine changing the root shell to anything you like.

Here’s how it went:

Quick answer:

cfgadm -al

Searching the web for this information took a few more minutes than I expected it to, so I’m posting this article with as may relevant keywords as I can think of. Thanks to Pascal Gienger for the clear answer to this question.

The situation is pretty common for system administrators; you have a production server that’s running out of storage space and you remedy the situation by allocating a new LUN on your back end SAN.

In Linux, I’d typically echo ‘- – -’ > /sys/class/scsi_host/host1/scan in order to issue a rescan, then run multipath -v2, then add the resulting /dev/mpath/foobar device to LVM.

In Solaris 10, this process has been greatly simplified. One command even re-populates your scsi_vhci multipath controller for you.

Consider before scanning:

[jmccune@otto ~]$ sudo format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c4t00D0B2202E001900d0 <DEFAULT cyl 8352 alt 2 hd 255 sec 63>
          /scsi_vhci/disk@g00d0b2202e001900
Specify disk (enter its number): ^D
[jmccune@otto ~]$

And consider the online re-scan of the fiber channel storage system.

[jmccune@otto ~]$ sudo cfgadm -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c0                             fc-fabric    connected    configured   unknown
c0::212000d0b202e201           disk         connected    configured   unknown
c1                             fc-fabric    connected    configured   unknown
c1::212000d0b202e201           disk         connected    configured   unknown
usb0/1                         unknown      empty        unconfigured ok
usb0/2                         unknown      empty        unconfigured ok
usb1/1                         usb-device   connected    configured   ok
usb1/2                         usb-device   connected    configured   ok
usb2/1                         unknown      empty        unconfigured ok
usb2/2                         unknown      empty        unconfigured ok
usb3/1                         unknown      empty        unconfigured ok
usb3/2                         usb-device   connected    configured   ok
usb3/3                         unknown      empty        unconfigured ok
usb3/4                         unknown      empty        unconfigured ok
usb3/5                         unknown      empty        unconfigured ok
usb3/6                         unknown      empty        unconfigured ok
[jmccune@otto ~]$ sudo format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c4t00D0B2202E000900d0 <DEFAULT cyl 50127 alt 2 hd 255 sec 63>
          /scsi_vhci/disk@g00d0b2202e000900
       1. c4t00D0B2202E001900d0 <DEFAULT cyl 8352 alt 2 hd 255 sec 63>
          /scsi_vhci/disk@g00d0b2202e001900
Specify disk (enter its number): ^D

Creating a new ZFS pool based on this new LUN is easy. Note, the new disk is ID zero from format, not id 1.

[jmccune@otto ~]$ sudo zpool create db1 c4t00D0B2202E000900d0
[jmccune@otto ~]$ zpool list
NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
db1     382G   112K   382G     0%  ONLINE  -
rpool  63.5G  38.1G  25.4G    60%  ONLINE  -

Keywords: solaris, sun MPxIO, scsi_vhci, san, lun, attach, online, resize, scan, rescan, luxadm.