North Star Labs

I’ve recently been struggling to track down a problem with my OpenSolaris xVM system. I’m running xvm in OpenSolaris b133. The issue is that my manual configuration of dom0_mem in /rpool/boot/grub/menu.lst seems to constantly be overwritten upon reboot. This is a problem since I need dom0 to be clamped down to prevent Xen’s balloon feature from fighting with the ZFS arc. In addition to this problem, there are bugs in the b132 and b133 of OpenSolaris which require the config/dom0-min-mem SMF property to be set to match dom0_mem.

I’ve also been running into the dom0-min-mem issues documented at My South – Sun xVM 3.4.2 available, dom0_min_mem. Pascal also mentions setting the dom0-min-mem propery, but doesn’t appear to be running into the issue I have with b132 and b133 where the property is consistently changes by the xvm-milestone service method script.

The problem is caused by the SMF xvm milestone ( svc:/milestone/xvm) constantly re-writing these properties and the menu.lst file. The solution is to disable the xvm milestone and re-enable all of the xvm services manually. This will allow you to make manual changes to the menu.lst file without the xvm milestone interfering with you.

OpenSolaris introduced the xvm milestone in b126 around October of 2009. Please see [xen-discuss] FYI: enable/disable the xVM hypervisor.

Here is the recipe to fix the problem. First, make a backup copy of your menu.lst file, then disable the xvm milestone, enable the other xvm SMF services, and finally restore your menu.lst file. We do this because disabling the xvm milestone disables all of xvm, where we really just want to prevent /lib/svc/method/xvm-milestone from executing.

This assumes you already have xVM enabled through the use of svcadn enable milestone/xvm.

cd /rpool/boot/grub
pfexec cp -p menu.lst menu.lst.milestone-xvm.enabled
pfexec svcadm disable milestone/xvm
pfexec svcadm enable -r svc:/system/xvm/domains:default
pfexec cp -p menu.lst menu.lst.milestone-xvm.disabled
pfexec cp -p menu.lst.milestone-xvm.enabled menu.lst


Before rebooting, ensure the dom0_mem setting is something reasonable. I find 1.5GB to be a good balance.

title os-133-xvm1
findroot (pool_rpool,0,a)
bootfs rpool/ROOT/os-133-xvm1
kernel$ /boot/$ISADIR/xen.gz console=vga dom0_mem=1536M dom0_vcpus_pin=false watchdog=false
module$ /platform/i86xpv/kernel/$ISADIR/unix /platform/i86xpv/kernel/$ISADIR/unix -B $ZFS-BOOTFS
module$ /platform/i86pc/$ISADIR/boot_archive


Finally, ensure SMF properties match the dom0_mem value:

svccfg -s svc:/milestone/xvm listprop hypervisor/dom0_mem
svccfg -s xend listprop config/dom0-min-mem


If they don’t match, they may be set using:

pfexec svccfg -s svc:/system/xvm/xend setprop config/dom0-min-mem = 1536
pfexec /usr/sbin/svccfg -s svc:/milestone/xvm setprop hypervisor/dom0_mem = 1536


I plan to diagnose just why the xvm-milestone service method script is misbehaving so much and file the appropriate bug reports. If anyone has any suggestions or ideas, please let me know.

I originally installed OpenSolaris 2008.11 on my home media server, and 2009.06 has some features I’d like to take advantage of. Running a pkg image-update, I ran my root pool out of space since it’s located on a relatively small compact flash card.

I decided to grow the root pool by using an available external disk I have. The process involves attaching the new, larger disk to the root pool as a mirror, waiting for the resilver process to complete, installing the boot loader onto the new disk, then detaching the old, small device from the root pool. This information is documented at sun.com in the document How to Replace a Disk in the ZFS Root Pool.

Attempting to attach the new device to the pool with zpool attach, I ran in the error message "cannot label 'c3t0d0': EFI labeled devices are not supported on root pools." I tried wiping the EFI label, but kept running into the same error. I noticed other people talking about this issue;
Removing EFI (format -e not working?) and
Please help need to remove EFI label: msg#00173

My problem was that I was not properly creating the root partition on the disk with an SMI label. I was properly using format -e, then executing “fdisk”, creating the VTOC on the entire disk, but I forgot the step where once the VTOC is created, you need to create partition 0, which will be used for the zpool vdev.

If you run into this error, make sure you use the “partition” option in fdisk -e, which will allow you to then define slice 0. Label the slice “root” and give it as much space as you’d like. Make sure it does not overlap with the boot slice which is automatically created when the VTOC is created.

Once slice 0 is present, use c1t0d0s0 rather than c1t0d0 when you attach the new disk to the root pool. For example:

Correct:
zpool attach rpool c4t0d0s0 c3t0d0s0
Incorrect:
zpool attach rpool c4t0d0s0 c3t0d0
If you receive an error about overlapping partitions, just use zpool attach -f to force the attach.

Once the device is in the pool and re-silvering, use installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c3t0d0s0 to install the boot block.

After testing the new boot device, use zpool detatch rpool c4t0d0s0 to remove the old device from the pool and complete the resize process.

Here’s my original partition layout:

Current partition table (original):
Total disk cylinders available: 3820 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders        Size            Blocks
  0       root    wm       1 - 3818        7.46GB    (3818/0/0) 15638528
  1 unassigned    wm       0               0         (0/0/0)           0
  2     backup    wu       0 - 3819        7.46GB    (3820/0/0) 15646720
  3 unassigned    wm       0               0         (0/0/0)           0
  4 unassigned    wm       0               0         (0/0/0)           0
  5 unassigned    wm       0               0         (0/0/0)           0
  6 unassigned    wm       0               0         (0/0/0)           0
  7 unassigned    wm       0               0         (0/0/0)           0
  8       boot    wu       0 -    0        2.00MB    (1/0/0)        4096
  9 unassigned    wm       0               0         (0/0/0)           0

Here’s my new, larger disk layout:

Current partition table (original):
Total disk cylinders available: 60797 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       boot    wm       1 - 60700      929.97GB    (60700/0/0) 1950291000
  1 unassigned    wm       0                0         (0/0/0)              0
  2     backup    wu       0 - 60796      931.46GB    (60797/0/0) 1953407610
  3 unassigned    wm       0                0         (0/0/0)              0
  4 unassigned    wm       0                0         (0/0/0)              0
  5 unassigned    wm       0                0         (0/0/0)              0
  6 unassigned    wm       0                0         (0/0/0)              0
  7 unassigned    wm       0                0         (0/0/0)              0
  8       boot    wu       0 -     0       15.69MB    (1/0/0)          32130
  9 unassigned    wm       0                0         (0/0/0)              0

After detaching the original, small disk from the mirror, the root pool expands to the size of the remaining vdev:

jmccune@rain:~$ zpool list rpool
NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
rpool   930G  5.57G   924G     0%  ONLINE  -

I cooked dinner for myself today and sat down at the table looking forward to streaming The Daily Show or The Colbert Report on Hulu since there’s no way I’m paying $65 a month for cable TV. As it turns out, there haven’t been any new episodes in awhile, and I like my fake news fresh off the wire, so I decided to catch up on my other fake news addiction; aggregated blog rss feeds.

I came across the TED talk for today, which is Michell Obama. Great speech by the way, check it out at: http://www.ted.com/talks/michelle_obama.html. The streaming video quality left something to be desired, so I looked around and found the HD podcast URL at podcasters.tv.

This works well with iTunes, and MediaLink is able to copy the movie file from my MacBook pro, but for some reason streaming the video usually quits part way through playback with an obscure error code.

I have my OpenSolaris, Intel Atom based file server running on a gigabit network connected up to the Playstation 3 and HDTV using MediaLink, so I decided to look for some unix tool to download the podcast which could easily be run from cron.

After some searching and research into different options, I downloaded Peapod, a wonderful python command line application, and gave it a whirl.

To my complete satisfaction, peapod runs from my home directory without requiring any piece of itself being installed into the system. The only missing dependency I ran into was urlgrabber for python 2.4. Luckily, I have easy_install installed so it was a simple matter of:
pfexec /usr/bin/easy_install-2.4 urlgrabber
Once urlgrabber was installed, setup of the podcast client was a breeze:

jmccune@rain:~$ cd ~/bin
jmccune@rain:~/bin/$ ln -s ../apps/peapod/peapod.py peapod
jmccune@rain:~/bin/$ cd ~
jmccune@rain:~$ peapod
Creating user directory: /home/jmccune/.peapod
Created a default configuration file in :
/home/jmccune/.peapod/peapod.xml
Please edit this file to contain your feeds and options.

I commented out the sample podcast and added TED in HD.
(Note: I found the feed URL by doing a “Get Info”, or clicking on the little i next to the podcast title in the podcast section of iTunes.)

For the title I made it “TED Talks (HD)” and for the URL, I used http://feeds.feedburner.com/TedtalksHD.

Finally, running peapod simply works.

jmccune@rain:~$ peapod
...Spawning thread 0 for feed url http://feeds.feedburner.com/TedtalksHD
Fetching feed for TED Talks (HD)
Downloading TED Talks (HD) -- http://video.ted.com/talks/podcast/MichelleObama_2009P_480.mp4
Trackname MichelleObama_2009P_480.mp4
Savename /export/dozer/podcasts/jmccune/TED Talks (HD)/MichelleObama_2009P_480.mp4
Mime-type video/mp4


This will be added to cron to run every day a few hours before I get home from work, and MediaTomb should pickup the new content.

And now to figure out how to manually kick off a MediaTomb scan of the folder once downloading is complete.

Here are some decent feeds I’ve found so far:

peapod --addnew=http://www.hbo.com/podcasts/billmaher/podcast.xml --title="Bill Maher"


Please post more video feed URL’s, especially 720p and higher, in the comments if you have some good video podcasts worth watching on my TV.

Using OpenSolaris 2008.11, it appears the most complete way to obtain a full-featured development tool chain is to install the ss-dev and gcc-dev package clusters.

pfexec pkg install gcc-dev ss-dev

References:

• Sun Studio Express 11/08 Packages

I spent the better part of the evening attempting to get a reliable, responsive and otherwise unobtrusive DLNA media server running on my new OpenSolaris home file server. I finally stumbled upon PMS which “just works” after using X11 forwarding over ssh once in order to get at the GUI configuration screen. I went ahead and tried the Linux tarball. None of the included binaries execute on solaris, but the jar file appears to run great.

Once running, my PS3 sees the media player quickly and easily and streams my MP3’s nicely.

I’m planning on cooking up a SMF profile to keep this running as a service and figure out the mplayer calls in order to stream my favorite web streams directly to the PS3.

Other DLNA media servers I tried were Coherence (no documentation, didn’t work out of box), Mediatomb (needed to hack the source to get it to run on solaris. When running, rarely showed up in the XMB), fuppes (compile issues).

Contrary to recommendations from seasoned Unix admins, it’s perfectly acceptable to change the root shell from the bourne shell to something like bash. The most common reason to leave the root shell alone usually goes something like, “you need a valid and statically linked shell defined in /etc/passwd to boot into single user mode if you need to recover your system.”

There’s a really nice list of Solaris root shell misconceptions published at http://www.roble.com/docs/sol_root_shell.html.

Fortunately for me, this isn’t the case in Solaris 10. While setting up a new Solaris 10 system today, I accidentally set root’s shell to /sbin/bash instead of /usr/bin/bash. /sbin/bash doesn’t exist, so I could no longer log into the system.

Luckily, this is a system with a Dell RAC card setup for remote console access. I logged into the RAC and issued a “graceful shutdown” power off command, which Solaris responded to nicely and brought the system entirely down. Once I powered the system back on, it’s simply a matter of booting into single user mode by passing the -s flag to the kernel.

Solaris 10 is smart enough to fall back to /sbin/sh if it can’t invoke the shell defined in /etc/passwd booted into single user mode. So long as you don’t horribly mangle /sbin/sh and the libraries it’s linked to, you’ll be fine changing the root shell to anything you like.

Here’s how it went:

Quick answer:

cfgadm -al

Searching the web for this information took a few more minutes than I expected it to, so I’m posting this article with as may relevant keywords as I can think of. Thanks to Pascal Gienger for the clear answer to this question.

The situation is pretty common for system administrators; you have a production server that’s running out of storage space and you remedy the situation by allocating a new LUN on your back end SAN.

In Linux, I’d typically echo ‘- – -’ > /sys/class/scsi_host/host1/scan in order to issue a rescan, then run multipath -v2, then add the resulting /dev/mpath/foobar device to LVM.

In Solaris 10, this process has been greatly simplified. One command even re-populates your scsi_vhci multipath controller for you.

Consider before scanning:

[jmccune@otto ~]$ sudo format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c4t00D0B2202E001900d0 <DEFAULT cyl 8352 alt 2 hd 255 sec 63>
          /scsi_vhci/disk@g00d0b2202e001900
Specify disk (enter its number): ^D
[jmccune@otto ~]$

And consider the online re-scan of the fiber channel storage system.

[jmccune@otto ~]$ sudo cfgadm -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c0                             fc-fabric    connected    configured   unknown
c0::212000d0b202e201           disk         connected    configured   unknown
c1                             fc-fabric    connected    configured   unknown
c1::212000d0b202e201           disk         connected    configured   unknown
usb0/1                         unknown      empty        unconfigured ok
usb0/2                         unknown      empty        unconfigured ok
usb1/1                         usb-device   connected    configured   ok
usb1/2                         usb-device   connected    configured   ok
usb2/1                         unknown      empty        unconfigured ok
usb2/2                         unknown      empty        unconfigured ok
usb3/1                         unknown      empty        unconfigured ok
usb3/2                         usb-device   connected    configured   ok
usb3/3                         unknown      empty        unconfigured ok
usb3/4                         unknown      empty        unconfigured ok
usb3/5                         unknown      empty        unconfigured ok
usb3/6                         unknown      empty        unconfigured ok
[jmccune@otto ~]$ sudo format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c4t00D0B2202E000900d0 <DEFAULT cyl 50127 alt 2 hd 255 sec 63>
          /scsi_vhci/disk@g00d0b2202e000900
       1. c4t00D0B2202E001900d0 <DEFAULT cyl 8352 alt 2 hd 255 sec 63>
          /scsi_vhci/disk@g00d0b2202e001900
Specify disk (enter its number): ^D

Creating a new ZFS pool based on this new LUN is easy. Note, the new disk is ID zero from format, not id 1.

[jmccune@otto ~]$ sudo zpool create db1 c4t00D0B2202E000900d0
[jmccune@otto ~]$ zpool list
NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
db1     382G   112K   382G     0%  ONLINE  -
rpool  63.5G  38.1G  25.4G    60%  ONLINE  -

Keywords: solaris, sun MPxIO, scsi_vhci, san, lun, attach, online, resize, scan, rescan, luxadm.