North Star Labs

As a system administrator with a very large music collection, I’ve always been mildly irritated at the difficulty accessing my “master” music library while away from home.

Enter Simply Media a free, small application which does just as the name promises.

My iTunes library back home just shows up in my shared iTunes listing, regardless of where I am. No firewall hackery, nothing to configure, it just works, and works well.

The iTunes integration is fantastic.

I’ve run into several problems backing up our central file servers with Bacula, mostly centered around the sheer number of files (~6 million) a single job must process and store into the MySQL catalog.

I ran into the following error last night, attempting to back up the entire 6TB array as a single job:

  07-Nov 18:10 backup-dir JobId 3: Fatal error: sql_create.c:732 sql_create.c:732 insert INSERT INTO batch VALUES (1580771,3,'/Volumes/0/export/users/kodama/Desktop/GAP/gap4r4/small/small2/','sml800.z','OAAAD DkeW IGk B ih C+ A KZn BAA BY BHLtzL 1sNQO BFnqZZ A A C','0') failed:
  Incorrect key file for table '/tmp/#sql2459_94_0.MYI'; try to repair it

After doing a bit of research, I’ve concluded the /tmp volume, which is only a 256M tmpfs partition is filling to capacity before the job is able to complete.

Restarting the job this morning confirms MySQL is spooling data into /tmp.

  [root@backup tmp]# ls -l /tmp/
  total 332
  -rw-rw---- 1 mysql mysql 319276 Nov  8 09:48 #sql511e_3_0.MYD
  -rw-rw---- 1 mysql mysql   1024 Nov  8 09:48 #sql511e_3_0.MYI
  -rw-rw---- 1 mysql mysql   8722 Nov  8 09:48 #sql511e_3_0.frm

My solution for the time being is to reconfigure mysql to use /var/tmp for it’s temporary storage, rather than /tmp. This places the data on a much larger file system.

# /etc/my.cnf
[mysqld]
tmpdir=/var/tmp

I’m also planning to split the job into smaller jobs, using regular expressions to include only pieces of the home directory tree at a time. This will keep the number of files each job needs to handle under a reasonable threshold.

I’ve been doing a lot of subversion branch, test, merge cycles against our main Puppet configuration repository. I’ve run into issues when both the trunk and my testing branches are modified after I’ve forked off my branch.

This creates merge conflicts when I’m done testing, and need to merge my changes back into the production branch.

In an effort to reduce the overhead associated with manually resolving each conflict that arises from the divergence, I’ve started employing the use of svnmerge.py.

It’s great.

  svn copy cluster-orange-server cluster-orange-server-test01
  svn checkin cluster-orange-server-test01 -m 'Branched testing off.'
  cd cluster-orange-server-test01

  svnmerge.py init
  svn ci -F svnmerge-commit-message.txt

Now that my pristine branch of the production code has been initialized with svnmerge, I’m free to make changes to my testing copy. Once I need to merge back into production, I just need:

  svnmerge.py merge

The DVD drive on my desktop Mac is broken, making it difficult to install the operating system. I wanted to put Leopard on this machine today, so I tried installing to the machine from my MacBook Pro.

Since Leopard is now Universal for both PowerPC and Intel, this ended up working nicely. The one note, however, is that installer will complain about the partition table when it’s executing on an intel machine, but installing to a PowerPC disk. The installer thinks it’ll be booting from the drive, so it doesn’t like the Apple Partition Map, demanding a GPT table instead.

The solution is to set the CM_BUILD variable, allowing installation to the target disk.

  export CM_BUILD=CM_BUILD
  export COMMAND_LINE_INSTALL=1
  export SRC="/Volumes/Mac OS X Install DVD"

  installer -verbose \
    -pkg "$SRC"/System/Installation/Packages/OSInstall.mpkg \
    -target "/Volumes/Macintosh HD 1/" \
    -lang en | tee /tmp/installer.log

One of the small, but incredibly useful features for me in Leopard is that ssh-agent is automatically running for each user account. This relatively small change allows me to log into remote machines without entering my password each time.

Using the SOCKS proxy built into ssh, we’re also able to setup a quick and easy secure tunnel. I wanted to check some sensitive information this morning, but I’m at a coffee shop that doesn’t pass VPN traffic, so I quickly hacked together the following:

Setup a new Location in the Network System Preference Pane to configure the SOCKS proxy at 127.0.0.1, port 4088. This connects most Apple applications to the secure and encrypted tunnel.

Next, I configured ssh to automatically setup the SOCKS proxy whenever I type “ssh ford”, which is an alias for my workstation back at the office.

# ~/.ssh/config
host ford
  User mccune
  HostName ford.math.ohio-state.edu
  # Handle sleep/wake robustly with TCPKeepAlive
  TCPKeepAlive no
  Port 22
  # DynamicForward is a SOCKS proxy server.
  DynamicForward 4088
  ForwardX11 no

With this configuration, I’m able to load my SSH public key into the ssh-agent running by default on Leopard, type “ssh ford” to setup the encrypted SOCKS proxy, then change location to “SSH Socks Proxy” to automatically have Mail.app, iChat, Safari and Camino use the secure proxy.

An easy way to verify the proxy is working is to add an IP Address gadget to your personal google home page:

Finally, with the Network Location module for Quicksilver, you can easily switch back and forth between the encrypted proxy.

I quickly noticed that the “mate” command line tool does not work as expected under GNU Screen on Leopard. The error I get is:

mate: failed to establish connection with TextMate.

I suspect this is a side-effect of the Leopard sandbox feature. I’ve found a quick shell alias to be a nice work-around:

alias mate='open -a TextMate.app'

This works inside of GNU Screen, where the mate command line utility does not.

So I’m finally running Mac OS X 10.5 Leopard on my portable. I’ve decided to migrate to the new sparsebundle style FileVault image, and here’s how I did it:

First, make sure you’ve created a FileVault master certificate by setting a master password in the Security preference pane.

Manually create the sparse bundle:

umask 077
export NAME="mccune"
hdiutil create -size 300g \
  -encryption -agentpass \
  -certificate /Library/Keychains/FileVaultMaster.cer \
  -uid 502 -gid 20 -mode 0700 \
  -fs "HFS+J" \
  -type SPARSEBUNDLE \
  -layout SPUD \
  -volname "$NAME" \
  "$NAME".sparsebundle;
chown -R "$NAME":staff "$NAME".sparsebundle

Make sure to set the password on the disk image the same as the password used with the user account, otherwise the system won’t be able to decrypt the image from the loginwindow.

Mount the sparsebundle:

hdiutil mount -owners on -mountrandom /tmp -stdinpass "$NAME".sparsebundle

Copy the contents of your home directory:

rsync -avxHE --progress /Users/mccune/ /tmp/dmg.TYSCwg/

After I did the initial pass with rsync, I logged out of my user account, and logged in using the administrator account in order to run the rsync process a second time, while my profile was in a steady state.

The quick problem and fix of the day deals with iSCSI storage, CentOS 5, RHEL5, and LVM. As previously mentioned, I’m using LVM tagging to arbitrate logical volume activation among a set of physical hosts all hitting the same storage. This has been working quite well, and appears to a simple and effective solution to the clustered Xen host problem.

We recently installed a new iSCSI target, and my boss complained that it’s LVM logical volumes weren’t active on boot, despite being properly tagged. This is because all block devices are scanned for LVM signatures from within the initial ram disk, not later in the boot process. At this stage, there’s no networking, and the iSCSI initiator hasn’t been brought online yet.

Nothing necessary for boot lives on the iSCSI target, it’s really just a large pool of bits for our backup system, so I decided the most simple solution is to just activate all volumes a second time from /etc/rc.local. This appears to work well and reliably.

  # Append to /etc/rc.local, executed after all other init scripts.
  # Activate all logical volumes tagged with the local machine's hostname.
  lvchange -ay @$(uname -n)

I tried retrieving the password for a wireless network I’d joined sometime in the past. To do this, I went into Keychain Access, selected the System keychain, and asked it to Show Password for the networking in question.

As it turns out, the system keychain is protected with a shared secret which I believe is maintained in /var/db/SystemKey. This is a binary file, so it’s not very easy to pull out the password in a form that can be entered in the dialog box Keychain Access presents.

As a work around, I found a nice hint [1] that uses Keychain Scripting to copy the password to the clipboard. Here’s the Apple Script I’m using. Please not that the first character of a WEP key appears to be $, which should be discarded.

display dialog "What key?" default answer ""
set theKeyName to the text returned of the result
tell application "Keychain Scripting"
	tell keychain "System.keychain"
		set theKeyList to every key
		repeat with k in theKeyList
			if the name of k is theKeyName then
				set TheKeyValue to the password of k
			end if
		end repeat
	end tell
end tell

set the clipboard to TheKeyValue

display dialog "Copied " & length of TheKeyValue & " chars to clipboard." with icon note buttons {"OK"} default button "OK"

[1] 10.4: AirPort and System.keychain password solution

Now available for download, Mozilla SeaMonkey 1.1.4. This package is helpful if you have customers who are familiar with the Composer feature of Netscape and Mozilla when creating quick web pages.

• Sea Monkey 1.1.4