North Star Labs

Digging around in a NetBoot-Install.dmg file created by NetRestore Helper, I found a nice little gem.

In Leopard, and perhaps earlier versions of Mac OS X, we’re able to start a VNC server with the machine serial number as a password. This is particularly interesting for a managed network or lab environment.

As an example, I’m starting a VNC server in my NetBoot-Install image with the following shell script:

# Credit to Mike Bombich for this snippet

VNC="/System/Library/CoreServices/RemoteManagement/AppleVNCServer.bundle/Contents/MacOS/AppleVNCServer"

if [ -x "$VNC" ]; then
    "$VNC" -noRegister -serialNumber &
fi

I’m then able to quickly connect with Cmd+K in the finder:

If you’re scripting this, here’s a quick way to snag the serial number. I do this before I bless a client machine to netboot, so I have the serial number to connect back up once it’s in the NetRestore system.

system_profiler SPHardwareDataType | \
  grep -i 'serial number' | \
  perl -ple 's/.*:\s+(\w+).*?/$1/'

I’ve been using the find command for over a decade now, and I’m ashamed to say I never really learned how to properly exclude directories. Dealing with with subversion working copies that litter “.svn” folders everywhere, I finally sorted it all out this afternoon.

To exclude “.svn” folders and all contents:

$ find . '!' '(' -name '.svn' -prune ')'

This, combined with find -print0 and xargs -0 to execute arbitrary commands on every filesystem object found is a wonderful tool to keep handy.

Consider the following statement in a puppet manifest (think of a manifest as a script).

node "subversion.math.ohio-state.edu" {
    subversion::server::webrepository {
        "support": path => "/var/svn/support";
        "test":    path => "/var/svn/test";
    }
}

Without describing the problem this puppet snippet addresses, one might guess that I need to configure two subversion repositories, available via HTTP on the host “subversion.math.ohio-state.edu”.

The reason I absolutely *love* Puppet is the above code is all there is to this entire problem. Think about all the work that actually needs to happen to setup a subversion repository on a SSL enabled web server:

• Install apache
• Setup SSL certificates
• Install subversion and dependencies
• Setup apache virtual host with mod_dav_svn
• Setup apache htaccess for access control to the repositry
• Punch holes in the firewall (80, 443)
• Create the blank repository with svnadmin
• Ensure the repository is owned by apache
• Ensure post-commit hooks are put in the right place and executable

Now, this is a lot of work, and I’ve already had the need to create new subversion repositories on other hosts. Because I’ve already modeled this problem in puppet, it’s trivial for me to bring up subversion servers on arbitrary hosts. I just re-use the block you see above.

Now, for the tricky part… Here are the modules that actually model the subversion repository in question.

Note that I’ve left out the classes which model other aspects of the host in question. For example, web::baseserver::ssl, firewall::input-port, and site-files::certificates (SSL Certs).

# Subversion Module.

class subversion::server inherits subversion {
    File {
        mode => 0640,
        owner => "apache",
        group => 0,
        require => [ User["apache"], Package["subversion"] ]
    }

    define webrepository ($path = false) {
        File {
            owner => "apache",
            group => "0",
            mode => 0660
        }
        $path_real = $path ? {
            false => "$name",
            default => "$path"
        }
        include subversion::server
        repository {
            "$name": path => "$path_real";
        }
        file {
            "$path_real":
                recurse => true,
                require => [ User["apache"], Repository["$name"] ];
            "$path_real/hooks":
                ensure => directory;
            "$path_real/hooks/bin":
                ensure => directory;
            "$path_real/hooks/bin/commit-email.pl":
                content => template("subversion/hooks/bin/commit-email.pl"),
                mode => 0770;
            "$path_real/hooks/post-commit":
                content => template("subversion/hooks/post-commit"),
                mode => 0770;
        }
    }

    include web::baseserver::ssl

    file {
        "/var/svn":
            ensure => directory;
        "/etc/httpd/htaccess/authz_svn.htaccess":
            content => template("subversion/htaccess/authz_svn.htaccess.erb");
        "/etc/httpd/htaccess/authz_svn.users":
            content => template("subversion/htaccess/htpasswd.mathsvn.erb");
    }
    web::vhost {
        "subversion":
            template => "subversion.conf.erb";
    }
    package {
        "subversion-perl":;
        "mod_dav_svn":;
    }
}

class subversion {
    $authz_svn_access_file = "/etc/httpd/auth_SVNAccessFile.math"
    $auth_svn_users_file = "/etc/httpd/auth_htpasswd.mathsvn"
    $svn_base_parent_repo = "/var/svn"
    Package {
        ensure => present
    }
    package {
        "subversion":;
    }

    define repository ($path = false) {
        $path_real = $path ? {
            false => "$name",
            default => "$path"
        }
        include subversion
        # Create a blank repository.
        exec {
            "svnadmin_create_$path_real":
                command => "/usr/bin/svnadmin create '$path_real'",
                require => [ Package["subversion"] ],
                creates => "$path_real";
        }
    }
}

We experienced a power outage today, caused by someone tripping the emergency power off relay to our server room. Unfortunately, emergency power off really means “power off” so our UPS did the right thing and completely cut power rather than fall back to battery backup.

It was a little bit stressful getting everything back up, but everything appears to be working fine now.

The one serious error message we ran into is the following, when bring our OpenLDAP server back up:

[root@ldap ldap]# /etc/init.d/ldap restart
Stopping slapd:                                            [FAILED]
Checking configuration files for slapd:  bdb_db_open: unclean shutdown detected; attempting recovery.
bdb_db_open: Recovery skipped in read-only mode. Run manual recovery if errors are encountered.
bdb(dc=math,dc=ohio-state,dc=edu): PANIC: fatal region error detected; run recovery
bdb_db_open: Database cannot be opened, err -30974. Restore from backup!
bdb(dc=math,dc=ohio-state,dc=edu): DB_ENV->lock_id_free interface requires an environment configured for the locking subsystem
backend_startup_one: bi_db_open failed! (-30974)
slap_startup failed (test would succeed using the -u switch)
                                                           [FAILED]
stale lock files may be present in /var/lib/ldap           [WARNING]

Fortunately, the solution to this problem is easy enough. Just run slapd_db_recover -v in the Berkeley Database directory.

cd /var/lib/ldap
slapd_db_recover -v

Finding last valid log LSN: file: 4 offset 4818337
Recovery starting from [4][4815752]
Recovery complete at Wed Feb  6 15:33:42 2008
Maximum transaction ID 80000ba7 Recovery checkpoint [4][4818337]

After that, slapd should startup just fine.

[root@ldap lib]# /etc/init.d/ldap start
Checking configuration files for slapd:  bdb_db_open: unclean shutdown detected; attempting recovery.
bdb_db_open: Recovery skipped in read-only mode. Run manual recovery if errors are encountered.
config file testing succeeded
                                                           [  OK  ]
Starting slapd:                                            [  OK  ]

Nigel has posted slides from our Macworld 2008 presentation on Puppet.

Please see: Puppet Macworld 2008 Project

I’ll post additional information once I find out the details of distribution of any audio/video recordings taken during the presentation.

I haven’t posted in awhile, mainly because I’ve been preoccupied with a relatively long and relaxing vacation over the winter break where I largely ignored all things technology.

I’ve been preparing for Macworld 2008, where Nigel Kersten and I will be presenting some demonstrations and technical details about our respective Puppet deployments at Google and Ohio State University.

If you’ll be attending Macworld, feel free to follow my Twitter feed. I don’t post much at the moment, though I believe it’ll really come in handy during the fast and furious pace of a week long conference like Macworld.

Some other links for gratuitous self promotion:

• IT824 Mac OS X Laptop Deployments with Puppet
• My Macworld Social Network Profile
• Macworld 2008 Puppet Site(Check here after Thursday’s Session.)

Please leave a comment if you’ll be attending Macworld this year.

Working with Apache today, I ran into an issue where the process would appear to start OK, returning a zero exit status, yet strace was showing a SIGCHLD being caught.

Needless to say, the server wasn’t actually running for any length of time, but I found the following strace command immensely helpful in figuring out the problem.

  strace -o /tmp/httpd.strace -ff /usr/sbin/httpd

Because apache spawns a number of children, strace with -ff attaches to each child and recorded the system calls in /tmp/httpd.strace.$PID

As it turns out, I was receiving the following error in the child processes:

    bind(5, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("0.0.0.0")}, 16) \
    = -1 EADDRINUSE (Address already in use)

Joel just posted a great article titled SSL and LDAP in Leopard about the pitfalls of using SSL secured LDAP servers in Leopard. As Joel mentions, Leopard now refuses to trust any and all SSL protected LDAP servers out of the box.

A few people have complained that Directory.app, and the the LDAPv3 plugin for DirectoryService don’t honor the same purchased certificates that work just fine in their web browsers. Nor do they honor certificates signed by authorities listed in the x509Anchors keychain.

I’m not sure exactly where I personally stand on these very reasonable gripes, but I do know that it’s relatively trivial to configure all of your clients to honor “legitimate” certificates signed by authorities such as VeriSign, GeoTrust, etc…

It’s as simple as:

echo "TLS_CACERT /usr/share/curl/curl-ca-bundle.crt" >> \
  /etc/openldap/ldap.conf

You may need to give DirectoryService a kick, with killall DirectoryService.

This works because Apple already distributes a long PEM encoded list of certificate authorities for use with the curl command line utility. We’re able to leverage it’s trusted certificate store.

Also be warned your Leopard workstations are now slightly more vulnerable than if you were to configure only the certification authorities you need to get your LDAP server trusted.

James Reynolds mentioned on the Mac Enterprise mailing list that /etc/rc.local scripting is no longer a viable option to execute scripts at boot time, before the loginwindow presents itself to the user.

In Mac OS X 10.4, I’ve relied heavily on /etc/rc.local to execute a number of management scripts, and /etc/rc.local has always irritated me as a solution because of the parallel nature of sub-systems coming online during the Mac OS X boot process. Note; I didn’t say “boot sequence.” For example, I have some pretty extensive code to simply detect if DNS resolution is working or not.

Now that loginwindow is started from launchd, we’re able to replace it with our own scripts, and ultimately call it when we’re finished. I have yet to try this as a replacement for my current system, but I’m hopeful there aren’t as many issues figuring out what’s available and what isn’t at boot time, now that many more processes are started form launchd.

I’ll report back with any problems I’m sure to encounter migrating my startup scripts to launchd in Leopard.

I’ve run into several problems backing up our central file servers with Bacula, mostly centered around the sheer number of files (~6 million) a single job must process and store into the MySQL catalog.

I ran into the following error last night, attempting to back up the entire 6TB array as a single job:

  07-Nov 18:10 backup-dir JobId 3: Fatal error: sql_create.c:732 sql_create.c:732 insert INSERT INTO batch VALUES (1580771,3,'/Volumes/0/export/users/kodama/Desktop/GAP/gap4r4/small/small2/','sml800.z','OAAAD DkeW IGk B ih C+ A KZn BAA BY BHLtzL 1sNQO BFnqZZ A A C','0') failed:
  Incorrect key file for table '/tmp/#sql2459_94_0.MYI'; try to repair it

After doing a bit of research, I’ve concluded the /tmp volume, which is only a 256M tmpfs partition is filling to capacity before the job is able to complete.

Restarting the job this morning confirms MySQL is spooling data into /tmp.

  [root@backup tmp]# ls -l /tmp/
  total 332
  -rw-rw---- 1 mysql mysql 319276 Nov  8 09:48 #sql511e_3_0.MYD
  -rw-rw---- 1 mysql mysql   1024 Nov  8 09:48 #sql511e_3_0.MYI
  -rw-rw---- 1 mysql mysql   8722 Nov  8 09:48 #sql511e_3_0.frm

My solution for the time being is to reconfigure mysql to use /var/tmp for it’s temporary storage, rather than /tmp. This places the data on a much larger file system.

# /etc/my.cnf
[mysqld]
tmpdir=/var/tmp

I’m also planning to split the job into smaller jobs, using regular expressions to include only pieces of the home directory tree at a time. This will keep the number of files each job needs to handle under a reasonable threshold.