10 1 * * 0 /usr/local/bin/replica.py yubaba/shared safe/shared

The yubaba pool is my main RAID-Z file system and the safe pool is on a USB hard drive located in a fire/water-proof safe with a USB pass-through. You can use any two filesystems that you like, as long as there is sufficient space on the destination.

The script will maintain at most two snapshots on both the source and destination filesystems in order to support sending incremental snapshots. It is a zero-maintenance script, there is nothing that you need to do beyond the one-time setup of the cron job. Be sure not to make any changes in the destination filesystem, as they will be automatically rolled back during the next incremental replication.

The script is located on Google Code: replica.py — I’m running OpenIndiana build 148 and the version of Python there 2.6, so the script most likely requires that version.

Luckily enough I found a blog post that discussed, among other things, how to inject a newline using sed on Mac. Although the example is rather complicated given that he’s solving a different problem, the crux of the matter is this expression: $'\n/g'

$ echo 'foo bar baz quux' | sed -e 's/ /\'$'\n/g'
foo
bar
baz
quux

All that is really doing is taking advantage of the bash extquote option where $'string' quoting is performed on the enclosed string. In this case it’s a \n which expands to a newline character, followed by /g which goes through as-is. The baskslash (\) before the $'\n/g' tells sed to escape the newline character in the replacement string. I’m no expert here, but my understanding is that the argument to sed consists of two parts, the s/ /\ and the [newline]/g, where the latter resulted from the $'\n/g' evaluated by bash. Together this forms the sed expression s/ /\[newline]/g. How the \' doesn’t escape the quote and go through as-is to sed I’m not sure. Maybe someone can explain in the comments.

Thanks for reading!

Finding myself having to use an SCM that is frozen in time, like a caveman in an ice block, I searched high and low to find a way to get the Perforce command line client provide me with the ability to discover unversioned files. Alas, I found none. Being that I know how to write halfway passable Python I wrote a script that does exactly what I want: p4unknown.py

Admittedly it’s not the prettiest name for a script, but it works and I used it for several months while I was at Gracenote. Hopefully you may find it useful as well.

# gem list
*** LOCAL GEMS ***
actionmailer (2.3.5, 2.2.2, 1.3.6)
actionpack (2.3.5, 2.2.2, 1.13.6)
actionwebservice (1.2.6)
activerecord (2.3.5, 2.2.2, 1.15.6)
...
# gem cleanup
Cleaning up installed gems...
Attempting to uninstall fastthread-1.0.1
Unable to uninstall fastthread-1.0.1:
	Gem::InstallError: cannot uninstall, check `gem list -d fastthread`
...

It seems that (Snow) Leopard comes bundled with a plethora of Ruby gems, all of which are installed in /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8. Meanwhile, the gem install command puts new gems in /Library/Ruby/Gems/1.8, as shown with the -d option to gem list:

# gem list -d fastthread
*** LOCAL GEMS ***
fastthread (1.0.7, 1.0.1)
    Author: MenTaLguY
    Rubyforge: http://rubyforge.org/projects/mongrel
    Installed at (1.0.7): /Library/Ruby/Gems/1.8
                 (1.0.1): /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8
    Optimized replacement for thread.rb primitives

Some folks have said to simply leave the old gems in place, while others have suggested deleting the old gems from the /System/Library location using rm. Neither solution is all that appealing to me.

The solution I was looking for, it seems, is rather simple and involves nothing more than setting the GEM_HOME environment variable before running the gem cleanup command.

# export GEM_HOME=/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8
# gem cleanup
...
Attempting to uninstall rake-0.8.3
Executables and scripts will remain installed.
Successfully uninstalled rake-0.8.3
Clean Up Complete
# unset GEM_HOME

And that will do it, all of the outdated gems are removed and I didn’t have to perform file system surgery. Yay!

Assuming you’ve got the hardware set up and the operating system is in place, it’s time to set up the ZFS pool and file system for the Time Machine backups. The values shown below will almost certainly differ for your system, so use pfexec format to get a list of disk device names; look for those that are not part of a zpool already (use zpool status to see which devices are in which pools).

$ pfexec bash
# zpool create yubaba raidz c4t0d0 c4t1d0 c5t0d0 c5t1d0
# zfs create yubaba/mac_backup
# zfs set compression=on yubaba/mac_backup
# zfs set com.sun:auto-snapshot=true yubaba/mac_backup

I chose to use RAID-Z (striping with parity) so as to maximize my disk space while still getting some redundancy. The two zfs set commands are used to enable ZFS supported features for the new file system. The first turns on compression, which if your system is fast enough, is probably a good idea. The second enables the automatic snapshots that OpenSolaris makes possible via a feature called Time Slider. This is a good way to roll back unwanted changes if the need arises.

Now that the file system is ready, set up the netatalk daemon so your Mac can talk to the file server. There are other options (e.g. NFS) but this seems the most straightforward approach to me. With netatalk in place, add the following line to the /usr/local/etc/netatalk/AppleVolumes.default file so that the backup volume is mountable on the Mac, then restart the netatalk daemon.

/yubaba/mac_backup "Mac Backup" allow:@staff cnidscheme:dbd options:usedots,invisibledots,upriv perm:0770

By default, earlier versions of Mac OS X do not allow unsupported servers to be the destination for Time Machine backups. To convince it otherwise, simply run the following command in Terminal. This is only necessary the first time, as the change is saved permanently.

$ defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

The next step is to create the sparse bundle disk image on the server. If simply pointing Time Machine to the backup server doesn’t work (i.e. it reports that it could not create the backup disk image), then you will have to create the disk image manually and copy it over to the server. There is a procedure that makes good use of the graphical utilities in OS X, outlined on kremalicious, but since we already have the Terminal open, let’s do it from the command line. After all, if you’re a Solaris fan you probably won’t mind seeing how this is really done.

Time Machine puts some cryptic pieces of information into the disk image name that it creates on any remote device, be it Time Capsule or our storage server. These are nothing more than the MAC address and the host name, and can be gathered with the following commands.

$ hostname -s

This command shows the host name of your Mac in shortened form, which is the first piece of information.

$ ifconfig en0 | awk '/ether/{gsub(/:/, "", $2); print $2}'

This command sequence does nothing more than return the MAC address in the format used by Time Machine. You could get this from Network Utility, but then you’d have to type the numbers by hand.

Now we can create the sparse bundle disk image in which Time Machine will store the backups. This is done with a single, albeit long, command. Where you see the “300g” below, feel free to replace that with whatever size specification you like. The ‘g’ stands for gigabytes and the 300 indicates the number of gigabytes. Obviously this is a sparse bundle so it won’t immediately consume 300GB from the start, that’s just the maximum size to which the image will grow.

$ sudo hdiutil create -nospotlight -type SPARSEBUNDLE -imagekey sparse-band-size=131072 -size 300g -fs "HFS+J" -volname "MacBackup" hostname_MACADDR.sparsebundle

That’s a rather long command, so let’s take a look at each part. First, the hdiutil command is used to manage disk images of all sorts, and it has a subcommand called create that will create just about any kind of image you could want. The -nospotlight option simply sets an attribute that tells Spotlight to ignore this volume. The -type option is obvious, but the -imagekey is a bit mysterious. That sets up the disk image to use larger files for the “bands” that make up the sparse bundle. According to this discussion, Jon Stevens (latchkey) says the larger the band size the better the performance over the network. The -size was alluded to above; use whatever value you like but it should at least be as large as your OS X system disk. The -fs option indicates the type of file system to use within the disk image, in this case it’s HFS+ with journaling.

The last parts of the command are machine specific and you will need to change their values accordingly. First off, for the -volname option, use whatever name you like, it’s there for ease of identification, nothing more. The hostname should be replaced with the value returned by hostname -s from the commands shown above. Likewise, the MACADDR must be replaced with the MAC address revealed with the ifconfig command shown above. Note that the underscore (_) and the extension (.sparsebundle) are important, so keep this in the disk image name.

A lot of what I’ve learned about Time Machine came from using BackMyFruitUp, created by Jon Stevens. It includes an AppleScript scriptlet that creates the sparse bundle disk image for you, if you like.

Now that the disk image has been created, you can copy it to the volume you created on your storage box. You can use cp in Terminal, or simply drag and drop it using Finder.

At this point you can open the Time Machine preferences and choose the volume on the storage server as the backup “disk”. At first nothing will happen, so you may want to start a backup immediately. If that fails mysteriously, try unmounting the volume in Finder, then start the backup again. From this point onward, Time Machine will automatically mount the volume whenever it performs a backup, and any time you “enter” Time Machine to browse its contents.

Parallelized

The first major change was to introduce a parallelized version of burstsort. In this initial attempt, it only parallelizes the bucket sorting. The building of the trie and buckets still happens in a single thread. However, once the structure is built, the multi-threaded version creates a thread pool of size equal to the number of available processors, and creates sort jobs that are then run in parallel. These jobs run independently of one another, copying the sorted output to the original array without any unnecessary synchronization. As a result, the overall runtime is shortened considerably on a dual-core CPU, which in my case is the Intel Core 2 Duo in my MacBook Pro.

Engineered Burstsort

With the publication of the WEA 2008 paper by Ranjan Sinha and Anthony Wirth came a newly engineered variation of the original burstsort. In particular, this algorithm made much better use of memory, while still being nearly as fast as the original algorithm. It primarily makes a change in the structure of the buckets, where instead of a single dynamic array they now have an array that points to other arrays of pointers to strings. These sub-buckets, as they are called, are grown at a slower pace and stop growing at a much lower threshold than the original algorithm. Once a sub-bucket is filled, a new sub-bucket is created, and so on until the overall bucket size reaches a threshold equal to that of the original algorithm. As a result, the memory usage is dramatically improved.

Needless to say, my excitement was very high at this point. I desperately wanted to implement this redesigned burstsort in Java as soon as I could. But, certain other obligations got in the way for a time, and after a few months I finally wrote the Java version of the engineered burstsort. After fixing one small mistake it was working and it was better than I could have imagined. Not only did the memory efficiency go from about 25 percent to 95 percent, it was often a little bit faster than my original implementation.

What’s Next

There are yet more improvements to make. First of all, the WEA 2008 paper offers a second improvement, which is to copy the string tails from a bucket to a string buffer and sort them there. That is, the string buffer would only be used during the bucket sorting phase and would be re-used after each bucket is sorted. I have an idea to use a large character array and an implementation of CharSequence to create lightweight strings.

Secondly, I want to experiment with the parallel version of burstsort. In particular, try out the suggestions made by Kimo to parallelize the building of the trie/bucket structure. I think it can definitely be done, the only question will be how much contention there will be on the trie nodes. As a means to test these parallel algorithms I’ve bought a quad-core AMD Phenom CPU and mainboard to use as my development machine. I’m really looking forward to seeing the results.

1. Install OpenSolaris on the first disk as usual.
2. Make sure the second disk has an SMI label instead of an EFI label so it can be attached to the rpool. Invoke the format -e command (the -e is important, it enables expert options like setting the label type), choose the second disk, type  “label”, and choose the SMI option. If it does not allow this change because you have to delete the partitions, then type “fdisk” and delete the partitions and create a new one (Solaris2 type), save the changes to disk, and now you can change the label type. Type “quit” to save your changes to disk. [opensolaris.org]
3. The partition table of the second disk needs to be made identical to the first one. As the root user, type prtvtoc /dev/rdsk/disk1 | fmthard -s - /dev/rdsk/disk2 where disk1 and disk2 might look like c0d0s2 and c0d1s2 (note the use of slice 2 here, the “whole disk” slice). [opensolaris.org]
4. Attach the second disk to the root pool: zpool attach rpool disk1 disk2 (where disk1 and disk2 are the device names, typically including the slice, such as c7d0s0 and c7d1s0). You may find it necessary to force the attach as ZFS may complain about slices overlapping. If s0 overlaps s2 that’s actually normal, so just add the -f flag.
5. Make sure the boot loader is copied to the second disk so it is bootable in the event the first disk becomes unbootable: installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/disk2 [opensolaris.org]

See, it was a cinch after all. That is, once you knew what to do.

[Update: Removed the remark about using s0 or s2 interchangably. So far the posts I've seen mostly lean toward s2, but the prtvtoc output for s0 and s2 on my disks were identical, so it would not have made a difference in my case. Nonetheless, go with the flow. Also added references to original sources in case you were to think I was some kind of genius or something.]

In an earlier post I compared a Drobo to a custom built storage system based on OpenSolaris, with the conclusion that while a Drobo is convenient, building your own server offers many advantages. In this post I want to show how I went about building a new storage system based on an article that came out at the end of last year. Among the improvements over my old server are lower power consumption and a smaller form factor. In fact, the case that I’ve chosen has the same footprint as the Drobo, and only about five inches taller. In addition, this particular case has hot swappable drive bays, so disks can be replaced while the system is powered on.

Before I get into the particulars, I want to share some useful links that may help if you’re new to OpenSolaris and/or building a system from parts. First of all, there’s the First-Timer’s Guide to Building a Computer from Scratch at LifeHacker. I find that the hardest part of building your own system is getting the parts list right. To that end, check out Ars Technica’s excellent system buyer’s guide, which offers recipes for various types of systems and provides advice on choosing suitable parts. Once the system is put together you might be asking yourself which operating system to choose. For me, OpenSolaris with ZFS is a no-brainer — it really is an excellent software stack that gives me piece of mind knowing my that data is as safe as it can be (for the amount of money I’m willing to spend). But you don’t have to take my word for it, see what Simon and Scott have to say about OpenSolaris and ZFS. If setting up and administering OpenSolaris is too daunting for you, then check out FreeNAS — it’s based on FreeBSD with a port of ZFS and is designed to be easier to set up and maintain.

Power Consumption

One of the primary goals with building a new storage server was to minimize power consumption. According to a recently purchased Kill-A-Watt device, my original web server box consumes about 100W while idle, and 120W while the disks are active. During power-on, the consumption peaks at around 260W. This is actually not too bad for a server-class machine. As for the new server, it starts off at around 36W, then hits 117 when everything spins up at once, then levels out at 62. During active reading and writing, the consumption peaks at 71W and typically hovers around 67W. Not too bad, around half of the consumption of my original machine.

Bill of Materials

These prices are based on what was available in March of 2009, so these may have changed by now. Nonetheless, it shows just how little there is to buy, with half of the parts available from a single retailer.

That adds up to $420 for everything you need to equal a Drobo and DroboShare. That’s $150 less than the price of the Drobo plus DroboShare on Amazon, and this system is faster and the components can be sourced from multiple vendors. That, by the way, was one of my goals in building a new system; I didn’t want to be locked in to a particular hardware vendor.

Regarding the selection of the Chenbro case, it was a luxury. It’s rather pricey for an ITX case, but the LogicSupply review convinced me it was worth it. Another option would be to use the Enlight PR-42A1 which can be had for around $60. The major difference is the Enlight is an ATX case, so you would have to get a different mainboard, and chances are it would not have a low power chipset.

You may have noticed that the mainboard does not support ECC (error-checking and correcting) memory. Yes, that was a bit of a let down, but to get ECC you typically have to go with server-grade parts, which cost more and often consume more power.

Operating System

Unlike with pre-built systems, you get to choose your operating system when building your own system. There are many people who go with Windows Home Server, but obviously that costs money. Free options include FreeBSD, FreeNAS, Linux and OpenSolaris. Being a long time user of Solaris, I went with OpenSolaris. It has the advantage of including the reference implementation of ZFS. While I had been a Linux user for 10 years, and ran a software project/web server on Linux for half that time, I know a better solution when I see one.

Disk Performance

Given that this system was built to replace a Drobo, I wanted to compare their performance. However, there is no reasonable way to compare the performance of a Drobo with a storage server such as the one I’ve built here. Firstly, the Drobo does not have native Ethernet, so it has to rely on an external device connected over USB. That alone is going to add delay and turn the comparison into an apples and oranges argument. What I can say though is that the new server feels faster than the Drobo, and is certainly fast enough for my needs. It is on par with the server-grade equipment that this box is replacing, and meets our file sharing and Time Machine backup requirements perfectly.

Conclusion

If you’re new to building a storage server, I hope that I’ve given you some inspiration to learn more. If you have the time and a handful of tools, building your own system is pretty easy. And if you haven’t spent a lot of time learning how to configure an operating system, then let FreeNAS come to your aid. In short, you can choose your own parts from any of a number of suppliers, install whatever operating system you like, and set up a pretty reliable vault for your data.

Update: Check out the media storage system George Drapeau built based on OpenSolaris and ZFS.

The good news is, there’s a simple solution. Just install the SUNWscp package. This is the “source compatibility package”, which installs numerous commands that help OpenSolaris behave more like other Unix systems. The Logwatch installer script prepends the /usr/ucb directory to the PATH when it runs, so it finds the install script that it is expecting, and thus it installs Logwatch perfectly. The only thing left is to add the crontab entry, as shown at the end of the install output.

One last note about Logwatch, and it concerns that crontab entry. It seems that the default configuration for Logwatch is to print the report rather than sending an email to the default recipient, root. However, the example crontab entry is redirecting all output to /dev/null. So how exactly is one supposed to get a daily report? The answer is to edit the /etc/logwatch/conf/logwatch.conf file, adding Print = no at the end of the file. That tells Logwatch to email the report rather than printing. It’s a mystery to me why that’s the default given the example crontab entry they display during the installation process. But at least it’s easy to fix, and nicely demonstrates how easy it is to customize Logwatch without touching the default configuration files.

Start by installing netatalk, if you haven’t already, as described in an earlier post. Next, create the SMF manifest file that will register the AFP daemon as a discoverable service. Name the file dnssd-afp.xml and place it in the /var/svc/manifest/site directory.

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<service_bundle type="manifest" name="dnssd-afp">
  <service
     name="site/dnssd-afp"
     type="service"
     version="1">

    <single_instance/>

    <dependency
       name="filesystem-local"
       grouping="require_all"
       restart_on="none"
       type="service">
      <service_fmri value="svc:/system/filesystem/local:default"/>
    </dependency>

   <dependency
       name="dns-multicast"
       grouping="require_all"
       restart_on="none"
       type="service">
      <service_fmri value="svc:/network/dns/multicast:default"/>
    </dependency>

    <exec_method
       type="method"
       name="start"
       exec="/lib/svc/method/dnssd-afp"
       timeout_seconds="60">
      <method_context>
        <method_credential user="root" group="root"/>
      </method_context>
    </exec_method>

    <exec_method
       type="method"
       name="stop"
       exec=":kill"
       timeout_seconds="60">
    </exec_method>

    <instance name="default" enabled="false" />

    <stability value="Unstable" />

    <template>
      <common_name>
        <loctext xml:lang="C">
          dns-sd registration of afp daemon
        </loctext>
      </common_name>
      <documentation>
        <manpage title="dns-sd" section="1M" manpath="/usr/man"/>
      </documentation>
    </template>
  </service>
</service_bundle>

Change the ownership of the manifest to root:sys and make it read-only by all but the root user. For this service, we’ll need to write a shell script that spawns dns-sd as a background process, otherwise SMF will timeout waiting for the service to start (the SMF documentation is better at explaining this than I am). Note that chihiro in the script is the hostname that will be advertised; you may want to change this to suit your needs.

#!/sbin/sh
#
# Registers the AFP daemon with dns-sd.
#
/usr/bin/dns-sd -R chihiro _afpovertcp._tcp local 548 &
/usr/bin/dns-sd -R chihiro _device-info._tcp. local 548 model=Xserve &
# Sleep to ensure service has enough time to start up,
# otherwise SMF will timeout waiting for it to be ready.
sleep 5

Place the shell script in the /lib/svc/method directory, give it the name dnssd-afp, change the ownership to root:bin, and make the file executable by all and writable only by root. Now we’re ready to import the service configuration and start the service. Import it using the command pfexec svccfg -v import /var/svc/manifest/site/dnssd-afp.xml, then start the service using pfexec svcadm enable dnssd-afp, and finally check that the service is running with svcs dnssd-afp. At this point, if the dns-sd process were to unexpectedly die, SMF will immediately restart it. That’s one of the many advantages of SMF over initd, which does not monitor the processes that it initiates. With the AFP service now registered, any Mac on your network should see your storage server as a Mac-compatible file share, which will appear automatically in the Finder sidebar. If you’ve added a shared volume to your Login Items previously, you can remove it, you won’t need it any more.