Symbolic links with trailing newlines?

Recently I encountered a weird situation where the software we were developing was producing symbolic links that didn’t link.

$ ls -l
total 40
-rw-r--r--  1 nfiedler  staff  101 Feb 27 13:53 a1
-rw-r--r--  1 nfiedler  staff  101 Feb 27 13:53 a2
-rw-r--r--  1 nfiedler  staff  101 Feb 27 13:53 a3
lrwxr-xr-x  1 nfiedler  staff    3 Feb 27 13:53 link1 -> a1?
lrwxr-xr-x  1 nfiedler  staff    2 Feb 27 13:58 link2 -> a1

Right, so the length is different, and the link apparently points to something that starts with “a1″, but what’s going on? I used the readlink command to dump the contents of the two links.

$ readlink link1
a1
$ readlink link2
a1

Uh, yeah, so what gives? I read through the readlink man page and found the -n option, which tells readlink not to print a newline after the link contents. Lo and behold, there was now indeed a difference. To be precisely sure what the difference was, I dumped the readlink output to a file and ran hexdump -C on that.

$ readlink -n link1 > foo1
$ hexdump -C foo1
00000000  61 31 0a                                          |a1.|
00000003

$ readlink -n link2 > foo2
$ hexdump -C foo2
00000000  61 31                                            |a1|
00000002

Finally, now we can see it was indeed a newline at the end of the symbolic link. Don’t ask how that got there, that requires a whole ‘nother story.

Posted in Computing | Leave a comment

Building Git on Mac, with docs!

If you have built Git from source on Mac OS X you probably didn’t have much trouble. That is, unless you were trying to also build the documentation, in which case you probably ran into problems, right? I’m guessing it happened somewhere around getopt, gettext, or some other library or tool you didn’t have installed. And after several attempts to get the nth dependency to build you gave up. Look no further, as these steps will build Git and its documentation on Mac OS X, at least as of today.

This guide was mostly inspired by Setting up the Git documentation build chain on Mac OS X Leopard which was mostly correct at the time it was written, I assume. As of today it’s missing a couple of steps and it deserves an update for Lion and the latest versions of the libraries.

So without further ado, here are the steps.

A. Install AsciiDoc using the usual configure, make, sudo make install, which you probably did already since this one is easy.

B. Download gettext, extract the tarball and open the file gettext-tools/gnulib-lib/stpncpy.c in an editor. Find the #define __stpncpy stpncpy line and comment it out (hat tip to biow0lf). Now the usual configure, make, sudo make install will suffice.

C. Download getopt, extract the tarball, open the Makefile in an editor and add -lintl to the LDFLAGS variable (as described in the original Wincent wiki), at which point configure, make, sudo make install should work.

D. Install Docbook XSL and Docbook XML, which is a bit more involved. This is a streamlined version of what you will find on the Wincent wiki.

  1. sudo mkdir -p /usr/local/share/docbook
  2. cd /usr/local/share/docbook
  3. sudo tar jxf ~/Downloads/docbook-xsl-1.77.1.tar.bz2
  4. sudo unzip ~/Downloads/docbook-xml-4.5.zip -d docbook-xml-4.5
  5. sudo mkdir /etc/xml
  6. sudo xmlcatalog --noout --add nextCatalog '' file:///usr/local/share/docbook/docbook-xsl-1.77.1/catalog.xml --create /etc/xml/catalog
  7. sudo xmlcatalog --noout --add nextCatalog '' file:///usr/local/share/docbook/docbook-xml-4.5/catalog.xml --create /etc/xml/catalog

E. Install Docbook2X which apparently wasn’t necessary when the Wincent page was written.

  1. ./configure
  2. make
  3. sudo make install
  4. cd /usr/local/bin
  5. sudo ln -s docbook2texi docbook2x-texi
  6. sudo xmlcatalog --noout --add nextCatalog '' file:///usr/local/share/docbook2X/xslt/catalog.xml --create /etc/xml/catalog

F. Install xmlto by first ensuring that /usr/local/bin is in your PATH before /usr/bin (sudo vi /etc/paths may help). Now the usual configure, make, sudo make install and you are nearly there.

G. Install Git, finally!

  1. git clone https://github.com/git/git.git
  2. git checkout v1.7.11.5 (or whatever the latest release happens to be)
  3. make prefix=/usr/local all doc info
  4. sudo make prefix=/usr/local install install-doc install-html install-info

At this point you should have a freshly built Git in /usr/local along with all of the documentation. Hurray! If that doesn’t work for you, feel free to write to me or post a comment. I would like to keep these instructions up to date as best I can.

Posted in Compiling, Computing, HowTo | Tagged , , | 1 Comment

Running ZFS filesystem backup

If you’ve read some of my earlier posts regarding ZFS, you’ll likely know that it is at the heart of my home file server. In some of my other posts, you’ll notice that I take backups pretty seriously. So combining both of those interests I developed a small Python script that replicates a ZFS filesystem. It works both for a one-time replication as well as ongoing incremental replications. All that’s needed is to set up the cron job to run the script. Here is an example entry from the root user’s crontab:

10 1 * * 0 /usr/local/bin/replica.py yubaba/shared safe/shared

The yubaba pool is my main RAID-Z file system and the safe pool is on a USB hard drive located in a fire/water-proof safe with a USB pass-through. You can use any two filesystems that you like, as long as there is sufficient space on the destination.

The script will maintain at most two snapshots on both the source and destination filesystems in order to support sending incremental snapshots. It is a zero-maintenance script, there is nothing that you need to do beyond the one-time setup of the cron job. Be sure not to make any changes in the destination filesystem, as they will be automatically rolled back during the next incremental replication.

The script is located on Google Code: replica.py — I’m running OpenIndiana build 148 and the version of Python there 2.6, so the script most likely requires that version.

Posted in Computing, Storage | Tagged , , , , | Leave a comment

Newlines in sed on Mac

For whatever reason, this is harder than it should have been. All I wanted to do was replace a particular expression with a newline character (0x10). My preference is typically to script such tasks, and the sed command is the perfect fit. This would be simple on just about any system, except for Mac OS X, where apparently all the standard advice is difficult to apply. Worse, the sed man page leads you to believe it’s a very simple matter of putting a newline in the replacement string. Of course, there’s no explanation as to how you are expected to do that. And not being a bash expert, I was at a loss.

Luckily enough I found a blog post that discussed, among other things, how to inject a newline using sed on Mac. Although the example is rather complicated given that he’s solving a different problem, the crux of the matter is this expression: $'\n/g'

$ echo 'foo bar baz quux' | sed -e 's/ /\'$'\n/g'
foo
bar
baz
quux

All that is really doing is taking advantage of the bash extquote option where $'string' quoting is performed on the enclosed string. In this case it’s a \n which expands to a newline character, followed by /g which goes through as-is. The baskslash (\) before the $'\n/g' tells sed to escape the newline character in the replacement string. I’m no expert here, but my understanding is that the argument to sed consists of two parts, the s/ /\ and the [newline]/g, where the latter resulted from the $'\n/g' evaluated by bash. Together this forms the sed expression s/ /\[newline]/g. How the \' doesn’t escape the quote and go through as-is to sed I’m not sure. Maybe someone can explain in the comments.

Thanks for reading!

Posted in Computing, HowTo | Tagged , | 11 Comments

Finding unversioned files in Perforce

For many years I’ve used CVS, and my experience with Subversion started during the pre-alpha days, so I am accustomed to the features they provide. In particular, I make heavy use of the Subversion status command to list not only the files I’ve modified locally, but also the files I’ve created but not yet added to the list of pending changes. It helps tremendously in remembering to add files to the change list before committing.

Recently I was working for Gracenote, which had been using Perforce for the last 10 years. Finding myself using Perforce again after several years, I searched high and low to find a way to get the Perforce command line client to provide me with the ability to discover unversioned files. Alas, I found none. Being that I know how to write halfway passable Python I wrote a script that does exactly what I want: p4unknown.py

Admittedly it’s not the prettiest name for a script, but it works and I used it for several months while I was at Gracenote. Hopefully you may find it useful as well.

Update: Recently the p4 command line has introduced a new subcommand called reconcile which makes my script irrelevant. Hurray for progress!

Posted in Computing, SCM, Scripting | Tagged , | 2 Comments

Cleaning up Ruby Gems on Mac OS X

This has come up a few times for me and searching the Internet failed to turn up any satisfying solutions. In particular, I wanted to update the Ruby Gems on my Mac and subsequently remove the old versions. I found that certain gems would not go away when the gem cleanup command was invoked.

# gem list
*** LOCAL GEMS ***
actionmailer (2.3.5, 2.2.2, 1.3.6)
actionpack (2.3.5, 2.2.2, 1.13.6)
actionwebservice (1.2.6)
activerecord (2.3.5, 2.2.2, 1.15.6)
...
# gem cleanup
Cleaning up installed gems...
Attempting to uninstall fastthread-1.0.1
Unable to uninstall fastthread-1.0.1:
	Gem::InstallError: cannot uninstall, check `gem list -d fastthread`
...

It seems that (Snow) Leopard comes bundled with a plethora of Ruby gems, all of which are installed in /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8. Meanwhile, the gem install command puts new gems in /Library/Ruby/Gems/1.8, as shown with the -d option to gem list:

# gem list -d fastthread
*** LOCAL GEMS ***
fastthread (1.0.7, 1.0.1)
    Author: MenTaLguY
    Rubyforge: http://rubyforge.org/projects/mongrel
    Installed at (1.0.7): /Library/Ruby/Gems/1.8
                 (1.0.1): /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8
    Optimized replacement for thread.rb primitives

Some folks have said to simply leave the old gems in place, while others have suggested deleting the old gems from the /System/Library location using rm. Neither solution is all that appealing to me.

The solution I was looking for, it seems, is rather simple and involves nothing more than setting the GEM_HOME environment variable before running the gem cleanup command.

# export GEM_HOME=/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8
# gem cleanup
...
Attempting to uninstall rake-0.8.3
Executables and scripts will remain installed.
Successfully uninstalled rake-0.8.3
Clean Up Complete
# unset GEM_HOME

And that will do it, all of the outdated gems are removed and I didn’t have to perform file system surgery. Yay!

Posted in Computing, HowTo | Tagged , , | 1 Comment

Using OpenSolaris and ZFS with Time Machine

In earlier posts I described how to set up a file server using OpenSolaris and ZFS. This proved to be a very good choice, at least for me, and it made sense to make this file server be the destination for my Mac’s Time Machine backups. In this post I’ll try to cover everything I did to make that possible.

Assuming you’ve got the hardware set up and the operating system is in place, it’s time to set up the ZFS pool and file system for the Time Machine backups. The values shown below will almost certainly differ for your system, so use pfexec format to get a list of disk device names; look for those that are not part of a zpool already (use zpool status to see which devices are in which pools).

$ pfexec bash
# zpool create yubaba raidz c4t0d0 c4t1d0 c5t0d0 c5t1d0
# zfs create yubaba/mac_backup
# zfs set compression=on yubaba/mac_backup
# zfs set com.sun:auto-snapshot=true yubaba/mac_backup

I chose to use RAID-Z (striping with parity) so as to maximize my disk space while still getting some redundancy. The two zfs set commands are used to enable ZFS supported features for the new file system. The first turns on compression, which if your system is fast enough, is probably a good idea. The second enables the automatic snapshots that OpenSolaris makes possible via a feature called Time Slider. This is a good way to roll back unwanted changes if the need arises.

Now that the file system is ready, set up the netatalk daemon so your Mac can talk to the file server. There are other options (e.g. NFS) but this seems the most straightforward approach to me. With netatalk in place, add the following line to the /usr/local/etc/netatalk/AppleVolumes.default file so that the backup volume is mountable on the Mac, then restart the netatalk daemon.

/yubaba/mac_backup "Mac Backup" allow:@staff cnidscheme:dbd options:usedots,invisibledots,upriv perm:0770

By default, earlier versions of Mac OS X do not allow unsupported servers to be the destination for Time Machine backups. To convince it otherwise, simply run the following command in Terminal. This is only necessary the first time, as the change is saved permanently.

$ defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

The next step is to create the sparse bundle disk image on the server. If simply pointing Time Machine to the backup server doesn’t work (i.e. it reports that it could not create the backup disk image), then you will have to create the disk image manually and copy it over to the server. There is a procedure that makes good use of the graphical utilities in OS X, outlined on kremalicious, but since we already have the Terminal open, let’s do it from the command line. After all, if you’re a Solaris fan you probably won’t mind seeing how this is really done.

Time Machine puts some cryptic pieces of information into the disk image name that it creates on any remote device, be it Time Capsule or our storage server. These are nothing more than the MAC address and the host name, and can be gathered with the following commands.

$ hostname -s

This command shows the host name of your Mac in shortened form, which is the first piece of information.

$ ifconfig en0 | awk '/ether/{gsub(/:/, "", $2); print $2}'

This command sequence does nothing more than return the MAC address in the format used by Time Machine. You could get this from Network Utility, but then you’d have to type the numbers by hand.

Now we can create the sparse bundle disk image in which Time Machine will store the backups. This is done with a single, albeit long, command. Where you see the “300g” below, feel free to replace that with whatever size specification you like. The ‘g’ stands for gigabytes and the 300 indicates the number of gigabytes. Obviously this is a sparse bundle so it won’t immediately consume 300GB from the start, that’s just the maximum size to which the image will grow.

$ sudo hdiutil create -nospotlight -type SPARSEBUNDLE -imagekey sparse-band-size=131072 -size 300g -fs "HFS+J" -volname "MacBackup" hostname_MACADDR.sparsebundle

That’s a rather long command, so let’s take a look at each part. First, the hdiutil command is used to manage disk images of all sorts, and it has a subcommand called create that will create just about any kind of image you could want. The -nospotlight option simply sets an attribute that tells Spotlight to ignore this volume. The -type option is obvious, but the -imagekey is a bit mysterious. That sets up the disk image to use larger files for the “bands” that make up the sparse bundle. According to this discussion, Jon Stevens (latchkey) says the larger the band size the better the performance over the network. The -size was alluded to above; use whatever value you like but it should at least be as large as your OS X system disk. The -fs option indicates the type of file system to use within the disk image, in this case it’s HFS+ with journaling.

The last parts of the command are machine specific and you will need to change their values accordingly. First off, for the -volname option, use whatever name you like, it’s there for ease of identification, nothing more. The hostname should be replaced with the value returned by hostname -s from the commands shown above. Likewise, the MACADDR must be replaced with the MAC address revealed with the ifconfig command shown above. Note that the underscore (_) and the extension (.sparsebundle) are important, so keep this in the disk image name.

A lot of what I’ve learned about Time Machine came from using BackMyFruitUp, created by Jon Stevens. It includes an AppleScript scriptlet that creates the sparse bundle disk image for you, if you like.

Now that the disk image has been created, you can copy it to the volume you created on your storage box. You can use cp in Terminal, or simply drag and drop it using Finder.

At this point you can open the Time Machine preferences and choose the volume on the storage server as the backup “disk”. At first nothing will happen, so you may want to start a backup immediately. If that fails mysteriously, try unmounting the volume in Finder, then start the backup again. From this point onward, Time Machine will automatically mount the volume whenever it performs a backup, and any time you “enter” Time Machine to browse its contents.

Posted in Computing, HowTo, Storage | Tagged , , , | 1 Comment