Mivok.net

From Solaris to FreeBSD

Wed, 31 Mar 2010 00:00:00

Less than one week after I switched my hosting over to Solaris 10, with all its ZFS/dtrace goodness, Oracle quietly makes a license change that everybody dealing with Solaris is likely now familiar with, and Solaris 10 is no longer free to use. Emails to Sun/Oracle's licensing department result only in form letters repeating instructions on the website, and then nothing.

I can't really blame Oracle for this, Sun didn't make enough money to survive, and Oracle has this radical idea that you need to actually charge people in order to make money. I can blame them for not providing more clarity regarding the issue (so far they haven't announced anything), and for leaving customers unsure about what's going to happen next. However, this is mostly besides the point. I now needed to look into a good alternative.

OpenSolaris is the obvious candidate, and I've played around with it a little previously, but I can't make myself like some of the changes made to it. The biggest annoyances being related to the new packaging system and some of the poor choices made in its design (e.g. no --nodeps option). That is an entire post (or rather, rant) in itself however. In addition to this, I can't help but believe that Oracle is going to make some change to OpenSolaris that makes it not a realistic option.

This is where FreeBSD comes in. With release 8.0, ZFS has become a fully supported filesystem. It has dtrace support, jails (just like zones), even virtual networking so you can have a full network stack inside the jail.

For my personal server, the main feature I was interested in was ZFS, specifically ZFS root/boot. With ZFS it is trivial to set up mirrored drives, and I wanted to avoid doing software raid with UFS as well as ZFS. Thankfully there is extensive documentation on how to do this. It isn't in the standard install, but if you need a repeatable procedure for many servers, it's a (relatively) simple matter to script the installation, and you would probably want to do this anyway for an automated install.

There were a few gotchas, as with any new system you're not familiar with, but so far it looks quite nice. I'll be looking further into jails (especially the vimage jails) and other nice features. Hopefully, FreeBSD will turn out to be a good replacement for Solaris.

A skeleton template for new python projects

Sat, 06 Mar 2010 00:00:00

Whenever I start a new programming project, I find myself implementing the same initial functionality over and over: config files, command line options, some form of debug/logging output and so on. I hesitate to call this 'boilerplate' code, but I can't really think of a better word to describe it. Faced with this, I've come up with a template that I can build on that gives me some skeleton code to implement features I nearly always require. It's not very big, but saves time when starting a new project.

Most of my projects start with some wacky idea that just pops into my head (usually with the assistance of a higher than normal level of caffeine), then out through my fingers as quickly as possible to get something that kinda sorta works before I forget, or worse - get pulled off on some other task (my free time is very limited). Before starting the skeleton code, I would be half way through, suddenly realizing that it would be really useful to have somewhere to store configuration options right now, and waste time implementing that code instead of what I'm really doing. Using python with its large amount of built-in libraries reduces, but doesn't eliminate the problem.

I suspect a template of this sort is a somewhat personal thing, but if anybody would find such a thing useful, the git repository is git://git.mivok.net/pytemplate.git/ (browse).

These are the things I implemented/included in the template:

Setting up of logging (used for debuggin and informational output) using the logging module. This does some basic setup with timestamps, outputting to stdout, and then I can just do logging.info("something") or logging.debug("debug only output") and things work out how I expect.

Configuration file parsing using ConfigParser. There is code to search for several common config file locations (~/.prognamerc, /etc/prognamerc), and to load a default configuration from data/defaults.

ConfigParser isn't used directly. There is a separate config.py module that adds default variants of the various get methods. For example, getintdefault, which will fetch an integer from the config file, falling back to a default value if the option doesn't exist rather than raising an exception.

Command line option processing with some default options specifying whether to turn debugging output on and to specify an alternative config file location. New options can be quickly added, and the two options I add in every project are already in place.

A project layout based on that mentioned at http://sayspy.blogspot.com/2010/03/various-ways-of-distributing-python.html and a predone setup.py script, allowing simple distribution of the app.

A use_template.sh script that goes through and replaces 'progname' with the actual name of the project. Edit this script, fill in the values, and run it once. Then it can be deleted, with the template code already filled in.

This is being updated as I change how I do things. For instance the repository layout was changed very recently. If you find yourself starting multiple projects and implementing similar features each time, try building your own template. Even if it only contains a license file, skeleton readme file, and something to process command line arguments, it is work that you don't have to do each time.

Gitosis - manage git repositories sanely

Fri, 05 Mar 2010 00:00:00

I've finally made all my projects available publicly via git at http://git.mivok.net/ thanks to gitosis. Before that, I kind of just thrown everything in a git directory under my home directory and accessed it over ssh, which worked fine for private repositories, but fell flat whenever I wanted to make something available to somebody else.

Gitosis promised to make it easy to add new repositories and set up access for new people as needed, and once everything is set up, it is really easy - everything is contained in a config file inside a git repository, so you can make changes locally and push. You also have the benefit that your changes themselves are under version control. However, there were a few hiccups along the way, so I'm going to describe what I did in case others try and hit the same problems I did.

Gitosis uses python and setuptools, which I already had available. I'm running Ubuntu, so installing any requirements is as simple as running:

aptitude install python python-setuptools

Of course, git itself is a requirement. For now we'll use the Ubuntu package, but it's a good idea to build from source if you want the latest version:

aptitude install git-core

Next, get the gitosis source:

git clone git://eagain.net/gitosis.git

and install:

cd gitosis
sudo python ./setup.py install

So far, everything is pretty straightforward. Next we need to add a user that everyone will connect as in order to access repositories. The main method gitosis uses for accessing repositories, is to have a single user that everyone connects to over ssh. Logins are only allowed via ssh keys, and anyone who connects is restricted to running gitosis commands, preventing them from accessing anything they shouldn't.

sudo adduser --system --shell /bin/sh --gecos 'git user' \
    --group --disabled-password --home /srv/git git

Here I've set the home directory to /srv/git. This directory will hold all repositories and gitosis files. Next we need to initialize this directory with all of the gitosis configuration files:

sudo -H -u git gitosis-init < your-ssh-key.pub

(the -H option to sudo sets the HOME variable to the user you are running commands as. In this case - /srv/git).

The your-ssh-key.pub file should be your ssh public key for the computer you are working on now. You will use this to access the administration repository and any other repositories you create later. If you don't have an ssh key set up already, make one now and copy the id_rsa.pub file to the server before running the above command:

ssh-keygen -t rsa -b 4096

For more information on ssh-keys, see the ssh-keygen man page.

Note: by default, gitosis takes the comment field of your ssh key to be your username. In my case, it was mark@laptop, and I would have had to use mark@laptop as my username whenever editing permissions. If you want something nicer, edit the copy of your public key before running the gitosis-init command and change the comment field to something a little nicer.

Now you have the basic server set up. To edit the configuration, clone the administration repository:

git clone git@your-server.example.com:gitosis-admin.git

Then you can edit the gitosis.conf file, commit it, and push back to the server.

At this point I hit my first snag. Any changes pushed back to the server didn't take effect. The magic updating of settings wasn't working. After some hunting around (read: typing stuff into Google and clicking frantically), I found that all of the magic is done via a hook on the gitosis-admin repository. For some reason, the hook script wasn't executable, and so never ran. Before committing any configuration changes, make sure to fix the permissions on the repository hook:

sudo chmod +x /srv/git/repositories/gitosis-admin.git/hooks/post-update

The basic gitosis setup at this point is complete. Aside from adding repositories and new users, the other steps are optional. However, we are talking about making repositories publicly accessible, and the other two steps - setting up git:// access via git-daemon and setting up gitweb will do this.

First though, here's a quick overview on adding users/repositories.

To add a new user, get a copy of their ssh public key (ssh keys are what makes the whole thing work), and copy it to your gitosis-admin checkout inside the keydir directory. Name the file USERNAME.pub, replacing the username with the name of the user you wish to add - this is the username you will use when setting permissions. For example, if you add joe.pub, then you will use joe as the username in the configuration below.

To add a repository, you just give somebody permission to access it and then push. This involves editing the gitosis.conf and adding a few lines:

[group foo]
writable = myrepository
members = joe

This allows user joe to write to myrepository.git. You then add this as a remote in your local repository and push to create the repository on the server:

cd path/to/my-repository
git remote add origin git@your-server.example.com:myrepository.git
git push

This assumes you actually have something to push. In practice this isn't an issue - you start with a blank local repository (using git init), commit your first changes, and push. The first person to push actually creates the repository.

Setting up git:// access

This part allows people to clone a repository without needing to authenticate, and without having to generate ssh keys. You can't push to repositories in this way however - you have to use ssh if you want to push back to a repository. Chances are, you don't want all repositories to be public, and gitosis allows you to pick and choose which you make public and which you make private using (wait for it...) the gitosis.conf file.

Setting up git:// access is as simple as running the git-daemon command:

sudo -u git git-daemon --base-path=/srv/git/repositories/

If you're running Ubuntu however, gitosis comes with a nice script that you just drop in to /etc/event.d, edit to change the path, and it will start the git-daemon automatically on boot:

sudo cp gitosis/etc-event.d-local-git-daemon /etc/event.d/local-git-daemon
sudo sed -i s+/srv/example.com/git+/srv/git+ /etc/event.d/local-git-daemon
sudo initctl start local-git-daemon

The initctl script starts the daemon without rebooting, which is usually a good thing.

By default, no repositories are made public. To make them public, you need to add a daemon = yes option to your gitosis.conf:

[repo myrepository]
daemon = yes

Here we have made a new repo section for myrepository. Save the gitosis.conf file, commit, push, and you should be able to clone myrepository using git://your-server.example.com/myrepository.git.

Gitweb - making everything look pretty

The final step is getting gitweb working. For this you need a copy of gitweb.cgi and associated files. I built git from source, and gitweb.cgi was built as part of this, but if you didn't do this, there is an Ubuntu package available called gitweb. I also use lighttpd on my server, with pages stored under /srv/www/domain.example.com/pages/ so I'll be describing a configuration for that server and layout.

First, copy gitweb.cgi, gitweb.css, and all of the images to /srv/www/domain.example.com/. I put the css files and images inside a pages subdirectory (the document root), and put gitweb.cgi inside a separate cgi-bin directory outside of the document root.

Next, configure lighttpd. I have simple-vhost set up which sets the document root based on the domain name requested, so we only need to do special set up for the git/cgi parts:

$HTTP["host"] =~ "^git\.your-server\.example\.com$" {
     url.redirect = (
         "^/$" => "/gitweb/",
         "^/gitweb$" => "/gitweb/"
     )
     alias.url = (
         "/gitweb/" => "/srv/www/git.your-server.example.com/cgi-bin/gitweb.cgi",
     )
     setenv.add-environment = (
         "GITWEB_CONFIG" => "/srv/www/git.your-server.example.com/gitweb.conf",
     )
     $HTTP["url"] =~ "^/gitweb/" { cgi.assign = ("" => "") }
 }

Gitosis does provide a config file for lighttpd, but it wasn't appropriate for my setup. Note that the above needs the following modules loaded: mod_alias, mod_cgi, mod_redirect, mod_setenv.

Gitweb.cgi needs editing slightly in the above configuration, by default it looks for the css file in the same location as the gitweb.cgi file (i.e. in a /gitweb/ dir), but they are stored at the root of the site. Open up gitweb.cgi and search for gitweb.css. Add a slash before the filename and save the file.

Next is creating a gitweb.conf file. Again, gitosis helps out here, providing a gitweb.conf file that just needs some tweaking with the right paths. Copy the gitweb.conf file from the gitosis source distribution to /srv/www/git.your-server.example.com/, and open it up for editing.

Edit the $projects_list, $projectroot, and the @git_base_url_list lines and save:

$projects_list = '/srv/git/gitosis/projects.list';
$projectroot = "/srv/git/repositories";
@git_base_url_list = ('git://your-server.example.com');

By default, gitosis creates repositories that are only accessible by the git user and users in the git group, so we need to give the web server permissions to access repositories. If this isn't done, gitweb will say that there are no repositories available even when you configure web access in gitosis for the repository.

usermod -G git www-data

You will need to restart the web server after this in order for the group change to take effect. If your web server is running as someone else other than www-data, change the above command appropriately.

Finally, to give access to a repository via gitweb, the process is similar to setting up git:// access. Edit gitosis.conf, and add a gitweb = yes line next to the daemon = yes line for the repository. Commit, push, and the repository should now show up in gitweb. In the default configuration, you need to have both daemon = yes and gitweb = yes for a repository to be made available via gitweb. See the gitweb.conf file if you want to change this.

Bash function renaming and overriding

Sun, 20 Sep 2009 00:00:00

One annoyance I found when writing bash scripts is the lack of function references. This became apparent when overriding a function, but when I wanted to change the behavior only slightly. I had a library of functions, and wanted to add some commands before the start of the function, and some cleanup code immediately after it finished.

This being a library function that was called elsewhere, I couldn't edit the function in the library itself. Nor could I edit the calling code and add the steps before and after - the calling code was itself another library function. This left the option of copying and pasting the entire function, and adding my extra code to the beginning and end.

In python (and many other languages), I would have done something like the following:

old_foo = foo

def foo():
    initialization_code()
    old_foo()
    cleanup_code()

but bash doesn't seem to support function references in that manner. After much searching however, I finally found a way to save a function under a new name, which gives the same kind of functionality using bash's declare builtin.

The declare command prints out the values of declared variables, and more importantly, declared functions - declare -f foo will print out the code for function foo. So all you need to do is execute the output of the declare -f command, after substituting the name of the function. The following bash function does just this:

save_function() {
    local ORIG_FUNC=$(declare -f $1)
    local NEWNAME_FUNC="$2${ORIG_FUNC#$1}"
    eval "$NEWNAME_FUNC"
}

Add that to your scripts, and you have a simple way to copy/rename a function, and a simple way to add a step before/after an existing function. To copy the python example above:

save_function foo old_foo
foo() {
    initialization_code()
    old_foo()
    cleanup_code()
}

Now any code calling foo in the script will get the new behavior.

Bash quoting and whitespace

Sat, 18 Apr 2009 00:00:00

A common thing when writing shell scripts is to allow the user to specify options to commands in a variable. Something like the following:

$ OPTS="--some-option --some-other-option"
$ my_command $OPTS

We can set my_command to the following script to see exactly what gets passed:

#!/bin/bash
for t; do
    echo "'$t'"
done

Running the above prints the following output:

'--some-option'
'--some-other-option'

This works fine, until you want to include options with whitespace in them:

$ OPTS="--libs='-L/usr/lib -L/usr/local/lib'"
$ my_command $OPTS
'--libs='-L/usr/lib'
'-L/usr/local/lib''

This output clearly isn't what we want. We want a single parameter passed with the entire content of $OPTS. The culprit here is Word Splitting. Bash will split the value of $OPTS into individual parameters based on whitespace. One way to get around this is to put $OPTS in double quotes:

$ OPTS="--libs='-L/usr/lib -L/usr/local/lib'"
$ my_command "$OPTS"
'--libs='-L/usr/lib -L/usr/local/lib''

$ OPTS="--libs=-L/usr/lib -L/usr/local/lib"
$ my_command "$OPTS"
'--libs=-L/usr/lib -L/usr/local/lib'

Putting $OPTS in double quotes suppresses word expansion. In the above example, that works as expected. The second command has the single quotes removed as they were passed directly to the command, which isn't what we wanted. So far, so good. The problem, as you may have spotted by the removal of the single quotes, comes when we want to pass more than one parameter in $OPTS:

$ OPTS="--cflags=O3 --libs=-L/usr/lib -L/usr/local/lib"
$ my_command "$OPTS"
'--cflags=O3 --libs=-L/usr/lib -L/usr/local/lib'

Here, the entire $OPTS variable gets passed as a single parameter, which isn't what we want. We want --cflags to be passed as one parameter, and --libs (and everything that comes with it) to be passed as another parameter. Adding more quotes, backslash escaped or not, does nothing to help.

The solution? Use bash arrays:

$ OPTS=("--cflags=O3" "--LIBS=-L/usr/lib -L/usr/local/lib")
$ my_command "${OPTS[@]}"
'--cflags=O3'
'--LIBS=-L/usr/lib -L/usr/local/lib'

Perfect. But what about backward compatibility? If you have hundreds of scripts that use a string for $OPTS, how does it work if you change to using arrays? Let's try it out:

$ OPTS="--some-option --some-other-option"
$ my_command "${OPTS[@]}"
'--some-option --some-other-option'

So it works if your old scripts only have single options, but if multiple scripts are needed, then they will need to be changed to use arrays instead. This however seems to be the best option for passing multiple arguments with whitespace.