As anyone who has tried this will no doubt have found, there is a large amount of conflicting information out there as to how exactly this can be achieved. To ease the burdens of my fellow sysadmins, I thought I’d gather all the necessary info together into one easy post.

The first thing we’re going to need to do is to generate a GPG key to sign our repository and all of our packages with. If you’ve already done this, please skip this section. If not, you can follow these simple steps!

Creating a GPG Key for Signing Stuff

• Run the following command: gpg --gen-key
• Select Option 5 to generate an RSA Key
• Choose your keysize from the options given (doesn’t really matter for our purposes
• Enter your name and email address when prompted

There, we now have a GPG key we’re going to use to sign everything. The next stage is to produce a ASCII formatted public key file which we can distribute to people so they’ll be able to tell apt to trust our repository.

Creating a ASCII Formatted Public Key File

By default, the gpg utility’s export function produces a binary formatted output of our public key. If you’re planning to distribute your public key via the web, this isn’t very handy so we’re going to use the gpg utility’s --armor option to produce an ASCII formatted version.

You’ll want to substitute the email address of the key you’re exporting, and the output filename as appropriate in the following command.

• gpg --armor --export jon@aframe.com --output jon@aframe.com.gpg.key

Save this keyfile somewhere, we’ll be making it available over the web for people to add to their apt keychains – this is what’ll say that our repository is trusted for package installs.

Signing Some .deb Packages

I’m going to assume that you already know how to create .deb packages, by way of keeping this blogpost short. This section will simply cover signing a package you’re creating and resigning an existing package.

The good news is that if you’ve already generated a GPG key as detailed above, your packages will be automatically signed as long as the name and email address in your package’s changelog file are the same as that of the GPG key you created. This means that simply running dpkg-buildpackage will now give you signed packages. If you’re unsure of how to build debian packages at all, there’s plenty of information out there on doing this. I might write a blog post on that soon

If you want to resign an existing debian package, for example if you’re setting up your own backport of a package (as with my usecase, backporting Chef 0.9.16 into debian), then this is very easy too if you already have a GPG key set up. We use a tool called dpkg-sig.

Install the dpkg-sig tool by running the command

Then run the following command to sign the package, substituting the name of the deb package as appropriate.

The “builder” name mentioned is a debian convention, indicating that the builder of the package signed it. The GPG key used will be the one you set up above, providing you’re running the command as the same user you set up the key with.

Creating a Signed APT Repository

OK, so you’ve got a bunch of signed packages, but now you need some way to make them available to the world. Preferably one which doesn’t throw up authentication warnings every time they try to install from your repository. What you need is a signed repository!

The first step to creating your repository is to create a base directory for it to live in. Doesn’t matter where, any directory will do. Now inside that directory, create a folder called conf.

Inside your conf folder, you’re going to create a text file called distributions. Below is the one I’ve used for my repository:

Origin: apt.aframe.com
Label: apt repository
Codename: lenny
Architectures: amd64 source
Components: main
Description: Aframe debian package repo
SignWith: yes
Pull: lenny

Some of the above will be self explanatory, for example if you’re not packaging for debian lenny, you’d replace all occurrences of lenny with squeeze, for example. Additionally, if you’re going to package for more than one distribution, you’ll want to copy the above section for each distro. It all stays in the same file, you just change the “lenny” parts as applicable.

Also, since I’m only creating 64-bit packages, I’ve said that my repository will only contain the amd64 and source architectures. If you’re packaging for i386 or other architectures, remember to add them!

The important line in the above example is the one that says SignWith: yes. That line states that we want to sign the repository with our GPG key. Again, if you’re running commands as the same user you used to create your GPG key, this should work fine.

Now that we have the descriptions file all ready to go, we’re going to use a tool called reprepro to add our packages to the repository. You can install reprepro by running the following command:

The nice thing about this tool is that it will create all the structure it needs inside the repository automatically so you don’t need to worry about it. Here’s an example of how to add a package to our repository. PLEASE NOTE you have to run the below command from your repository directory, ie the directory containing the conf folder.

So what does this command mean? Well, the --ask-passphrase part tells it to prompt you for the passphrase for your GPG Key. You did set one, right? The -Vb . includedeb lenny part tells the command to be verbose, and set the base directory for the command as the current directory. It then says we’re going to be passing the command a deb file (that’s the includedeb part) and then says that we’re going to be adding it to the lenny distribution in our repository. The last part is the absolute path to the .deb package we’re adding.

When you run this command, you’ll see a bunch of output telling you that various folders have been created, and once you’ve entered the password for your GPG key your package will be tucked away nicely in your properly structured, properly signed debian repository. Running the above command on a further package or packages will add the new package(s) into the existing structure. You’ll probably now notice that your repository directory is structured much more like the official debian repositories – that’s because it’s now structured “The Debian Way”.

Making your Repository Available to the World

The final section in this blog post is how to make your wonderful new repository available to the rest of the world. This comes in two parts. The first is making your repository directory available over the web. I’m going to assume if you’re creating packages you can probably do this part by yourself, so I’m going to skip over it

The second part is making our public GPG key available to the world so they can mark our repository as trusted. I’d suggest keeping the public GPG key we created above in the root of your repository, to make it easy for people to find. Just make sure you only store the *public* part of the key there. That’s the part we created using the gpg --armor --export command.

I’d also recommend publishing a simple command for people to download your public key and import it into their apt keychain in one step – this makes it nice and easy for them to get up and running with your repo. This command is as follows (change the URL to your key as appropriate):

wget -O - http://apt.aframe.com/jon@aframe.com.gpg.key | sudo apt-key add -

Once your users have run the above command, they can add your nicely-formatted repo to their /etc/apt/sources.list file by adding the following line (change URL and distribution as appropriate):

deb http://apt.aframe.com/ lenny main

Then they just run apt-get update and they’re all ready to use your repository containing all of your signed packages – and not an authentication warning in sight.


OUTPUTFILE="/tmp/mailman_report"
CURRMONTH=`date +%m-%Y`
LISTS=`/usr/local/mailman/bin/list_lists | awk '{print $1}' | grep -v [!0-9]`
rm ${OUTPUTFILE}
echo "Mailman Report for ${CURRMONTH}" > ${OUTPUTFILE}
echo >> ${OUTPUTFILE}
for x in ${LISTS}
do
echo "Members of List ${x}:" >> ${OUTPUTFILE}
LIST_MEMBERS=`/usr/local/mailman/bin/list_members ${x}`
for mems in ${LIST_MEMBERS}
do
echo ${mems} >> ${OUTPUTFILE}
done
echo >> ${OUTPUTFILE}
done
/bin/mail -s "Mailman_Report_for_${CURRMONTH}" foo@foo.com -c blah@blah.com < ${OUTPUTFILE}


I knew that filesystems designed for exactly this sort of sharing were available but hadn’t done much with them but after investigating for a bit, it seemed like the Oracle Clustered File System (linky) was the best option as it was already supported by the Linux kernel and was pretty mature code. The main problem I had in setting all of this up was that the available documentation was very much geared towards people who already had in-depth experience of OCFS whereas I’d never used it before. Hence this blog post, which details setting up iSCSI and then configuring both servers to talk to the same OCFS partition. The instructions are written for Ubuntu Server, but will work on any distro which used apt. Packages are also available for rpm distros, the only instructions you need to change are the package fetching ones.

Setting up iSCSI

* Install Open-iSCSI

apt-get install open-iscsi

* Edit the Open-iSCSI configuration file

The default configuration file could be located at /etc/openiscsi/iscsid.conf or ~/.iscsid.conf. Open the file and set the parameters as required by your iSCSI device. I’ve included the (mostly default) options I used for reference

node.startup = automatic
node.session.timeo.replacement_timeout = 120
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.noop_out_interval = 10
node.conn[0].timeo.noop_out_timeout = 15
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 65536

* Save and close the file. Restart the open-iscsi service:

/etc/init.d/open-iscsi restart

Now you need to run a discovery against the iscsi target host which basically finds all the iSCSI targets the SAN can give us:

iscsiadm -m discovery -t sendtargets -p ISCSI-SERVER-IP-ADDRESS

Finally restart the service again:

/etc/init.d/open-iscsi restart

Now you should see an additional drive on the system such as /dev/sdc. Look in the /var/log/messages file to find out device name:

Next, you need to use fdisk to create a blank partition on the device. This is pretty well documented so I’ll skip these steps, other than to say that I’ll assume the device was called /dev/sdc, and the new blank partition is called /dev/sdc1 for the remainder of this post. So now we’re talking to our iSCSI device and we’ve got a blank partition all ready to format as an OCFS drive. Next, how exactly we do that!

To be continued…

Run the following commands:

# This gets and builds a patched version of Apple's diskdev_cmds package which will work on Linux
wget http://www.mythic-beasts.com/resources/appletv/mb_boot_tv/diskdev_cmds-332.14.tar.gz
wget http://www.ecl.udel.edu/~mcgee/diskdev_cmds/diskdev_cmds-332.14.patch.bz2
tar xzf diskdev_cmds-332.14.tar.gz
bunzip2 -c diskdev_cmds-332.14.patch.bz2 | patch -p0
cd diskdev_cmds-332.14
make -f Makefile.lnx

# Create symlinks to the mkfs and fsck commands for HFS+
sudo cp newfs_hfs.tproj/newfs_hfs /sbin/mkfs.hfsplus
sudo cp fsck_hfs.tproj/fsck_hfs /sbin/fsck.hfsplus

# Get and enable the hfsplus kernel module
sudo apt-get install hfsplus
sudo modprobe hfsplus

Now that's done, you can use the following handy bash script (must be run as root) I've written to create a DMG file which contains the contents of a directory you specify on the command line.

#!/bin/bash

# DMG Creation Script
# Usage: makedmg <imagename> <imagetitle> <imagesize (MB)> <contentdir>
#
# imagename: The output file name of the image, ie foo.dmg
# imagetitle: The title of the DMG File as displayed in OS X
# imagesize: The size of the DMG you're creating in MB (Blame Linux for the fixed size limitation!!)
# contentdir: The directory containing the content you want the DMG file to contain
#
# Example: makedmg foo.dmg "Script Test" 50 /home/jon/work/scripts/content
#
# Author: Jon Cowie
# Creation Date: 02/04/2008

if [ ! $# == 4 ]; then
	echo "Usage: makedmg <imagename> <imagetitle> <imagesize (MB)> <contentdir>"
else
	OUTPUT=$1
	TITLE=$2
	FILESIZE=$3
	CONTENTDIR=$4
	USER=`whoami`
	TMPDIR="/tmp/dmgdir"

	if [ ${USER} != "root" ]; then
		echo "makedmg must be run as root!"
	else
		echo "Creating DMG File..."
		dd if=/dev/zero of=${OUTPUT} bs=1M count=$FILESIZE
		mkfs.hfsplus -v "${TITLE}" ${OUTPUT}

		echo "Mounting DMG File..."
		mkdir -p ${TMPDIR}
		mount -t hfsplus -o loop ${OUTPUT} ${TMPDIR} 

		echo "Copying content to DMG File..."
		cp -R ${CONTENTDIR}/* ${TMPDIR}

		echo "Unmounting DMG File..."
		umount ${TMPDIR}
		rm -rf ${TMPDIR}

		echo "All Done!"
	fi
fi

Hope it’s useful!

Ee Link

Being a rather geeky type, I’m quite excited to see what all these clever hardware types come up with in the next few years…wouldn’t it be nice to be able to buy a server full of VM aware kit which lets you run multiple VMs as quickly as if you were using just one host OS? I can’t see virtualisation going away any time soon, it’s just too damn useful – so I reckon this might be something just around the corner.

And now for some stuff I like…