Setting Up a Package Mirror on FreeBSD

Welcome back! Today we’ll look briefly into how to setup a pkg(8) mirror on FreeBSD. Essentially, we’ll only need a bunch of binary packages for the mirror, pkg-create(8), a running HTTP server and a configuration file for the new mirror on your target system. You must know how to build packages for your target system — if that is the case you are ready to dive in.

Assuming you have those prebuilt packages lying around in /tmp/packages, the following script will create a layout that is reminiscent of the upstream FreeBSD package mirror. I’ll explain why that’s important in a bit. First the script with a couple of annotations:

#!/bin/sh

PACKAGESDIR=/tmp/packages
STAGEDIR=/tmp/stage
SETSDIR=/tmp/sets

mkdir -p ${SETSDIR}
rm -f ${SETSDIR}/packages-*.tar

rm -rf ${STAGEDIR}
mkdir -p ${STAGEDIR}

# rebuild expected FreeBSD structure
mkdir -p ${STAGEDIR}/Latest
mkdir -p ${STAGEDIR}/All

# push packages to home location
cp ${PACKAGESDIR}/* ${STAGEDIR}/All

# needed bootstrap glue when no packages are on the system
cd ${STAGEDIR}/Latest && ln -s ../All/pkg-*.txz pkg.txz

# generate index files
cd ${STAGEDIR} && pkg repo .

tar -cf ${SETSDIR}/packages-`date '+%Y%m%d'`.tar *

Unfortunately, the bootstrap binary pkg-static(8) expects the mirror to point to a file Latest/pkg.txz, but pkg-create(8) itself only creates the index files digests.txz, packagesite.txz and meta.txz (new since version 1.3) for a given directory and its subdirectories. That also means you can shuffle your package mirror around and use custom layouts — it doesn’t matter where the packages are as long as there is a link to the current pkg package for bootstrapping. If you don’t provide a pkg package, you’ll either have a complimentary mirror with extra packages for FreeBSD, or you won’t be able to operate your package mirror replacement. ;)

The archive itself is not compressed since all the files already use a good compression algorithm. It would only waste time to try to compress it again. On the mirror, unpack the archive into an exposed HTTP directory. Your mirror is already fully set up!

Last stage is to add repository awareness to your target system. This is done by setting up a simple config file, either under /etc/pkg (assuming it’s your base system) or /usr/local/etc/pkg/repos (package-shipping-friendly). The file name itself doesn’t really matter, but good style is your repo name followed by the idiomatic dot-conf:

cat > /usr/local/etc/pkg/repos/my_repo.conf <<EOF
my_repo: {
  url: "pkg+http://pkg.doe.it/${ABI}/latest",
  mirror_type: "srv",
  enabled: yes
}
EOF

It’s important to set the repository name in the file, because the file name itself is not authoritative. If you are trying to replace the stock mirror, either set enabled to no in /etc/pkg/FreeBSD.conf or delete the file. ${ABI} and latest aren’t really necessary in the URL: you’ll want ${ABI} when you need to support different architectures on the same mirror, and you’ll want latest if you have another type of packages that denote a more conservative update behaviour (or you simply wish to follow the current FreeBSD style). Generally, these are just names you may choose to use even for alternate meanings in your domain.

An invoke of pkg-update(8) will reconfigure your system to adjust to your repository settings. pkg-install(8) and friends will then be able to manage packages from your mirror. One use case idea is shipping a package that embeds a custom mirror for external plugins so that FreeBSD doesn’t have to worry about delivering them — they are most likely not essential but still nice to have for the user. Have some fun on your own. See you again soon.

From here on out things get a little funky with regard to package mirror priority in case (seemingly identical) packages are found on multiple mirrors. It is especially bad when you have different options for these same packages. Upstream has just released 1.4.3 to address two issues: priorities for repo configs and a CONSERVATIVE_UPGRADE switch that retains packages from the same mirror even when other mirrors have newer versions. Unfortunately, this also has security implications like hijacking a package… There are ways to only talk to a single mirror (option “-r”), but having a consistent mirror (or multiple complementary mirrors) is probably the most sensible course at this point.