Setting Up a Package Mirror on FreeBSD

Welcome back! Today we’ll look briefly into how to setup a pkg(8) mirror on FreeBSD. Essentially, we’ll only need a bunch of binary packages for the mirror, pkg-create(8), a running HTTP server and a configuration file for the new mirror on your target system. You must know how to build packages for your target system — if that is the case you are ready to dive in.

Assuming you have those prebuilt packages lying around in /tmp/packages, the following script will create a layout that is reminiscent of the upstream FreeBSD package mirror. I’ll explain why that’s important in a bit. First the script with a couple of annotations:

#!/bin/sh

PACKAGESDIR=/tmp/packages
STAGEDIR=/tmp/stage
SETSDIR=/tmp/sets

mkdir -p ${SETSDIR}
rm -f ${SETSDIR}/packages-*.tar

rm -rf ${STAGEDIR}
mkdir -p ${STAGEDIR}

# rebuild expected FreeBSD structure
mkdir -p ${STAGEDIR}/Latest
mkdir -p ${STAGEDIR}/All

# push packages to home location
cp ${PACKAGESDIR}/* ${STAGEDIR}/All

# needed bootstrap glue when no packages are on the system
cd ${STAGEDIR}/Latest && ln -s ../All/pkg-*.txz pkg.txz

# generate index files
cd ${STAGEDIR} && pkg repo .

tar -cf ${SETSDIR}/packages-`date '+%Y%m%d'`.tar *

Unfortunately, the bootstrap binary pkg-static(8) expects the mirror to point to a file Latest/pkg.txz, but pkg-create(8) itself only creates the index files digests.txz, packagesite.txz and meta.txz (new since version 1.3) for a given directory and its subdirectories. That also means you can shuffle your package mirror around and use custom layouts — it doesn’t matter where the packages are as long as there is a link to the current pkg package for bootstrapping. If you don’t provide a pkg package, you’ll either have a complimentary mirror with extra packages for FreeBSD, or you won’t be able to operate your package mirror replacement. ;)

The archive itself is not compressed since all the files already use a good compression algorithm. It would only waste time to try to compress it again. On the mirror, unpack the archive into an exposed HTTP directory. Your mirror is already fully set up!

Last stage is to add repository awareness to your target system. This is done by setting up a simple config file, either under /etc/pkg (assuming it’s your base system) or /usr/local/etc/pkg/repos (package-shipping-friendly). The file name itself doesn’t really matter, but good style is your repo name followed by the idiomatic dot-conf:

cat > /usr/local/etc/pkg/repos/my_repo.conf <<EOF
my_repo: {
  url: "pkg+http://pkg.doe.it/${ABI}/latest",
  mirror_type: "srv",
  enabled: yes
}
EOF

It’s important to set the repository name in the file, because the file name itself is not authoritative. If you are trying to replace the stock mirror, either set enabled to no in /etc/pkg/FreeBSD.conf or delete the file. ${ABI} and latest aren’t really necessary in the URL: you’ll want ${ABI} when you need to support different architectures on the same mirror, and you’ll want latest if you have another type of packages that denote a more conservative update behaviour (or you simply wish to follow the current FreeBSD style). Generally, these are just names you may choose to use even for alternate meanings in your domain.

An invoke of pkg-update(8) will reconfigure your system to adjust to your repository settings. pkg-install(8) and friends will then be able to manage packages from your mirror. One use case idea is shipping a package that embeds a custom mirror for external plugins so that FreeBSD doesn’t have to worry about delivering them — they are most likely not essential but still nice to have for the user. Have some fun on your own. See you again soon.

From here on out things get a little funky with regard to package mirror priority in case (seemingly identical) packages are found on multiple mirrors. It is especially bad when you have different options for these same packages. Upstream has just released 1.4.3 to address two issues: priorities for repo configs and a CONSERVATIVE_UPGRADE switch that retains packages from the same mirror even when other mirrors have newer versions. Unfortunately, this also has security implications like hijacking a package… There are ways to only talk to a single mirror (option “-r”), but having a consistent mirror (or multiple complementary mirrors) is probably the most sensible course at this point.

5 thoughts on “Setting Up a Package Mirror on FreeBSD

  1. Franco Post author

    Hi Daniel, good catch. The code is copied from a larger base so I had to make a few simplifications and zapped too much it seems. Fixed up the article. Thanks!

  2. darkfader

    Hi!

    Sorry to ask questions when you already laid out such a nice article… but:
    One thing I just can’t figure:

    I’ve a build jail for old 9.1, say another for 9.2 and one for 9.3 and when it comes out I would roll one for 9.4.
    (I think 9.1 and 9.2 are dead already, they’re here for sake of example)

    $ABI will resolve to the very same for all of them, yet a build of i.e. bash in a 9.3 jail will produce a different binary than the one in the 9.4 jail.

    I’d like to lay out my network pkg server as nicely as possible but right now I don’t see how…
    I know there’s also going to be a 9.3-compat package once 9.4 is out. Is that the answer?

    Would appreciate your thoughts, especially if I’m just not groking the thing and there’s nothing to worry :)

  3. Franco Post author

    Hi there,

    well, every build will produce different binaries even on the same 9.x. The whole idea of STABLE is that the ABI doesn’t change, so that you can use all binaries on all of these versions interchangeably, hence ${ABI} is the same.

    You want to, however, always build your packages against the latest STABLE you can get and use them for all earlier versions as well, so in your case use 9.4 exclusively for ports builds and use them on older 9.x versions.

    For dynamic linking 9.x doesn’t matter, but if you have a port that statically links against a vulnerable library it’ll inherit this vulnerability. The latest STABLE is always your best bet for ports builds.

    Hope that helps! :)

    Cheers,
    Franco

  4. darkfader

    Thanks a lot!

    This makes things easier. I’m only doing FreeBSD on the side and had forgotten some basics :)
    I’ll set up one more build running on latest STABLE, makes sense.

    And your script is stored now, will check it out, probably only for the custom packages, since I just finished moving away from tinderbox to poudriere and don’t wanna replace everything once more.

    Greets
    Florian

Leave a Reply

Your email address will not be published. Required fields are marked *