FreeBSD: Your own package repository with poudriere

While using the extremely powerful and flexible ports collection is the traditional BSD
way of installing software, fetching binary packages is also an option. Binary packages
are just precompiled ports. Most people switching from other operating systems aren’t
used to using ports, and binary packages can make the upgrade to BSD a bit more
comfortable for them at first.

Poudriere is a BSD-licensed utility for creating and testing FreeBSD packages. It uses
FreeBSD jails to set up isolated compilation environments. These jails can be used to
build packages for versions of FreeBSD that are different from the system on which it
is installed, and also to build packages for i386 if the host is an amd64 system. Once
the packages are built, they are in a layout identical to the official mirrors. These
packages are usable by pkg(8) and other package management tools.
(https://www.freebsd.org/doc/handbook/ports-poudriere.html)

Also, to use ports and packages seems a bit broken. How to finde out which application
was installed via ports and which by package is tricky and not really possible.

In the past I was only using Ports and als also src for the base system to install application
and/or update applications and the base system. But nowadays its state of the art to
use packages and binary upgrades for the base system. It is much more comfortable and a lot
faster.

But when using packages (pkg) for applications on a FreeBSD system you often need to set
other options for an application/packages which will end up using ports.
But this will take extra time and also mix up ports and packages.

For example: If you can install the nginx webserver as a package, but if you want to use
the SPDY option or IMAP Proxy you end up using the ports to set the option.

So, why not use poudriere on a system to have your own FreeBSD package repository with
individual configuration of the applications/packages.

Here is a short description how you can setup poudriere as a local package repository
for your FreeBSD Systems (even Jails).

First of all, install poudriere on your FreeBSD System:

pkg install poudriere

Configure poudriere application.
Those are the parameters I have changed in /usr/local/etc/poudriere.conf

ZPOOL=zroot
ZROOTFS=/poudriere
FREEBSD_HOST=http://ftp.ch.freebsd.org/
RESOLV_CONF=/etc/resolv.conf
BASEFS=/poudriere/base
POUDRIERE_DATA=${BASEFS}/data
USE_PORTLINT=no
DISTFILES_CACHE=/usr/ports/distfiles
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
WRKDIR_ARCHIVE_FORMAT=txz
NOLINUX=yes
URL_BASE=http://pkg.example.de/poudriere/
BUILDER_HOSTNAME=pkg.example.de
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/pkg.example.de.key

When we build packages with poudriere, we want to be able to sign them with a private key.
This will ensure all of our machines that the packages created are legitimate and that
nobody is intercepting the connection to the build machine to serve malicious packages.

mkdir /usr/local/etc/ssl/keys
mkdir /usr/local/etc/ssl/certs
cd /usr/local/etc/ssl
openssl genrsa -out keys/pkg.example.de.key 4096
openssl rsa -in keys/pkg.example.de.key -pubout > certs/pkg.example.de.cert

Configure the package repository in /etc/pkg/FreeBSD.conf

example: { 
url : "http://pkg.example.de/",
mirror_type : "HTTP",
    enabled         : yes
    signature_type  : "PUBKEY",
    pubkey: "/usr/local/etc/ssl/certs/pkg.example.de.cert",
  }

Now we install the ports for poudriere.

poudriere ports -c

Time to create the poudriere jail for the target version of FreeBSD. FreeBSD 10.1 is
the latest release, so I will use this. Also I use the official notation.

poudriere jail -c -j freebsd:10:x86:64 -v 10.1-RELEASE -a amd64

Some global make options for the packages we create with poudriere ini /usr/local/etc/poudriere.d/freebsd:10:x86:64-make.conf

WITH_PKGNG="yes"
WITHOUT_X11="yes"
WITHOUT_HAL="yes"
WITHOUT_NLS="yes"

Now its time to specify which ports we want to build (pkglist) as a package with poudriere.
We can add more later, but imho, poudriere is a system for a standard environment where
you do not change that much in the poudriere pkglist: /usr/local/etc/poudriere.freebsd:10:x86:64.pkglist

For example:

dns/bind99
sysutils/cpdup
dns/dnstop
security/easy-rsa
security/expiretable
www/lynx
sysutils/monit
net-mgmt/nfdump
net-mgmt/nfsen
www/nginx
net/openntpd
security/openvpn
sysutils/pftop
ports-mgmt/pkg
ports-mgmt/poudriere
ports-mgmt/portmaster
sysutils/tmux
shells/zsh
mail/postfix
mail/dovecot2
mail/roundcube
www/nginx
www/spawn-fcgi
security/clamav
security/clamsmtp
databases/postgresql94-server
mail/postfixadmin
mail/spamassassin

As I mentioned before, you can set your individual options for your ports/packages.
Yes, this is what you want to do. If not, there is no need to use poudriere.
Setting the option is a one-off todo.

poudriere options -f /usr/local/etc/poudriere.freebsd:10:x86:64.pkglist -j freebsd:10:x86:64

After setting the options, building the packages will be the next step.

poudriere bulk -f /usr/local/etc/poudriere.freebsd:10:x86:64.pkglist -j freebsd:10:x86:64

There is a simple web frontend for poudriere which can be started without running
a websever like nginx.

root@example:/poudriere/base/data/logs/bulk/freebsd:10:x86:64-default # python2.7 -m SimpleHTTPServer 8080

To setup nginx do the following:

server {
        listen       8080;
        server_name  pkg.example.de;
        access_log /var/log/poudriere.log;

        location / {
            root /poudriere/base/data/logs/bulk/freebsd:10:x86:64-default;
            index  index.html index.htm;
            autoindex on;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;
        }
    }

server {
        listen       80;
        server_name  pkg.example.de;
        access_log /var/log/poudriere.log;

        location / {
            root /poudriere/base/data/packages/freebsd:10:x86:64-default;
            index  index.html index.htm;
            autoindex on;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;
        }
    }

For automatic fetching ports and building your packages:

## poudriere update script
#!/bin/sh

JAILNAME=$1
PORTSTREE=$2

POUDRIERE_DIR="/usr/local/etc"

PATH="$PATH:/usr/local/bin"

if [ -z "$JAILNAME" ]; then
    printf "Provide a jail name please.n"
    exit 1
fi

## check that there's a list of packages for that jail
if [ ! -f "$POUDRIERE_DIR/poudriere.$JAILNAME.pkglist" ]; then
    printf "No such file (list of packages: $POUDRIERE_DIR$PORTSTREE)n"
    exit 3
fi

# check that the jail is there
poudriere jails -l | grep "^$JAILNAME" > /dev/null
if [ $? -gt 0 ]; then
    printf "No such jail ($JAILNAME)n"
    exit 4
fi

Activate cronjob

30      1       *       *       *       root    /usr/local/etc/poudriere.update.sh freebsd:10:x86:64
Posted at 22.59PM on 03.15.15 | Add a comment | Filed under: FreeBSD

FreeBSD Upgrade Jail Base

# freebsd-update -b /path/to/jail fetch install
Posted at 20.21PM on 03.11.15 | Add a comment | Filed under: FreeBSD

FreeBSD + ZFS + Jails

FreeBSD+ZFS+Jails

## ZFS dataset for jail

#zfs create -o compress=lz4 zroot/jails/base101x64
#cd /var/tmp

## Fetch dist files

#fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/10.1-RELEASE/base.txz 
#fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/10.1-RELEASE/src.txz
#fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/10.1-RELEASE/lib32.txz 
#fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/10.1-RELEASE/games.txz

## untar distfiles to dataset

#tar -JxvC /jails/base101x64/ -f base.txz
#tar -JxvC /jails/base101x64/ -f src.txz
#tar -JxvC /jails/base101x64/ -f lib32.txz
#tar -JxvC /jails/base101x64/ -f games.txz

##Update Jail to patchlevel

#freebsd-update -b /jails/base101x64 fetch install

##config jail

#cp /etc/resolv.conf /jails/base101x64/etc/

#vi /jails/base101×64/etc/rc.conf

rpcbind_enable="NO"
font8x8="iso15-8x8"
font8x14="iso15-8x14"
font8x16="iso15-8x16"
keymap="german.iso"
sendmail_enable="NONE"
clear_tmp_enable="YES"
syslogd_flags="-ss"
sshd_enable="YES"

##chroot and config jail

#chroot /jails/base101x64
#newaliases
#passwd root
#pkg 
#exit

##Snapshot of jail

#zfs snapshot zroot/jails/base101x64@p6

##show all datasets

#zfs list -t all
...
zroot/jails/base101x64@p6                         0      -   770M  -
...
#cd /jails/base101x64/.zfs/snapshot
#ls -l
total 9
drwxr-xr-x  17 root  wheel  21 Nov 11 22:03 p6

##Clone des Snapshots for new jail
##Clones are related to snapshots
##you can not delete a snapshot if there is still a clonse based
##on the snapshots

#zfs clone zroot/jails/bas101x64@p6 zroot/jails/j_new

##Copy ZFS Datasets
##Not related to snapshot
##you can do this also copying the jail over the net with ssh

#zfs send -v zroot/jails/base101x64@p6| zfs receive zroot/jails/j_new
...
zroot/jails/j_new                        1.42G   157G  1.42G  /jails/j_new
...
Posted at 20.12PM on 03.11.15 | Add a comment | Filed under: FreeBSD