From: Ingo Schwarze (schwarzeusta.de)
Date: Wed Nov 01 2006 - 17:36:12 CST

• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Nearly a year ago, Christian Weisgerber considered integrating
sysutils/mergemaster into the base system [1], but postponed it
because he intended to evaluate mergeslave first [2].

I should like to suggest the solution implemented by the patch
below. The basic idea is taken from mergeslave [3]. That is,
diff the old and new default configurations and apply the
resulting patch to the root file system. See the man page
and the source included below for details.

For testing the patch, i set up 3.9-stable, 4.0-release and
4.0-current source trees, applied the patch to each of them,
made release for each of them, installed the resulting 3.9-stable
on a i386 test box, changed various configuration files,
upgraded to 4.0-release, and upgraded to 4.0-current.

In case you like the idea, i am interested in expanding this to
handle configuration files from the ports tree, such that the
upgrade workflow will become:

 - reboot some bsd.rd and run ./upgrade
 - reboot the new /bsd
 - pkg_add -u (copying its default config files
               to /var/etc-dist/running, too)
 - upgrade_etc
 - check /var/etc-dist/todo
 - upgrade_etc

But let's do one step at a time and start with the base system.

The code is short on purpose. My design goals were simplicity
and efficiency. Any feedback is welcome.

The patch below is also available from
http://www.studis.de/Software/upgrade_etc.patch

References:
[1] http://marc.theaimsgroup.com/?l=openbsd-misc&m=113572636605434&w=2
[2] http://marc.theaimsgroup.com/?l=openbsd-misc&m=114010382708699&w=2
[3] http://www.xs4all.nl/~hanb/software/OpenBSD-binary-upgrade/

 ----- 8< ----- schnipp ----- >8 ----- 8< ----- schnapp ----- >8 -----

Index: distrib/miniroot/install.sub
===================================================================
RCS file: /cvs/src/distrib/miniroot/install.sub,v
retrieving revision 1.406
diff -u -r1.406 install.sub
--- distrib/miniroot/install.sub 29 Aug 2006 01:02:49 -0000 1.406
+++ distrib/miniroot/install.sub 1 Nov 2006 21:32:44 -0000
-971,6 +971,17
                         DEFAULTSETS=$(rmel $_f $DEFAULTSETS)
                 fi
         done
+
+ if isin etc${VERSION}.tgz $_files ; then
+ echo "Saving a copy of the default configuration ..."
+ ftp $_ftp_active -o - -V -m "$_src/etc${VERSION}.tgz" \
+ | tar zxphf - -C /mnt/var/etc-dist/installing
+ fi
+ [[ -e /mnt/etc/X11 ]] && if isin xetc${VERSION}.tgz $_files ; then
+ echo "Saving a copy of the default X configuration ..."
+ ftp $_ftp_active -o - -V -m "$_src/xetc${VERSION}.tgz" \
+ | tar zxphf - -C /mnt/var/etc-dist/installing
+ fi
 }
 
 # Encode $1 as specified for usercodes and passwords in RFC 1738
-1357,6 +1368,18
         [[ -x /sbin/mount_nfs ]] && _locs="$_locs nfs"
         [[ -n $MTDEVS && -x /bin/mt ]] && _locs="$_locs tape"
 
+ # Save the old default configuration files.
+ # Usually, this will only happen in upgrade mode.
+ if [[ -e /mnt/var/etc-dist/running ]]; then
+ rm -rf /mnt/var/etc-dist/old /mnt/var/etc-dist/done
+ mv /mnt/var/etc-dist/running /mnt/var/etc-dist/old
+ fi
+
+ # Do not extract the default configuration directly to
+ # /mnt/var/etc-dist/running - otherwise, the old ones
+ # might be clobbered in case an upgrade is interrupted
+ mkdir -p /mnt/var/etc-dist/installing
+
         echo "\nLet's $MODE the sets!"
         while :; do
                 umount -f /mnt2 >/dev/null 2>&1
-1623,6 +1646,8
         populateusrlocal
 
         [ -x /mnt/$MODE.site ] && /mnt/usr/sbin/chroot /mnt /$MODE.site
+
+ mv /mnt/var/etc-dist/installing /mnt/var/etc-dist/running
 
         # Pat on the back.
         cat <<__EOT
Index: usr.sbin/upgrade/upgrade_etc
===================================================================
--- usr.sbin/upgrade/upgrade_etc Thu Jan 1 01:00:00 1970
+++ usr.sbin/upgrade/upgrade_etc Wed Nov 1 22:20:08 2006
-0,0 +1,119
+#! /bin/sh
+#
+# $OpenBSD$
+#
+# Copyright (c) 2006 Ingo Schwarze <schwarzeusta.de>
+# inspired by the mergeslave script by Han Boetes
+#
+# Permission to use, copy, modify, and distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+PATH=
+WORKDIR=/var/etc-dist
+
+if [[ -e ${WORKDIR}/done ]]; then
+ echo "${WORKDIR}/done exists"
+ exit 1
+fi
+
+if [[ ! -e ${WORKDIR}/todo ]]; then
+ /bin/mkdir ${WORKDIR}/todo
+ echo 'Finding new files ... '
+ cd ${WORKDIR}/running
+ /usr/bin/find . -exec /bin/test ! -e /{} \; \
+ -print -prune > ${WORKDIR}/todo/find.out
+ case $? in
+ 0) if [[ -s ${WORKDIR}/todo/find.out ]]; then
+ echo -n 'New files:'
+ /usr/bin/wc -l ${WORKDIR}/todo/find.out
+ rerun=1
+ else
+ echo 'No new files.'
+ rerun=0
+ fi
+ ;;
+ *) echo 'find FAILED'
+ exit 1
+ ;;
+ esac
+
+ echo 'Diffing old files ... '
+ cd ${WORKDIR}
+ /usr/bin/diff -ru old running > todo/diff.out
+ case $? in
+ 0) echo 'No files changed.'
+ ;;
+ 1) rerun=`/usr/bin/grep -c ^--- todo/diff.out`
+ echo "$rerun files changed."
+ ;;
+ *) echo 'diff FAILED'
+ exit 1
+ ;;
+ esac
+
+ if [[ $rerun -eq 0 ]]; then
+ /bin/mv todo done
+ else
+ echo "Check ${WORKDIR}/todo and rerun $0."
+ fi
+ exit 0
+fi
+
+TIMESTAMP=`/usr/bin/mktemp` && /usr/bin/touch ${TIMESTAMP} || exit 1
+/bin/sleep 1
+
+if [[ -s ${WORKDIR}/todo/diff.out ]]; then
+ echo 'Patching / ... '
+ cd ${WORKDIR}/todo
+ /usr/bin/patch -u -d / -p1 -b -z .old -s < diff.out 2> patch.err
+ case $? in
+ 0) echo 'Patching succeeded.'
+ ;;
+ 1) echo 'The following files could not be patched:'
+ /bin/cat patch.err
+ ;;
+ *) echo 'patch FAILED'
+ /bin/rm -f ${TIMESTAMP}
+ exit 1
+ ;;
+ esac
+fi
+
+while read filename; do
+ /bin/cp -pR "${WORKDIR}/running/$filename" "/$filename"
+ case $? in
+ 0) echo "Installed $filename"
+ ;;
+ *) echo "cp $filename FAILED"
+ ;;
+ esac
+done < ${WORKDIR}/todo/find.out
+
+/bin/mv ${WORKDIR}/todo ${WORKDIR}/done
+
+if [[ /etc/master.passwd -nt ${TIMESTAMP} ]]; then
+ echo 'Rebuilding password database ...'
+ /usr/sbin/pwd_mkdb -p /etc/master.passwd
+fi
+
+if [[ -e /etc/login.conf.db && /etc/login.conf -nt ${TIMESTAMP} ]]; then
+ echo 'Rebuilding login class capability database ...'
+ /usr/bin/cap_mkdb /etc/login.conf
+fi
+
+if [[ /etc/mail/aliases -nt ${TIMESTAMP} ]]; then
+ echo 'Rebuilding mail aliases database ...'
+ /usr/bin/newaliases
+fi
+
+/bin/rm -f ${TIMESTAMP}
+exit 0
Index: usr.sbin/upgrade/upgrade_etc.8
===================================================================
--- usr.sbin/upgrade/upgrade_etc.8 Thu Jan 1 01:00:00 1970
+++ usr.sbin/upgrade/upgrade_etc.8 Wed Nov 1 22:20:08 2006
-0,0 +1,176
+.\" $OpenBSD$
+.\"
+.\" Ingo Schwarze, 2006. Public Domain.
+.\"
+.Dd November 01, 2006
+.Dt UPGRADE_ETC 8
+.Os
+.Sh NAME
+.Nm upgrade_etc
+.Nd upgrade configuration files after upgrading the system
+.Sh SYNOPSIS
+.Nm upgrade_etc
+.Nm vi Pa /var/etc-dist/todo/find.out
+.Nm vi Pa /var/etc-dist/todo/diff.out
+.Nm upgrade_etc
+.Sh DESCRIPTION
+The
+.Ox
+installation procedure copies the default configuration files from the
+.Pa etcNN.tgz
+and optionally from the
+.Pa xetcNN.tgz
+file sets to the new root file system, and it saves an additional copy to
+the directory
+.Pa /var/etc-dist/running .
+The
+.Ox
+upgrade procedure does not change any configuration files in the root file
+system, but it renames
+.Pa /var/etc-dist/running
+to
+.Pa /var/etc-dist/old
+and saves a copy of the new default configuration files to a new directory
+.Pa /var/etc-dist/running .
+.Pp
+After completing the upgrade procedure and booting the upgraded system, run
+the
+.Nm
+utility
+.Em twice
+to upgrade the system configuration files.
+.Ss First upgrade_etc pass: creating diffs
+If
+.Pa /var/etc-dist/todo
+does not exist when
+.Nm
+is started, the program will compare the default configurations for the old
+and the running versions of the operating system. First, it will place a
+list of files and directories into the file
+.Pa /var/etc-dist/todo/find.out .
+This list will consist of the names of all the files and directories
+contained in the default configuration for the running version which are not
+yet installed in the root file system. Additionally,
+.Nm
+will place input for
+.Xr patch 1
+into the file
+.Pa /var/etc-dist/todo/diff.out .
+For each file contained in the default configuration for the running version
+that changed since the old version, this patch file will contain one or more
+hunks.
+.Pp
+After the first pass of
+.Nm ,
+you should read all the files in the directory
+.Pa /var/etc-dist/todo .
+In case you do not want to install one particular new configuration file,
+delete its name from the file
+.Pa find.out .
+In case you do not want to apply one particular hunk, delete it from the file
+.Pa diff.out .
+.Pp
+Deleting lines from the files in
+.Pa /var/etc-dist/todo
+may be useful, but manually adding lines to these files is usually
+unnecessary. In case you want to apply additional configuration updates,
+apply them manually after the second
+.Nm
+pass completed.
+.Pp
+Note that files you manually delete from
+.Pa find.out
+will be proposed again during the next upgrade, while patch hunks you
+manually delete from
+.Pa diff.out
+will never again be considered.
+.Ss Second upgrade_etc pass: applying diffs
+If
+.Pa /var/etc-dist/todo
+is present when
+.Nm
+is started, the program will apply the changes described therein. First,
+it will apply the
+.Xr patch 1
+input file
+.Pa /var/etc-dist/todo/diff.out
+to the root file system. Original files will be preserved by adding the
+suffix '.old' to their name. Any error messages will be saved to a file
+.Pa /var/etc-dist/todo/patch.err .
+After that,
+.Nm
+will copy the files named in
+.Pa /var/etc-dist/todo/find.out
+from the directory
+.Pa /var/etc-dist/running
+to the root file system. Finally, it will rename
+.Pa /var/etc-dist/todo
+to
+.Pa /var/etc-dist/done
+and rebuild configuration database files as required.
+.Pp
+After the second pass of
+.Nm ,
+you should check any rejects listed in the file
+.Pa /var/etc-dist/done/patch.err
+and manually repair any files that were incorrectly patched.
+.Ss Third upgrade_etc pass: problem recovery
+If
+.Pa /var/etc-dist/done
+is already present when
+.Nm
+is started, the program will not do anything, but just exit. Usually, this
+means that the upgrade had already been completed earlier.
+.Pp
+In case the execution of
+.Nm
+was interrupted for whatever reason, you can try to manually
+restore the contents of the directory
+.Pa /var/etc-dist
+to some useful state before restarting
+.Nm .
+This may involve renaming
+.Pa /var/etc-dist/done
+back to
+.Pa /var/etc-dist/todo .
+Of course, better avoid interrupting
+.Nm
+once you started the process.
+.Sh FILES
+In the following, all filenames are relative to the directory
+.Pa /var/etc-dist .
+.Pp
+.Bl -tag -width "done/patch.err" -compact
+.It installing
+Default configuration files for the operating system install or upgrade
+currently being in progress.
+.It running
+Default configuration files for the running version of the operating system.
+.It old
+Default configuration files for the old version of the operating system,
+before the previous upgrade.
+.It todo/find.out
+New configuration files to be installed.
+.It todo/diff.out
+Patches for the upgrade in progress.
+.It done
+Changes applied during the previous upgrade.
+.It done/patch.err
+Changes rejected during the previous upgrade.
+.El
+.\" .Sh DIAGNOSTICS
+.Sh SEE ALSO
+.Xr find 1 ,
+.Xr diff 1 ,
+.Xr vi 1 ,
+.Xr patch 1
+.Sh HISTORY
+The
+.Nm
+utility appeared in
+.Ox 4.0 .
+.Sh AUTHORS
+The
+.Nm
+utility was written by
+.An Ingo Schwarze Aq schwarzeusta.de .
Index: usr.sbin/upgrade/Makefile
===================================================================
--- usr.sbin/upgrade/Makefile Thu Jan 1 01:00:00 1970
+++ usr.sbin/upgrade/Makefile Wed Nov 1 22:23:09 2006
-0,0 +1,14
+# $OpenBSD$
+#
+# Ingo Schwarze, 2006.
+# This Makefile is in the Public Domain.
+
+.include <bsd.own.mk>
+
+MAN= upgrade_etc.8
+
+beforeinstall:
+ ${INSTALL} ${INSTALL_COPY} -o ${BINOWN} -g ${BINGRP} -m ${BINMODE} \
+ ${.CURDIR}/upgrade_etc ${DESTDIR}/usr/sbin/upgrade_etc
+
+.include <bsd.prog.mk>
Index: usr.sbin/Makefile
===================================================================
RCS file: /cvs/src/usr.sbin/Makefile,v
retrieving revision 1.129
diff -u -r1.129 Makefile
--- usr.sbin/Makefile 29 Oct 2006 18:06:07 -0000 1.129
+++ usr.sbin/Makefile 1 Nov 2006 21:32:44 -0000
-13,7 +13,8
         ripctl ripd \
         rmt rpc.bootparamd rpc.lockd rwhod sa sasyncd sensorsd sliplogin \
         slstats spamdb spray syslogc syslogd tcpdrop tcpdump \
- timed tokenadm tokeninit traceroute trpt usbdevs user vipw vnconfig \
+ timed tokenadm tokeninit traceroute trpt upgrade \
+ usbdevs user vipw vnconfig \
         watchdogd wsconscfg wsfontload wsmoused zdump zic ztsscale
 
 # IPv6

• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]