Proposal: NetBSD System Installation Packages ============================================= CONTENTS -------- 0. Introduction 1. System Packages 1.1 Package Format 1.2 Package Granularity 1.2.1 Root/User/Share separation 2. Package Sets 2.1 Set format 3. Creation of Packages and Sets 4. Modifications to the NetBSD installation process A. Working Plan ------------------------------------------------------------------------ 0. Introduction The current NetBSD installation process involves the downloading of binary `sets', which the user can choose among at install time. A set is a tarred, gzipped set of files, to be untarred relative to '/'. No facility exists to choose convenient subsets of the files in a set to be installed, or to remove a set which has been installed. The current granularity of sets is very large, being divided into: base -- general system binaries comp -- compilers and related tools etc -- system configuration files games -- games and other amusements man -- system manual pages misc -- items not falling into other categories secr -- items not exportable under US law text -- text processing tools xbase -- general X11R6 binaries xcomp -- X11R6 development items xcontrib - random binaries from the X11R6 `contrib' tree xfont -- X11R6 fonts xserver -- X11R6 servers for various video hardware Users who wish to install part of a set need to either install the full set and then determine which files they need to remove, or abandon the normal install process, and figure out which files to unpack by hand. Similarly, if a set is later determined to be unnecessary, the only way to remove it is to figure out which files on the system belonged to that set, and remove them by hand. When it comes time to upgrade a system which has been installed this way, the usual procedure is to unpack a new version of each installed set over the previous version. When a file is moved, renamed, or removed in a newer version of a set, the old version often remains on the system for some time. In at least one recent instance (the move of /sbin/mountd to /usr/sbin/mountd) this has resulted in much confusion, and large amounts of traffic on the relevant mailing lists. The remainder of this document describes a proposed method of handling these and other problems with the current install set system by moving to the use of fine-grained `system packages', based on the currently existing package system for third-party software, and allowing users to choose among either `package sets' at the same granularity as our current install sets, or individual `packages' at a much finer level of granularity. In either case, the new system would also greatly simplify upgrading or removal of such packages and sets at a later time, and would allow tracking of dependencies between the various sets and packages distributed as part of NetBSD. First, the format of system packages in the proposed system is discussed, followed by the format of package sets, which will serve as a replacement for the current install sets. The creation of packages in an automated fashion from a NetBSD source tree is discussed as is the effect of this system on the NetBSD installation process. An appendix discusses my work plan to implement this new system. It is hoped that this document will serve as a basis for discussion of what is involved in changing NetBSD to use system packages for system installation and upgrades, and that after several iterations of discussion and revision, it will serve as a plan for the actual implementation of this system. ------------------------------------------------------------------------ 1. System Packages System packages will be the basic building blocks of a NetBSD system. At install time, the user will choose which system packages to install, subject to dependencies between packages. After system install, users will be able to install additional packages or remove installed packages. When it comes time to upgrade the system, packages can be removed and reinstalled in a reliable fashion. All of this functionality is already available for third-party software via the use of the software package system in /usr/pkgsrc. This proposal extends that functionality to the NetBSD system itself. 1.1 Package Format System packages will be identical in format to the binary packages used by the current third-party package system. This will allow the same tools to be used for working with system packages as are currently used for working with third-party packages. This will also also allow the system to benefit from the fact that the workings of the current package system are well understood. 1.2 Package Granularity System packages will be at the granularity of groups of related tools and their support files. Thus, `Kerberos', `UUCP', `Text formatting' and `amd' might each be packages which depended on nothing but a few base packages, while `C Development' and `Fortran development' might be separate packages which each depended upon `Binutils' and `Base EGCS utilities' packages. Packages sets, described below, would add the ability to choose entire broad categories of software to install, like todays install sets, while maintaining the ability to remove individual packages later. 1.2.1 Root/User/Share separation In order to support a variety of system configurations, it is crucial that the new package system support the possibility of some part of a system residing on a server and possibly being shared between multiple machines on a network. A machine which has some filesystems local and some shared must, at the very least, be able to add and remove packages from local filesystems, and should be able to determine what packages have been added or removed from the volumes mounted over the network. The most common shared configurations are to have a system share /usr/share from the network, and have all other filesystems local, or to share the entirety of /usr from the network, and maintain local root and /var hierarchies, possibly as a single filesystem. Other commonly shared hierarchies include /usr/X11R6 and /usr/pkg. Two steps are necessary to support this type of sharing: the system must be able to check separate repositories for packages installed on different filesystems, and packages must be designed so as to allow a client to install only those parts of the system which reside on local filesystems. The first of these is addressed by a set of patches described by Alistair Crooks in a post to the netbsd-current mailing list on Friday, September 18, 1998. These patches, which have not yet been committed cause third-party software packages installed in /usr/pkg to be registered in /usr/pkg/etc/pkg, and packages installed in /usr/X11R6 to be registered in /usr/X11R6/etc/pkg. This could be extended easily to allow sharing of system package installations by having the new system X11R6 packages also use /usr/X11R6/etc/pkg for package registration, to have system packages installed in /usr use /usr/etc/pkg for package registration, and to have system packages installed in / and /var use /etc/pkg for package registration. This would allow all of the types of filesystem sharing described above, without introducing too much complication into the package system. The second step, that of insuring that a client can choose to install only the parts of the system which reside on local volumes can be most easily addressed by careful consideration of package contents. A look through the contents of the current install sets suggests that relatively few packages will in fact need to install in more than one of /, /usr, /usr/share and /usr/X11R6. Were such packages split into separate components, based on filesystem boundaries, users would easily be able to install only the parts which are local in their particular configuration. ------------------------------------------------------------------------ 2. Package Sets In moving to fine-grained system packages, it is important that beginning users still be able to select broad categories of software to install at once. The introduction of `package sets', analogous in granularity, but not mechanism, to the current binary install sets addresses this concern, while maintaining the ability of more advanced users to choose among individual packages at install time, and maintaining the ability to remove, upgrade, or add individual packages at a later time. These package sets will maintain the same layout as the current install sets, so that a user who chooses the same sets as he would have chosen now will see the same results. In the new system, however, these sets will be made up of binary packages, and installing a set will simply result in the installation of the constituent packages. 2.1 Set format A set will be a tar archive containing the packages which make up the set plus a contents file. At the least, the index file will contain the name of each included package, plus a one line description of each package's contents. Installation utilities will offer the option of installing the whole set, or choosing among individual packages, based on the descriptions in the contents file. It is expected that the contents file itself will be automatically generated from the one-line descriptions provided in each package's pkg/COMMENT file. When a set is installed, the contents file will be recorded in a manner similar to the registration of package information in the current third-party package system. This will allow users to remove an entire set at a later date, without needing to know what individual packages came from that set. ------------------------------------------------------------------------ 3. Creation of Packages and Sets Under the current distribution-building system, the Makefile in /usr/src/etc creates binary install sets from an installed system, based on the set lists in /usr/src/distrib/sets/lists. In the new system, a new directory hierarchy, /usr/src/distrib/pkg, will contain Makefiles and data files relevant to the creation of system packages and package sets. The directory /usr/src/distrib/pkg/sets will contain a directory for each package set, and each of these directories will contain a directory for each package in that set. The Makefile in /usr/src/distrib/pkg/sets will recurse into these set directories to build each set. The individual set Makefiles will recurse into each package directory to build the individual packages, and will then create a set file from the constituent packages and from the contents file, which will be automatically generated from the package directories. The package directories will resemble the package directories for third-party software packages in /usr/pkgsrc, except that they will probably rely on the files making up the package already being present in ${DESTDIR}, rather than building them directly. This assumption is already present in the current distribution package Makefile code, and is probably reasonable to keep. ------------------------------------------------------------------------ 4. Modifications to the NetBSD installation process Once the NetBSD system is available as system packages and package sets, it will be possible to modify the various installation tools to use these sets to install the system. It is expected that installation tools will default to allow users to choose among package sets at install time, but allow an `advanced mode' in which packages could be selected and deselected on an individual basis. This will require that the various package tools (at least pkg_add) be present on install media to be used with system packages. Modifications to sysinst and other install tools are beyond the current scope of this proposal, but will be necessary to take advantage of the new capabilities provided by this system. ------------------------------------------------------------------------ A. Working Plan My current plan for implementing system packages and package sets for NetBSD consists of four steps. All of these steps should be taken in the CVS source tree (segregated into src/distrib/pkg, of course), and hopefully will involve other contributors in addition to myself: 1.) Hammer this proposal into a more detailed specification I am submitting this proposal now in the hopes that it will spark discussion which will lead to a refinement of the planned system package system. Once some sort of consensus is reached on the relevant mailing lists, I will begin work in earnest on implementing this. 2.) Create the /usr/src/distrib/pkg hierarchy, and a template package The first step in actually implementing this system will be to create either an actual or mocked-up system package which can be used as a template for creation of the remaining system packages. 3.) Create system packages I expect that this step will involve most of the actual work in implementing the new system. Packages will have to be created for each functional group of binaries currently shipped with NetBSD. A lot of discussion and design will have to go into the decisions as to how many packages should make up each set and what files belong in which packages. 4.) Create Package Sets Once all system packages exist, it will be necessary to put together some code to automatically generate set contents files and to create sets from each directory of packages in /usr/src/distrib/pkg/sets. Once these steps are complete, NetBSD will have system packages, and it will be possible to begin looking at modifying the NetBSD install process to use them. It is important to note that none of these changes will require modifying the current installation set building code in any way, so the use of the current system can continue unhindered while the new system is being implemented. ------------------------------------------------------------------------ $Id: PROPOSAL,v 1.1 2002/01/15 03:47:34 jwise Exp $