Add index.html, rename documentation to doc, clean up

3 files changed, 373 insertions, 0 deletions

diff --git a/doc/DEPENDENCIES-BUILD.md b/doc/DEPENDENCIES-BUILD.md
new file mode 100644
index 0000000..e1d97de
--- /dev/null
+++ b/doc/DEPENDENCIES-BUILD.md
@@ -0,0 +1,175 @@
+
+# lh-bootstrap: software built for the BUILD machine
+
+Laurent Bercot
+2016-03-31
+
+
+This file documents the software installed and run on the BUILD
+machine prior to building the HOST image.
+
+Please read the INTERNALS.md file first, for the general organization
+of the build, and basic definitions.
+
+
+## BUILD tools
+
+### Linux kernel headers
+
+Makefile directory: sub/kernel
+
+The Linux kernel is downloaded and will be configured and compiled
+to boot a qemu image for the HOST. Since it will be downloaded
+anyway, we reuse the source to process and install the kernel headers
+for the BUILD.
+Those kernel headers, coupled with the musl libc's headers, are
+necessary to compile Linux-specific software such as util-linux and skarnet-org.
+
+
+### musl libc
+
+Makefile directory: sub/musl
+
+ We have no control over the BUILD's native compiler and libc. Most
+likely, it's gcc and produces binaries that are dynamically linked
+against the glibc - but we're not certain; we would like certainty,
+even for the build tools. We do not want our tools' behaviour to
+depend on external factors such as a misconfigured libc or dynamic
+linker.
+
+ So, we download the musl libc (which we would download for use in
+the HOST anyway) and compile it for the BUILD. We then link all our
+BUILD tools against it.
+
+
+### skarnet.org packages
+
+Makefile directory: sub/skarnet.org
+
+ The HOST uses s6-rc as its service manager. We provide a template
+for the database in source format in `layout/rootfs/etc/s6-rc/source-base`;
+this template is preprocessed and added to the rootfs at layout
+installation time, at the beginning of the HOST build.
+ However, in order to boot, the HOST needs the database in compiled
+form, not in source form: so we must run s6-rc-compile before the HOST
+boots. Since the source and compiled formats are platform-independent,
+we just run s6-rc-compile on the BUILD. Which means we need to compile
+s6-rc for the BUILD, with the same settings that the HOST is using.
+So we end up compiling most of the skarnet.org stack.
+
+ Since we have to compile skalibs anyway, which is by far the heaviest
+component of the stack, we also use the opportunity to compile
+s6-portable-utils for the BUILD: the time spent compiling this package
+is negligible once skalibs is built, and it contains
+alternative tools that we use subsequently in the build, because their
+behaviour is more predictible than the tools provided by the BUILD's
+distribution.
+
+ Note: since we need to mirror the HOST's layout for s6-rc-compile
+to work properly, we compile the skarnet.org stack following the
+slashpackage convention, with --enable-slashpackage. However, we
+obviously don't install a slashpackage hierarchy on the BUILD's root
+filesystem, we use the $(OUTPUT)/build-build staging directory.
+The consequence is that skarnet.org binaries that exec other binaries
+via slashpackage paths will not work. This is ok for our use since
+the main tool we need is s6-rc-compile, which does not exec anything
+else, but it's something to keep in mind. It's the reason why we do
+not use s6-setuidgid even after building s6: we stick to the hackish
+and inefficient bin/setuidgid script to drop privileges, because our
+temporary installation of s6-setuidgid simply does not work.
+
+
+#### skalibs
+
+ The library which all other skarnet.org packages depend on.
+
+
+#### execline
+
+ The scripting language used by s6 and s6-rc.
+
+
+#### s6
+
+ The supervision suite used by s6-rc.
+
+
+#### s6-rc
+
+ The service manager used by the HOST. We compile it for the BUILD in
+order to use s6-rc-compile to compile the service database before
+booting the HOST.
+
+
+#### s6-portable-utils
+
+ Some utilities are akin to POSIX tools, but will have reproducible behavior
+no matter what distribution is used. We have had trouble with
+differences across BUILD distributions, with some distributions
+slightly deviating from the standard (looking at you, Ubuntu); using
+our own tools is insurance against that.
+
+
+### util-linux
+
+Makefile directory: sub/util-linux
+
+ To make the qemu image, we need losetup -P, to set up a loopback
+mount that supports partitions. But the -P option to losetup only
+appear in latest versions of util-linux, and not all distributions
+ship a recent enough version. (Looking at you, Ubuntu and Debian
+stable.)
+ So we download and build util-linux. Except the util-linux
+build system is a bloated plate of noodles, that can have a lot
+of dependencies - in particular a dependency to ncurses, and we
+DO NOT want to build ncurses if it can be avoided. Fortunately,
+none of the tools we need require ncurses. So we end up building
+those individual binaries from util-linux and avoid pulling in
+the kitchen sink.
+ Currently, the binaries we build are: losetup, fdisk, mkswap,
+mount, umount. This list can change as the package evolves; the
+current list is described in the UTLX_PROGLIST variable definition
+in the sub/util-linux/Makefile file.
+
+
+### xz-utils
+
+Makefile directory: sub/xz
+
+ xz-utils includes another compression library (liblzma), which
+is also a dependency of kmod - actually, this is the one that
+interests us. So we have to build the xz-utils package for
+BUILD.
+
+
+### kmod
+
+Makefile directory: sub/kmod
+
+ Ah, kmod.
+
+ We build the Linux kernel for HOST with module support, for
+practicality. Modules are compressed, to save storage space.
+Traditionally, there are compressed with gzip (and have extension
+`.ko.gz`), but xz is generally a better compressor than gzip:
+it decompresses faster and the compressed data is smaller. So
+we use xz to compress the modules (extension `.ko.xz`). On the
+HOST, we load the modules with busybox modprobe, which supports
+both extensions. So far, so good.
+
+ Except that xz support for kmod is relatively recent, and some
+distributions insist on providing an ancient version of kmod,
+which *does not* allows modules to be compressed with xz.
+(And the kernel's build system does not report the error - the
+modules silently fail to be installed, which makes diagnostic
+fun!)
+
+ So, we have to provide our own version of kmod.
+
+ I have to say that kmod is the single worst package that appears
+in this whole build. The software itself works, but the
+build system is *extremely* buggy and requires several workarounds,
+that have all been implemented in the Makefile. Please do not
+attempt to "simplify" this Makefile by using "correct" configure
+options and eliminating make variables: that will not work.
+
diff --git a/doc/INTERNALS.md b/doc/INTERNALS.md
new file mode 100644
index 0000000..ab7aa54
--- /dev/null
+++ b/doc/INTERNALS.md
@@ -0,0 +1,106 @@
+
+
+
+# Internals of lh-bootstrap
+
+Laurent Bercot
+last modified: 2017-04-07
+
+## Definitions
+
+BUILD is the machine you're running this set of scripts on.
+HOST is the machine you're building the image for.
+TARGET has the same meaning as HOST. See the `README.md` file.
+
+
+## General organization
+
+### ./lh-config and ./make
+
+All your configuration should happen in the `lh-config` file. It is a series
+of variable assignments, in shell syntax.
+
+`./make` is a shell script that reads `lh-config`, provides reasonable
+defaults for variables not specified in `lh-config`, exports a set of
+variables into the environment then executes into `make` with the exact
+same command line that it was given. So you can use make options, specify
+a make target, etc.
+
+
+### Other files and directories
+
+The main `Makefile` includes sub-Makefiles that are in the subdirectories
+of the `sub` directory, roughly one per software package. Those subdirectories
+also may contain various scripts and patches necessary to make the software
+package compile and/or run properly.
+
+The `bin` directory contains scripts that are used throughout the whole
+build process. They have been designed for maximum portability, not efficiency.
+
+The `sysdeps` directory contains "system dependencies" for the various HOSTs
+that lh-bootstrap supports, i.e. a textual description of the
+HOST architecture's capabilities and quirks, such as endianness, sizes of
+certain types, etc. These descriptions are used when cross-compiling the
+skarnet.org packages.
+
+The `layout` directory contains the base layout for all the filesystems
+that are used to build our image:
+
+- *rootfs* is the basic one, the root filesystem. It will be mounted
+read-only.
+- *rwfs* is a read-write directory we use to store our configuration
+and other read-write data for normal operation of the machine. It is
+not accessible to "normal" users. For instance, `/var` is a part of
+rwfs.
+- *userfs* is a read-write directory that will be used to store user
+data. For instance, `/home` is a part of userfs.
+- *stagingfs* is unused for now. It will be used for safe firmware
+updates.
+
+ All the files under `layout` must be text or otherwise editable
+files: there must be absolutely no binary files in it. Currently,
+the timezone files (`/etc/zoneinfo`) are an exception to that rule;
+at some point I will remove them and make a sub-package script to
+install them.
+
+
+## Dependencies and build order
+
+`./make` first builds a set of tools for the BUILD. The goal is to make the
+build work on as many BUILD machines as possible, with as few dependencies
+as possible. For instance, parts of a recent `util-linux` package are
+installed because the build needs the `-P` option to the `losetup` binary
+and not all distributions ship a `util-linux` version with `losetup -P`.
+
+Then `./make` builds the software for the HOST, installing it into the
+`rootfs` subdirectory of the output.
+
+
+### Topological list of the BUILD tools
+
+- Linux kernel headers
+- musl
+- skarnet.org packages
+- some binaries from util-linux
+- xz-utils
+- kmod
+
+
+### Topological list of the HOST packages
+
+ For now, the HOST packages are a mix of "production" bootstrap packages,
+i.e. software that is needed to get a host up and running without
+aiming for development on that host, and "development" bootstrap packages,
+i.e. software that is needed to turn the host into a native development
+platform (in order to build and install software that is difficult to
+cross-compile). A better separation between those two sets of packages
+is planned for future versions of lh-bootstrap.
+
+- Linux kernel
+- musl
+- bearssl
+- skarnet.org packages
+- busybox
+- dnscache (from djbdns)
+- dropbear
+- a native host toolchain (for now just copied from a location)
diff --git a/doc/index.html b/doc/index.html
new file mode 100644
index 0000000..d17ff15
--- /dev/null
+++ b/doc/index.html
@@ -0,0 +1,92 @@
+<html>
+  <head>
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+    <meta http-equiv="Content-Language" content="en" />
+    <title>lh-bootstrap - a tool to build images running s6 and s6-rc</title>
+    <meta name="Description" content="lh-bootstrap - a tool to build images running s6 and s6-rc" />
+    <meta name="Keywords" content="lh-bootstrap bootstrap VM qemu linux s6 s6-rc virtual machine disk image" />
+    <!-- <link rel="stylesheet" type="text/css" href="http://skarnet.org/default.css" /> -->
+  </head>
+<body>
+
+<p>
+<a href="http://skarnet.org/software/">Software</a><br />
+<a href="http://skarnet.org/">skarnet.org</a>
+</p>
+
+<h1> lh-bootstrap </h1>
+
+<h2> What is it&nbsp;? </h2>
+
+<p>
+ <tt>lh-bootstrap</tt> is a set of scripts that build a disk image
+for a virtual machine such as <a href="http://www.qemu.org/">QEMU</a>.
+The image uses <a href="https://kernel.org/">Linux</a>,
+<a href="http://musl-libc.org/">musl</a>,
+<a href="http://busybox.net/">BusyBox</a>, and skarnet.org tools,
+and similar small and efficient software. It provides the necessary
+minimum amount of tools to securely connect to the Internet and
+download more software while having a stable init system and service
+infrastructure.
+</p>
+
+<hr />
+
+<h2> Installation </h2>
+
+<h3> Requirements </h3>
+
+<p>
+ Read the
+<a href="https://github.com/skarnet/lh-bootstrap/blob/master/README.md">README.md</a>
+file (on GitHub, for Markdown formatting).
+</p>
+
+<h3> Licensing </h3>
+
+<p>
+ lh-bootstrap is free software. It is available under the
+<a href="https://opensource.org/licenses/GPL-2.0">GNU General Public License
+version 2</a>.
+</p>
+
+<h3> Download </h3>
+
+<p>
+<tt>lh-bootstrap</tt> does not have regular numbered releases; you should always
+pull the latest version from Git.
+</p>
+
+<ul>
+ <li> Pull from the
+<a href="http://git.skarnet.org/cgi-bin/cgit.cgi/lh-bootstrap/">main
+lh-bootstrap git repository</a>:
+<pre> git clone git://git.skarnet.org/lh-bootstrap </pre> </li>
+ <li> Pull from the
+<a href="https://github.com/skarnet/lh-bootstrap">GitHub mirror</a>. </li>
+</ul>
+
+<h3> Compilation </h3>
+
+<ul>
+ <li> Read the <a href="https://github.com/skarnet/lh-bootstrap/blob/master/README.md">README.md</a>
+file carefully </li>
+ <li> Copy <tt>lh-config.dist</tt> to <tt>lh-config</tt> and edit it to suit your needs </li>
+ <li> Run <tt>./make</tt> </li>
+</ul>
+
+<a name="related">
+<h2> Related resources </h2>
+</a>
+
+<h3> lh-bootstrap discussion </h3>
+
+<ul>
+ <li> <tt>lh-bootstrap</tt> is discussed on the
+<a href="http://skarnet.org/lists.html#skaware">skaware</a> mailing-list. </li>
+ <li> It can also be discussed on the <tt>#s6</tt> IRC channel on Freenode. </li>
+</ul>
+
+</body>
+</html>