From a44a71d3b95c42604f15316c2fd5d2d78d4dae9b Mon Sep 17 00:00:00 2001 From: Laurent Bercot Date: Tue, 24 Mar 2015 12:13:53 +0000 Subject: - Added overview.html - Added clarifications on s6-svwait --- doc/index.html | 1 + doc/overview.html | 462 +++++++++++++++++++++++++++++++++++++++++++++++++++++ doc/s6-svwait.html | 12 ++ 3 files changed, 475 insertions(+) create mode 100644 doc/overview.html diff --git a/doc/index.html b/doc/index.html index ed62e41..26fd229 100644 --- a/doc/index.html +++ b/doc/index.html @@ -62,6 +62,7 @@ supervision that might help you understand the basics.

A high-level overview of s6
Why another supervision suite? Isn't runit good enough?
What is instant notification? What does the ftrigr library do exactly?

s6: an overview

+s6
+Software
+skarnet.org +

An overview of s6

+ s6 is a collection of utilities revolving around process supervision and +management, logging, and system initialization. This page is a high-level +description of the different parts of s6. +

Process supervision

+ At its core, s6 is a process supervision suite, like its ancestor +daemontools and its +close cousin +runit. +

Concept

+ The concept of process supervision comes from several observations: +

Unix systems, even minimalistic ones, need to run +long-lived processes, aka daemons. Sometimes, they +need to run a lot of them.
Daemons can die unexpectedly. Maybe they are missing a vital +resource and cannot handle a certain failure; maybe they tripped on a bug; +maybe a misconfigured administration program killed them; maybe the +kernel killed them.
Automatically restarting daemons when they die is generally a good +thing. In any case, sysadmin intervention is necessary, but at least the +daemon is providing service, or trying to, until the sysadmin can log in +and investigate the underlying problem.
Ad-hoc shell scripts that restart daemons suck, for +several reasons that would each justify their own page. The difficulty of +keeping track of the PID, explained below, is one of those reasons.
It is sometimes necessary to send signals to a daemon. To kill it, +of course, but also to make it read its config file again, for instance; +signalling a daemon is a natural and very common way of sending it +simple commands.
Generally, to send a signal to a daemon, you need to know its PID. +Without a supervision suite, knowing the proper PID is hard. Most +non-supervision systems use a hack known as .pid files, i.e. +the script that starts the daemon stores its PID into a file, and other +scripts read that file. This is a bad mechanism for several reasons, and +the case against .pid files would also justify its own page; the most +important drawback of .pid files is that they create race conditions +and management scripts may kill the wrong process.
Non-supervision systems provide scripts to start and stop daemons, +but those scripts may fail at boot time even though they work when run +manually, +and vice versa. If a sysadmin logs in and runs the script to restart a +daemon that has died, the result might not be the same as if the whole +system had been rebooted, and the daemon may exhibit strange behaviours! +This is because the boot-time environment and the restart-time environment +are not the same when the script is run; and a non-supervision system +just cannot ensure reproducibility of the environment. This is a core +problem of non-supervision systems: countless bugs have been falsely +reported because of simple environment differences or configuration errors, +countless man-hours have been wasted to try and understand what was +going on.

+ A process supervision system organizes the process hierarchy in a +radically different way. +

A process supervision system starts an independent hierarchy of +processes at boot time, called a supervision tree. This +supervision tree never dies: when one of its components dies, it is +restarted automatically. To ensure availability of the supervision +tree at all times, it should be rooted in process 1, which cannot die.
A daemon is never started, either manually or in a script, as a +scion of the script that starts it. + Instead, to start a daemon, you configure a +specific directory which contains all the information about your daemon; +then you send a command to the supervision tree. The supervision tree +will start the daemon as a leaf. In a process supervision +system, daemons are always spawned by the supervision tree, and +never by an admin's shell.
The parent of your daemon is a supervisor. Since your +daemon is its direct child, the supervisor always knows the +correct PID of your daemon.
The supervisor watches your daemon and can restart it when it +dies, automatically.
The supervision tree always has the same environment, so starting +conditions are reproducible. Your daemon will always be started with the +same environment, whether it is at boot time via init scripts or for the +100th automatic - or manual - restart.
To send signals to your daemon, you send a command to its +supervisor, which will then send a signal to the daemon on your behalf. +Your daemon is identified by the directory containing its information, +which is stable, instead of by its PID, which is not stable; the supervisor +maintains the correct association without a race condition or the other +problems of .pid files.

Implementation

+ s6 is a straightforward implementation of those concepts. +

The s6-svscan and +s6-supervise programs are the components +of the supervision tree. They are long-lived programs. +
- s6-supervise is a daemon's +supervisor, its direct parent. For every long-lived process on a +system, there is a corresponding s6-supervise +process watching it. This is okay, because every instance of +s6-supervise uses very few resources.
- s6-svscan is, in a manner of speaking, +a supervisor for the supervisors. It watches and maintains a collection of +s6-supervise processes: it is the branch +of the supervision tree that all supervisors are stemming from. It can be +run and +supervised +by your regular init process, or it can +run as +process 1 itself. Running s6-svscan as process 1 currently requires +some manual effort from the user, because of the inherent non-portability +of init processes; future versions of s6 will automate that effort and +provide users with ready-to-run "init" commands.
- The configuration of a daemon to be supervised by +s6-supervise is done via a +service directory.
- The place to gather all service directories to be watched by a +s6-svscan instance is called a +scan directory.
+
The command that controls a single supervisor, and allows you to +send signals to a daemon, is +s6-svc. It is a short-lived program.
The command that controls a set of supervisors, and allows you to +start and stop supervision trees, is +s6-svscanctl. It is a short-lived +program.

+ These four programs, +s6-svscan, +s6-supervise, +s6-svscanctl and +s6-svc, +are the very core of s6. Technically, once you have them, you have a +functional s6 installation, and the other utilities are just a bonus. +

Practical usage

+ To use s6's supervision features, you need to perform the following steps: +

For every daemon you potentially want supervised, write a +service directory. Make sure that +your daemon does not background itself when started in the +./run script! Auto-backgrounding is a historical hack +that was implemented when supervision suites did not exist; since +you're using a supervision suite, auto-backgrounding is unnecessary +and in this case detrimental.
Write a single scan directory for +the set of daemons you want to actually run. This set can be modified +at run time.
At some point in your initialization scripts, run +s6-svscan on the scan directory. This will +start the supervision tree, including your set of daemons. The exact +way of running s6-svscan depends on your system: it is not quite the same +when you want to run it as process 1 on a real machine, or under another +init on a real machine, or as process 1 in a +Docker container, or in another +context entirely.
Alternatively, you can start s6-svscan +on an empty scan directory, then populate it step by step and send an +update command to s6-svscan via +s6-svscanctl whenever the supervision +tree should pick up the differences and start the services you added.
That's it, your services are running. To control them manually, +you can use the s6-svc command.
At the end of the system's lifetime, you can use +s6-svscanctl to bring down the supervision +tree.

Service-specific logging

+s6-svscan can monitor a supervision tree, +but it can also do one more thing. It can ensure that a daemon's log, +i.e. what the daemon outputs to its stdout (or stderr if you redirect it), +gets processed by another, supervised, long-lived process, called a +logger; and it can make sure that the logs are never lost +between the daemon and the logger - even if the daemon dies, even if the +logger dies. +

+ If your daemon is outputting messages, you have a decision to make +about where to send them. +

You can do as non-supervision systems do, and send the messages +to syslog. It's entirely possible with a supervision system too. +However, like auto-backgrounding, syslog is a historical mechanism that +predates supervision suites, and is technically inferior; it is +recommended that you do not use it whenever you can avoid it.
You can send them to the daemon's stdout/stderr and do nothing special +about it. The logs will then be sent to s6-svscan's stdout/stderr; +what mechanism will read them depends on how you started s6-svscan.
You can use s6-svscan's service-specific logging mechanism and +dedicate a logger process to your daemon's messages.

+ s6 provides you with a long-lived process to use as a logger: +s6-log. It will store your logs in one (or +more) specific directory of your choice, and rotate them automatically. +

Helpers for run scripts

+ Creating a working +service directory, and especially a good +run script, is the most important part of the work when +adapting a daemon to a supervision framework. +

+ If you can find your daemon's invocation script on a non-supervision system, +for instance a System V-style init script, you can see the exact +options that the daemon is being run with: environment variables, +uid and gid, open descriptors, etc. This is what you +need to replicate in your run script. +

+ (Do not replicate the auto-backgrounding, or things like +start-stop-daemon +invocation: start-stop-daemon and its friends are hideous and kludgy +attempts to work around the lack of proper supervision mechanisms. Now +that you have s6, you should remove them from your system, throw them +into a bonfire, and dance and laugh while they burn.) +

+ The vast majority of the tools provided by s6 are meant to be used in +run scripts: they help you control the process state and +environment in your script before it executes into your daemon. Or, +sometimes, they are daemons themselves, designed to be supervised. +

+ s6, like other skarnet.org +software, makes heavy use of +chain +loading, also known as "Bernstein chaining": a lot of s6 tools will +perform some action that changes the process state, then execute into the +rest of their command line. This allows the user to change the process state +in a very flexible way, by combining the right components in the right +order. Very often, a run script can be reduced to a single command line - +likely a long one, but still a single one. (That is the main reason why +using the +execline language +to write run scripts is recommended: execline makes it natural to handle +long command lines made of massive amounts of chain loading. This is by no +means mandatory, though: a run script can be any executable file you want, +provided that running it eventually results in a long-lived process with +the same PID.) +

+ Some examples of s6 programs meant to be used in run scripts: +

The s6-log program is a long-lived +process. It is meant to be executed into by a ./log/run +script: it will be supervised, and will process what it reads on +its stdin (i.e. the output of the ./run daemon).
The s6-envdir program is a +short-lived process that will update its current environment according +to what it reads in a given directory, then execute into the rest of its +command line. It is meant to be used in a run script to adjust the +environment with which the final daemon will be executed into.
Similarly, the s6-softlimit program +adjusts its resource limits, then executes into the rest of its command +line: it is meant to set the resources the final daemon will have +access to.
The s6-applyuidgid program, +part of the s6-*uidgid family, drops root privileges before +executing into the rest of its command line: it is meant to be used +in run scripts that need root privileges when starting but do not +need it for the execution of the long-lived process.
s6-ipcserverd is a daemon that +listens to a Unix socket and spawns a program for every connection. +It is meant to be supervised, so it should be used in a run script, +and it's also meant to be a flexible super-server that you can use +for different applications: so it is a building block that may appear in +several of your run scripts defining +local services.

Readiness notification and dependency management

+ Now that you have a supervision tree, and long-lived processes running +supervised, you may want to introduce dependencies between them: do not +perform an action (e.g. start (with s6-svc -u) +the Web server connecting to a database) +before a given daemon is up and running (e.g. the database server). +s6 provides tools to do that: +

The s6-svwait, +s6-svlisten1 and +s6-svlisten programs will wait until a set of +daemons is up, ready or down.
Unfortunately, a daemon being up does not mean that it is +ready: +this page goes into the details. s6 +gives you a tool to use in your run +scripts so the supervisor knows when a service is ready.

+ For now, s6 does not provide a complete dependency management framework, +i.e. a program to automatically start (or stop) a set of services in a +specific order - that order being automatically computed from a graph of +dependencies between services. + That functionality is planned for a future version of s6. +

Additional utilities

+ The other programs in the s6 package are various utilities that may be +useful in designing servers, and more generally multi-process software. +They can be used with or without a supervision environment, although +it is of course recommended to have one; but they are not part of the core s6 +functionality, and you may safely ignore them for now if you are just getting +into the supervision world. +

Generic inter-process notification

+ The s6-ftrig* family of programs allows notifications between +unrelated processes: a set of processes can subscribe to a certain +channel - identified by a directory in the filesystem - and ask to be +notified of certain events on that channel; another set of processes can +send events to the channel. +

+ The underlying mechanism is the same as the one used by the supervision +tree for readiness notification, but the s6-ftrig* tools provide +a more generic access to that mechanism. +

Helpers for designing local services

+ Local services, i.e. daemons listening to a Unix domain socket, are a +powerful and flexible mechanism, especially with modern Unix systems +that allow client authentication. s6 includes tools to take advantage +of that mechanism. +

The s6-ipc* family of programs is about designing clients +or servers that communicate over Unix domain sockets.
The s6-*access* and s6-connlimit +family of programs is about client access control.
The s6-sudo* family of programs is about using a local +service in order to give selected +clients the ability to run a command line with the privileges of the +server, without using suid programs.

Keeping file descriptors open

+ Sometimes you want to keep a file descriptor open, even if the program +normally using it dies - so the program can restart and use the same +file descriptor without losing any data. To do that, you need to +hold the descriptor in another process, i.e. that process +should have it open but do nothing with it. +

+s6-svscan, for instance, holds the pipe +existing between a supervised daemon and its logger, so even if the +daemon or the logger dies while there are logs in the pipe, the pipe +remains open and the logs are not lost. +

+ s6 provides a mechanism to store and retrieve open file descriptors +in a totally generic way: the s6-fdholder* family of programs. +

The s6-fdholder-daemon program +is a daemon (or, rather, executes into the +s6-fdholderd daemon), meant to be +supervised, that will hold file descriptors on its clients' behalf.
Other programs in the family, such as +s6-fdholder-store, are client +programs that interact with this daemon to store and retrieve file +descriptors.

+ Note that "socket activation", one of the main advertised benefits of the +systemd +init system, sounds similar to fd-holding. +The reality is that socket activation is a mixture of several different +mechanisms, one of which is fd-holding; s6 allows you to implement the +healthy parts of socket activation. +

Other miscellaneous utilities

+ This page does not list or classify every s6 tool. Please +explore the "Reference" section of the +main s6 page for details on a specific program. +

timeout

Notes

+ +
+
s6-svwait should be given one or more service directories as +arguments, not a scan directory. If you need to wait for a whole +scan directory, give all its contents as arguments to s6-svwait.
+
s6-svwait will only work on service directories that are already +active, i.e. have a s6-supervise process +running on them. It will not work on a service directory where +s6-supervise has not been started yet.
+
+

Internals

-- cgit v1.2.3