More doc; unify tipidee-config exit codes

Signed-off-by: Laurent Bercot <ska@appnovation.com>

9 files changed, 177 insertions, 23 deletions

diff --git a/doc/tipidee-config-preprocess.html b/doc/tipidee-config-preprocess.html
index 86fb22e..9b6ea38 100644
--- a/doc/tipidee-config-preprocess.html
+++ b/doc/tipidee-config-preprocess.html
@@ -49,17 +49,55 @@ appropriately included files. </li>
  <li> tipidee-config-preprocess then exits 0. </li>
 </ul>
 
+<h2> Exit codes </h2>
+
 <p>
- TODO: write this page.
+If tipidee-config-preprocess exits nonzero, 
+<a href="tipidee-config.html">tipidee-config</a> will immediately exit
+with the same error code.
 </p>
 
-<h2> Exit codes </h2>
-
-<h2> Options </h2>
+<dl>
+ <dt> 0 </dt> <dd> success </dd>
+ <dt> 1 </dt> <dd> syntax error </dd>
+ <dt> 2 </dt> <dd> invalid inclusion (cycle or unauthorized duplicate) </dd>
+ <dt> 100 </dt> <dd> wrong usage </dd>
+ <dt> 111 </dt> <dd> system call failed </dd>
+</dl>
 
 <h2> Detailed operation </h2>
 
+<ul>
+ <li> tipidee-config-preprocess reads <em>file</em>, looking for lines
+that start with a <tt>!</tt> (bang). It prints other lines as is to its
+stdout. </li>
+ <li> A line that starts with <tt>!</tt> is not printed. Instead, it
+is parsed for an <tt>!include</tt>, <tt>!includedir</tt> or <tt>!included:</tt>
+directive. </li>
+ <li> <tt>!included:</tt> directives immediately inform the file currently
+being preprocessed. </li>
+ <li> <tt>!include <em>foo</em></tt> and <tt>!includedir <em>bar</em></tt>
+directives cause tipidee-config-preprocess to immediately start preprocessing
+file <em>foo</em> or all files in directory <em>bar</em>; this can happen
+recursively. When it's done preprocessing <em>foo</em> or <em>bar</em>,
+tipidee-config-preprocess resumes its current file where it had left it. </li>
+ <li> Around inclusions, tipidee-config-preprocess prints special lines
+starting with a single <tt>!</tt> (bang) and containing line and file
+information. This helps <a href="tipidee-config.html">tipidee-config</a>
+track which file the data it's reading comes from and accurately report
+errors. </li>
+</ul>
+
 <h2> Notes </h2>
 
+<ul>
+ <li> tipidee-config-preprocess does not know anything about the
+<a href="tipidee.conf.html"><tt>/etc/tipidee.conf</tt></a> file format.
+It was purposefully written to be generic: it only reads lines and
+includes files based on <tt>!include</tt> information, and inserts
+<tt>!</tt> lines into the output stream to help with error reporting.
+It can be used as a preprocessor for other tools. </li>
+</ul>
+
 </body>
 </html>
diff --git a/doc/tipidee-config.html b/doc/tipidee-config.html
index d6a6f56..66953ab 100644
--- a/doc/tipidee-config.html
+++ b/doc/tipidee-config.html
@@ -27,7 +27,7 @@ web server.
 <h2> Interface </h2>
 
 <pre>
-     tipidee-config [ -i <em>ifile</em> ] [ -o <em>ofile</em> ]
+     tipidee-config [ -i <em>textfile</em> ] [ -o <em>cdbfile</em> ] [ -m <em>mode</em> ]
 </pre>
 
 <ul>
@@ -36,17 +36,98 @@ configuration file, parses it, and outputs a cdb file to <tt>/etc/tipidee.conf.c
  <li> It then exits 0. </li>
 </ul>
 
-<p>
- TODO: write this page.
-</p>
-
 <h2> Exit codes </h2>
 
+<dl>
+ <dt> 0 </dt> <dd> success </dd>
+ <dt> 1 </dt> <dd> syntax error </dd>
+ <dt> 2 </dt> <dd> invalid inclusion (cycle or unauthorized duplicate) </dd>
+ <dt> 100 </dt> <dd> wrong usage </dd>
+ <dt> 111 </dt> <dd> system call failed </dd>
+ <dt> 129+ </dt> <dd> <a href="tipidee-config-preprocess.html">tipidee-config-preprocess</a> was killed </dd>
+</dl>
+
 <h2> Options </h2>
 
+<dl>
+ <dt> -i <em>textfile</em> </dt>
+  <dd> Use <em>textfile</em> as input instead of <tt>/etc/tipidee.conf</tt> </dd>
+ <dt> -o <em>cdbfile</em> </dt>
+  <dd> Use <em>cdbfile</em> as output instead of <tt>/etc/tipidee.conf.cdb</tt>.
+You can then use the <tt>-f <em>cdbfile</em> option to
+<a href="tipideed.html">tipideed</>. </dd>
+ <dt> -m <em>mode</em> </dt>
+  <dd> Create the output file with permissions <em>mode</em> (given in octal).
+Default is <strong>0644</strong>. Note that the output file should be readable
+by the user <a href="tipideed.html">tipideed</> is started as. If
+<a href="tipideed.html">tipideed</> is started as root and drops its privileges
+itself, the file can be made private. </dd>
+</dl>
+
 <h2> Detailed operation </h2>
 
+<ul>
+ <li> tipidee-config spawns a
+<a href="tipidee-config-preprocess.html">tipidee-config-preprocess</a> helper
+that reads <tt>/etc/tipidee.conf</tt>, takes care of all the inclusions, and
+feeds it a single stream of data. If
+<a href="tipidee-config-preprocess.html">tipidee-config-preprocess</a> dies
+with a nonzero exit code at any point, tipidee-config exits with the same
+error code, or 128 plus the signal number if
+<a href="tipidee-config-preprocess.html">tipidee-config-preprocess</a> was
+killed by a signal. </li>
+ <li> It reads the data and parses it, expecting it to follow the
+<a href="tipidee.conf.html">/etc/tipidee.conf file format</a>. </li>
+ </li> On failure, it exits nonzero with an error message. </li>
+ <li> It supplies sane defaults for configuration values that have not
+been provided. </li>
+ <li> It writes the data as a <a href="https://cr.yp.to/cdb/cdb.txt">cdb file</a>,
+<tt>/etc/tipidee.conf.cdb</tt>. A previously existing file is replaced
+atomically. </li>
+ <li> Running instances of <a href="tipideed.html">tipideed</a> will keep
+using the old <tt>/etc/tipidee.conf.cdb</tt> data until their connection is closed;
+new instances will use the new one. </li>
+</ul>
+
 <h2> Notes </h2>
 
+<ul>
+ <li> It is by design that tipidee uses this unconventional "compile the
+configuration file" approach. There are several benefits to it:
+  <ul>
+   <li> Parsing a configuration file is not very efficient. Every instance of
+<a href="tipideed.html">tipideed</a> would have to do it on startup, and
+there is an instance of <a href="tipideed.html">tipideed</a> for every
+HTTP connection. Pre-parsing the configuration makes the initial server
+response faster. </li>
+   <li> Data parsed by <a href="tipideed.html">tipideed</a> needs to use
+<em>private dirty</em> memory for every instance, even if the data is
+static &mdash; and that means incompressible RAM. By contrast, a cdb file
+is mapped read-only, so its pages are <em>shared clean</em>, which means it's
+essentially free. </li>
+   <li> <a href="tipideed.html">tipideed</a> is exposed to the network. You
+want to its attack surface to be as small as possible. Taking the parsing code
+out of it goes a long way &mdash; admittedly, having to parse HTTP in the
+first place is more attack surface than a simple config file can ever hope
+to be, but every little bit helps. </li>
+   <li> Run time is the worst time to detect errors. Nobody wants their
+service to go down because Bob edited the live config file and made a typo.
+Having the parsing done <em>offline</em> prevents that: tipidee-config
+doubles as a syntax checker, and when it runs successfully, you know the
+service will pick up the new config and be fine. </li>
+   <li> In general, decoupling the <em>live configuration</em>, which is
+the one used by live services (here, <tt>/etc/tipidee.conf.cdb</tt>), from
+the <em>working configuration</em>, which is the one that humans can
+tinker with (here, <tt>/etc/tipidee.conf</tt>), is a good idea. Don't
+touch production until you're ready to flip the switch atomically;
+tipidee-config is the switch. </li>
+  </ul> </li>
+</ul>
+
+<p>
+ Just remember to run <tt>tipidee-config</tt> whenever you make
+a modification to your config file. It not insurmountable.
+</p>
+
 </body>
 </html>
diff --git a/doc/tipidee.conf.html b/doc/tipidee.conf.html
index eb2b7f4..64e3d30 100644
--- a/doc/tipidee.conf.html
+++ b/doc/tipidee.conf.html
@@ -366,7 +366,7 @@ Except for <tt>domain</tt>, they can only be used after a <tt>domain</tt> direct
 </p>
 
 <ul>
- <tt>domain</tt> is a special directive in that it is stateful. Instead of
+ <li> <tt>domain</tt> is a special directive in that it is stateful. Instead of
 having a direct effect on the configuration, it merely defines the domain that
 the next local directives will apply to. <tt>domain example.com</tt> means
 that a subsequent <tt>cgi /cgi-bin/</tt> line will declare that a resource 
diff --git a/doc/tipideed.html b/doc/tipideed.html
index 97102c5..ce8f5e5 100644
--- a/doc/tipideed.html
+++ b/doc/tipideed.html
@@ -48,6 +48,18 @@ current working directory, one subdirectory for every domain it hosts. </li>
 
 <h2> Exit codes </h2>
 
+<dl>
+ <dt> 0 </dt> <dd> clean exit. The client closed the connection after a stream of
+HTTP exchanges. </dd>
+ <dt> 100 </dt> <dd> bad usage. tipideed has been run in an incorrect way: bad command
+line options, or missing environment variables, etc. </dd>
+ <dt> 101 </dt> <dd> cannot happen. This signals a bug in tipideed, and comes with an
+error message asking you to report the bug. Please do so. </dd>
+ <dt> 111 </dt> <dd> system call failed. If this happens while serving a request,
+tipideed likely has sent a 500 (Internal Server Error) response to the
+client before exiting. </dd>
+</dl>
+
 <h2> Environment variables </h2>
 
 <h2> Options </h2>
@@ -56,5 +68,11 @@ current working directory, one subdirectory for every domain it hosts. </li>
 
 <h2> Notes </h2>
 
+<ul>
+ <li> <tt>tipideed</tt> is pronounced <em>tipi-deed</em>. You can also say
+<em>tipi-dee-dee</em>, but only if you're the type of person who says
+<em>PC computer</em>, <em>NIC card</em> or <em>ATM machine</em>. </li>
+</ul>
+
 </body>
 </html>
diff --git a/src/config/confnode.c b/src/config/confnode.c
index 758e79d..2d5aeea 100644
--- a/src/config/confnode.c
+++ b/src/config/confnode.c
@@ -8,7 +8,6 @@
 
 #include "tipidee-config-internal.h"
 
-#define dienomem() strerr_diefu1sys(111, "stralloc_catb")
 #define diestorage() strerr_diefu2x(100, "add node to configuration tree", ": too much data")
 #define diefilepos() strerr_diefu2x(100, "add node to configuration tree", ": file too large")
 
diff --git a/src/config/conftree.c b/src/config/conftree.c
index fc0b5bc..4eac28e 100644
--- a/src/config/conftree.c
+++ b/src/config/conftree.c
@@ -11,8 +11,6 @@
 
 #include "tipidee-config-internal.h"
 
-#define dienomem() strerr_diefu1sys(111, "stralloc_catb")
-
 static void *confnode_dtok (uint32_t d, void *data)
 {
   return g.storage.s + GENSETDYN_P(confnode, (gensetdyn *)data, d)->key ;
diff --git a/src/config/lexparse.c b/src/config/lexparse.c
index da9d7f5..99693a5 100644
--- a/src/config/lexparse.c
+++ b/src/config/lexparse.c
@@ -15,7 +15,6 @@
 #include <tipidee/config.h>
 #include "tipidee-config-internal.h"
 
-#define dienomem() strerr_diefu1sys(111, "stralloc_catb")
 #define dietoobig() strerr_diefu1sys(100, "read configuration")
 
 typedef struct mdt_s mdt, *mdt_ref ;
diff --git a/src/config/tipidee-config-internal.h b/src/config/tipidee-config-internal.h
index 154c6ff..7ffee8a 100644
--- a/src/config/tipidee-config-internal.h
+++ b/src/config/tipidee-config-internal.h
@@ -10,6 +10,8 @@
 #include <skalibs/stralloc.h>
 #include <skalibs/cdbmake.h>
 
+#define dienomem() strerr_diefu1sys(111, "stralloc_catb")
+
 typedef struct confnode_s confnode, *confnode_ref ;
 struct confnode_s
 {
diff --git a/src/config/tipidee-config-preprocess.c b/src/config/tipidee-config-preprocess.c
index 03b54f1..a5a47c2 100644
--- a/src/config/tipidee-config-preprocess.c
+++ b/src/config/tipidee-config-preprocess.c
@@ -149,23 +149,22 @@ static void includefromhere (char const *file)
   uint32_t d ;
   uint32_t line = 1 ;
   char buf[4096] ;
-  char linefmt[UINT32_FMT] = "1" ;
   unsigned char state = 0 ;
 
   if (!stralloc_catb(&namesa, "\004", 1) || sarealpath(&namesa, file) < 0 || !stralloc_0(&namesa)) dienomem() ;
   if (avltree_search(&namemap, namesa.s + namesabase + 1, &d))
   {
     if (namesa.s[d] & 0x04)
-      strerr_dief3x(3, "file ", namesa.s + namesabase + 1, " is included in a cycle") ;
+      strerr_dief3x(2, "file ", namesa.s + namesabase + 1, " is included in a cycle") ;
     if (!(namesa.s[d] & 0x02))
-      strerr_dief3x(3, "file ", namesa.s + namesabase + 1, " is included twice but does not declare !included: unique or !included: multiple") ;
+      strerr_dief3x(2, "file ", namesa.s + namesabase + 1, " is included twice but does not declare !included: unique or !included: multiple") ;
     namesa.len = namesabase ;
     if (namesa.s[d] & 0x01) return ;
   }
   else
   {
     if (namesabase > UINT32_MAX)
-      strerr_dief3x(3, "in ", namesa.s + d + 1, ": too many, too long filenames") ;
+      strerr_dief3x(100, "in ", namesa.s + d + 1, ": too many, too long filenames") ;
     d = namesabase ;
     if (!avltree_insert(&namemap, d)) dienomem() ;
   }
@@ -197,7 +196,11 @@ static void includefromhere (char const *file)
       if (!stralloc_0(&sa)) dienomem() ;
       cmd = idcmd(sa.s + sastart) ;
       if (cmd == -1)
-        strerr_dief6x(2, "in ", namesa.s + d + 1, " line ", linefmt, ": unrecognized directive: ", sa.s + sastart) ;
+      {
+        char linefmt[UINT32_FMT] ;
+        linefmt[uint32_fmt(linefmt, line)] = 0 ;
+        strerr_dief6x(1, "in ", namesa.s + d + 1, " line ", linefmt, ": unrecognized directive: ", sa.s + sastart) ;
+      }
       sa.len = sastart ;
     }
     if (what & 0x0080)
@@ -207,17 +210,28 @@ static void includefromhere (char const *file)
       {
         case 2 :
           if (namesa.s[d] & 2)
-            strerr_dief5x(3, "in ", namesa.s + d + 1, " line ", linefmt, ": extra !included: directive") ;
+          {
+            char linefmt[UINT32_FMT] ;
+            linefmt[uint32_fmt(linefmt, line)] = 0 ;
+            strerr_dief5x(1, "in ", namesa.s + d + 1, " line ", linefmt, ": extra !included: directive") ;
+          }
           if (!strcmp(sa.s + sastart, "unique")) namesa.s[d] |= 3 ;
           else if (!strcmp(sa.s + sastart, "multiple")) namesa.s[d] |= 2 ;
-          else strerr_dief6x(3, "in ", namesa.s + d + 1, " line ", linefmt, ": invalid !included: argument: ", sa.s + sastart) ;
+          else
+          {
+            char linefmt[UINT32_FMT] ;
+            linefmt[uint32_fmt(linefmt, line)] = 0 ;
+            strerr_dief6x(1, "in ", namesa.s + d + 1, " line ", linefmt, ": invalid !included: argument: ", sa.s + sastart) ;
+          }
           break ;
         case 1 :
         case 0 :
         {
           int fdhere = open2(".", O_RDONLY | O_DIRECTORY) ;
+          char linefmt[UINT32_FMT] ;
           if (fdhere == -1)
             strerr_dief3sys(111, "in ", namesa.s + d + 1, ": unable to open base directory: ") ;
+          linefmt[uint32_fmt(linefmt, line)] = 0 ;
           if (cmd & 1)
           {
             if (chdir(sa.s + sastart) == -1)
@@ -239,9 +253,14 @@ static void includefromhere (char const *file)
       }
       sa.len = sastart ;
     }
-    if (c == '\n' && state <= 8) linefmt[uint32_fmt(linefmt, ++line)] = 0 ;
+    if (c == '\n' && state <= 8) line++ ;
+  }
+  if (state > 8)
+  {
+    char linefmt[UINT32_FMT] ;
+    linefmt[uint32_fmt(linefmt, line)] = 0 ;
+    strerr_dief5x(1, "in ", namesa.s + d + 1, " line ", linefmt, ": syntax error: invalid ! line") ;
   }
-  if (state > 8) strerr_dief5x(2, "in ", namesa.s + d + 1, " line ", linefmt, ": syntax error: invalid ! line") ;
   fd_close(fd) ;
   sa.len = sabase ;
   namesa.s[d] &= ~0x04 ;