diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/conf.py | 4 | ||||
| -rw-r--r-- | doc/connections.rst | 29 | ||||
| -rw-r--r-- | doc/data.rst | 20 | ||||
| -rw-r--r-- | doc/hosts.rst | 6 | ||||
| -rw-r--r-- | doc/ideas.rst | 30 | ||||
| -rw-r--r-- | doc/index.rst | 1 | ||||
| -rw-r--r-- | doc/installation.rst | 2 | ||||
| -rw-r--r-- | doc/introduction.rst | 9 | ||||
| -rw-r--r-- | doc/pitfalls.rst | 31 | ||||
| -rw-r--r-- | doc/properties.rst | 5 | ||||
| -rw-r--r-- | doc/tutorial/os_installation.rst | 80 |
11 files changed, 166 insertions, 51 deletions
diff --git a/doc/conf.py b/doc/conf.py index 0bf86c6..653cb12 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -22,7 +22,7 @@ copyright = '2020-2021, Sean Whitton' author = 'Sean Whitton' # The full version, including alpha/beta/rc tags -release = '0.8.0' +release = '0.9.0' # -- General configuration --------------------------------------------------- @@ -49,7 +49,7 @@ highlight_language = 'common-lisp' # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # -html_theme = 'haiku' +html_theme = 'sphinx_rtd_theme' # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, diff --git a/doc/connections.rst b/doc/connections.rst index be3ed80..6e20500 100644 --- a/doc/connections.rst +++ b/doc/connections.rst @@ -114,22 +114,17 @@ in that saved image. Typically a ``:SUDO`` connection hop is used before hops which start up remote Lisp images, so these issues will not arise for most users. -``:CHROOT.FORK`` -~~~~~~~~~~~~~~~~ - -Since forking is typically only possible when it is not the case that multiple -threads are running, it is better to avoid using this connection type as the -first hop, i.e., directly out of the root Lisp (this is not much of a -restriction, since typically the root Lisp is running under a uid which cannot -use the ``chroot(2)`` system call anyway). More generally, you should avoid -using this connection type within a Lisp image which might try to execute -other deployments in parallel. Typical usage would be something like:: +Connections which fork: ``:CHROOT.FORK``, ``:SETUID`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +These connection types cannot be used as the first hop, i.e., directly out of +the root Lisp. This is because they must call fork(2), and Consfigurator only +makes this system call in contexts in which there shouldn't ever be more than +one thread (excluding Lisp implementation finaliser threads and the like). +The root Lisp is not such a context, because it is often multithreaded due to +the use of SLIME. This is, however, not much of a restriction, because +typically the root Lisp is running under a UID which cannot use system calls +like chroot(2) and setuid(2) anyway. Thus, typical usage on localhost would +be something like:: (deploy (:sudo :sbcl (:chroot.fork :into "...")) ...) - -In some situations you might want to have a connection chain which effectively -uses a connection type like ``:SBCL`` twice in a row, so that the first Lisp -image can execute deployments in parallel while the second forks into the -chroot (typically by having a ``DEPLOYS`` property with connection type -``:SBCL`` as one of the properties applied by a deployment whose connection -chain itself ends with ``:SBCL``). diff --git a/doc/data.rst b/doc/data.rst index faba8a3..a32acb1 100644 --- a/doc/data.rst +++ b/doc/data.rst @@ -21,7 +21,14 @@ not use prerequisite data identified by strings matching these conditions for other purposes. - ``(HOSTNAME . PATH)`` means the data that should be uploaded to ``PATH`` on - ``HOSTNAME`` (and nowhere else) + ``HOSTNAME`` (and usually nowhere else, except in the case of, e.g., a + public key). ``PATH`` must be absolute, not relative. + +- ``(_CONTEXT . ITEM)`` is an arbitrary prerequisite data context named + ``CONTEXT``; typically ``CONTEXT`` will be a network or grouping name, + rather than referring to a single host. ``ITEM`` might be a path or some + other identifier. Reserved for consfigs; will not be used by property + definitions included with Consfigurator. - ``("--lisp-system" . SYSTEM)`` means the data is Lisp code which, when loaded, defines the packages and symbols contained in the ASDF system @@ -34,14 +41,17 @@ other purposes. identified by ``NAME``; see ``DATA.GIT-SNAPSHOT`` - ``("--pgp-pubkey" . FINGERPRINT)`` means the/a OpenPGP public key with - fingerprint FINGERPRINT + fingerprint FINGERPRINT, ASCII-armoured + +- ``("--pgp-seckey" . FINGERPRINT)`` means the/a OpenPGP secret key with + fingerprint FINGERPRINT, ASCII-armoured - ``("--luks-passphrase" . VOLUME-LABEL)`` means a LUKS passphrase for volume with label ``VOLUME-LABEL``. -(Proposed convention: Except for the first item above, these reserved names -should start with ``--`` and use ``--`` to separate parameter values within -the string. Hostnames cannot start with a hyphen.) +(Proposed convention: Except for the first two items above, these reserved +names should start with ``--`` and use ``--`` to separate parameter values +within the string. Hostnames cannot start with a hyphen.) Mechanics --------- diff --git a/doc/hosts.rst b/doc/hosts.rst index 1f8a56c..25829aa 100644 --- a/doc/hosts.rst +++ b/doc/hosts.rst @@ -31,6 +31,12 @@ keyword symbols. The semantics of these attributes are documented here: - ``:HOSTNAME``: the host's hostname -- if the host has a domain name, then the FQDN, not just the part before the first dot +- ``:ALIASES``: see ``NETWORK:ALIASES`` + +- ``:IPV4``: the host's public IPv4 addresses + +- ``:IPV6``: the host's public IPv6 addresses + - ``:DATA``: items of prerequisite data required by the host - ``:OS``: the operating system of the host diff --git a/doc/ideas.rst b/doc/ideas.rst index 688d040..a98fa42 100644 --- a/doc/ideas.rst +++ b/doc/ideas.rst @@ -12,14 +12,6 @@ Properties Connections ----------- -- :SBCL could (fork and) SAVE-LISP-AND-DIE. That way, we have something that - a cronjob can call to re-run the deployment to ensure that all properties - remain applied. Need to think about how the property which sets up the - cronjob will be specified in consfigs -- does it make sense to allow passing - in arbitrary deployments, or do we only allow re-running exactly the same - thing? If the former, the saved image will need to take some sort of - command line input telling it what arguments to pass to DEPLOY*. - - Basic infrastructure for connections which work with just input and output streams connected to an interactive POSIX sh somewhere, like TRAMP, and probably using ``base64 -d`` for WRITEFILE. Probably the basic connection @@ -35,13 +27,12 @@ Connections temporary name so that rsync can do an incremental update, and then rename the file to the new version. -Data sources ------------- - -- It might be useful to have a data source which can just provide a single - item of data when registered. Then in your consfig you can just register - this data source to make a particular file you have on your system available - to deployments. +- It would sometimes be useful to have the SSH connection pass + ``-oHostName=<known IP address>`` when ``NETWORK:IPV4`` and/or + ``NETWORK:IPV6`` have been specified for the host, so that DNS propagation + is less likely to get in the way of configuring the host. Some hosts' SSH + daemons might only be accessible over VPNs and the like, however, so it will + need to be easy to override this. Core ---- @@ -54,10 +45,11 @@ Core - A CONCURRENTLY combinator for property application specifications, which means to apply each of the enclosed properties in parallel. Particularly useful surrounding a set of DEPLOYS applications, to concurrently deploy a - number of hosts. We use ``WITH-CURRENT-DIRECTORY`` in various places, so we - may not be able to do this using threads. But if we want to do it with lots - of forking, then practically speaking usage of this combinator will be - restricted to connection chains which start up remote Lisp images. + number of hosts. Now that we don't call fork(2) while executing + deployments, we ought to be able to do this using threads, and so it can + work in the root Lisp too. However, we still use ``WITH-CURRENT-DIRECTORY`` + in various places. Perhaps that macro could be changed to only affect RUN, + MRUN etc. for the sake of enabling multithreading. - It might be useful to have a restart for the case where an attempt is made to apply a list of properties containing some ``:LISP`` properties with a diff --git a/doc/index.rst b/doc/index.rst index 680f406..2c62559 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -8,6 +8,7 @@ Consfigurator user's manual introduction installation tutorial/disk_image + tutorial/os_installation connections properties hosts diff --git a/doc/installation.rst b/doc/installation.rst index 4a2d0b2..c71836c 100644 --- a/doc/installation.rst +++ b/doc/installation.rst @@ -6,7 +6,7 @@ Installation Debian and Debian derivatives ----------------------------- -The most recent tagged release of Consfigurator is included in `Debian +The most recent tagged release of Consfigurator is usually included in `Debian unstable`_, and the .deb there should work fine on Debian stable and testing, and derivatives like Ubuntu. After adding an apt source for unstable, if necessary, ``apt-get install cl-consfigurator/unstable``. diff --git a/doc/introduction.rst b/doc/introduction.rst index 656e7eb..b6d2f0c 100644 --- a/doc/introduction.rst +++ b/doc/introduction.rst @@ -83,7 +83,7 @@ Try it out / quick start ;; These two properties are not for debootstrap(1) for but apt ;; inside the chroot. (apt:uses-parent-proxy) ; use the apt-cacher-ng set up outside chroot - (apt:uses-parent-mirror))) ; use the apt mirror set up above + (apt:uses-parent-mirrors))) ; use the apt mirror set up above Here, "spwhitton" is my username on athena; we have to tell Consfigurator what user it will be when it tries to sudo, so it knows whose password it @@ -290,10 +290,9 @@ Portability and stability - All of the code in the core library should be portable ANSI Common Lisp, though optional packages providing properties and connection types might use - implementation-specific functionality. There is one exception: we require - an implementation of ``MACROEXPAND-ALL``, but most Lisps in use today - provide this. Little to no testing is done by the author on implementations - other than SBCL, so testing and portability patches are welcome. + implementation-specific functionality. Little to no testing is done by the + author on implementations other than SBCL, so testing and portability + patches are welcome. - Little attempt is made by the author to support systems other than Debian GNU/Linux, but again, portability patches are welcome, and the design of diff --git a/doc/pitfalls.rst b/doc/pitfalls.rst index 7a7bbae..b0610eb 100644 --- a/doc/pitfalls.rst +++ b/doc/pitfalls.rst @@ -48,3 +48,34 @@ serialisable, so you can't pass anonymous functions or objects containing those. You can work around the latter restriction by defining a new property which passes in the desired anonymous function, and then adding the new property to your property application specification. + +Code-walking limitations +------------------------ + +The preprocessing of propspecs, and the conversion of unevaluated propspecs +into propspecs, both require code walking. Consfigurator's implementation of +this is in the function ``MAP-PROPSPEC-PROPAPPS``. However, due to +limitations in the Common Lisp standard, it is not possible to implement the +work of that function in a way which is both always correct and fully +portable. I have not found a general purpose code walker which hooks into +implementation-specific functionality and that is currently maintained, and so +at present we use a best-effort portable code walker, Agnostic Lizard. + +This will almost always generate the correct expansions, but if you have +particularly advanced macro property combinators then it is possible that +``MAP-PROPSPEC-PROPAPPS`` will return incorrectly expanded forms. For full +details see Michael Raskin. 2017. "Writing a best-effort portable code +walker in Common Lisp." In *Proceedings of 10th European Lisp Symposium*, +Vrije Universiteit Brussel, Belgium, April 2017 (ELS2017). DOI: +10.5281/zenodo.3254669. + +It is possible to implement the work of ``MAP-PROPSPEC-PROPAPPS`` in terms of +``MACROEXPAND-ALL``, whose semantics are conventionally well-understood and +for which fully correct implementations are available in most implementations +of Common Lisp (the trivial-macroexpand-all library can be used to get at +these). However, note that we cannot just call ``MACROEXPAND-ALL`` on +propspecs because unquoted lists appearing as arguments to properties in +atomic property applications will look like invalid function calls to the code +walker. Avoiding this would seem to require wrapping the propspec in one +macrolet for each known property, and this makes ``MACROEXPAND-ALL`` too slow, +even if the macrolet forms are precomputed. diff --git a/doc/properties.rst b/doc/properties.rst index b21f51b..3273da6 100644 --- a/doc/properties.rst +++ b/doc/properties.rst @@ -7,8 +7,9 @@ Names The names of properties may not end in the character ``.``, because that has a special meaning in unevaluated property application specifications. -Properties occupy the function cells of symbols, so do not try to define an -ordinary function with the same name as a property. +Properties with ``:APPLY`` subroutines occupy the function cells of symbols, +so except in the case of properties with no ``:APPLY`` subroutine, do not try +to define an ordinary function with the same name as a property. Working directories ------------------- diff --git a/doc/tutorial/os_installation.rst b/doc/tutorial/os_installation.rst new file mode 100644 index 0000000..7c114f0 --- /dev/null +++ b/doc/tutorial/os_installation.rst @@ -0,0 +1,80 @@ +Tutorial: OS installation +========================= + +Consfigurator implements at least the basic elements of a number of methods +for installing operating systems. + +.. include:: conventions.rst + +Build and write out a raw disk image +------------------------------------ + +This is the simplest method, and Consfigurator has decent support built-in: +see the previous tutorial. It is less practical for systems which have large +disks and/or complex, nested partitioning schemes, such as ext4 on LVM on +LUKS, as is common for GNU/Linux laptops. In such cases it is nontrivial to +expand the partitions to fill the whole physical disk after the first +successful boot, so the disk image has to be the same size as the target disk, +which can be unwieldy. + +Live replacement of provider cloud images +----------------------------------------- + +See the docstring of the INSTALLER:CLEANLY-INSTALLED-ONCE property. This is +an efficient way to handle machines in faraway datacentres. Consfigurator's +support for installing Debian stable this way has been fairly well tested, and +the technique should work for other operating systems too, once Consfigurator +has been taught how to bootstrap them. + +Build a specialised live image +------------------------------ + +This third approach is more experimental; Consfigurator has all the necessary +capabilities, at least for Debian, but at present you'll need to string them +together yourself in your consfig. With this approach you build a live image +containing everything you need to run Consfigurator on the hardware to which +you want to install. After booting up the live system, you can either run +Consfigurator manually, or you can set things up to have it run automatically +upon boot. + +Consfigurator's ability to bootstrap fresh root filesystems typically requires +Internet access, but an alternative is to build and customise a chroot +corresponding to the root filesystem of the target system, and include that in +the live image, such that after boot Consfigurator just needs to partition the +disk, copy in the contents of the prebuilt chroot, and update /etc/fstab and +/etc/crypttab with UUIDs. Here is a sketch of how to do something like that:: + + (try-register-data-source + :git-snapshot :name "consfig" + :repo #P"src/cl/consfig/" :depth 1 :branch "master") + + (defproplist live-installer-built-for :lisp (with-chroot-for) + "Build a custom Debian Live system at /srv/live/installer.iso. + + Typically this property is not applied in a DEFHOST form, but rather run as + needed at the REPL. The reason for this is that otherwise the whole image will + get rebuilt each time a commit is made to ~/src/cl/consfig/." + (:desc "Debian Live system image built") + (disk:debian-live-iso-built. nil "/srv/live/installer.iso" + (os:debian-stable "bullseye" :amd64) + (apt:installed "task-english" "live-config" "lvm2" "cryptsetup") + (git:snapshot-extracted "/etc/skel/src/cl" "consfig") + (chroot:os-bootstrapped-for + nil + (merge-pathnames (get-hostname with-chroot-for) "/srv/chroot/") + with-chroot-for))) + +Supposing we've a DEFHOST form for test.silentflame.com, on our laptop we +could then use:: + + CONSFIG> (hostdeploy-these laptop.silentflame.com + (live-installer-built-for test.silentflame.com)) + +Then once the live system has booted on the target host, you'd use the +DISK:HOST-VOLUMES-CREATED and INSTALLER:CHROOT-INSTALLED-TO-VOLUMES properties +to complete the installation. + +To prepare a live image that is capable of installing more than one system +without an Internet connection, you'd probably need to investigate including +an apt repo, or equivalent, in the live system, and point Consfigurator's OS +bootstrapping properties at that. |