path: root/blog/entry/delivering-lisp-images.mdwn
diff options
authorSean Whitton <>2021-07-21 13:23:44 -0700
committerSean Whitton <>2021-07-21 13:23:44 -0700
commitc8a308bd983dc0d477bc0baf74b30b0ccccab0bc (patch)
tree2cab5d72ef0a6b9f7de784064376ddb11c5d8ec1 /blog/entry/delivering-lisp-images.mdwn
parentd6a67474c6e643e9b9558222bd360612772c731e (diff)
new blog entry
Diffstat (limited to 'blog/entry/delivering-lisp-images.mdwn')
1 files changed, 87 insertions, 0 deletions
diff --git a/blog/entry/delivering-lisp-images.mdwn b/blog/entry/delivering-lisp-images.mdwn
new file mode 100644
index 0000000..6402660
--- /dev/null
+++ b/blog/entry/delivering-lisp-images.mdwn
@@ -0,0 +1,87 @@
+[[!meta title="Delivering Common Lisp executables using Consfigurator"]]
+[[!tag tech gnu+linux consfigurator]]
+I realised this week that my recent efforts to improve how Consfigurator makes
+the fork(2) system call have also created a way to install executables to
+remote systems which will execute arbitrary Common Lisp code. Distributing
+precompiled programs using free software implementations of the Common Lisp
+standard tends to be more of a hassle than with a lot of other high level
+programming languages. Executables will often be hundreds of megabytes in
+size even if your codebase is just a few megabytes, because the whole
+interactive Common Lisp environment gets bundled along with your program's
+code. Commercial Common Lisp implementations manage to do better, as I
+understand it, by knowing how to shake out unused code paths. Consfigurator's
+new mechanism uploads only changed source code, which might only be kilobytes
+in size, and updates the executable on the remote system. So it should be
+useful for deploying Common Lisp-powered web services, and the like.
+Here's how it works. When you use Consfigurator you define an ASDF system --
+analagous to a Python package or Perl distribution -- called your "consfig".
+This defines HOST objects to represent the machines that you'll use
+Consfigurator to manage, and any custom properties, functions those properties
+call, etc.. An ASDF system can depend upon other systems; for example, every
+consfig depends upon Consfigurator itself. When you execute Consfigurator
+deployments, Consfigurator uploads the source code of any ASDF systems that
+have changed since you last deployed this host, starts up Lisp on the remote
+machine, and loads up all the systems. Now the remote Lisp image is in a
+similarly clean state to when you've just started up Lisp on your laptop and
+loaded up the libraries you're going to use. Only then are the actual
+deployment instructions are sent on stdin.
+What I've done this week is insert an extra step for the remote Lisp image in
+between loading up all the ASDF systems and reading the deployment from stdin:
+the image calls fork(2) and establishes a pipe to communicate with the child
+process. The child process can be sent Lisp forms to evaluate, but for each
+Lisp form it receives it will actually fork again, and have *its* child
+process evaluate the form. Thus, going into the deployment, the original
+remote Lisp image has the capability to have arbitrary Lisp forms evaluated in
+a context in which all that has happened is that a statically defined set of
+ASDF systems has been loaded -- the child processes never see the full
+deployment instructions sent on stdin. Further, the child process responsible
+for actually evaluating the Lisp form received from the first process first
+forks off another child process and sets up its own control pipe, such that it
+too has the capacbility to have arbitrary Lisp forms evaluated in a cleanly
+loaded context, no matter what else it might put in its memory in the
+meantime. (Things are set up such that the child processes responsible for
+actually evaluating the Lisp forms never see the Lisp forms received for
+evaluation by other child processes, either.)
+So suppose now we have an ASDF system ````,
+and there is a function ``(start-server PORT)`` which we should call to start
+listening for connections. Then we can make our consfig depend upon that ASDF
+system, and do something like this:
+``` {.lisp}
+CONSFIG> (deploy-these ((:ssh :user "root") :sbcl)
+ ;; Set up Apache to proxy requests to our service.
+ (apache:https-vhost ...)
+ ;; Now apply a property to dump the image.
+ (image-dumped "/usr/local/bin/cool-web-service"
+ '(cool-web-service:start-server 1234)))
+Consfigurator will: SSH to; upload all the ASDF source for
+your consfig and its dependencies; compile and load that code into a remote
+SBCL process; call fork(2) and set up the control pipe; receive the
+applications of APACHE:HTTPS-VHOST and IMAGE-DUMPED shown above from your
+laptop, on stdin; apply the APACHE:HTTPS-VHOST property to ensure that Apache
+is proxying connections to port 1234; send a request into the control pipe to
+have the child process fork again and dump an executable which, when started,
+will evaluate the form ``(cool-web-service:start-server 1234)``. And that
+form will get evaluated in a pristine Lisp image, where the only meaningful
+things that have happened is that some ASDF systems have been loaded and a
+single fork(2) has taken place. You'd probably need to add some other
+properties to add some mechanism for actually invoking
+``/usr/local/bin/cool-web-service`` and restarting it when the executable is
+(Background: The primary reason why Consfigurator's remote Lisp images need to
+call fork(2) is that they need to do things like setuid from root to other
+accounts and enter chroots without getting stuck in those contexts.
+Previously we forked right before entering such contexts, but that meant that
+Consfigurator deployments could never be multithreaded, because it might later
+be necessary to fork, and you can't usually do that once you've got more than
+one thread running. So now we fork before doing anything else, so that the
+parent can then go multithreaded if desired, but can still execute
+subdeployments in contexts like chroots by sending Lisp forms to evaluate in
+those contexts into the control pipe.)