diff options
| -rw-r--r-- | blog/entry/delivering-lisp-images.mdwn | 87 |
1 files changed, 87 insertions, 0 deletions
diff --git a/blog/entry/delivering-lisp-images.mdwn b/blog/entry/delivering-lisp-images.mdwn new file mode 100644 index 0000000..6402660 --- /dev/null +++ b/blog/entry/delivering-lisp-images.mdwn @@ -0,0 +1,87 @@ +[[!meta title="Delivering Common Lisp executables using Consfigurator"]] +[[!tag tech gnu+linux consfigurator]] + +I realised this week that my recent efforts to improve how Consfigurator makes +the fork(2) system call have also created a way to install executables to +remote systems which will execute arbitrary Common Lisp code. Distributing +precompiled programs using free software implementations of the Common Lisp +standard tends to be more of a hassle than with a lot of other high level +programming languages. Executables will often be hundreds of megabytes in +size even if your codebase is just a few megabytes, because the whole +interactive Common Lisp environment gets bundled along with your program's +code. Commercial Common Lisp implementations manage to do better, as I +understand it, by knowing how to shake out unused code paths. Consfigurator's +new mechanism uploads only changed source code, which might only be kilobytes +in size, and updates the executable on the remote system. So it should be +useful for deploying Common Lisp-powered web services, and the like. + +Here's how it works. When you use Consfigurator you define an ASDF system -- +analagous to a Python package or Perl distribution -- called your "consfig". +This defines HOST objects to represent the machines that you'll use +Consfigurator to manage, and any custom properties, functions those properties +call, etc.. An ASDF system can depend upon other systems; for example, every +consfig depends upon Consfigurator itself. When you execute Consfigurator +deployments, Consfigurator uploads the source code of any ASDF systems that +have changed since you last deployed this host, starts up Lisp on the remote +machine, and loads up all the systems. Now the remote Lisp image is in a +similarly clean state to when you've just started up Lisp on your laptop and +loaded up the libraries you're going to use. Only then are the actual +deployment instructions are sent on stdin. + +What I've done this week is insert an extra step for the remote Lisp image in +between loading up all the ASDF systems and reading the deployment from stdin: +the image calls fork(2) and establishes a pipe to communicate with the child +process. The child process can be sent Lisp forms to evaluate, but for each +Lisp form it receives it will actually fork again, and have *its* child +process evaluate the form. Thus, going into the deployment, the original +remote Lisp image has the capability to have arbitrary Lisp forms evaluated in +a context in which all that has happened is that a statically defined set of +ASDF systems has been loaded -- the child processes never see the full +deployment instructions sent on stdin. Further, the child process responsible +for actually evaluating the Lisp form received from the first process first +forks off another child process and sets up its own control pipe, such that it +too has the capacbility to have arbitrary Lisp forms evaluated in a cleanly +loaded context, no matter what else it might put in its memory in the +meantime. (Things are set up such that the child processes responsible for +actually evaluating the Lisp forms never see the Lisp forms received for +evaluation by other child processes, either.) + +So suppose now we have an ASDF system ``:com.silentflame.cool-web-service``, +and there is a function ``(start-server PORT)`` which we should call to start +listening for connections. Then we can make our consfig depend upon that ASDF +system, and do something like this: + +``` {.lisp} +CONSFIG> (deploy-these ((:ssh :user "root") :sbcl) server.example.org + ;; Set up Apache to proxy requests to our service. + (apache:https-vhost ...) + ;; Now apply a property to dump the image. + (image-dumped "/usr/local/bin/cool-web-service" + '(cool-web-service:start-server 1234))) +``` + +Consfigurator will: SSH to server.example.org; upload all the ASDF source for +your consfig and its dependencies; compile and load that code into a remote +SBCL process; call fork(2) and set up the control pipe; receive the +applications of APACHE:HTTPS-VHOST and IMAGE-DUMPED shown above from your +laptop, on stdin; apply the APACHE:HTTPS-VHOST property to ensure that Apache +is proxying connections to port 1234; send a request into the control pipe to +have the child process fork again and dump an executable which, when started, +will evaluate the form ``(cool-web-service:start-server 1234)``. And that +form will get evaluated in a pristine Lisp image, where the only meaningful +things that have happened is that some ASDF systems have been loaded and a +single fork(2) has taken place. You'd probably need to add some other +properties to add some mechanism for actually invoking +``/usr/local/bin/cool-web-service`` and restarting it when the executable is +updated. + +(Background: The primary reason why Consfigurator's remote Lisp images need to +call fork(2) is that they need to do things like setuid from root to other +accounts and enter chroots without getting stuck in those contexts. +Previously we forked right before entering such contexts, but that meant that +Consfigurator deployments could never be multithreaded, because it might later +be necessary to fork, and you can't usually do that once you've got more than +one thread running. So now we fork before doing anything else, so that the +parent can then go multithreaded if desired, but can still execute +subdeployments in contexts like chroots by sending Lisp forms to evaluate in +those contexts into the control pipe.) |