summaryrefslogtreecommitdiffhomepage
path: root/blog/entry/delivering-lisp-images.mdwn
blob: 6402660fd3b2befcb1eb932441ae7e5cbd327cb7 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
[[!meta title="Delivering Common Lisp executables using Consfigurator"]]
[[!tag  tech gnu+linux consfigurator]]

I realised this week that my recent efforts to improve how Consfigurator makes
the fork(2) system call have also created a way to install executables to
remote systems which will execute arbitrary Common Lisp code.  Distributing
precompiled programs using free software implementations of the Common Lisp
standard tends to be more of a hassle than with a lot of other high level
programming languages.  Executables will often be hundreds of megabytes in
size even if your codebase is just a few megabytes, because the whole
interactive Common Lisp environment gets bundled along with your program's
code.  Commercial Common Lisp implementations manage to do better, as I
understand it, by knowing how to shake out unused code paths.  Consfigurator's
new mechanism uploads only changed source code, which might only be kilobytes
in size, and updates the executable on the remote system.  So it should be
useful for deploying Common Lisp-powered web services, and the like.

Here's how it works.  When you use Consfigurator you define an ASDF system --
analagous to a Python package or Perl distribution -- called your "consfig".
This defines HOST objects to represent the machines that you'll use
Consfigurator to manage, and any custom properties, functions those properties
call, etc..  An ASDF system can depend upon other systems; for example, every
consfig depends upon Consfigurator itself.  When you execute Consfigurator
deployments, Consfigurator uploads the source code of any ASDF systems that
have changed since you last deployed this host, starts up Lisp on the remote
machine, and loads up all the systems.  Now the remote Lisp image is in a
similarly clean state to when you've just started up Lisp on your laptop and
loaded up the libraries you're going to use.  Only then are the actual
deployment instructions are sent on stdin.

What I've done this week is insert an extra step for the remote Lisp image in
between loading up all the ASDF systems and reading the deployment from stdin:
the image calls fork(2) and establishes a pipe to communicate with the child
process.  The child process can be sent Lisp forms to evaluate, but for each
Lisp form it receives it will actually fork again, and have *its* child
process evaluate the form.  Thus, going into the deployment, the original
remote Lisp image has the capability to have arbitrary Lisp forms evaluated in
a context in which all that has happened is that a statically defined set of
ASDF systems has been loaded -- the child processes never see the full
deployment instructions sent on stdin.  Further, the child process responsible
for actually evaluating the Lisp form received from the first process first
forks off another child process and sets up its own control pipe, such that it
too has the capacbility to have arbitrary Lisp forms evaluated in a cleanly
loaded context, no matter what else it might put in its memory in the
meantime.  (Things are set up such that the child processes responsible for
actually evaluating the Lisp forms never see the Lisp forms received for
evaluation by other child processes, either.)

So suppose now we have an ASDF system ``:com.silentflame.cool-web-service``,
and there is a function ``(start-server PORT)`` which we should call to start
listening for connections.  Then we can make our consfig depend upon that ASDF
system, and do something like this:

``` {.lisp}
CONSFIG> (deploy-these ((:ssh :user "root") :sbcl) server.example.org
           ;; Set up Apache to proxy requests to our service.
           (apache:https-vhost ...)
		   ;; Now apply a property to dump the image.
		   (image-dumped "/usr/local/bin/cool-web-service"
		                 '(cool-web-service:start-server 1234)))
```

Consfigurator will: SSH to server.example.org; upload all the ASDF source for
your consfig and its dependencies; compile and load that code into a remote
SBCL process; call fork(2) and set up the control pipe; receive the
applications of APACHE:HTTPS-VHOST and IMAGE-DUMPED shown above from your
laptop, on stdin; apply the APACHE:HTTPS-VHOST property to ensure that Apache
is proxying connections to port 1234; send a request into the control pipe to
have the child process fork again and dump an executable which, when started,
will evaluate the form ``(cool-web-service:start-server 1234)``.  And that
form will get evaluated in a pristine Lisp image, where the only meaningful
things that have happened is that some ASDF systems have been loaded and a
single fork(2) has taken place.  You'd probably need to add some other
properties to add some mechanism for actually invoking
``/usr/local/bin/cool-web-service`` and restarting it when the executable is
updated.

(Background: The primary reason why Consfigurator's remote Lisp images need to
call fork(2) is that they need to do things like setuid from root to other
accounts and enter chroots without getting stuck in those contexts.
Previously we forked right before entering such contexts, but that meant that
Consfigurator deployments could never be multithreaded, because it might later
be necessary to fork, and you can't usually do that once you've got more than
one thread running.  So now we fork before doing anything else, so that the
parent can then go multithreaded if desired, but can still execute
subdeployments in contexts like chroots by sending Lisp forms to evaluate in
those contexts into the control pipe.)