Add portable dumper

Add a new portable dumper as an alternative to unexec.  Use it by default.

* src/dmpstruct.awk: New file.
* src/doc.c (get_doc_string): use will_dump_p().
* src/editfns.c (styled_format): silence compiler warning
with UNINIT.
* src/emacs-module.c (syms_of_module): staticpro ltv_mark.
* src/emacs.c (gflags): new variable.
(init_cmdargs): unwrap
(string_starts_with_p, find_argument, dump_error_to_string)
(load_pdump): new functions.
(main): detect pdumper and --temacs invocation; actually load
portable dump when detected; set gflags as appropriate; changes to
init functions throughout to avoid passing explicit
'initialized' argument.
* src/eval.c (inhibit_lisp_code): remove unused variable.
(init_eval_once_for_pdumper): new function.
(init_eval_once): call it.
* src/filelock.c: CANNOT_DUMP -> will_dump_p()
* src/fingerprint-dummy.c: new file
* src/fingerprint.h: new file
* src/fns.c: CANNOT_DUMP -> will_dump_p(), etc.
(weak_hash_tables): remove
(hashfn_equal, hashfn_eql): un-staticify
(make_hash_table): set new 'next_weak' hash table field; drop
global weak_hash_tables logic.
(copy_hash_table): drop global weak_hash_tables logic.
(hash_table_rehash): new function.
(hash_lookup, hash_put, hash_remove_from_table, hash_clear):
rehash if needed.
(sweep_weak_table): un-staticify; explain logic; bool-ify.
(sweep_weak_hash_tables): remove function.
* src/font.c (syms_of_font): remember pdumper stuff.
* src/fontset.c (syms_of_fontset): remember pdumper stuff.
* src/frame.c (make_initial_frame): don't reset Vframe_list.
(init_frame_once_for_pdumper, init_frame_once): new functions.
(syms_of_frame): remove redundant staticpro.
* src/fringe.c (init_fringe_once_for_pdumper): new functin.
(init_fringe_once): call it.
* src/ftcrfont.c (syms_of_ftcrfont_for_pdumper): new function.
(syms_of_ftcrfont): call it.
* src/ftfont.c (syms_of_ftfont_for_pdumper): new function.
(syms_of_ftfont): call it.
* src/ftxont.c (syms_of_ftxfont_for_pdumper): new function.
(syms_of_ftxfont): call it.
* src/gmalloc.c: adjust for pdumper througout
(DUMPED): remove weird custom dumped indicator.
* src/gnutls.c (syms_of_gnutls): pdumper note for
gnutls_global_initialized.
* src/image.c (syms_of_image): add pdumper comment,
initializer note.
* src/insdel.c (prepare_to_modify_buffer_1): account
for buffer contents possibly being in dump image.
* src/keyboard.c (syms_of_keyboard_for_pdumper): new function.
(syms_of_keyboard): staticpro more; call pdumper syms function.
* src/lisp.h: add comments throughout
(gflags): declare.
(will_dump_p, will_bootstrap_p, will_dump_with_pdumper_p)
(dumped_with_pdumper_p, will_dump_with_unexec_p)
(dumped_with_unexec_p, definitely_will_not_unexec_p): new
functions.
(POWER_OF_2, ROUNDUP): move macros.
(PSEUDOVECTOR_TYPE, PSEUDOVECTOR_TYPEP): take vectorlike header
pointer instead of vector; constify.
(Lisp_Hash_Table): add comment about need to rehash on access; add
comment for next_weak.
(HASH_KEY, HASH_VALUE, HASH_HASH, HASH_TABLE_SIZE): const-ify.
(hash_table_rehash): declare.
(hash_rehash_needed_p, hash_rehash_if_needed): new functions.
(finalizers, doomed_finalizers): declare extern.
(SUBR_SECTION_ATTRIBUTE): new macro.
(staticvec, staticidx): un-static-ify.
(sweep_weak_hash_tables): remove declaration.
(sweep_weak_table): declare.
(hashfn_eql, hashfn_equal): declare.
(number_finalizers_run): new variable.
(Vdead): externify when ENABLE_CHECKING.
(gc_root_type): new enumeration.
(gc_root_visitor): new struct.
(visit_static_gc_roots): declare.
(vectorlike_nbytes): declare.
(vector_nbytes): define as trivial inline function wrapper for
vectorlike_nbytes.
(init_obarray_once): change signature.
(primary_thread): extern-ify.
(init_buffer): change signature.
(init_frame_once): declare.
* src/lread.c (readevalloop): adjust for new dumped predicates.
(init_obarray_once): new function.
(ndefsubr): new variable.
(defsubr): increment it.
(load_path_check): adjust for pdumper.
(load_path_default): use pdumper functions; adjust for
dump search.
* src/macfont.m (macfont_init_font_change_handler): avoid
shadowing global.
(syms_of_macfont_for_pdumper): new function.
(syms_of_macfont): call it.
* src/menu.c (syms_of_menu): staticpro more stuff.
* src/minibuf.c (Ftry_completion): rehash if needed.
(init_minibuf_once_for_pdumper): new function.
(init_minibuf_once): call it.
* src/nsfont.m (syms_of_nsfns): staticpro more.
* src/nsfont.m (syms_of_nsfont_for_pdumper): new function.
(syms_of_nsfont): call it.
* src/nsterm.m (syms_of_nsfont): remember pdumper stuff.
* src/pdumper.c: new file.
* src/pdumper.h: new file.
* src/process.c (init_process_emacs): use new pdumper functions
instead of CANNOT_DUMP.
* src/profiler.c (syms_of_profiler_for_pdumper): new function.
(syms_of_profiler_for_pdumper): call it.
* src/search.c (syms_of_search_for_pdumper): new function.
(syms_of_search_for_pdumper): call it.
* src/sheap.c (bss_sbrk_did_unexec): remove.
* src/sheap.h (bss_sbrk_did_unexec): remove.
* src/syntax.c (syms_of_syntax): don't redundantly staticpro
re_match_object.
* src/sysdep.c: use will_dump_with_unexec_p() instead of bss
hack thing.
* src/syssignals.h (init_sigsegv): declare.
* src/systime.h (init_timefns): remove bool from signature.
* src/textprop.c (syms_of_textprop): move staticpro.
* src/thread.c (main_thread_p): constify.
* src/thread.h (main_thread_p): constify.
* src/timefns.c (init_timefns): remove bool from signature.
(syms_of_timefns_for_pdumper): new function.
(syms_of_timefns): call it.
* src/w32.c: rearrange code.
* src/w32.h (w32_relocate): declare.
* src/w32fns.c (syms_of_w32fns): add pdumper note.
* src/w32font.c (syms_of_w32font_for_pdumper): new function.
(syms_of_w32font): call it.
* src/w32heap.c (using_dynamic_heap): new variable.
(init_heap): use it.
* src/w32menu.c (syms_of_w32menu): add pdumper note.
* src/w32proc.c
(ctrl_c_handler, mainCRTStartup, _start, open_input_file)
(rva_to_section, close_file_data): move here.
* src/w32uniscribe.c (syms_of_w32uniscribe_for_pdumper):
new function.
(syms_of_w32uniscribe): call it.
* src/window.c (init_window_once_for_pdumper): new function.
(init_window_once): call it; staticpro more stuff.
* src/xfont.c (syms_of_xfont_for_pdumper): new function.
(syms_of_xfont): call it.
* src/xftfont.c (syms_of_xftfont_for_pdumper): new function.
(syms_of_xftfont): call it.
* src/xmenu.c (syms_of_xmenu_for_pdumper): new function.
(syms_of_xmenu): call it.
* src/xselect.c (syms_of_xselect_for_pdumper): new function.
(syms_of_xselect): call it.
* src/xsettings.c (syms_of_xsettings): add more pdumper notes.
* src/term.c (syms_of_xterm): add pdumper note.

* src/dispnew.c (init_faces_initial): new function.
(init_display_interactive): rename from init_display; use
will_dump_p instead of !initialized.  Initialize faces early for
pdumper if needed.
(init_display): new function.
(syms_of_display_for_pdumper): new function.
(syms_of_display): call it.

* src/dbusbind.c (syms_of_dbusbind): Add TODO for bus reset
on pdumper load.

* src/data.c (Fdefalias): Use will_dump_p
instead of Vpurify_flag.
(Fmake_variable_buffer_local): silence compiler warning with -Og
by making valcontents UNINIT.
(arith_driver): silence compiler warning with UNINIT.

* src/conf_post.h (ATTRIBUTE_SECTION): new macro.

* src/composite.c (composition_gstring_put_cache): rehash hash
table if needed.

* src/coding.c (init_coding_once, syms_of_coding): remember
pdumper stuff.

* src/charset.h (charset_table_size, charset_table_user): declare.

* src/charset.c (charset_table_used, charset_table_size): un-static.
(init_charset_oncem, syms_of_charset): remember pdumper stuff.

* src/category.c (category_table_version): remove obsolete
variable.

* src/callint.c (syms_of_callint): staticpro 'preserved_fns'
(init_callproc): use will_dump_p instead of !CANNOT_DUMP.

* src/bytecode.c (exec_byte_code): rehash table tables if needed

* src/buffer.c (alloc_buffer_text, free_buffer_text): account for
pdumper
(init_buffer_once): add TODO; remember stuff for pdumper.
(init_buffer): don't take initialized argument; adjust
for pdumper.

* src/atimer.c (init_atimer): initialize subr only if
!initialized.

* src/alloc.c: (vector_marked_p, set_vector_marked)
(vectorlike_marked_p, set_vectorlike_marked, cons_marked_p)
(set_cons_marked, string_marked_p, set_string_marked)
(symbol_marked_p, set_symbol_marked, interval_marked_p)
(set_interval_marked): new accessor routines.  Use them
instead of raw GC access throughout.
(Vdead): make non-static when ENABLE_CHECKING.
(vectorlike_nbytes): rename of 'vector_nbytes'; take a vectorlike
header as input instead of a vector.
(number_finalizers_run): new internal C variable.
(mark_maybe_object): check for pdumper objects.
(valid_pointer_p): don't be gratuitously inefficient under rr(1).
(make_pure_c_string): add support for size_byte = -2 mode
indicating that string data points into Emacs image rodata.
(visit_vectorlike_root): visits GC roots embedded in
vectorlike objects.
(visit_buffer_root): visits GC roots embedded in
our totally-not-a-buffer buffer global objects.
(visit_static_gc_roots): visit GC roots in the Emacs data section.
(mark_object_root_visitor): root callback used for conventional GC
marking
(weak_hash_tables): new internal variable for tracking found weak
hash tables during GC.
(mark_and_sweep_weak_table_contents): new weak hash table marking.
(garbage_collect_1): use new GC root visitor machinery.
(mark_vectorlike): accept a vectorlike_header instead of a
Lisp_Vector.
(mark_frame, mark_window, mark_hash_table): new functions.
(mark_object): initialize 'm'; check for pdumper objects and use
new mark-bit accessors throughout.  Remove some object-specific
marking code and move to helper functions above.
(survives_gc_p): check for pdumper objects.
(gc-sweep): clear pdumper mark bits.
(init_alloc_once_for_pdumper): new helper function for early init
called both during normal init and pdumper load.
(init_alloc_once): pdumper integration.

* src/Makefile.in: Rewrite dumping for pdumper; add pdumper.o;
invoke temacs with --temacs command line option; build dmpstruct.h
from dmpstruct.awk; stop relying on CANNOT_DUMP; clean up pdumper
intermediate files during build.

* nextstep/Makefile.in: build emacs.pdmp into NS packages

* lisp/startup.el: account for new '--temacs' and '--dump-file'
command line option.

* lisp/loadup.el: rewrite early init to account for pdumper; use
injected 'dump-mode' variable (set via the new '--temacs' option)
instead of parsing command line.

* lisp/cus-start.el: Check 'dump-mode' instead of 'purify-flag',
since the new 'dump-mode'

* lib-src/make-fingerprint.c: new program

* lib-src/Makefile.in: built make-fingerprint utility program

* configure.ac: Add --with-pdumper toggle to control pdumper
support; add --with-unexec toggle to control unexec support.
Add --with-dumping option to control which dumping strategy we use
by default.  Adjust for pdumper throughout.  Check for
posix_madvise.

* Makefile.in: Add @DUMPING@ substitution; add pdumper mode.

* .gitignore: Add make-fingerprint, temacs.in, fingerprint.c,
dmpstruct.h, and pdumper dump files.

1 files changed, 267 insertions, 0 deletions

diff --git a/src/pdumper.h b/src/pdumper.h
new file mode 100644
index 00000000000..8ed4fc1cb3b
--- /dev/null
+++ b/src/pdumper.h
@@ -0,0 +1,267 @@
+/* Header file for the portable dumper.
+
+Copyright (C) 2016 Free Software Foundation,
+Inc.
+
+This file is part of GNU Emacs.
+
+GNU Emacs is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or (at
+your option) any later version.
+
+GNU Emacs is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
+
+#ifndef EMACS_PDUMPER_H
+#define EMACS_PDUMPER_H
+
+#include "lisp.h"
+
+INLINE_HEADER_BEGIN
+
+#define PDUMPER_NO_OBJECT ((enum Lisp_Type) -1)
+
+/* Indicate in source code that we're deliberately relying on pdumper
+   not preserving the given value.  Compiles to nothing --- for humans
+   only.  */
+#define PDUMPER_IGNORE(thing) ((void) &(thing))
+
+/* The portable dumper automatically preserves the Lisp heap and any C
+   variables to which the Lisp heap points.  It doesn't know anything
+   about other C variables.  The functions below allow code from other
+   parts of Emacs to tell the portable dumper about other bits of
+   information to preserve in dumped images.
+
+   These memory-records are themselves preserved in the dump, so call
+   the functions below only on the !initialized init path, just
+   like staticpro.
+
+   There are no special functions to preserve a global Lisp_Object.
+   You should just staticpro these.  */
+
+/* Remember the value of THING in dumped images.  THING must not
+   contain any pointers or Lisp_Object variables: these values are not
+   valid across dump and load.  */
+#define PDUMPER_REMEMBER_SCALAR(thing)                  \
+  pdumper_remember_scalar (&(thing), sizeof (thing))
+
+extern void pdumper_remember_scalar_impl (void *data, ptrdiff_t nbytes);
+
+INLINE
+void
+pdumper_remember_scalar (void *data, ptrdiff_t nbytes)
+{
+#ifdef HAVE_PDUMPER
+  pdumper_remember_scalar_impl (data, nbytes);
+#else
+  (void) data;
+  (void) nbytes;
+#endif
+}
+
+extern void pdumper_remember_lv_ptr_raw_impl (
+  void *ptr, enum Lisp_Type type);
+
+/* Remember the pointer at *PTR.  *PTR must be null or point to a Lisp
+   object.  TYPE is the rough type of Lisp object to which *PTR
+   points.  */
+INLINE
+void
+pdumper_remember_lv_ptr_raw (void* ptr, enum Lisp_Type type)
+{
+#ifdef HAVE_PDUMPER
+  pdumper_remember_lv_ptr_raw_impl (ptr, type);
+#else
+  (void) ptr;
+  (void) type;
+#endif
+}
+
+typedef void (*pdumper_hook)(void);
+extern void pdumper_do_now_and_after_load_impl (pdumper_hook hook);
+
+INLINE void
+pdumper_do_now_and_after_load (pdumper_hook hook)
+{
+#ifdef HAVE_PDUMPER
+  pdumper_do_now_and_after_load_impl (hook);
+#else
+  hook ();
+#endif
+}
+
+/* Macros useful in pdumper callback functions.  Assign a value if
+   we're loading a dump and the value needs to be reset to its
+   original value, and if we're initializing for the first time,
+   assert that the value has the expected original value.  */
+
+#define PDUMPER_RESET(variable, value)         \
+  do {                                         \
+    if (dumped_with_pdumper_p ())              \
+      (variable) = (value);                    \
+    else                                       \
+      eassert ((variable) == (value));         \
+  } while (0)
+
+#define PDUMPER_RESET_LV(variable, value)         \
+  do {                                            \
+    if (dumped_with_pdumper_p ())                 \
+      (variable) = (value);                       \
+    else                                          \
+      eassert (EQ ((variable), (value)));         \
+  } while (0)
+
+/* Actually load a dump.  */
+
+enum pdumper_load_result
+  {
+    PDUMPER_LOAD_SUCCESS,
+    PDUMPER_NOT_LOADED /* Not returned: useful for callers */,
+    PDUMPER_LOAD_FILE_NOT_FOUND,
+    PDUMPER_LOAD_BAD_FILE_TYPE,
+    PDUMPER_LOAD_FAILED_DUMP,
+    PDUMPER_LOAD_OOM,
+    PDUMPER_LOAD_VERSION_MISMATCH,
+    PDUMPER_LOAD_ERROR,
+  };
+
+enum pdumper_load_result pdumper_load (const char *dump_filename);
+
+struct pdumper_loaded_dump {
+  uintptr_t start;
+  uintptr_t end;
+};
+
+#ifdef HAVE_PDUMPER
+extern struct pdumper_loaded_dump dump_public;
+#endif
+
+/* Return whether the OBJ points somewhere into the loaded dump image.
+   Works even when we have no dump loaded --- in this case, it just
+   returns false.  */
+INLINE _GL_ATTRIBUTE_CONST
+bool
+pdumper_object_p (const void *obj)
+{
+#ifdef HAVE_PDUMPER
+  uintptr_t obj_addr = (uintptr_t) obj;
+  return dump_public.start <= obj_addr && obj_addr < dump_public.end;
+#else
+  (void) obj;
+  return false;
+#endif
+}
+
+extern bool pdumper_cold_object_p_impl (const void *obj);
+
+/* Return whether the OBJ is in the cold section of the dump.
+   Only bool-vectors and floats should end up there.
+   pdumper_object_p() and pdumper_object_p_precise() must have
+   returned true for OBJ before calling this function.  */
+INLINE _GL_ATTRIBUTE_CONST
+bool
+pdumper_cold_object_p (const void *obj)
+{
+#ifdef HAVE_PDUMPER
+  return pdumper_cold_object_p_impl (obj);
+#else
+  (void) obj;
+  return false;
+#endif
+}
+
+
+extern enum Lisp_Type pdumper_find_object_type_impl (const void *obj);
+
+/* Return the type of the dumped object that starts at OBJ.  It is a
+   programming error to call this routine for an OBJ for which
+   pdumper_object_p would return false.  */
+INLINE _GL_ATTRIBUTE_CONST
+enum Lisp_Type
+pdumper_find_object_type (const void *obj)
+{
+#ifdef HAVE_PDUMPER
+  return pdumper_find_object_type_impl (obj);
+#else
+  (void) obj;
+  emacs_abort ();
+#endif
+}
+
+/* Return whether OBJ points exactly to the start of some object in
+   the loaded dump image.  It is a programming error to call this
+   routine for an OBJ for which pdumper_object_p would return
+   false.  */
+INLINE _GL_ATTRIBUTE_CONST
+bool
+pdumper_object_p_precise (const void *obj)
+{
+#ifdef HAVE_PDUMPER
+  return pdumper_find_object_type (obj) != PDUMPER_NO_OBJECT;
+#else
+  (void) obj;
+  emacs_abort ();
+#endif
+}
+
+extern bool pdumper_marked_p_impl (const void *obj);
+
+/* Return whether OBJ is marked according to the portable dumper.
+   It is an error to call this routine for an OBJ for which
+   pdumper_object_p_precise would return false.  */
+INLINE
+bool
+pdumper_marked_p (const void *obj)
+{
+#ifdef HAVE_PDUMPER
+  return pdumper_marked_p_impl (obj);
+#else
+  (void) obj;
+  emacs_abort ();
+#endif
+}
+
+extern void pdumper_set_marked_impl (const void *obj);
+
+/* Set the pdumper mark bit for OBJ.  It is a programming error to
+   call this function with an OBJ for which pdumper_object_p_precise
+   would return false.  */
+INLINE
+void
+pdumper_set_marked (const void *obj)
+{
+#ifdef HAVE_PDUMPER
+  pdumper_set_marked_impl (obj);
+#else
+  (void) obj;
+  emacs_abort ();
+#endif
+}
+
+extern void pdumper_clear_marks_impl (void);
+
+/* Clear all the mark bits for pdumper objects.  */
+INLINE
+void
+pdumper_clear_marks (void)
+{
+#ifdef HAVE_PDUMPER
+  pdumper_clear_marks_impl ();
+#endif
+}
+
+/* Handle a page fault that occurs when we access the portable dumper
+   mapping.  Return true iff the fault should be considered handled
+   and execution should resume.  */
+bool pdumper_handle_page_fault (void *fault_addr_ptr);
+
+void syms_of_pdumper (void);
+
+INLINE_HEADER_END
+#endif