diff options
Diffstat (limited to 'doc/misc/ert.texi')
| -rw-r--r-- | doc/misc/ert.texi | 271 |
1 files changed, 248 insertions, 23 deletions
diff --git a/doc/misc/ert.texi b/doc/misc/ert.texi index fafdb8c4eb4..71c423ad9c6 100644 --- a/doc/misc/ert.texi +++ b/doc/misc/ert.texi @@ -109,6 +109,7 @@ Appendix @end menu @end ifnottex + @node Introduction @chapter Introduction @cindex introduction to ERT @@ -123,7 +124,7 @@ commands to run them to verify whether the definitions that are currently loaded in Emacs pass the tests. Some Lisp files have comments like the following (adapted from the -package @code{pp.el}): +package @file{pp.el}): @lisp ;; (pp-to-string '(quote quote)) ; expected: "'quote" @@ -260,35 +261,103 @@ unexpected result. In the example above, there are two failures, both due to failed @code{should} forms. @xref{Understanding Explanations}, for more details. +The following key bindings are available in the ERT results buffer: + +@table @kbd + +@item @key{RET} +@kindex RET@r{, in ert results buffer} +Each name of a function or macro in this buffer is a button; moving +point to it and typing @kbd{@key{RET}} jumps to its definition. + +@item @key{TAB} +@itemx S-@key{TAB} @kindex TAB@r{, in ert results buffer} @kindex S-TAB@r{, in ert results buffer} -In the ERT results buffer, @kbd{@key{TAB}} and @kbd{S-@key{TAB}} cycle between -buttons. Each name of a function or macro in this buffer is a button; -moving point to it and typing @kbd{@key{RET}} jumps to its definition. +Cycle between buttons forward (@code{forward-button}) and backward +(@code{backward-button}). +@item r @kindex r@r{, in ert results buffer} +@findex ert-results-rerun-test-at-point +Re-run the test near point on its own +(@code{ert-results-rerun-test-at-point}). + +@item d @kindex d@r{, in ert results buffer} +@findex ert-results-rerun-test-at-point-debugging-errors +Re-run the test near point on its own with the debugger enabled +(@code{ert-results-rerun-test-at-point-debugging-errors}). + +@item R +@kindex R@r{, in ert results buffer} +@findex ert-results-rerun-all-tests +Re-run all tests (@code{ert-results-rerun-all-tests}). + +@item . @kindex .@r{, in ert results buffer} +@findex ert-results-find-test-at-point-other-window +Jump to the definition of the test near point +(@code{ert-results-find-test-at-point-other-window}). This has the +same effect as @kbd{@key{RET}}, but does not require point to be on +the name of the test. + +@item b @kindex b@r{, in ert results buffer} +@findex ert-results-pop-to-backtrace-for-test-at-point @cindex backtrace of a failed test -Pressing @kbd{r} re-runs the test near point on its own. Pressing -@kbd{d} re-runs it with the debugger enabled. @kbd{.} jumps to the -definition of the test near point (@kbd{@key{RET}} has the same effect -if point is on the name of the test). On a failed test, @kbd{b} shows -the backtrace of the failure. @xref{Debugging,, Backtraces, elisp, -GNU Emacs Lisp Reference Manual}, for more information about -backtraces. +Show the backtrace of a failed test +(@code{ert-results-pop-to-backtrace-for-test-at-point}). +@xref{Debugging,, Backtraces, elisp, GNU Emacs Lisp Reference Manual}, +for more information about backtraces. +@item l @kindex l@r{, in ert results buffer} -@kbd{l} shows the list of @code{should} forms executed in the test. -If any messages were generated (with the Lisp function @code{message}) -in a test or any of the code that it invoked, @kbd{m} will show them. - +@findex ert-results-pop-to-should-forms-for-test-at-point +Show the list of @code{should} forms executed in the test +(@code{ert-results-pop-to-should-forms-for-test-at-point}). + +@item m +@kindex m@r{, in ert results buffer} +@findex ert-results-pop-to-messages-for-test-at-point +Show any messages that were generated (with the Lisp function +@code{message}) in in a test or any of the code that it invoked +(@code{ert-results-pop-to-messages-for-test-at-point}). + +@item L @kindex L@r{, in ert results buffer} +@findex ert-results-toggle-printer-limits-for-test-at-point By default, long expressions in the failure details are abbreviated -using @code{print-length} and @code{print-level}. Pressing @kbd{L} -while point is on a test failure will increase the limits to show more -of the expression. +using @code{print-length} and @code{print-level}. Increase the limits +to show more of the expression by moving point to a test failure with +this command (@code{ert-results-toggle-printer-limits-for-test-at-point}). + +@item D +@kindex D@r{, in ert results buffer} +@findex ert-delete-test +@cindex delete test +Delete a test from the running Emacs session (@code{ert-delete-test}). + +@item h +@kindex h@r{, in ert results buffer} +@findex ert-describe-test +Show the documentation of a test (@code{ert-describe-test}). + +@item T +@kindex T@r{, in ert results buffer} +@findex ert-results-pop-to-timings +Display test timings for the last run (@code{ert-results-pop-to-timings}). + +@item M-x ert-delete-all-tests +@findex ert-delete-all-tests +@cindex delete all tests +Delete all tests from the running session. + +@item M-x ert-describe-test +@findex ert-results-describe-test-at-point +Prompt for a test and then show its documentation. + +@end table @node Running Tests in Batch Mode @@ -308,7 +377,7 @@ emacs -batch -l ert -l my-tests.el -f ert-run-tests-batch-and-exit @end example This command will start up Emacs in batch mode, load ERT, load -@code{my-tests.el}, and run all tests defined in it. It will exit +@file{my-tests.el}, and run all tests defined in it. It will exit with a zero exit status if all tests passed, or nonzero if any tests failed or if anything else went wrong. It will also print progress messages and error diagnostics to standard output. @@ -323,12 +392,37 @@ summary as shown below: emacs -batch -l ert -f ert-summarize-tests-batch-and-exit output.log @end example +@vindex ert-batch-print-level +@vindex ert-batch-print-length +ERT attempts to limit the output size for failed tests by choosing +conservative values for @code{print-level} and @code{print-length} +when printing Lisp values. This can in some cases make it difficult +to see which portions of those values are incorrect. Use +@code{ert-batch-print-level} and @code{ert-batch-print-length} +to customize that: + +@example +emacs -batch -l ert -l my-tests.el \ + --eval "(let ((ert-batch-print-level 10) \ + (ert-batch-print-length 120)) \ + (ert-run-tests-batch-and-exit))" +@end example + +@vindex ert-batch-backtrace-line-length +Even modest settings for @code{print-level} and @code{print-length} can +produce extremely long lines in backtraces, however, with attendant +pauses in execution progress. Set +@code{ert-batch-backtrace-line-length} to t to use the value of +@code{backtrace-line-length}, @code{nil} to stop any limitations on backtrace +line lengths (that is, to get full backtraces), or a positive integer to +limit backtrace line length to that number. + @vindex ert-quiet By default, ERT in batch mode is quite verbose, printing a line with result after each test. This gives you progress information: how many tests have been executed and how many there are. However, in some cases this much output may be undesirable. In this case, set -@code{ert-quiet} variable to a non-nil value: +@code{ert-quiet} variable to a non-@code{nil} value: @example emacs -batch -l ert -l my-tests.el \ @@ -347,10 +441,21 @@ emacs -batch -l ert -l my-tests.el \ -eval '(ert-run-tests-batch-and-exit "to-match")' @end example +@vindex EMACS_TEST_VERBOSE@r{, environment variable} By default, ERT test failure summaries are quite brief in batch -mode--only the names of the failed tests are listed. If the -EMACS_TEST_VERBOSE environment variable is set, the failure summaries -will also include the data from the failing test. +mode---only the names of the failed tests are listed. If the +@env{EMACS_TEST_VERBOSE} environment variable is set, the failure +summaries will also include the data from the failing test. + +@vindex EMACS_TEST_JUNIT_REPORT@r{, environment variable} +ERT can produce JUnit test reports in batch mode. If the environment +variable @env{EMACS_TEST_JUNIT_REPORT} is set, ERT will produce for +every test package @file{my-tests.el} a corresponding JUnit test +report @file{my-tests.xml}. The function +@code{ert-summarize-tests-batch-and-exit} collects all these package +test reports into a new JUnit test report, with the respective name of +that environment variable. + @node Test Selectors @section Test Selectors @@ -419,8 +524,10 @@ to find where a test was defined if the test was loaded from a file. * Expected Failures:: Tests for known bugs. * Tests and Their Environment:: Don't depend on customizations; no side effects. * Useful Techniques:: Some examples. +* erts files:: Files containing many buffer tests. @end menu + @node The @code{should} Macro @section The @code{should} Macro @@ -701,6 +808,121 @@ for testing. Usually, this makes the interfaces easier to use as well. +@node erts files +@section erts files + +@findex ert-test-erts-file +Many relevant Emacs tests depend on comparing the contents of a buffer +before and after executing a particular function. These tests can be +written the normal way---making a temporary buffer, inserting the +``before'' text, running the function, and then comparing with the +expected ``after'' text. However, this often leads to test code +that's pretty difficult to read and write, especially when the text in +question is multi-line. + +So ert provides a function called @code{ert-test-erts-file} that takes +two parameters: The name of a specially-formatted @dfn{erts} file, and +(optionally) a function that performs the transform. + +@findex erts-mode +These erts files can be edited with the @code{erts-mode} major mode. + +An erts file is divided into sections by the (@samp{=-=}) separator. + +Here's an example file containing two tests: + +@example +Name: flet + +=-= +(cl-flet ((bla (x) +(* x x))) +(bla 42)) +=-= +(cl-flet ((bla (x) + (* x x))) + (bla 42)) +=-=-= + +Name: defun + +=-= +(defun x () + (print (quote ( thingy great + stuff)))) +=-=-= +@end example + +A test starts with a line containing just @samp{=-=} and ends with a +line containing just @samp{=-=-=}. The test may be preceded by +freeform text (for instance, comments), and also name/value pairs (see +below for a list of them). + +If there is a line with @samp{=-=} inside the test, that designates +the start of the ``after'' text. Otherwise, the ``before'' and +``after'' texts are assumed to be identical, which you typically see +when writing indentation tests. + +@code{ert-test-erts-file} puts the ``before'' section into a temporary +buffer, calls the transform function, and then compares with the +``after'' section. + +Here's an example usage: + +@lisp +(ert-test-erts-file "elisp.erts" + (lambda () + (emacs-lisp-mode) + (indent-region (point-min) (point-max)))) +@end lisp + +A list of the name/value specifications that can appear before a test +follows. The general syntax is @samp{Name: Value}, but continuation +lines can be used (along the same lines as in mail---subsequent lines +that start with a space are part of the value). + +@example +Name: foo +Code: (indent-region + (point-min) (point-max)) +@end example + +@table @samp +@item Name +All tests should have a name. This name will appear in ERT output if +the test fails, and helps to identify the failing test. + +@item Code +This is the code that will be run to do the transform. This can also +be passed in via the @code{ert-test-erts-file} call, but @samp{Code} +overrides that. It's used not only in the following test, but in all +subsequent tests in the file (until overridden by another @samp{Code} +specification). + +@item No-Before-Newline +@itemx No-After-Newline +These specifications say whether the ``before'' or ``after'' portions +have a newline at the end. (This would otherwise be impossible to +specify.) + +@item Point-Char +Sometimes it's useful to be able to put point at a specific place +before executing the transform function. @samp{Point-Char: |} will +make @code{ert-test-erts-file} place point where @samp{|} is in the +``before'' form (and remove that character), and will check that it's +where the @samp{|} character is in the ``after'' form (and issue a +test failure if that isn't the case). (This is used in all subsequent +tests, unless overridden by a new @samp{Point-Char} spec.) + +@item Skip +If this is present and value is a form that evaluates to a +non-@code{nil} value, the test will be skipped. +@end table + +If you need to use the literal line single line @samp{=-=} in a test +section, you can quote it with a @samp{\} character. + + @node How to Debug Tests @chapter How to Debug Tests @@ -902,6 +1124,7 @@ For information on mocks, stubs, fixtures, or test suites, see below. * Fixtures and Test Suites:: How ERT differs from tools for other languages. @end menu + @node Mocks and Stubs @section Other Tools for Emacs Lisp @cindex mocks and stubs @@ -976,11 +1199,13 @@ e.g., to run quick tests during interactive development and slow tests less often. This can be achieved with the @code{:tag} argument to @code{ert-deftest} and @code{tag} test selectors. + @node Index @unnumbered Index @printindex cp + @node GNU Free Documentation License @appendix GNU Free Documentation License @include doclicense.texi |