Buttercup is a testing framework for emacs-lisp. It is used by large projects like Clojure's CIDER to write clean, concise, and descriptive tests.
I introduce Buttercup and build up to advanced usages with the faint, unlikely dream that some Emacs programmer decides to add tests to their library…
What is Buttercup?
Buttercup's entry points are:
describe a test suite with a name. Test cases within the possibly nested
suites are done with
it and assertions as
expect blocks within.
(describe "Four" (describe "comparisons" (it "is greater than one" (expect (> 4 1))) (it "and less than five" (expect 4 :to-be-less-than 5))) (it "is a number" (expect (numberp 4))))
passes with testing output:
Four comparisons is greater than one (0.24ms) and less than five (0.12ms) is a number (0.09ms)
Setup and Teardown
reduce boilerplate with setting up and tearing down test suites.
(describe "Lisp mode syntax" (before-all (set-syntax-table lisp-mode-syntax-table)) (after-each (delete-region (point-min) (point-max))) (it "sets comments" (insert ";; foo") (expect (nth 4 (syntax-ppss)))) (it "sets strings" (insert "\"foo\"") (backward-char) (expect (nth 3 (syntax-ppss)))))
expect has more utility than simple tests of truth. Matchers are
keywords that tailor the expectation.
Some example matcher expansions:
(eq foo bar)
(equal foo bar)
(member foo bar)
(foo bar precisision)
(expr &optional signal signal-args)
Some other more advanced matchers include:
These matchers may be combined too: eg.
(expect 4 :not :to-be-greater-than 5).
Matchers are more than just transforms+comparisons. They give information about the failure.
(describe "Example Matchers" (it "regexes" (expect (s-concat "foo" "bar") :to-match (rx word-start "foo" word-end))))
Expected `(s-concat "foo" "bar")' with value "foobar" to match the regexp "\\<foo\\>", but instead it was "foobar".
I recommend using
Cask and executing tests with
cask exec buttercup -L . in the project root.
For example, have a file named
Cask in the project root with:
(source gnu) (source melpa) (package-file "test-stuff-i-beg-you-mode.el") ;; Project Dependencies (depends-on "dash") ;; Additional Testing Dependencies (development (depends-on "buttercup") (depends-on "faceup"))
A folder named
test/ should be present and contain
This file should have your tests, set up the load path if needed, and require everything you need.
Lastly I will mention some other useful features before diving in to Buttercup:
Variables can be defined with let syntax with
Buttercup has good support for spying on function calls.
x, so it's
xdescribe, mark the test as pending so it won't be executed.
Case Study: Testing Indentation
You have written
yet-another-lisp-like-mode you affectionately call
yall-mode and want to test its indentation.
Lets write a skeleton to test the simplest cases:
;; Want to test these two cases: ;; (foo ;; bar) ;; (foo bar ;; baz) (describe "Indentation" (before-all (setq indent-line-function #'yall-indent-line)) (describe "standard cases" (it "opening line has one sexp - so indentation doesn't carry" (expect ???)) (it "opening line has two+ sexps - so indentation carries" (expect ???))))
To test indentation - all we need is the text we expect, as the text alone determines the indent.
Buttercup allows us to achieve this via custom matchers. We can bypass all boilerplate and write our expectations as simply as:
(expect " (foo bar) " :indented)
buttercup-define-matcher allows defining our own matcher, that will
perform transforms, assertions, and give descriptive failures.
Lets implement our
(defun yall-trim-indent (text) "Remove indentation from TEXT." (->> text s-lines (-map #'s-trim-left) (s-join "\n"))) (defun yall-buffer-string () "Return buffer as text with beginning and ending empty space trimmed." (s-trim (buffer-substring-no-properties (point-min) (point-max)))) (buttercup-define-matcher :indented (text) (let* ((text (s-trim (funcall text))) (text-no-indent (yall-trim-indent text))) (insert text-no-indent) (indent-region-line-by-line (point-min) (point-max)) (let ((text-with-indent (yall-buffer-string))) (delete-region (point-min) (point-max)) (if (string= text text-with-indent) t `(nil . ,(format "\nGiven indented text \n%s\nwas instead indented to \n%s\n" text text-with-indent))))))
Now we can see the power of buttercup when we accidentally write:
(describe "Indentation" (before-all (setq indent-line-function #'yall-indent-line)) (describe "standard cases" (it "opening line has two+ sexps - so indentation carries" (expect " (foo bar baz) " :indented))))
and are given the failure:
FAILED: Given indented text (foo bar baz) was instead indented to (foo bar baz)
We know exactly what went wrong, with nearly all the implementation details
separated from the testcase with boilerplate just
Testing Emacs programs doesn't have to be painful - buttercup is a great and battle-tested library for writing quality Emacs programs.