Tabularised feature comparison between Catch, doctest and lest C++ test frameworks.
Skip to the feature comparison.
Note: This is an initial draft, it is incomplete and likely contains errors.
Ideas, additions, corrections, signaling omissions, etc. welcome!
The frameworks in this comparison share the following properties:
- The frameworks are available as single-file header-only library.
- Test cases are described by a string, they are not identified by a function name.
- There is a single assertion macro for expressions.
Expressions are decomposed and original expression code can be shown alongside the values they represent. - Fixtures can be created inside a test case. They are called sections or sub cases.
Advent of Catch Expression-decomposing assertions and function-level fixtures (sections) were introduced by Kevlin Henney in his talk Rethinking Unit Testing in C++ at ACCU London in May 2010 [1]. At that time, Phil Nash was working on a testing framework called YACUTS (Yet Another C++ Unit Test System) that would bring similar capabilities, be it a little more intrusively [2]. Kevlin's talk inspired Phil to develop Catch [3].
Kevlin's emphasis on simplicity in testing and the desire to understand how expression decomposition works inspired Martin Moene to write lest [4]. The code of lest is intentionally kept small, so that it should be fairly easy to gain insight in how things work.
Accompanying test code with the code to test was one of the drivers for Viktor Kirilov to write doctest [5]. Others are to enable using doctest with many different compilers and to shine in compile-time and run-time performance. At CppCon 2017 Viktor gave a presentation that explains many techniques used in doctest and Catch [23].
| Feature | doctest | Catch-1 | Catch-2 | lest | Notes |
|---|---|---|---|---|---|
| License | MIT | BSL-1.0 | BSL-1.0 | BSL-1.0 | |
| C++ standard related | |||||
| C++ standard | C++98 | C++98 | C++11 | C++11 | |
| Can work without exceptions | ✓ | - | - | - | reduced functionality |
| Requires RTTI | - | - | - | - | |
| Fields of use | |||||
| Desktop | ✓ | ✓ | ✓ | ✓ | |
| Embedded (tiny) | ? | ? | ? | ? | |
| Embedded (largish) | ? | ? | ? | ✓ | |
| Application, tool | ? | ✓ | ✓ | ? | |
| Library, framework | ? | ✓ | ✓ | ✓ | |
| Operating System | ? | ? | ? | ✓ | lest: IncludeOS |
| Documentation of framework | see [22] | ||||
| First contact | ? | ? | ? | ? | new users |
| Education | ? | ? | ? | ? | new & existing users |
| Support | ? | ? | ? | ? | experienced users |
| Troubleshooting | ? | ? | ? | ? | annoyed users |
| Internals | ? | ? | ? | ? | fellow developers |
| Reference | ? | ? | ? | ? | everyone |
| Try it online | ✓ | ✓ | ✓ | ✓ | on wandbox |
| Forum | - | ✓ | ✓ | - | Catch: Google group |
| Chat | ✓ | - | - | - | doctest: Gitter |
| StackOverflow tag | - | ✓ | ✓ | - | see [19] |
| Test organisation | |||||
| Suites of tests | ✓ | - | - | - | |
| Fixtures (sections) | ✓ | ✓ | ✓ | ✓ | doctest: sub case |
| Fixtures (class-based) | ✓ | ✓ | ✓ | - | |
| Type-parameterised tests | ✓ | ✓ | ✓ | - | |
| Value-parameterised tests | - | - | - | - | |
| Auto-registration of tests | ✓ | ✓ | ✓ | ✓ | |
| Tabularised tests | - | - | - | ✓ | array of lambdas |
| Allow tests accompany code | ✓ | ✓ | ✓ | - | Catch: not recommended... |
| Allow tests in header files | ✓ | ✓ | ✓ | - | ...to avoid long compilation times |
| Assertions | |||||
| BDD style scenarios | ✓ | ✓ | ✓ | ✓ | |
| Assert expressions | ✓ | ✓ | ✓ | ✓ | |
| Assert exceptions | ✓ | ✓ | ✓ | ✓ | |
| Assert abortion (death) | - | - | - | - | |
| Assert assertions (death) | - | - | - | - | |
| Expression decomposition | ✓ | ✓ | ✓ | ✓ | see [2] |
| Floating point comparison, approx | ✓ | ✓ | ✓ | ✓ | |
| Floating point comparison, ULP | - | - | ✓ | - | see stf [10] |
| Hamcrest matchers (built-in) | - | some | some | • | see [12][13][14][15] |
| Hamcrest matchers (user-definable) | - | ✓ | ✓ | - | |
| Macros with and without prefix | ✓ | ✓ | ✓ | ✓ | CATCH_CHECK(), CHECK() |
| Other test facilities | |||||
| Mocking support | • | • | • | • | via 3rd party |
| Logging facility | ✓ | ✓ | ✓ | - | |
| Logging levels | - | - | - | - | |
| Checkpoints | - | - | - | - | see Boost.Test [11] |
| Test decorators | ✓ | - | - | - | see [16] |
| Test data generators | - | • | - | - | Cach-1: undocumented |
| Property-based testing | - | - | - | - | |
| Obtain name of current test | ? | • | ? | - | Catch-1: undocumented |
| API / seams | |||||
| Reporting user-defined types | ✓ | ✓ | ✓ | ✓ | |
| Reporter API | - | ✓ | ✓ | - | |
| Event listeners | - | ✓ | ✓ | - | |
| Run-time context | ✓ | ✓ | ✓ | - | |
| Reporting formats | |||||
| Console, multi-line | ✓ | ✓ | ✓ | - | |
| Console, single-line (compact) | - | ✓ | ✓ | ✓ | similar to compiler error |
| JUnit | - | ✓ | ✓ | - | result at end |
| XML | - | ✓ | ✓ | - | streaming |
| TeamCity | - | ✓ | ✓ | - | |
| TAP | - | ✓ | ✓ | - | |
| Automake | - | ✓ | ✓ | - | |
| Reporting options | |||||
| Colourised output (run-time) | ✓ | ✓ | ✓ | ||
| Colourised output (compile-time) | ✓ | - | - | ✓ | |
| Literal suffix u, l, f | - | - | - | ✓ | compile-time |
| Test execution | |||||
| Concurrent execution of tests | - | - | - | - | |
| Isolated execution of tests | - | - | - | - | |
| Limit test execution time | ✓ | - | - | - | doctest: via test decorator |
| Signal tests without assertions | - | ✓ | ✓ | - | |
| Supports floating point exceptions | - | - | - | - | |
| Supports Posix signals | ✓ | ✓ | ✓ | - | |
| Supports Windows SEH | ✓ | ✓ | ✓ | - | |
| Compile-time control | |||||
| Improve compilation speed | ✓ | ✓ | ✓ | - | reduced functionality |
| Omit macros without prefix | ✓ | ✓ | ✓ | ✓ | |
| Omit std::wstring | - | - | - | ✓ | |
| Omit std::cout, cerr, clog | - | ✓ | ✓ | - | |
| Specify terminal color system | ✓ | ✓ | ✓ | - | none, ansi, win32 |
| Specify terminal width | - | ✓ | ✓ | - | |
| Control handling of POSIX signals | ✓ | ✓ | ✓ | - | |
| Control handling of SE on Windows | ✓ | ✓ | ✓ | - | |
| Enable leak checking on Windows | - | ✓ | ✓ | - | uses CRT Debug Heap |
| Enable use of __COUNTER__ | - | ✓ | ✓ | - | for internal unique names |
| Run-time control, commandline | |||||
| Help screen | ✓ | ✓ | ✓ | ✓ | |
| List selected suites | ✓ | - | - | - | |
| List selected tests | ✓ | ✓ | ✓ | ✓ | |
| List tags of selected tests | - | ✓ | ✓ | ✓ | |
| Select tests by suite | ✓ | - | - | - | |
| Select tests by name | ✓ | ✓ | ✓ | ✓ | |
| Select tests by tag | ✓ | ✓ | ✓ | ✓ | |
| Select tests by file | ✓ | ✓ | ✓ | - | Catch: via filename as tags |
| Select section by name | ✓ | ✓ | ✓ | - | |
| Select supports regexp | ✓ | - | - | ✓ | |
| Omit assertions expected to throw | - | ✓ | ✓ | - | |
| Count selected tests | ✓ | - | - | ✓ | |
| Abort at N-th failure | ✓ | ✓ | ✓ | ✓ | lest: N is 1 |
| Break into debugger | ✓ | ✓ | ✓ | - | |
| Report passing tests | ✓ | ✓ | ✓ | ✓ | |
| Time duration of tests | ✓ | ✓ | ✓ | ✓ | |
| Control order of tests | ✓ | ✓ | ✓ | ✓ | |
| Repeat tests | - | ✓ | ✓ | ✓ | |
| IDE integration | see [17] | ||||
| Output-window-compatible reports | ✓ | ✓ | ✓ | ✓ | click through to source |
| C++ Builder | ? | ? | ? | ? | |
| CLion | ✓ | ✓ | ✓ | - | |
| Code::Blocks | ? | ? | ? | ? | |
| CodeLite | ? | ? | ? | ? | |
| Eclipse CDT | ? | ? | ? | ? | |
| KDevelop | ? | ? | ? | ? | |
| Qt Creator | ? | ? | ✓ | ? | |
| Visual-MinGW | ? | ? | ? | ? | |
| Visual Studio | • | • | • | - | Catch and doctest: via ReSharper C++ |
| Visual Studio Code | - | - | - | - | see issue |
| Platforms | |||||
| AIX / XL | ? | 11.1.0 | ? | ? | Catch-1: 2013 |
| macOS / clang | ? | ? | ? | ? | |
| macOS / gcc | 4.4-6 | ? | ? | ? | source: Travis |
| macOS / XCode / clang | 6.4-8.3 | 7.3-8 | 7.3-8 | ? | source: Travis |
| macOS / XCode / gcc | 6.4-7.3 | 7.3-8 | 7.3-8 | ? | source: Travis |
| Linux / clang | 3.4-4 | 3.4-3.8 | 3.5-3.8 | 3.6-3.7 | source: Travis |
| Linux / gcc | 4.4-6 | 4.4-6 | 4.8-6 | 4.9-5 | source: Travis |
| Solaris / SunPro | ? | 5.12.0 | ? | ? | Catch-1: 2013 |
| Windows / clang | ? | ? | ? | 3.2+ | |
| Windows / gcc | 7 | ? | ? | 4.8.1+ | |
| Windows / Visual C++ | 9-15 | 10-15 | 14-15 | 12-15 | Catch-1: possibly from 8 |
| Windows / Cygwin / clang | ? | ? | ? | ? | |
| Windows / Cygwin / gcc | ? | ? | ? | ? | |
| Windows / msys2 / clang | ? | ? | ? | ? | |
| Windows / msys2 / gcc | ? | ? | ? | ? | |
| Quality of project | note: challenge 42 | ||||
| Code of conduct | - | ✓ | ✓ | - | |
| Follows CII best practices | ✓ | partly | partly | partly | see [21] |
| Quality of code | |||||
| Unit testing | ✓ | ✓ | ✓ | ✓ | details below |
| CI testing | ✓ | ✓ | ✓ | ✓ | details below |
| CI code analysis | ✓ | - | - | - | details below |
| Testing of framework | |||||
| Unit tests | ✓ | ✓ | ✓ | ✓ | |
| Tests itself | ✓ | ✓ | ✓ | ✓ | |
| Baseline tests | ✓ | ✓ | ✓ | - | |
| CI testing | |||||
| Travis | ✓ | ✓ | ✓ | ✓ | |
| Appveyor | ✓ | ✓ | ✓ | ✓ | |
| Coverage scan | ✓ | - | ✓ | - | |
| Coverity scan | ✓ | - | - | - | |
| CI static code analysis | |||||
| clang-tidy | ✓ | - | - | - | |
| CppCheck | ✓ | - | - | - | |
| CppCoreCheck | - | - | - | - | |
| PVS-Studio | - | - | - | - | |
| OCLint | ✓ | - | - | - | |
| ReSharper C++ | - | - | - | - | |
| Visual Studio | ✓ | - | - | - | |
| Code-related | |||||
| Compile-time performance, relative | ☆☆☆ | ☆☆ | ? | ? | see [20] |
| Run-time performance, relative | ☆☆☆ | ☆☆ | ? | ? | see [20] |
| Compiler-warning friendly | ☆☆☆ | ☆☆ | ☆☆ | ☆☆ | |
| Compiles with -Werror -Wall -Wextra | ✓ | ✓ | ✓ | ✓ | |
| Can provide canned main() | ✓ | ✓ | ✓ | - | |
| Only depends on C++ std library | ✓ | ✓ | ✓ | ✓ | |
| Single-file header-only | ✓ | ✓ | ✓ | ✓ | |
| Size, LOC | 4,500 | 9,100 | 8,800 | 1,000 | see [18] |
| Distribution of framework | |||||
| GitHub single-file download | ✓ | ✓ | ✓ | ✓ | from the landing page |
| conan | ✓ | ✓ | ✓ | - | |
| hunter | ✓ | ? | - | - | |
| vcpkg | ✓ | ✓ | ✓ | - | |
| Linux and other (search) | search | → | search | search | on Repology |
| Feature | doctest | Catch-1 | Catch-2 | lest | Notes |
[1] Kevlin Henney. Rethinking Unit Testing in C++ (Video). Skills Matter, May 2010.
[2] Phil Nash. The Ultimate C++ Unit Test Framework. Blog, May 2010.
[3] Phil Nash. Unit Testing in C++ and Objective-C just got easier. Blog announcement, December 2010.
[4] Martin Moene. lest errors escape testing. Blog announcement, June 2013.
[5] Viktor Kirilov. doctest – the Lightest C++ Unit Testing Framework. Announcement in Overload 137, February 2017.
[6] Phil Nash. Catch-1 on GitHub. 2010.
[7] Phil Nash. Catch-2 on GitHub. 2017.
[8] Viktor Kirilov. doctest on GitHub. 2014.
[9] Martin Moene. lest on GitHub. 2013.
[10] Joel Falcou. Design rationale for using ULP with stf.
[11] Gennadiy Rozental and Raffi Enficiaud. Checkpoints in Boost.Test.
[12] Matchers in Catch-1.
[13] Matchers in Catch-2.
[14] Matchers in lest: via hamlest.
[15] Hamcrest 1.3 Quick Reference (PDF).
[16] Decorators in doctest.
[17] Wikipedia. List of C++ IDEs.
[18] CodeBlocks IDE. Non-empty, non-comment lines, via File > Properties.
[19] StackOverflow. Questions tagged catch-unit-test.
[20] Viktor Kirilov. doctest Benchmarks. May 2017.
[21] Linux Foundation (LF). CII Best Practices.
[22] Jacob Kaplan-Moss. Writing great documentation.
[23] Viktor Kirilov. Mix Tests and Production Code With Doctest - Implementing and Using the Fastest Modern C++ Testing Framework (slides). CppCon 2017.