Unit Testing

This page collects some common guidelines for writing unit tests in Gromacs.

• As stated on the Library Structure page, ideally each file in each module of libgromacs should have a tests/file.c(pp) file that tests the functionality in that file.  Tests for the public API of the module are the most important ones.
• Use the Google Test as the testing framework if/when you need one. It is already used in tests that exist in the source tree. For advanced C++ class testing, there is also Google Mock that works together with Google Test. Both are now included as part of the Gromacs source tree (under src/external/), as compiling them as part of the build process is the only supported way of using them. See examples in the existing tests for what these frameworks have to offer.
• There are CMake macros for conveniently declaring the unit tests; see the existing tests for examples on how to use it.
• Each module builds a single binary with all the tests for that module, and that binary is linked against all of libgromacs.
• Each test binary is visible as one test for CTest, and the tests are run as part of 'make test'. Each test binary produces a more detailed XML report with the individual test results. This XML file is parsed by Jenkins and it shows the results for individual failing tests automatically.
• There is a simple framework for building tests that check the results against reference data that is generated by the same test code. The code for this is located under src/testutils/, and stores the reference data in XML files (handled using libxml2) under src/module/tests/refdata/. The functionality it currently provides is quite basic, but can be extended if/when more control over, e.g., comparison tolerances is needed. See existing tests (e.g., the selection unit tests) for examples of how to use it.

Points for discussion:

• How to divide tests into binaries? Is the current approach sufficient?
  • A large number of binaries takes a substantial amount of time to link.
  • More tests there are in a binary, more careful we need to be about namespace conflicts between the tests.
  • I think that most testing frameworks don't deal well with one of the tests, e.g., segfaulting or asserting, but abort the whole binary in such cases.
  • We may need to link the test programs against things in other submodules. At least dependencies on really low-level utility modules like math routines cannot really be avoided. If all of these are in a header, the problem does not arise, but we should think how to handle these dependencies. Should all tests be linked against libgromacs, or should we try to have a more fine-grained approach?
• How to integrate the current tests with CTest/CDash? UPDATE: The current Jenkins integration more or less resolves this issue.
  • With Google Test (and with other unit testing frameworks as well), it is always possible to run just a subset of tests by providing command-line flags to the test binary. The binary, when run, also reports success/failure status per test. However, without hacking CTest will only report one success/failure per binary.
  • Google Test can create an XML report when running a test binary. This report contains the results of each individual test, so this could probably be used to get more fine-grained results into CTest. Would need additional investigation.
• The main issue with the current integration with Jenkins is that if one of the tests crashes (e.g., segfaults), then the XML file is not produced at all for tests in that binary. and the results in Jenkins are somewhat difficult to interpret. The error is visible somewhere in the console output, but there can be a lot of it. We could consider whether this can be improved with a reasonable amount of effort.