Chapter 11: Regression Suite

Table of Contents

• 11.1 Installing Google Test
  • 11.1.1 Checking out the Code from GitHub
  • 11.1.2 Compiling the Code
• 11.2 Running Google Test
• 11.3 Contributing New Tests
  • 11.3.1 Choosing the Right Test Case
  • 11.3.2 Writing Tests
  • 11.3.3 Writing Death Tests
• 11.4 Code Coverage

In order to ensure that the addition of new functionality did not compromise the existing code base, CARLsim comes with a regression suite, which relies on the Google Test framework.

11.1 Installing Google Test

11.1.1 Checking out the Code from GitHub

The Google Test framework is included in the CARLsim package as an external dependency. If you cloned the CARLsim repo using the –recursive flag, you already have a local copy of the Google Test codebase (see 1.2 Installation).

If you forgot to include the –recursive flag, you might encounter an empty directory when you navigate to external/googletest. If this is the case, you have to initialize the submodule yourself. Simply navigate to the CARLsim root directory and run the following:

$ git submodule update --init --recursive

11.1.2 Compiling the Code

Google Test can be installed with the help of CMake, per the following recipe:

1. Navigate to the Google Test folder and create a new directory called build:

      $ cd external/googletest
      $ mkdir build
   
   2. Change into the <tt>build</tt> directory and run <tt>cmake</tt>:
      \code
      $ cd build
      $ cmake ..

2. This will generate a Makefile. Run it:

   $ make

And done!

11.2 Running Google Test

Once you have downloaded and compiled the Google Test framework, all CARLsim test cases can be compiled and run from the CARLsim root directory:

$ make test
$ ./carlsim/test/carlsim_tests

Similary, if you installed CARLsim without GPU support, or if you want to exclude all GPU tests from the suite, run:

$ make test_nocuda
$ ./carlsim/test/carlsim_tests

Note

The test suite needs a results/ directory to run.

This should produce output like the following, possibly all green:

[==========] Running 100 tests from 17 test cases.
[----------] Global test environment set-up.
[----------] 7 tests from SpikeGen
[ RUN      ] SpikeGen.PeriodicSpikeGenerator
[       OK ] SpikeGen.PeriodicSpikeGenerator (339 ms)
[ RUN      ] SpikeGen.PeriodicSpikeGeneratorDeath
[       OK ] SpikeGen.PeriodicSpikeGeneratorDeath (351 ms)
[ RUN      ] SpikeGen.SpikeGeneratorFromFile
[       OK ] SpikeGen.SpikeGeneratorFromFile (1715 ms)

Note

If you installed CARLsim without GPU support (ch1s2s1s3_cpuonly), all GPU-related tests will be excluded from the test suite build.

Since

v3.0

11.3 Contributing New Tests

If you wish to contribute a new feature to CARLsim, you are expected to provide code that appropriately tests the new functionality.

11.3.1 Choosing the Right Test Case

Tests are grouped into different test cases, such as Interface, Core, CUBA, etc. Identify the right test case depending on the topic your new feature belongs to:

• core: All core functionality that involves running the whole CARLsim pipeline.
• CUBA/COBA: Functionality that is specific to either CUBA or COBA mode.
• Interface: Everything related to the CARLsim API.
• SpikeMon/ConnMon/GroupMon: Everything related to Spike, Connection, or Group Monitors.
• STDP/STP: Everything related to STDP or STP.
• Compartments: Everything related to compartmental models.
• etc.

11.3.2 Writing Tests

Every test is wrapped into some Google Test syntax:

TEST(testCase, testName) {
    
}

where the name of your test (testName) belongs to a certain test case (or group of tests, testCase).

Within this code block, you can write whatever CARLsim network you want.

After a network is run, you want to make sure its output is correct. This is achieved via Google Test-provided assertions such as:

• EXPECT_EQ: Expects to variables to be equal. For example:

  EXPECT_EQ(a, 4);

  This expects the variable a to be equal to 4.
• EXPECT_LT: Expects the first variable to be smaller than the second. For example:

  EXPECT_LT(a, 10);

  This expects the variable a to be smaller than 10.

All Google Test function calls are described here.

11.3.3 Writing Death Tests

If you write tests that expect the program to die (i.e., raise an Exception) in combination with EXPECT_DEATH, make sure the test is run in thread-safe mode by adding the following decorator:

TEST(testCase, testName) {
    ::testing::FLAGS_gtest_death_test_style = "threadsafe";

    EXPECT_DEATH(...)   
}

Since

v3.0

11.4 Code Coverage

If you are on a Linux machine, it is also fairly simple to generate a code coverage report using gcov. It's a bit harder on Mac OS X, but it can be done.

Simply follow the following procedure:

1. Uninstall CARLsim, because we will have to re-install it with some additional compile flags:

   $ make uninstall
   $ make clean

2. Enable code coverage generation by using a different target:

   $ make release_coverage -j4
   $ sudo -E make install

   You can also use release_nocuda_coverage, debug_coverage, or debug_nocuda_coverage.
3. Re-compile and run the test suite:

   $ make test_coverage
   $ ./carlsim/test/carlsim_tests

   If you ran the above with a nocuda option, make sure to run test_nocuda_coverage.
4. This should have generated a bunch of .gcda files, which can now be investigated. The easiest way to do this is to use lcov and generate a HTML report:

   $ lcov --directory ./ --capture --output-file ./code_coverage.info -rc lcov_branch_coverage=1
   $ genhtml code_coverage.info --branch-coverage --output-directory ./code_coverage_report/

5. Open the coverage report in a browser:

   $ google-chrome code_coverage_report/index.html &

Since

v3.1