CARLsim  6.1.0
CARLsim: a GPU-accelerated SNN simulator
Chapter 1: Getting Started

Table of Contents

1.1 Pre-Installation

Author
Kristofor D. Carlson
Michael Beyeler
Ting-Shuo Chou
Lars Niedermeier

CARLsim runs on both generic x86 CPUs and off-the-shelf NVIDIA GPUs. Full support of all CARLsim features requires that the NVIDIA CUDA parallel computing platform be installed. It is also possible to compile CARLsim without GPU support (see 1.2.1.3 Compiling the CARLsim Library).

CARLsim 6 requires GPUs with a compute capability of 5.0 (Maxwell) or higher. To find the compute capability of your device please refer to the CUDA article on Wikipedia.

CARLsim also requires CUDA Toolkit 11.0 or higher. For platform-specific CUDA installation instructions, please navigate to the NVIDIA CUDA Zone.

The rest of the chapter assumes you have successfully installed CUDA on appropriate hardware.

Note
Please make sure you install the CUDA SDK samples, as CARLsim relies on the file helper_cuda.h, which usually resides in /usr/local/cuda/samples/common/inc.

1.1.1 Supported Operating Systems

CARLsim has been tested primarily on the following platforms:

  • Ubuntu 20.04 LTS
  • Windows 10 Professional, Windows 11 Education
  • CUDA 11.2, 11.4, 11.5, 11.7
  • GPUs: Titan Xp, 1080ti, RTX 3090, A100

Due to the many possible combinations of operating systems, development stacks, CUDA versions, and GPUs, the GitHub wiki lists those that were tested by the community as well.

1.1.2 Source Directory Layout

Below is the directory layout of the CARLsim source code. All source code of the core library is contained in the directory carlsim. The sub-directories are key components to the CARLsim simulation library.

The doc directory contains doxygen-related source files in source and the pre-compiled HTML version of the documentation in html.

The projects directory contains a template for writing your first CARLsim program. Users will start here when they begin writing their first program.

The tools directory contains a number of CARLsim plugins that may be useful to users such as parameter tuning frameworks, MATLAB scripts, spike generators, and tools for visual stimuli.

├── carlsim # CARLsim source code directory
│   ├── interface # CARLsim interface (public user interface)
│   ├── kernel # CARLsim core functionality
│   ├── monitor # Monitors for groups, connections, spikes, etc.
│   ├── server # Utility to implement real-time server functionality
│   ├── test # Google test regression suite tests
│   ├── test6 # Google test suite for features of version 6
│   └── testadv # Tests that are long running or require specific hardware
├── doc # CARLsim documentation generation directory
│   ├── html # Generated documentation in html
│   └── source # Documentation source code
├── external # External dependencies
│   ├── googletest # Google Test framework
├── projects # User projects directory
│   └── hello_world # Project template for new users
└── tools # CARLsim tools that are not built-in
    ├── ecj_pti # Automated parameter-tuning interface using ECJ
    ├── eo_pti # Automated parameter-tuning interface using EO (deprecated)
    ├── offline_analysis_toolbox # Collection of MATLAB scripts for data analysis
    ├── simple_weight_tuner # Simple weight parameter-tuning tool
    ├── spike_generators # Collection of input spike generation tools
    └── visual_stimulus # Collection of MATLAB/CARLsim tools for visual stimuli
Since
v3.0

1.2 Installation

Author
Kristofor D. Carlson
Ting-Shuo Chou
Michael Beyeler
Lars Niedermeier

CARLsim is now available on GitHub.

If you are familiar with Git and GitHub, the preferred way to obtain the software is to fork and clone the GitHub repository. This will give you a way to access the latest stable and development versions of the code, and allow you to easily update your codebase later on.

This can be done in three steps:

  1. Navigate to CARLsim's GitHub page and click on the Fork box in the top-right corner.
  2. Switch to a terminal (or GitHub Desktop) and clone the repository:
    $ git clone --recursive https://github.com/UCI-CARL/CARLsim6.git
    $ cd CARLsim6
    where YourUsername is your GitHub user name. Note the –recursive option: It will make sure all external software dependencies get installed (e.g., Google Test).

Alternatively, you may download the latest stable release from the GitHub Release page (.zip or .tar.gz).

For installation instructions on Linux and Mac OS X platforms, please refer to 1.2.1 Linux / Mac OS X below. For installation instructions on any platform using CMake, refer to 1.2.2 Build and Install with cMake below.

1.2.1 Linux / Mac OS X

Instructions for Linux/Mac OS X installation assume you are using the Bash shell. Additionally, the GNU GCC compiler collection and GNU Make build environment should be installed. On most platforms, these are already installed by default.

1.2.1.1 Environment Variables

CARLsim 5 allows configuration via environment variables. The easiest way to set these is to add them to your ~/.bashrc file.

The following options are available:

  • Installation directory: By default, the CARLsim library lives in ~/CARL/lib, and CARLsim include files live in ~/CARL/include. You can overwrite these by exporting an evironment variable called CARLSIM5_INSTALL_DIR:
    $ export CARLSIM5_INSTALL_DIR=/path/to/your/preferred/location
    For advanced users or administrators, you could set the path to /usr/local
  • Custom CUDA paths: Set an environment variable called CUDA_PATH. By default, your CUDA library lives in /usr/local/cuda
    $ export CUDA_PATH=/path/to/CUDA
  • ECJ support: If you use the evolutionary computing based Parameter Tuning Interface, additional variables are needed. See ch10_ecj for the list of those additional variables.

Once you have made changes to your ~/.bashrc, make sure they go into effect by either typing:

$ source ~/.bashrc

or by closing the shell and opening another one.

Older versions of CARLsim used a configuration file called user.mk. This file is obsolete as of CARLsim 3.1.2.

1.2.1.2 Finding CUDA Toolkit Version and Compute Capability

The CUDA Toolkit version can be found via:

$ nvcc --version

You need only input the major number of the toolkit version (e.g. 11 for 11.5).

The compute capability of the GPU device can be found by compiling the deviceQuery sample in the directory 1_Utilities of the CUDA Toolkit.

# copy NVIDIA Toolkit to home directory
$ cd /usr/local/cuda/bin
$ ./cuda-install-samples-11.0.sh ~
$ cd ~/NVIDIA_CUDA-11.0_Samples/1_Utilities/deviceQuery
# compile and run deviceQuery
$ make
$ ./deviceQuery

For CUDA Toolkits other than version 11.0, the paths above need to be changed accordingly.

Note
Please make sure you install the CUDA SDK samples, as CARLsim relies on the file helper_cuda.h, which usually resides in /usr/local/cuda/samples/common/inc.

1.2.1.3 Setup for local development

The following information are intended for developers using Linux. Usually the models and experiments are developed on a workstation having a NVIDIA GeForce and to be evaluated later on a supercomputer like the DGX A100. The following preliminaries are derived from the NVIDIA documentation for CUDA 11.5:

  1. Linux: Ubuntu 20.04 LTS
  2. CMake: 3.22
  3. Google Test: 1.11

In this guide, the following file structure is used as a reference for the local development. Please replace the placeholder user1 with the actual user name:

/home/user1/
carlsim6/ # local installation CARLsim6
includes/
lib/
samples/
cmake-3.22/ # local installation of CMake
bin/
share/
gtest-1.11/ # local installation of Google Test
inclues/
lib/
git/ # cloned repositories from Github
CARLsim6/
googletest/

1.2.1.4 Setup Ubuntu

Ideally Ubuntu 20.04 LTS Desktop is installed from scratch on the workstation. For CUDA 11.5 please follow the official setup guide. Also the user should be setup from scratch to avoid any side effects. While it is technically possible to use multiple CUDA and CARLsim versions side by side and switching between them utilizing some kind of setenv.sh script, such szenarios also depend strongly on the specific requirements and are therefore out of scope. A dedicated environment is furthermore essential to find the root cause of potential issues.

Prepare the .bashrc like the following (replace user1 with the actual user name).

export PATH=/home/user1/cmake-3.22/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/home/user1/gtest-1.11/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export LD_LIBRARY_PATH=/home/user1/carlsim6/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

Validate that the CUDA 11.5 installation has added the following lines:

export PATH=/usr/local/cuda-11.5/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-11.5/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

Install pthreads from the distribution: sudo apt-get install libpthreads-dev

1.2.1.5 Setup CMake

Download the latest binary (e.g. 3.22) from Kitware and install it to /home/user1/CMake-3.22.

Restart the terminal and validate the installation with cmake --version.

$ cmake --version
cmake version 3.22.0-rc2
CMake suite maintained and supported by Kitware (kitware.com/cmake).

1.2.1.6 Setup Google Test

Clone the latest stable version (e.g. 1.11) of Googletest at GitHub and build it from source.

cd ~/git
git clone https://github.com/google/googletest.git
cd googletest
mkdir .build
cd .build
cmake -DCMAKE_INSTALL_PREFIX=/home/user1/gtest-1.11 -DBUILD_SHARED_LIBS=1 -DGTEST_HAS_PTHREAD=1 -DBUILD_GMOCK=OFF ../.
make install

1.2.2 Build and Install with cMake

Fast track for experienced users with cMake:

cd ~/git
git clone https://github.com/UCI-CARL/CARLsim6.git CARLsim6
cd CARLsim6
mkdir .build
cd .build
cmake -DCMAKE_INSTALL_PREFIX=/home/user1/carlsim6 -DCMAKE_BUILD_TYPE=Release ../.
make install

First you run cMake to configure with what options CARLsim is to be built. Then the project files for the desired development environment of the platform are generated, e.g. gcc on Linux or Visual Studio on Windows. The configuration is stored in a cache file and can be either changed by the command line interface (CLI) or with platform independent graphical user interface cmake-gui.

cmake-gui.png
Configure CARLsim 6 with CMake-GUI

Further information can be found at About CMake.

1.2.2.1 Run CMake

Create a build directory (you can make it anywhere)

$ mkdir <build-dir>

Proceed into build directory and do configuration:

$ cd <build-dir>
$ cmake \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=<carlsim-install-dir> \
-DCARLSIM_NO_CUDA=OFF \
<carlsim-source-dir>

As you can see cmake accepts several options -D<name>=<value>: they define cmake variables. CMAKE_BUILD_TYPE being assigned Release, which means that we are going to build release version of the library. If you need debug version then pass Debug. CMAKE_INSTALL_PREFIX specifies a directory which we are going to install the library into. CARLSIM_NO_CUDA switches on/off support of CUDA inside the library.

Build:

$ cmake --build <build-dir>

Install:

$ cmake --build <build-dir> --target install

Alternatively, the library can be built and installed from an IDE, in case you specified the IDE's CMake Generator during the configuration step.

1.2.2.2 Advanced installation for Linux

cd ~/git
git clone --recursive https://github.com/UCI-CARL/CARLsim6.git CARLsim6
cd CARLsim6
mkdir .build
cd .build
cmake -DCMAKE_INSTALL_PREFIX=/home/user1/carlsim6 -DCMAKE_BUILD_TYPE=Release ../.
make install

Hint: If cmake does not find the GTest_DIR set it manually in cmake-gui to /home/user1/gtest/lib/cmake/GTest. Also, setting the following environment variables in Linux in ~/.bashrc may help (replace user1 with username):

export GTEST_LIBRARY=/home/user1/gtest-1.11/lib/libgtest.a
export GTEST_MAIN_LIBRARY=/home/user1/gtest-1.11/lib/libgtest_main.a
export GTEST_ROOT=/home/user1/gtest-1.11/

Follow the following sequence to repeat builds

make clean
make -j8
make install

Hint: The defaults for CMake are configured to support the latest version of CUDA and the current generation of GeForce graphics card (Ampere achritecture). Check which compute capability your your GPU actually has and adopt CARLSIM_CUDA_GENCODE accordingly either in cmake-gui or by passing it as a parameter to the CLI. E.g. for a Titan Xp the parameter is set by -DCARLSIM_CUDA_GENCODE=-gencode arch=compute_61,code=sm_61

1.2.2.3 Validate the installation

Open a new terminal and validate the settings with env.

Start ~/carlsim6/samples/hello_world

Run the unit tests, e.g.

cd ~/git/CARLsim6/.build/carlsim/test
./carlsim-tests

To run all tests in parallel with monitoring the GPU utilization

gnome-terminal -- /bin/sh -c '~/git/CARLsim6/.build/carlsim/test/carlsim-tests;exec bash' &
gnome-terminal -- /bin/sh -c '~/git/CARLsim6/.build/carlsim/test6/carlsim-tests6;exec bash' &
gnome-terminal -- /bin/sh -c '~/git/CARLsim6/.build/carlsim/testadv/carlsim-testsadv --gtest_filter=-*GPU_MultiGPU*;exec bash' &
gnome-terminal -- /bin/sh -c 'nvidia-smi -l 1' &

1.2.2.4 Running the samples

The executables of the samples are installed to $CMAKE_INSTALL_PREFIX/carlsim6/samples. Add the path to the .bashrc to repetitive start it from the bash. As most of the samples create a result directory or write other files, create a new working directory to capture the indiviual runs.

Alternatively the samples can be run directly from the build directory.

1.2.1.3 Compiling the CARLsim Library

When it comes to compiling CARLsim, you have several options:

  1. Compile a release version:
    $ make -j4
    or
    $ make release -j4
    This will enable hardware accelerations using the -O3 and -ffast-math compile flags.
  2. Compile a debug version:
    $ make debug -j4
    This will disable hardware accelerations (via -O0) and enable extensive debug prints (via -g -Wall).

If your machine does not have a GPU card supporting CUDA framework, you could

  1. Compile a release version without GPU support:
    $ make release_nocuda -j4
  2. Compile a debug version without GPU support:
    $ make debug_nocuda -j4

After compiling CARLsim, you can install the library by typing:

  • $ make install
    or
  • $ sudo -E make install
    , if you are an administrator.
Note
Note the -E flag in the commands above, which will cause sudo to remember any environment variables you set above (such as CARLSIM5_INSTALL_DIR). You don't need this flag if you're not installing with sudo.
Also, note the -j4 flag, which will install CARLsim using four workers. You can increase the number on systems with more than four cores if you wish, or omit the flag if you're working on a single-core machine.

CARLsim comes with an optional automated parameter tuning framework. For more information about how to install the framework please see ch10_ecj. Additionally, CARLsim now comes with a regression suite that uses Google Test. For more information on how to use the regression suite, please see Chapter 11: Regression Suite.

Since
v3.1

1.2.1.4 Verifying the Installation

In order to make sure the installation was successful, you can compile the regression suite:

$ make test

or

$ make test_nocuda

, if you compiled CARLsim libarary without GPU support.

To run the regression suite, you can type:

$ ./carlsim/test/carlsim_tests

For more information on the regression suite, please refer to Chapter 11: Regression Suite.

1.3 Project Workflow

Author
Kristofor D. Carlson
Ting-Shuo Chou

A sample "Hello World" project is provided in the projects/hello_world directory. The project includes a single source file main_hello_world.cpp in the src directory that creates a network with two groups, connected with random weights, and can be used as a skeleton to create new projects.

Any output files created by the simulation will be automatically placed in the results/ directory.

All MATLAB scripts should be placed in the scripts/ directory. This directory already contains two MATLAB scripts to aid in using the OAT (see Chapter 9: MATLAB Offline Analysis Toolbox (OAT)). The script initOAT.m adds the OAT directory to the MATLAB path, whereas demoOAT.m will open a NetworkMonitor to visualize network activity. Note that for demoOAT.m to work, the executable must be run first (see 1.3.1.1 Compiling and Running the "Hello World" Project in Linux / Mac OS X below). In order to run the OAT, open MATLAB, change to projects/hello_world/scripts/, then type:

>> initOAT % adds OAT relative path to MATLAB paths
>> demoOAT % opens a NetworkMonitor on the simulation file

1.3.1 Linux / Mac OS X

1.3.1.1 Compiling and Running the "Hello World" Project in Linux / Mac OS X

The "Hello World" project comes with its own Makefile that compiles the file main_hello_world.cpp and links it with the CARLsim library. The project can be compiled and run with the following set of commands:

$ cd projects/hello_world
$ make
$ ./hello_world

If you compile CARLsim library without GPU support, please build your project by typing:

$ make nocuda

Any output files created by the simulation will be automatically placed in the results/ directory. This may include any spike files created by SpikeMonitor, a debug log file, and a network structure file.

All local objects and executables can be deleted via:

$ make clean

All output files, including local objects, executables, and files in the results/ directory can be deleted via:

$ make distclean
Warning
When make distclean is called, all data files in the results directory will be deleted!

1.3.1.2 Creating a New Project in Linux / Mac OS X

The easiest way to create a new project in Linux/Mac OS X is to make a copy of the projects/hello_world/ directory and all its corresponding subdirectories, rename the directory accordingly, and place it alongside hello_world/ in the projects/ directory. To do this, type

$ ./init.sh your_project_name

Only minimal changes to the Makefile must be made in order for the project to compile correctly. The Makefile provided in the directory was made so that users only have to modify a small portion of the file to build a custom project. Below is the modifiable portion of the Makefile:

# Name of the binary file to be created.
# NOTE: There must be a corresponding .cpp file named main_$(proj_target).cpp
proj_target := hello_world
# Directory where all include files reside. The Makefile will automatically
# detect and include all .h files within that directory.
proj_inc_dir := inc
# Directory where all source files reside. The Makefile will automatically
# detect and include all .cpp and .cu files within that directory.
proj_src_dir := src

The name of the project can be changed via variable project. Whatever string is assigned here will influence the name of the Makefile target as well as the name of the C++ source file. For example, setting project to "hello_world" will assume that a source file main_hello_world.cpp exists, and will create an executable called hello_world.

Note
The C++ source file must be named main_{project name}.cpp for the Makefile to compile correctly, where {project_name} is the string assigned to the project variable in the Makefile.

1.3.2 CMake

1.3.3.1 Compiling the "Hello World" Project using CMake

Make sure that CARLsim has been built and installed using CMake, because the "Hello World" project's cmake script searches for CARLsim's cmake package configuration file.

Create a build directory (you can make it anywhere)

$ mkdir <build-dir>

Proceed into the build directory and do configuration:

$ cd <build-dir>
$ cmake \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH=<carlsim-install-dir> \
<hello-world-source-dir>

CMAKE_BUILD_TYPE specifies whether we build Release of Debug version of the project. CMAKE_PREFIX_PATH must point to the CARLsim's installation directory in case we installed the library into a cutsom directory, so that cmake can find its configuration file. <hello-world-source-dir> must point to the directory holding the project's CMakeLists.txt file.

Build the project:

$ cmake --build <build-dir>

Alternatively, the project can be built from an IDE, in case you specified the IDE's CMake Generator during the configuration step.

1.4 Uninstallation

Author
Michael Beyeler

1.4.1 Linux / Mac OS X

To uninstall CARLsim on a Unix platform, open a terminal, simply remove the installed library and header files in CARLSIM5_INSTALL_DIR

Note
Any environments that have been added to ~/.bashrc must be removed manually.