X-Git-Url: http://info.iut-bm.univ-fcomte.fr/pub/gitweb/simgrid.git/blobdiff_plain/f3b7e5f4b4d7c87ee3e8827313ec966ea8fc8387..85bb22d801a4413fd0b2298a139b6207ad1ce1c5:/doc/doxygen/inside_tests.doc diff --git a/doc/doxygen/inside_tests.doc b/doc/doxygen/inside_tests.doc deleted file mode 100644 index e7aba36bbb..0000000000 --- a/doc/doxygen/inside_tests.doc +++ /dev/null @@ -1,258 +0,0 @@ -/*! -@page inside_tests Testing SimGrid - -This page will teach you how to run the tests, selecting the ones you -want, and how to add new tests to the archive. - -@tableofcontents - -SimGrid code coverage is usually between 70% and 80%, which is much -more than most projects out there. This is because we consider SimGrid -to be a rather complex project, and we want to modify it with less fear. - -We have two sets of tests in SimGrid: Each of the 10,000+ unit tests -check one specific case for one specific function, while the 500+ -integration tests run a given simulation specifically intended to -exercise a larger amount of functions together. Every example provided -in examples/ is used as an integration test, while some other torture -tests and corner cases integration tests are located in teshsuite/. -For each integration test, we ensure that the output exactly matches -the defined expectations. Since SimGrid displays the timestamp of -every logged line, this ensures that every change of the models' -prediction will be noticed. All these tests should ensure that SimGrid -is safe to use and to depend on. - -@section inside_tests_runintegration Running the tests - -Running the tests is done using the ctest binary that comes with -cmake. These tests are run for every commit and the result is publicly -available. - -@verbatim -ctest # Launch all tests -ctest -R msg # Launch only the tests which name match the string "msg" -ctest -j4 # Launch all tests in parallel, at most 4 at the same time -ctest --verbose # Display all details on what's going on -ctest --output-on-failure # Only get verbose for the tests that fail - -ctest -R msg- -j5 --output-on-failure # You changed MSG and want to check that you didn't break anything, huh? - # That's fine, I do so all the time myself. -@endverbatim - -@section inside_tests_rununit Running the unit tests - -All unit tests are packed into the unit-tests binary, that lives at the -source root. These tests are run when you launch ctest, don't worry. - -@verbatim -make unit-tests # Rebuild the test runner on need -./unit-tests # Launch all tests -./unit-tests --help # revise how it goes if you forgot -@endverbatim - - -@section inside_tests_add_units Adding unit tests - -Our unit tests are written using the Catch2 library, that is included -in the source tree. Please check for examples, listed at the end of -tools/cmake/Tests.cmake. - -It is important to keep your tests fast. We run them very very often, -and you should strive to make them as fast as possible, to not bother -the other developers. Do not hesitate to stress test your code, but -make sure that it runs reasonably fast, or nobody will run "ctest" -before committing code. - -@section inside_tests_add_integration Adding integration tests - -TESH (the TEsting SHell) is the test runner that we wrote for our -integration tests. It is distributed with the SimGrid source file, and -even comes with a man page. TESH ensures that the output produced by a -command perfectly matches the expected output. This is very precious -to ensure that no change modifies the timings computed by the models -without notice. - -To add a new integration test, you thus have 3 things to do: - - - Write the code exercising the feature you target. You should - strive to make this code clear, well documented and informative for - the users. If you manage to do so, put this somewhere under - examples/ and modify the cmake files as explained on this page: - @ref inside_cmake_examples. If you feel like you should write a - torture test that is not interesting to the users (because nobody - would sanely write something similar in user code), then put it under - teshsuite/ somewhere. - - - Write the tesh file, containing the command to run, the - provided input (if any, but almost no SimGrid test provide such an - input) and the expected output. Check the tesh man page for more - details.@n - Tesh is sometimes annoying as you have to ensure that the expected - output will always be exactly the same. In particular, your should - not output machine dependent information such as absolute data - path, nor memory addresses as they would change on each run. Several - steps can be used here, such as the obfucation of the memory - addresses unless the verbose logs are displayed (using the - #XBT_LOG_ISENABLED() macro), or the modification of the log formats - to hide the timings when they depend on the host machine.@n - The script located in /tools/tesh/generate_tesh can - help you a lot in particular if the output is large (though a smaller output is preferable). - There are also example tesh files in the /tools/tesh/ directory, that can be useful to understand the tesh syntax. - - - Add your test in the cmake infrastructure. For that, modify - the following file: - @verbatim - /teshsuite//CMakeLists.txt - @endverbatim - Make sure to pick a wise name for your test. It is often useful to - check a category of tests together. The only way to do so in ctest - is to use the -R argument that specifies a regular expression that - the test names must match. For example, you can run all MSG test - with "ctest -R msg". That explains the importance of the test - names. - -Once the name is chosen, create a new test by adding a line similar to -the following (assuming that you use tesh as expected). - -@verbatim -# Usage: ADD_TEST(test-name ${CMAKE_BINARY_DIR}/bin/tesh ) -# option --setenv bindir set the directory containing the binary -# --setenv srcdir set the directory containing the source file -# --cd set the working directory -ADD_TEST(my-test-name ${CMAKE_BINARY_DIR}/bin/tesh - --setenv bindir=${CMAKE_BINARY_DIR}/examples/my-test/ - --setenv srcdir=${CMAKE_HOME_DIRECTORY}/examples/my-test/ - --cd ${CMAKE_HOME_DIRECTORY}/examples/my-test/ - ${CMAKE_HOME_DIRECTORY}/examples/deprecated/msg/io/io.tesh -) -@endverbatim - -As usual, you must run "make distcheck" after modifying the cmake files, -to ensure that you did not forget any files in the distributed archive. - -@section inside_tests_ci Continuous Integration - -We use several systems to automatically test SimGrid with a large set -of parameters, across as many platforms as possible. -We use Jenkins on Inria -servers as a workhorse: it runs all of our tests for many -configurations. It takes a long time to answer, and it often reports -issues but when it's green, then you know that SimGrid is very fit! -We use Travis to -quickly run some tests on Linux and Mac. It answers quickly but may -miss issues. And we use AppVeyor -to build and somehow test SimGrid on windows. - -@subsection inside_tests_jenkins Jenkins on the Inria CI servers - -You should not have to change the configuration of the Jenkins tool -yourself, although you could have to change the slaves' configuration -using the CI interface of INRIA -- -refer to the CI documentation. - -The result can be seen here: https://ci.inria.fr/simgrid/ - -We have 2 interesting projects on Jenkins: -@li SimGrid - is the main project, running the tests that we spoke about.@n It is - configured (on Jenkins) to run the script tools/jenkins/build.sh -@li SimGrid-DynamicAnalysis - should be called "nightly" because it does not only run dynamic - tests, but a whole bunch of long lasting tests: valgrind (memory - errors), gcovr (coverage), Sanitizers (bad pointer usage, threading - errors, use of unspecified C constructs) and the clang static analyzer.@n It is configured - (on Jenkins) to run the script tools/jenkins/DynamicAnalysis.sh - -In each case, SimGrid gets built in -/builds/workspace/$PROJECT/build_mode/$CONFIG/label/$SERVER/build -with $PROJECT being for instance "SimGrid", $CONFIG "DEBUG" or -"ModelChecker" and $SERVER for instance "simgrid-fedora20-64-clang". - -If some configurations are known to fail on some systems (such as -model-checking on non-linux systems), go to your Project and click on -"Configuration". There, find the field "combination filter" (if your -interface language is English) and tick the checkbox; then add a -groovy-expression to disable a specific configuration. For example, in -order to disable the "ModelChecker" build on host -"small-netbsd-64-clang", use: - -@verbatim -(label=="small-netbsd-64-clang").implies(build_mode!="ModelChecker") -@endverbatim - -Just for the record, the slaves were created from the available -template with the following commands: -@verbatim -#debian/ubuntu -apt-get install gcc g++ gfortran automake cmake libboost-dev openjdk-8-jdk openjdk-8-jre libxslt-dev libxml2-dev libevent-dev libunwind-dev libdw-dev htop git python3 xsltproc libboost-context-dev -#for dynamicanalysis: -apt-get install jacoco libjacoco-java libns3-dev pcregrep gcovr ant lua5.3-dev sloccount - -#fedora -dnf install libboost-devel openjdk-8-jdk openjdk-8-jre libxslt-devel libxml2-devel xsltproc git python3 libdw-devel libevent-devel libunwind-devel htop lua5.3-devel - -#netbsd -pkg_add cmake gcc7 boost boost-headers automake openjdk8 libxslt libxml2 libunwind git htop python36 - -#opensuse -zypper install cmake automake clang boost-devel java-1_8_0-openjdk-devel libxslt-devel libxml2-devel xsltproc git python3 libdw-devel libevent-devel libunwind-devel htop binutils ggc7-fortran - -#freebsd -pkg install boost-libs cmake openjdk8 automake libxslt libxml2 libunwind git htop python3 automake gcc6 flang elfutils libevent -#+ clang-devel from ports - -#osx -brew install cmake boost libunwind-headers libxslt git python3 -@endverbatim - -@subsection inside_tests_travis Travis - -Travis is a free (as in free beer) Continuous Integration system that -open-sourced project can use freely. It is very well integrated in the -GitHub ecosystem. There is a plenty of documentation out there. Our -configuration is in the file .travis.yml as it should be, and the -result is here: https://travis-ci.org/simgrid/simgrid - -The .travis.yml configuration file can be useful if you fail to get -SimGrid to compile on modern mac systems. We use the @c brew package -manager there, and it works like a charm. - -@subsection inside_tests_appveyor AppVeyor - -AppVeyor aims at becoming the Travis of Windows. It is maybe less -mature than Travis, or maybe it is just that I'm less trained in -Windows. Our configuration is in the file appveyor.yml as it should -be, and the result is here: https://ci.appveyor.com/project/mquinson/simgrid - -We use @c Choco as a package manager on AppVeyor, and it is sufficient -for us. In the future, we will probably move to the ubuntu subsystem -of Windows 10: SimGrid performs very well under these settings, as -tested on Inria's CI servers. For the time being having a native -library is still useful for the Java users that don't want to install -anything beyond Java on their windows. - -@subsection inside_tests_debian Debian builders - -Since SimGrid is packaged in Debian, we benefit from their huge -testing infrastructure. That's an interesting torture test for our -code base. The downside is that it's only for the released versions of -SimGrid. That is why the Debian build does not stop when the tests -fail: post-releases fixes do not fit well in our workflow and we fix -only the most important breakages. - -The build results are here: -https://buildd.debian.org/status/package.php?p=simgrid - -@subsection inside_tests_sonarqube SonarQube - -SonarQube is an open-source code quality analysis solution. Their nice -code scanners are provided as plugin. The one for C++ is not free, but -open-source project can use it at no cost. That is what we are doing. - -Don't miss the great looking dashboard here: -https://sonarcloud.io/dashboard?id=simgrid_simgrid - -This tool is enriched by the script @c tools/internal/travis-sonarqube.sh -that is run from @c .travis.yml - -*/