The elements basic elements (with :ref:`pf_tag_host` and
:ref:`pf_tag_link`) are described first, and then the routes between
-any pair of hosts are explicitly given with :ref:`pf_tag_route`.
+any pair of hosts are explicitly given with :ref:`pf_tag_route`.
Any host must be given a computational speed in flops while links must
be given a latency and a bandwidth. You can write 1Gf for
-1,000,000,000 flops (full list of units in the reference guide of
-:ref:`pf_tag_host` and :ref:`pf_tag_link`).
+1,000,000,000 flops (full list of units in the reference guide of
+:ref:`pf_tag_host` and :ref:`pf_tag_link`).
Routes defined with :ref:`pf_tag_route` are symmetrical by default,
meaning that the list of traversed links from A to B is the same as
bandwidth and a relatively low diameter. To model this in SimGrid,
pass a ``topology="FAT_TREE"`` attribute to your cluster. The
``topo_parameters=#levels;#downlinks;#uplinks;link count`` follows the
-semantic introduced in `Figure 1B of this article
-<http://webee.eedev.technion.ac.il/wp-content/uploads/2014/08/publication_574.pdf>`_.
+semantic introduced in `Figure 1(b) of this article
+<https://ece.technion.ac.il/wp-content/uploads/2021/01/publication_776.pdf>`_.
Here is the meaning of this example: ``2 ; 4,4 ; 1,2 ; 1,2``
- Hosts are connected to only 1 router above them, while these routers
are connected to 2 routers above them (thus the ``1,2`` used as
``#uplink``).
-- Hosts have only one link to their router while every path between
+- Hosts have only one link to their router while every path between
level-1 routers and level-2 routers uses 2 parallel links. Thus the
``1,2`` used as ``link count``.
image. Once you `installed Docker itself
<https://docs.docker.com/install/>`_, simply do the following:
-.. code-block:: shell
+.. code-block:: console
- docker pull simgrid/tuto-smpi
- docker run -it --rm --name simgrid --volume ~/smpi-tutorial:/source/tutorial simgrid/tuto-smpi bash
+ $ docker pull simgrid/tuto-smpi
+ $ docker run --user $UID:$GID -it --rm --name simgrid --volume ~/smpi-tutorial:/source/tutorial simgrid/tuto-smpi bash
This will start a new container with all you need to take this
tutorial, and create a ``smpi-tutorial`` directory in your home on
``/source/simgrid-template-smpi`` in the image. You should copy it to
your working directory when you first log in:
-.. code-block:: shell
+.. code-block:: console
- cp -r /source/simgrid-template-smpi/* /source/tutorial
- cd /source/tutorial
+ $ cp -r /source/simgrid-template-smpi/* /source/tutorial
+ $ cd /source/tutorial
Using your Computer Natively
............................
traces. The provided code template requires ``make`` to compile. On
Debian and Ubuntu, you can get them as follows:
-.. code-block:: shell
+.. code-block:: console
- sudo apt install simgrid pajeng make gcc g++ gfortran python3 vite
+ $ sudo apt install simgrid pajeng make gcc g++ gfortran python3 vite
For R analysis of the produced traces, you may want to install R
and the `pajengr <https://github.com/schnorr/pajengr#installation/>`_ package.
-.. code-block:: shell
+.. code-block:: console
# install R and necessary packages
- sudo apt install r-base r-cran-devtools r-cran-tidyverse
+ $ sudo apt install r-base r-cran-devtools r-cran-tidyverse
# install pajengr dependencies
- sudo apt install git cmake flex bison
+ $ sudo apt install git cmake flex bison
# install the pajengr R package
- Rscript -e "library(devtools); install_github('schnorr/pajengr');"
+ $ Rscript -e "library(devtools); install_github('schnorr/pajengr');"
To take this tutorial, you will also need the platform files from the
previous section as well as the source code of the NAS Parallel
Benchmarks. Just clone `this repository
<https://framagit.org/simgrid/simgrid-template-smpi>`_ to get them all:
-.. code-block:: shell
+.. code-block:: console
- git clone https://framagit.org/simgrid/simgrid-template-smpi.git
- cd simgrid-template-smpi/
+ $ git clone https://framagit.org/simgrid/simgrid-template-smpi.git
+ $ cd simgrid-template-smpi/
If you struggle with the compilation, then you should double-check
your :ref:`SimGrid installation <install>`. On need, please refer to
:ref:`SimGrid installation <install>` if you get an error message):
-.. code-block:: shell
+.. code-block:: console
$ smpicc -O3 roundtrip.c -o roundtrip
Once compiled, you can simulate the execution of this program on 16
nodes from the ``cluster_crossbar.xml`` platform as follows:
-.. code-block:: shell
+.. code-block:: console
$ smpirun -np 16 -platform cluster_crossbar.xml -hostfile cluster_hostfile ./roundtrip
<https://www.nas.nasa.gov/publications/npb_problem_sizes.html>`_) with
4 nodes.
-.. code-block:: shell
+.. code-block:: console
$ make lu NPROCS=4 CLASS=S
(compilation logs)
visualization tracing, and convert the produced trace for later
use:
-.. code-block:: shell
+.. code-block:: console
- smpirun -np 4 -platform ../cluster_backbone.xml -trace --cfg=tracing/filename:lu.S.4.trace bin/lu.S.4
+ $ smpirun -np 4 -platform ../cluster_backbone.xml -trace --cfg=tracing/filename:lu.S.4.trace bin/lu.S.4
You can then produce a Gantt Chart with the following R chunk. You can
either copy/paste it in an R session, or `turn it into a Rscript executable
Now compile and execute the LU benchmark, class A, with 32 nodes.
-.. code-block:: shell
+.. code-block:: console
$ make lu NPROCS=32 CLASS=A
You can even generate the trace during the live simulation as follows:
-.. code-block:: shell
+.. code-block:: console
$ smpirun -trace-ti --cfg=tracing/filename:LU.A.32 -np 32 -platform ../cluster_backbone.xml bin/lu.A.32
``LU.A.32_files``. You can replay this trace with SMPI thanks to ``smpirun``.
For example, the following command replays the trace on a different platform:
-.. code-block:: shell
+.. code-block:: console
$ smpirun -np 32 -platform ../cluster_crossbar.xml -hostfile ../cluster_hostfile -replay LU.A.32
:language: cpp
:lines: 9-24
-.. code-block:: shell
+.. code-block:: console
$ smpicxx -O3 gemm_mpi.cpp -o gemm
- $ time smpirun -np 16 -platform cluster_crossbar.xml -hostfile cluster_hostfile --cfg=smpi/display-timing:yes --cfg=smpi/running-power:1000000000 ./gemm
+ $ time smpirun -np 16 -platform cluster_crossbar.xml -hostfile cluster_hostfile --cfg=smpi/display-timing:yes --cfg=smpi/host-speed:1000000000 ./gemm
-This should end quite quickly, as the size of each matrix is only 1000x1000.
+This should end quite quickly, as the size of each matrix is only 1000x1000.
But what happens if we want to simulate larger runs?
Replace the size by 2000, 3000, and try again.
The simulation time increases a lot, while there are no more MPI calls performed, only computation.
-The ``--cfg=smpi/display-timing`` option gives more details about execution
+The ``--cfg=smpi/display-timing`` option gives more details about execution
and advises using sampling if the time spent in computing loops seems too high.
-The ``--cfg=smpi/host-speed:1000000000`` option sets the speed of the processor used for
-running the simulation. Here we say that its speed is the same as one of the
-processors we are simulating (1Gf), so that 1 second of computation is injected
+The ``--cfg=smpi/host-speed:1000000000`` option sets the speed of the processor used for
+running the simulation. Here we say that its speed is the same as one of the
+processors we are simulating (1Gf), so that 1 second of computation is injected
as 1 second in the simulation.
-.. code-block:: shell
+.. code-block:: console
[5.568556] [smpi_kernel/INFO] Simulated time: 5.56856 seconds.
You may want to use sampling functions or trace replay to reduce this.
So in our case (size 3000), the simulation ran for 25 seconds, and the simulated time was 5.57s at the end.
-Computation by itself took 24 seconds, and can quickly grow with larger sizes
+Computation by itself took 24 seconds, and can quickly grow with larger sizes
(as computation is really performed, there will be variability between similar runs).
-SMPI provides sampling macros to accelerate simulation by sampling iterations
-of large computation loops, and skip computation after a certain amount of iterations,
+SMPI provides sampling macros to accelerate simulation by sampling iterations
+of large computation loops, and skip computation after a certain amount of iterations,
or when the sampling is stable enough.
The two macros only slightly differ :
- ``SMPI_SAMPLE_GLOBAL`` : the specified number of samples is produced by all processors
- ``SMPI_SAMPLE_LOCAL`` : each process executes a specified number of iterations
-So if the size of the computed part varies between processes (imbalance),
+So if the size of the computed part varies between processes (imbalance),
it's safer to use the LOCAL one.
To use one of them, replacing the external for loop of the multiply routine:
SMPI_SAMPLE_GLOBAL(int i = istart, i <= iend, ++i, 10, 0.005)
The first three parameters are the ones from the loop, while the two last ones are for sampling.
-They mean that at most 10 iterations will be performed and that the sampling phase can be exited
+They mean that at most 10 iterations will be performed and that the sampling phase can be exited
earlier if the expected stability is reached after fewer samples.
-Now run the code again with various sizes and parameters and check the time taken for the
+Now run the code again with various sizes and parameters and check the time taken for the
simulation, as well as the resulting simulated time.
-.. code-block:: shell
+.. code-block:: console
[5.575691] [smpi_kernel/INFO] Simulated time: 5.57569 seconds.
The simulation took 1.23698 seconds (after parsing and platform setup)
0.0319454 seconds were actual computation of the application
-In this case, the simulation only took 1.2 seconds, while the simulated time
+In this case, the simulation only took 1.2 seconds, while the simulated time
remained almost identical.
The computation results will obviously be altered since most computations are skipped.
With 85 processes and class C, the DT simulated benchmark will try to allocate 35GB of memory,
which may not be available on the node you are using.
-To avoid this we can simply replace the largest calls to malloc and free by calls
+To avoid this we can simply replace the largest calls to malloc and free by calls
to ``SMPI_SHARED_MALLOC`` and ``SMPI_SHARED_FREE``.
This means that all processes will share one single instance of this buffer.
As for sampling, results will be altered, and this should not be used for control structures.
Once done, you can now run
-.. code-block:: shell
+.. code-block:: console
$ make dt NPROCS=85 CLASS=C
(compilation logs)
And simulation should finish without swapping/crashing (Ignore the warning about the return value).
-If control structures are also problematic, you can use ``SMPI_PARTIAL_SHARED_MALLOC(size, offsets, offsetscount)``
-macro, which shares only specific parts of the structure between processes,
+If control structures are also problematic, you can use ``SMPI_PARTIAL_SHARED_MALLOC(size, offsets, offsetscount)``
+macro, which shares only specific parts of the structure between processes,
and use specific memory for the important parts.
It can be freed afterward with SMPI_SHARED_FREE.
If allocations are performed with malloc or calloc, SMPI (from version 3.25) provides the option
``--cfg=smpi/auto-shared-malloc-shared:n`` which will replace all allocations above size n bytes by
-shared allocations. The value has to be carefully selected to avoid smaller control arrays,
+shared allocations. The value has to be carefully selected to avoid smaller control arrays,
containing data necessary for the completion of the run.
Try to run the (non modified) DT example again, with values going from 10 to 100,000 to show that
too small values can cause crashes.
A useful option to identify the largest allocations in the code is ``--cfg=smpi/display-allocs:yes`` (from 3.27).
It will display at the end of a (successful) run the largest allocations and their locations, helping pinpoint the
targets for sharing, or setting the threshold for automatic ones.
-For DT, the process would be to run a smaller class of problems,
+For DT, the process would be to run a smaller class of problems,
-.. code-block:: shell
+.. code-block:: console
$ make dt NPROCS=21 CLASS=A
$ smpirun --cfg=smpi/display-allocs:yes -np 21 -platform ../cluster_backbone.xml bin/dt.A.x BH
Which should output:
-.. code-block:: shell
+.. code-block:: console
[smpi_utils/INFO] Memory Usage: Simulated application allocated 198533192 bytes during its lifetime through malloc/calloc calls.
Largest allocation at once from a single process was 3553184 bytes, at dt.c:388. It was called 3 times during the whole simulation.
