3 Simulating MPI Applications
4 ===========================
9 SimGrid can not only :ref:`simulate algorithms <usecase_simalgo>`, but
10 it can also be used to execute real MPI applications on top of
11 virtual, simulated platforms with the SMPI module. Even complex
12 C/C++/F77/F90 applications should run out of the box in this
13 environment. In fact, almost all proxy apps provided by the `ExaScale
14 Project <https://proxyapps.exascaleproject.org/>`_ only require minor
15 modifications to `run on top of SMPI
16 <https://github.com/simgrid/SMPI-proxy-apps/>`_.
18 This setting permits to debug your MPI applications in a perfectly
19 reproducible setup, with no Heisenbugs. Enjoy the full Clairevoyance
20 provided by the simulator while running what-if analysis on platforms
21 that are still to be built! Several `production-grade MPI applications
22 <https://framagit.org/simgrid/SMPI-proxy-apps#full-scale-applications>`_
23 use SimGrid for their integration and performance testing.
25 MPI 2.2 is already partially covered: over 160 primitives are
26 supported. Some parts of the standard are still missing: MPI-IO, MPI3
27 collectives, spawning ranks, and some others. If one of the functions
28 you use is still missing, please drop us an email. We may find the
29 time to implement it for you.
31 Multi-threading support is very limited in SMPI. Only funneled
32 applications are supported: at most one thread per rank can issue any
33 MPI calls. For better timing predictions, your application should even
34 be completely mono-threaded. Using OpenMP (or pthreads directly) may
35 greatly decrease SimGrid predictive power. That may still be OK if you
36 only plan to debug your application in a reproducible setup, without
37 any performance-related analysis.
42 In SMPI, communications are simulated while computations are
43 emulated. This means that while computations occur as they would in
44 the real systems, communication calls are intercepted and achived by
47 To start using SMPI, you just need to compile your application with
48 ``smpicc`` instead of ``mpicc``, or with ``smpiff`` instead of
49 ``mpiff``, or with ``smpicxx`` instead of ``mpicxx``. Then, the only
50 difference between the classical ``mpirun`` and the new ``smpirun`` is
51 that it requires a new parameter ``-platform`` with a file describing
52 the virtual platform on which your application shall run.
54 Internally, all ranks of your application are executed as threads of a
55 single unix process. That's not a problem if your application has
56 global variables, because ``smpirun`` loads one application instance
57 per MPI rank as if it was another dynamic library. Then, MPI
58 communication calls are implemented using SimGrid: data is exchanged
59 through memory copy, while the simulator's performance models are used
60 to predict the time taken by each communications. Any computations
61 occuring between two MPI calls are benchmarked, and the corresponding
62 time is reported into the simulator.
64 .. image:: /tuto_smpi/img/big-picture.svg
67 Describing Your Platform
68 ------------------------
70 As a SMPI user, you are supposed to provide a description of your
71 virtual platform, that is mostly a set of simulated hosts and network
72 links with some performance characteristics. SimGrid provides a plenty
73 of :ref:`documentation <platform>` and examples (in the
74 `examples/platforms <https://framagit.org/simgrid/simgrid/tree/master/examples/platforms>`_
75 source directory), and this section only shows a small set of introductory
78 Feel free to skip this section if you want to jump right away to usage
81 Simple Example with 3 hosts
82 ...........................
84 At the most basic level, you can describe your simulated platform as a
85 graph of hosts and network links. For instance:
87 .. image:: /tuto_smpi/3hosts.png
90 .. literalinclude:: /tuto_smpi/3hosts.xml
93 Note the way in which hosts, links, and routes are defined in
94 this XML. All hosts are defined with a speed (in Gflops), and links
95 with a latency (in us) and bandwidth (in MBytes per second). Other
96 units are possible and written as expected. Routes specify the list of
97 links encountered from one route to another. Routes are symmetrical by
100 Cluster with a Crossbar
101 .......................
103 A very common parallel computing platform is a homogeneous cluster in
104 which hosts are interconnected via a crossbar switch with as many
105 ports as hosts, so that any disjoint pairs of hosts can communicate
106 concurrently at full speed. For instance:
108 .. literalinclude:: ../../examples/platforms/cluster_crossbar.xml
112 One specifies a name prefix and suffix for each host, and then give an
113 integer range. In the example the cluster contains 262145 hosts (!),
114 named ``host-0.simgrid.org`` to ``host-262144.simgrid.org``. All hosts
115 have the same power (1 Gflop/sec) and are connected to the switch via
116 links with same bandwidth (125 MBytes/sec) and latency (50
123 Cluster with a Shared Backbone
124 ..............................
126 Another popular model for a parallel platform is that of a set of
127 homogeneous hosts connected to a shared communication medium, a
128 backbone, with some finite bandwidth capacity and on which
129 communicating host pairs can experience contention. For instance:
132 .. literalinclude:: ../../examples/platforms/cluster_backbone.xml
136 The only differences with the crossbar cluster above are the ``bb_bw``
137 and ``bb_lat`` attributes that specify the backbone characteristics
138 (here, a 500 microseconds latency and a 2.25 GByte/sec
139 bandwidth). This link is used for every communication within the
140 cluster. The route from ``node-0.acme.org`` to ``node-1.acme.org``
141 counts 3 links: the private link of ``node-0.acme.org``, the backbone
142 and the private link of ``node-1.acme.org``.
151 Many HPC facilities use torus clusters to reduce sharing and
152 performance loss on concurrent internal communications. Modeling this
153 in SimGrid is very easy. Simply add a ``topology="TORUS"`` attribute
154 to your cluster. Configure it with the ``topo_parameters="X,Y,Z"``
155 attribute, where ``X``, ``Y`` and ``Z`` are the dimension of your
158 .. image:: ../../examples/platforms/cluster_torus.svg
161 .. literalinclude:: ../../examples/platforms/cluster_torus.xml
164 Note that in this example, we used ``loopback_bw`` and
165 ``loopback_lat`` to specify the characteristics of the loopback link
166 of each node (i.e., the link allowing each node to communicate with
167 itself). We could have done so in previous example too. When no
168 loopback is given, the communication from a node to itself is handled
169 as if it were two distinct nodes: it goes twice through the private
170 link and through the backbone (if any).
175 This topology was introduced to reduce the amount of links in the
176 cluster (and thus reduce its price) while maintaining a high bisection
177 bandwidth and a relatively low diameter. To model this in SimGrid,
178 pass a ``topology="FAT_TREE"`` attribute to your cluster. The
179 ``topo_parameters=#levels;#downlinks;#uplinks;link count`` follows the
180 semantic introduced in the `Figure 1B of this article
181 <http://webee.eedev.technion.ac.il/wp-content/uploads/2014/08/publication_574.pdf>`_.
183 Here is the meaning of this example: ``2 ; 4,4 ; 1,2 ; 1,2``
185 - That's a two-level cluster (thus the initial ``2``).
186 - Routers are connected to 4 elements below them, regardless of its
187 level. Thus the ``4,4`` component that is used as
188 ``#downlinks``. This means that the hosts are grouped by 4 on a
189 given router, and that there is 4 level-1 routers (in the middle of
191 - Hosts are connected to only 1 router above them, while these routers
192 are connected to 2 routers above them (thus the ``1,2`` used as
194 - Hosts have only one link to their router while every path between a
195 level-1 routers and level-2 routers use 2 parallel links. Thus the
196 ``1,2`` that is used as ``link count``.
198 .. image:: ../../examples/platforms/cluster_fat_tree.svg
201 .. literalinclude:: ../../examples/platforms/cluster_fat_tree.xml
209 This topology was introduced to further reduce the amount of links
210 while maintaining a high bandwidth for local communications. To model
211 this in SimGrid, pass a ``topology="DRAGONFLY"`` attribute to your
214 .. literalinclude:: ../../examples/platforms/cluster_dragonfly.xml
219 Add the image, and the documuentation of the topo_parameters.
224 We only glanced over the abilities offered by SimGrid to describe the
225 platform topology. Other networking zones model non-HPC platforms
226 (such as wide area networks, ISP network comprising set-top boxes, or
227 even your own routing schema). You can interconnect several networking
228 zones in your platform to form a tree of zones, that is both a time-
229 and memory-efficient representation of distributed platforms. Please
230 head to the dedicated :ref:`documentation <platform>` for more
236 It is time to start using SMPI yourself. For that, you first need to
237 install it somehow, and then you will need a MPI application to play with.
242 The easiest way to take the tutorial is to use the dedicated Docker
243 image. Once you `installed Docker itself
244 <https://docs.docker.com/install/>`_, simply do the following:
246 .. code-block:: shell
248 docker pull simgrid/tuto-smpi
249 docker run -it --rm --name simgrid --volume ~/smpi-tutorial:/src/tutorial simgrid/tuto-smpi bash
251 This will start a new container with all you need to take this
252 tutorial, and create a ``smpi-tutorial`` directory in your home on
253 your host machine that will be visible as ``/src/tutorial`` within the
254 container. You can then edit the files you want with your favorite
255 editor in ``~/smpi-tutorial``, and compile them within the
256 container to enjoy the provided dependencies.
260 Any change to the container out of ``/src/tutorial`` will be lost
261 when you log out of the container, so don't edit the other files!
263 All needed dependencies are already installed in this container
264 (SimGrid, a C/C++ compiler, a Fortran compiler, make, pajeng and
265 R). Vite being only optional in this tutorial, it is not installed to
266 reduce the image size.
268 The code template is available under ``/src/simgrid-template-smpi`` in
269 the image. You should copy it to your working directory when you first
272 .. code-block:: shell
274 cp -r /src/simgrid-template-smpi/* /src/tutorial
277 Using your Computer Natively
278 ............................
280 To take the tutorial on your machine, you first need to :ref:`install
281 SimGrid <install>`, the C/C++/Fortran compilers and also ``pajeng`` to
282 visualize the traces. You may want to install `Vite
283 <http://vite.gforge.inria.fr/>`_ to get a first glance at the
284 traces. The provided code template requires make to compile. On
285 Debian and Ubuntu for example, you can get them as follows:
287 .. code-block:: shell
289 sudo apt install simgrid pajeng make gcc g++ gfortran vite
291 An initial version of the source code is provided on framagit. This
292 template compiles with cmake. If SimGrid is correctly installed, you
293 should be able to clone the `repository
294 <https://framagit.org/simgrid/simgrid-template-smpi>`_ and recompile
295 everything as follows:
297 .. code-block:: shell
299 git clone git@framagit.org:simgrid/simgrid-template-smpi.git
300 cd simgrid-template-smpi/
304 If you struggle with the compilation, then you should double check
305 your :ref:`SimGrid installation <install>`. On need, please refer to
306 the :ref:`Troubleshooting your Project Setup
307 <install_yours_troubleshooting>` section.
310 .. LocalWords: SimGrid
