2 @defgroup SMPI_API SMPI: Simulate real MPI applications
3 @brief Programming environment for the simulation of MPI applications
9 SMPI enables the study of MPI application by emulating them on top of
10 the SimGrid simulator. This is particularly interesting to study
11 existing MPI applications within the comfort of the simulator. The
12 SMPI reference article is available at
13 https://hal.inria.fr/hal-01415484. You should also read the
14 <a href="http://simgrid.org/tutorials/simgrid-smpi-101.pdf">SMPI
15 introductory slides</a>.
17 Our goal is to enable the study of **unmodified MPI applications**.
18 Some constructs and features are still missing, but we can probably
19 add them on demand. If you already used MPI before, SMPI should sound
20 very familiar to you: Use smpicc instead of mpicc, and smpirun instead
21 of mpirun. The main difference is that smpirun takes a virtual
22 platform as extra parameter (see @ref platform).
24 If you are new to MPI, you should first take our online [SMPI
25 CourseWare](https://simgrid.github.io/SMPI_CourseWare/). It consists
26 in several projects that progressively introduce the MPI concepts. It
27 proposes to use SimGrid and SMPI to run the experiments, but the
28 learning objectives are centered on MPI itself.
30 For **further scalability**, you may modify your code to speed up your
31 studies or save memory space. Maximal **simulation accuracy**
32 requires some specific care from you.
35 - @ref SMPI_use_compile
38 - @ref SMPI_use_colls_algos
39 - @ref SMPI_use_colls_tracing
41 - @ref SMPI_what_coverage
42 - @ref SMPI_what_globals
44 - @ref SMPI_adapting_size
45 - @ref SMPI_adapting_speed
47 - @ref SMPI_troubleshooting
48 - @ref SMPI_trouble_buildchain
51 @section SMPI_use Using SMPI
53 @subsection SMPI_use_compile Compiling your code
55 If your application is in C, then simply use <tt>smpicc</tt> as a
56 compiler just like you use mpicc with other MPI implementations. This
57 script still calls your default compiler (gcc, clang, ...) and adds
58 the right compilation flags along the way. If your application is in
59 C++, Fortran 77 or Fortran 90, use respectively <tt>smpicxx</tt>,
60 <tt>smpiff</tt> or <tt>smpif90</tt>.
62 @subsection SMPI_use_exec Executing your code on the simulator
64 Use the <tt>smpirun</tt> script as follows for that:
67 smpirun -hostfile my_hostfile.txt -platform my_platform.xml ./program -blah
70 - <tt>my_hostfile.txt</tt> is a classical MPI hostfile (that is, this
71 file lists the machines on which the processes must be dispatched, one
73 - <tt>my_platform.xml</tt> is a classical SimGrid platform file. Of
74 course, the hosts of the hostfile must exist in the provided
76 - <tt>./program</tt> is the MPI program to simulate, that you
77 compiled with <tt>smpicc</tt>
78 - <tt>-blah</tt> is a command-line parameter passed to this program.
80 <tt>smpirun</tt> accepts other parameters, such as <tt>-np</tt> if you
81 don't want to use all the hosts defined in the hostfile, <tt>-map</tt>
82 to display on which host each rank gets mapped of <tt>-trace</tt> to
83 activate the tracing during the simulation. You can get the full list
90 @subsection SMPI_use_colls Simulating collective operations
92 MPI collective operations are crucial to the performance of MPI
93 applications and must be carefully optimized according to many
94 parameters. Every existing implementation provides several algorithms
95 for each collective operation, and selects by default the best suited
96 one, depending on the sizes sent, the number of nodes, the
97 communicator, or the communication library being used. These
98 decisions are based on empirical results and theoretical complexity
99 estimation, and are very different between MPI implementations. In
100 most cases, the users can also manually tune the algorithm used for
101 each collective operation.
103 SMPI can simulate the behavior of several MPI implementations:
105 <a href="http://star-mpi.sourceforge.net/">STAR-MPI</a>, and
106 MVAPICH2. For that, it provides 115 collective algorithms and several
107 selector algorithms, that were collected directly in the source code
108 of the targeted MPI implementations.
110 You can switch the automatic selector through the
111 \c smpi/coll-selector configuration item. Possible values:
113 - <b>ompi</b>: default selection logic of OpenMPI (version 1.7)
114 - <b>mpich</b>: default selection logic of MPICH (version 3.0.4)
115 - <b>mvapich2</b>: selection logic of MVAPICH2 (version 1.9) tuned
116 on the Stampede cluster
117 - <b>impi</b>: preliminary version of an Intel MPI selector (version
118 4.1.3, also tuned for the Stampede cluster). Due the closed source
119 nature of Intel MPI, some of the algorithms described in the
120 documentation are not available, and are replaced by mvapich ones.
121 - <b>default</b>: legacy algorithms used in the earlier days of
122 SimGrid. Do not use for serious perform performance studies.
125 @subsubsection SMPI_use_colls_algos Available algorithms
127 You can also pick the algorithm used for each collective with the
128 corresponding configuration item. For example, to use the pairwise
129 alltoall algorithm, one should add \c --cfg=smpi/alltoall:pair to the
130 line. This will override the selector (if any) for this algorithm.
131 It means that the selected algorithm will be used
133 Warning: Some collective may require specific conditions to be
134 executed correctly (for instance having a communicator with a power of
135 two number of nodes only), which are currently not enforced by
136 Simgrid. Some crashes can be expected while trying these algorithms
137 with unusual sizes/parameters
141 Most of these are best described in <a href="http://www.cs.arizona.edu/~dkl/research/papers/ics06.pdf">STAR-MPI</a>
143 - default: naive one, by default
144 - ompi: use openmpi selector for the alltoall operations
145 - mpich: use mpich selector for the alltoall operations
146 - mvapich2: use mvapich2 selector for the alltoall operations
147 - impi: use intel mpi selector for the alltoall operations
148 - automatic (experimental): use an automatic self-benchmarking algorithm
149 - bruck: Described by Bruck et.al. in <a href="http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=642949">this paper</a>
150 - 2dmesh: organizes the nodes as a two dimensional mesh, and perform allgather
152 - 3dmesh: adds a third dimension to the previous algorithm
153 - rdb: recursive doubling : extends the mesh to a nth dimension, each one
155 - pair: pairwise exchange, only works for power of 2 procs, size-1 steps,
156 each process sends and receives from the same process at each step
157 - pair_light_barrier: same, with small barriers between steps to avoid
159 - pair_mpi_barrier: same, with MPI_Barrier used
160 - pair_one_barrier: only one barrier at the beginning
161 - ring: size-1 steps, at each step a process send to process (n+i)%size, and receives from (n-i)%size
162 - ring_light_barrier: same, with small barriers between some phases to avoid contention
163 - ring_mpi_barrier: same, with MPI_Barrier used
164 - ring_one_barrier: only one barrier at the beginning
165 - basic_linear: posts all receives and all sends,
166 starts the communications, and waits for all communication to finish
167 - mvapich2_scatter_dest: isend/irecv with scattered destinations, posting only a few messages at the same time
171 - default: naive one, by default
172 - ompi: use openmpi selector for the alltoallv operations
173 - mpich: use mpich selector for the alltoallv operations
174 - mvapich2: use mvapich2 selector for the alltoallv operations
175 - impi: use intel mpi selector for the alltoallv operations
176 - automatic (experimental): use an automatic self-benchmarking algorithm
177 - bruck: same as alltoall
178 - pair: same as alltoall
179 - pair_light_barrier: same as alltoall
180 - pair_mpi_barrier: same as alltoall
181 - pair_one_barrier: same as alltoall
182 - ring: same as alltoall
183 - ring_light_barrier: same as alltoall
184 - ring_mpi_barrier: same as alltoall
185 - ring_one_barrier: same as alltoall
186 - ompi_basic_linear: same as alltoall
190 - default: naive one, by default
191 - ompi: use openmpi selector for the gather operations
192 - mpich: use mpich selector for the gather operations
193 - mvapich2: use mvapich2 selector for the gather operations
194 - impi: use intel mpi selector for the gather operations
195 - automatic (experimental): use an automatic self-benchmarking algorithm
196 which will iterate over all implemented versions and output the best
197 - ompi_basic_linear: basic linear algorithm from openmpi, each process sends to the root
198 - ompi_binomial: binomial tree algorithm
199 - ompi_linear_sync: same as basic linear, but with a synchronization at the
200 beginning and message cut into two segments.
201 - mvapich2_two_level: SMP-aware version from MVAPICH. Gather first intra-node (defaults to mpich's gather), and then exchange with only one process/node. Use mvapich2 selector to change these to tuned algorithms for Stampede cluster.
205 - default: naive one, by default
206 - ompi: use openmpi selector for the barrier operations
207 - mpich: use mpich selector for the barrier operations
208 - mvapich2: use mvapich2 selector for the barrier operations
209 - impi: use intel mpi selector for the barrier operations
210 - automatic (experimental): use an automatic self-benchmarking algorithm
211 - ompi_basic_linear: all processes send to root
212 - ompi_two_procs: special case for two processes
213 - ompi_bruck: nsteps = sqrt(size), at each step, exchange data with rank-2^k and rank+2^k
214 - ompi_recursivedoubling: recursive doubling algorithm
215 - ompi_tree: recursive doubling type algorithm, with tree structure
216 - ompi_doublering: double ring algorithm
217 - mvapich2_pair: pairwise algorithm
221 - default: naive one, by default
222 - ompi: use openmpi selector for the scatter operations
223 - mpich: use mpich selector for the scatter operations
224 - mvapich2: use mvapich2 selector for the scatter operations
225 - impi: use intel mpi selector for the scatter operations
226 - automatic (experimental): use an automatic self-benchmarking algorithm
227 - ompi_basic_linear: basic linear scatter
228 - ompi_binomial: binomial tree scatter
229 - mvapich2_two_level_direct: SMP aware algorithm, with an intra-node stage (default set to mpich selector), and then a basic linear inter node stage. Use mvapich2 selector to change these to tuned algorithms for Stampede cluster.
230 - mvapich2_two_level_binomial: SMP aware algorithm, with an intra-node stage (default set to mpich selector), and then a binomial phase. Use mvapich2 selector to change these to tuned algorithms for Stampede cluster.
234 - default: naive one, by default
235 - ompi: use openmpi selector for the reduce operations
236 - mpich: use mpich selector for the reduce operations
237 - mvapich2: use mvapich2 selector for the reduce operations
238 - impi: use intel mpi selector for the reduce operations
239 - automatic (experimental): use an automatic self-benchmarking algorithm
240 - arrival_pattern_aware: root exchanges with the first process to arrive
241 - binomial: uses a binomial tree
242 - flat_tree: uses a flat tree
243 - NTSL: Non-topology-specific pipelined linear-bcast function
244 0->1, 1->2 ,2->3, ....., ->last node: in a pipeline fashion, with segments
246 - scatter_gather: scatter then gather
247 - ompi_chain: openmpi reduce algorithms are built on the same basis, but the
248 topology is generated differently for each flavor
249 chain = chain with spacing of size/2, and segment size of 64KB
250 - ompi_pipeline: same with pipeline (chain with spacing of 1), segment size
251 depends on the communicator size and the message size
252 - ompi_binary: same with binary tree, segment size of 32KB
253 - ompi_in_order_binary: same with binary tree, enforcing order on the
255 - ompi_binomial: same with binomial algo (redundant with default binomial
257 - ompi_basic_linear: basic algorithm, each process sends to root
258 - mvapich2_knomial: k-nomial algorithm. Default factor is 4 (mvapich2 selector adapts it through tuning)
259 - mvapich2_two_level: SMP-aware reduce, with default set to mpich both for intra and inter communicators. Use mvapich2 selector to change these to tuned algorithms for Stampede cluster.
260 - rab: <a href="https://fs.hlrs.de/projects/par/mpi//myreduce.html">Rabenseifner</a>'s reduce algorithm
264 - default: naive one, by default
265 - ompi: use openmpi selector for the allreduce operations
266 - mpich: use mpich selector for the allreduce operations
267 - mvapich2: use mvapich2 selector for the allreduce operations
268 - impi: use intel mpi selector for the allreduce operations
269 - automatic (experimental): use an automatic self-benchmarking algorithm
270 - lr: logical ring reduce-scatter then logical ring allgather
271 - rab1: variations of the <a href="https://fs.hlrs.de/projects/par/mpi//myreduce.html">Rabenseifner</a> algorithm: reduce_scatter then allgather
272 - rab2: variations of the <a href="https://fs.hlrs.de/projects/par/mpi//myreduce.html">Rabenseifner</a> algorithm: alltoall then allgather
273 - rab_rsag: variation of the <a href="https://fs.hlrs.de/projects/par/mpi//myreduce.html">Rabenseifner</a> algorithm: recursive doubling
274 reduce_scatter then recursive doubling allgather
275 - rdb: recursive doubling
276 - smp_binomial: binomial tree with smp: binomial intra
277 SMP reduce, inter reduce, inter broadcast then intra broadcast
278 - smp_binomial_pipeline: same with segment size = 4096 bytes
279 - smp_rdb: intra: binomial allreduce, inter: Recursive
280 doubling allreduce, intra: binomial broadcast
281 - smp_rsag: intra: binomial allreduce, inter: reduce-scatter,
282 inter:allgather, intra: binomial broadcast
283 - smp_rsag_lr: intra: binomial allreduce, inter: logical ring
284 reduce-scatter, logical ring inter:allgather, intra: binomial broadcast
285 - smp_rsag_rab: intra: binomial allreduce, inter: rab
286 reduce-scatter, rab inter:allgather, intra: binomial broadcast
287 - redbcast: reduce then broadcast, using default or tuned algorithms if specified
288 - ompi_ring_segmented: ring algorithm used by OpenMPI
289 - mvapich2_rs: rdb for small messages, reduce-scatter then allgather else
290 - mvapich2_two_level: SMP-aware algorithm, with mpich as intra algoritm, and rdb as inter (Change this behavior by using mvapich2 selector to use tuned values)
291 - rab: default <a href="https://fs.hlrs.de/projects/par/mpi//myreduce.html">Rabenseifner</a> implementation
293 #### MPI_Reduce_scatter
295 - default: naive one, by default
296 - ompi: use openmpi selector for the reduce_scatter operations
297 - mpich: use mpich selector for the reduce_scatter operations
298 - mvapich2: use mvapich2 selector for the reduce_scatter operations
299 - impi: use intel mpi selector for the reduce_scatter operations
300 - automatic (experimental): use an automatic self-benchmarking algorithm
301 - ompi_basic_recursivehalving: recursive halving version from OpenMPI
302 - ompi_ring: ring version from OpenMPI
303 - mpich_pair: pairwise exchange version from MPICH
304 - mpich_rdb: recursive doubling version from MPICH
305 - mpich_noncomm: only works for power of 2 procs, recursive doubling for noncommutative ops
310 - default: naive one, by default
311 - ompi: use openmpi selector for the allgather operations
312 - mpich: use mpich selector for the allgather operations
313 - mvapich2: use mvapich2 selector for the allgather operations
314 - impi: use intel mpi selector for the allgather operations
315 - automatic (experimental): use an automatic self-benchmarking algorithm
316 - 2dmesh: see alltoall
317 - 3dmesh: see alltoall
318 - bruck: Described by Bruck et.al. in <a href="http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=642949">
319 Efficient algorithms for all-to-all communications in multiport message-passing systems</a>
320 - GB: Gather - Broadcast (uses tuned version if specified)
321 - loosely_lr: Logical Ring with grouping by core (hardcoded, default
323 - NTSLR: Non Topology Specific Logical Ring
324 - NTSLR_NB: Non Topology Specific Logical Ring, Non Blocking operations
327 - rhv: only power of 2 number of processes
329 - SMP_NTS: gather to root of each SMP, then every root of each SMP node
330 post INTER-SMP Sendrecv, then do INTRA-SMP Bcast for each receiving message,
331 using logical ring algorithm (hardcoded, default processes/SMP: 8)
332 - smp_simple: gather to root of each SMP, then every root of each SMP node
333 post INTER-SMP Sendrecv, then do INTRA-SMP Bcast for each receiving message,
334 using simple algorithm (hardcoded, default processes/SMP: 8)
335 - spreading_simple: from node i, order of communications is i -> i + 1, i ->
336 i + 2, ..., i -> (i + p -1) % P
337 - ompi_neighborexchange: Neighbor Exchange algorithm for allgather.
338 Described by Chen et.al. in <a href="http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=1592302">Performance Evaluation of Allgather Algorithms on Terascale Linux Cluster with Fast Ethernet</a>
339 - mvapich2_smp: SMP aware algorithm, performing intra-node gather, inter-node allgather with one process/node, and bcast intra-node
344 - default: naive one, by default
345 - ompi: use openmpi selector for the allgatherv operations
346 - mpich: use mpich selector for the allgatherv operations
347 - mvapich2: use mvapich2 selector for the allgatherv operations
348 - impi: use intel mpi selector for the allgatherv operations
349 - automatic (experimental): use an automatic self-benchmarking algorithm
350 - GB: Gatherv - Broadcast (uses tuned version if specified, but only for
351 Bcast, gatherv is not tuned)
354 - ompi_neighborexchange: see allgather
355 - ompi_bruck: see allgather
356 - mpich_rdb: recursive doubling algorithm from MPICH
357 - mpich_ring: ring algorithm from MPICh - performs differently from the one from STAR-MPI
361 - default: naive one, by default
362 - ompi: use openmpi selector for the bcast operations
363 - mpich: use mpich selector for the bcast operations
364 - mvapich2: use mvapich2 selector for the bcast operations
365 - impi: use intel mpi selector for the bcast operations
366 - automatic (experimental): use an automatic self-benchmarking algorithm
367 - arrival_pattern_aware: root exchanges with the first process to arrive
368 - arrival_pattern_aware_wait: same with slight variation
369 - binomial_tree: binomial tree exchange
370 - flattree: flat tree exchange
371 - flattree_pipeline: flat tree exchange, message split into 8192 bytes pieces
372 - NTSB: Non-topology-specific pipelined binary tree with 8192 bytes pieces
373 - NTSL: Non-topology-specific pipelined linear with 8192 bytes pieces
374 - NTSL_Isend: Non-topology-specific pipelined linear with 8192 bytes pieces, asynchronous communications
375 - scatter_LR_allgather: scatter followed by logical ring allgather
376 - scatter_rdb_allgather: scatter followed by recursive doubling allgather
377 - arrival_scatter: arrival pattern aware scatter-allgather
378 - SMP_binary: binary tree algorithm with 8 cores/SMP
379 - SMP_binomial: binomial tree algorithm with 8 cores/SMP
380 - SMP_linear: linear algorithm with 8 cores/SMP
381 - ompi_split_bintree: binary tree algorithm from OpenMPI, with message split in 8192 bytes pieces
382 - ompi_pipeline: pipeline algorithm from OpenMPI, with message split in 128KB pieces
383 - mvapich2_inter_node: Inter node default mvapich worker
384 - mvapich2_intra_node: Intra node default mvapich worker
385 - mvapich2_knomial_intra_node: k-nomial intra node default mvapich worker. default factor is 4.
387 #### Automatic evaluation
389 (Warning: This is still very experimental)
391 An automatic version is available for each collective (or even as a selector). This specific
392 version will loop over all other implemented algorithm for this particular collective, and apply
393 them while benchmarking the time taken for each process. It will then output the quickest for
394 each process, and the global quickest. This is still unstable, and a few algorithms which need
395 specific number of nodes may crash.
397 #### Adding an algorithm
399 To add a new algorithm, one should check in the src/smpi/colls folder how other algorithms
400 are coded. Using plain MPI code inside Simgrid can't be done, so algorithms have to be
401 changed to use smpi version of the calls instead (MPI_Send will become smpi_mpi_send). Some functions may have different signatures than their MPI counterpart, please check the other algorithms or contact us using <a href="http://lists.gforge.inria.fr/mailman/listinfo/simgrid-devel">SimGrid developers mailing list</a>.
403 Example: adding a "pair" version of the Alltoall collective.
405 - Implement it in a file called alltoall-pair.c in the src/smpi/colls folder. This file should include colls_private.hpp.
407 - The name of the new algorithm function should be smpi_coll_tuned_alltoall_pair, with the same signature as MPI_Alltoall.
409 - Once the adaptation to SMPI code is done, add a reference to the file ("src/smpi/colls/alltoall-pair.c") in the SMPI_SRC part of the DefinePackages.cmake file inside buildtools/cmake, to allow the file to be built and distributed.
411 - To register the new version of the algorithm, simply add a line to the corresponding macro in src/smpi/colls/cools.h ( add a "COLL_APPLY(action, COLL_ALLTOALL_SIG, pair)" to the COLL_ALLTOALLS macro ). The algorithm should now be compiled and be selected when using --cfg=smpi/alltoall:pair at runtime.
413 - To add a test for the algorithm inside Simgrid's test suite, juste add the new algorithm name in the ALLTOALL_COLL list found inside teshsuite/smpi/CMakeLists.txt . When running ctest, a test for the new algorithm should be generated and executed. If it does not pass, please check your code or contact us.
415 - Please submit your patch for inclusion in SMPI, for example through a pull request on GitHub or directly per email.
417 @subsubsection SMPI_use_colls_tracing Tracing of internal communications
419 By default, the collective operations are traced as a unique operation
420 because tracing all point-to-point communications composing them could
421 result in overloaded, hard to interpret traces. If you want to debug
422 and compare collective algorithms, you should set the
423 \c tracing/smpi/internals configuration item to 1 instead of 0.
425 Here are examples of two alltoall collective algorithms runs on 16 nodes,
426 the first one with a ring algorithm, the second with a pairwise one:
429 <a href="smpi_simgrid_alltoall_ring_16.png" border=0><img src="smpi_simgrid_alltoall_ring_16.png" width="30%" border=0 align="center"></a>
430 <a href="smpi_simgrid_alltoall_pair_16.png" border=0><img src="smpi_simgrid_alltoall_pair_16.png" width="30%" border=0 align="center"></a>
434 @section SMPI_what What can run within SMPI?
436 You can run unmodified MPI applications (both C/C++ and Fortran) within
437 SMPI, provided that you only use MPI calls that we implemented. Global
438 variables should be handled correctly on Linux systems.
440 @subsection SMPI_what_coverage MPI coverage of SMPI
442 Our coverage of the interface is very decent, but still incomplete;
443 Given the size of the MPI standard, we may well never manage to
444 implement absolutely all existing primitives. Currently, we have
445 almost no support for I/O primitives, but we still pass a very large
446 amount of the MPICH coverage tests.
448 The full list of not yet implemented functions is documented in the
449 file @ref include/smpi/smpi.h, between two lines containing the
450 <tt>FIXME</tt> marker. If you really miss a feature, please get in
451 touch with us: we can guide you though the SimGrid code to help you
452 implementing it, and we'd glad to integrate your contribution to the
453 main project afterward.
455 @subsection SMPI_what_globals Privatization of global variables
457 Concerning the globals, the problem comes from the fact that usually,
458 MPI processes run as real UNIX processes while they are all folded
459 into threads of a unique system process in SMPI. Global variables are
460 usually private to each MPI process while they become shared between
461 the processes in SMPI. The problem and some potential solutions are
462 discussed in this article: "Automatic Handling of Global Variables for
463 Multi-threaded MPI Programs", available at
464 http://charm.cs.illinois.edu/newPapers/11-23/paper.pdf (note that this
465 article does not deal with SMPI but with a competing solution called
466 AMPI that suffers of the same issue). This point used to be
467 problematic in SimGrid, but the problem should now be handled
468 automatically on Linux.
470 Older versions of SimGrid came with a script that automatically
471 privatized the globals through static analysis of the source code. But
472 our implementation was not robust enough to be used in production, so
473 it was removed at some point. Currently, SMPI comes with two
474 privatization mechanisms that you can @ref options_smpi_privatization
475 "select at runtime". At the time of writing (v3.18), the dlopen
476 approach is considered to be very fast (it's used by default) while
477 the mmap approach is considered to be rather slow but very robust.
479 With the <b>mmap approach</b>, SMPI duplicates and dynamically switch
480 the \c .data and \c .bss segments of the ELF process when switching
481 the MPI ranks. This allows each ranks to have its own copy of the
482 global variables. No copy actually occures as this mechanism uses \c
483 mmap for efficiency. This mechanism is considered to be very robust on
484 all systems supporting \c mmap (Linux and most BSDs). Its performance
485 is questionable since each context switch between MPI ranks induces
486 several syscalls to change the \c mmap that redirects the \c .data and
487 \c .bss segments to the copies of the new rank. The code will also be
488 copied several times in memory, inducing a slight increase of memory
491 Another limitation is that SMPI only accounts for global variables
492 defined in the executable. If the processes use external global
493 variables from dynamic libraries, they won't be switched
494 correctly. The easiest way to solve this is to statically link against
495 the library with these globals. This way, each MPI rank will get its
496 own copy of these libraries. Of course you should never statically
497 link against the SimGrid library itself.
499 With the <b>dlopen approach</b>, SMPI loads several copies of the same
500 executable in memory as if it were a library, so that the global
501 variables get naturally duplicated. It first requires the executable
502 to be compiled as a relocatable binary, which is less common for
503 programs than for libraries. But most distributions are now compiled
504 this way for security reason as it allows to randomize the address
505 space layout. It should thus be safe to compile most (any?) program
506 this way. The second trick is that the dynamic linker refuses to link
507 the exact same file several times, be it a library or a relocatable
508 executable. It makes perfectly sense in the general case, but we need
509 to circumvent this rule of thumb in our case. To that extend, the
510 binary is copied in a temporary file before being re-linked against.
511 `dlmopen()` cannot be used as it only allows 256 contextes, and as it
512 would also dupplicate simgrid itself.
514 This approach greatly speeds up the context switching, down to about
515 40 CPU cycles with our raw contextes, instead of requesting several
516 syscalls with the \c mmap approach. Another advantage is that it
517 permits to run the SMPI contexts in parallel, which is obviously not
518 possible with the \c mmap approach. It was tricky to implement, but we
519 are not aware of any flaws, so smpirun activates it by default.
521 In the future, it may be possible to further reduce the memory and
522 disk consumption. It seems that we could <a
523 href="https://lwn.net/Articles/415889/">punch holes</a> in the files
524 before dl-loading them to remove the code and constants, and mmap
525 these area onto a unique copy. If done correctly, this would reduce
526 the disk- and memory- usage to the bare minimum, and would also reduce
527 the pressure on the CPU instruction cache. See
528 <a href="https://github.com/simgrid/simgrid/issues/137">the relevant
529 bug</a> on github for implementation leads.\n
531 Also, currently, only the binary is copied and dlopen-ed for each MPI
532 rank. We could probably extend this to external dependencies, but for
533 now, any external dependencies must be statically linked into your
534 application. As usual, simgrid itself shall never be statically linked
535 in your app. You don't want to give a copy of SimGrid to each MPI rank:
536 that's ways too much for them to deal with.
538 @section SMPI_adapting Adapting your MPI code for further scalability
540 As detailed in the reference article (available at
541 http://hal.inria.fr/hal-01415484), you may want to adapt your code
542 to improve the simulation performance. But these tricks may seriously
543 hinder the result quality (or even prevent the app to run) if used
544 wrongly. We assume that if you want to simulate an HPC application,
545 you know what you are doing. Don't prove us wrong!
547 @subsection SMPI_adapting_size Reducing your memory footprint
549 If you get short on memory (the whole app is executed on a single node when
550 simulated), you should have a look at the SMPI_SHARED_MALLOC and
551 SMPI_SHARED_FREE macros. It allows to share memory areas between processes: The
552 purpose of these macro is that the same line malloc on each process will point
553 to the exact same memory area. So if you have a malloc of 2M and you have 16
554 processes, this macro will change your memory consumption from 2M*16 to 2M
555 only. Only one block for all processes.
557 If your program is ok with a block containing garbage value because all
558 processes write and read to the same place without any kind of coordination,
559 then this macro can dramatically shrink your memory consumption. For example,
560 that will be very beneficial to a matrix multiplication code, as all blocks will
561 be stored on the same area. Of course, the resulting computations will useless,
562 but you can still study the application behavior this way.
564 Naturally, this won't work if your code is data-dependent. For example, a Jacobi
565 iterative computation depends on the result computed by the code to detect
566 convergence conditions, so turning them into garbage by sharing the same memory
567 area between processes does not seem very wise. You cannot use the
568 SMPI_SHARED_MALLOC macro in this case, sorry.
570 This feature is demoed by the example file
571 <tt>examples/smpi/NAS/dt.c</tt>
573 @subsection SMPI_adapting_speed Toward faster simulations
575 If your application is too slow, try using SMPI_SAMPLE_LOCAL,
576 SMPI_SAMPLE_GLOBAL and friends to indicate which computation loops can
577 be sampled. Some of the loop iterations will be executed to measure
578 their duration, and this duration will be used for the subsequent
579 iterations. These samples are done per processor with
580 SMPI_SAMPLE_LOCAL, and shared between all processors with
581 SMPI_SAMPLE_GLOBAL. Of course, none of this will work if the execution
582 time of your loop iteration are not stable.
584 This feature is demoed by the example file
585 <tt>examples/smpi/NAS/ep.c</tt>
587 @section SMPI_accuracy Ensuring accurate simulations
589 Out of the box, SimGrid may give you fairly accurate results, but
590 there is a plenty of factors that could go wrong and make your results
591 inaccurate or even plainly wrong. Actually, you can only get accurate
592 results of a nicely built model, including both the system hardware
593 and your application. Such models are hard to pass over and reuse in
594 other settings, because elements that are not relevant to an
595 application (say, the latency of point-to-point communications,
596 collective operation implementation details or CPU-network
597 interaction) may be irrelevant to another application. The dream of
598 the perfect model, encompassing every aspects is only a chimera, as
599 the only perfect model of the reality is the reality. If you go for
600 simulation, then you have to ignore some irrelevant aspects of the
601 reality, but which aspects are irrelevant is actually
602 application-dependent...
604 The only way to assess whether your settings provide accurate results
605 is to double-check these results. If possible, you should first run
606 the same experiment in simulation and in real life, gathering as much
607 information as you can. Try to understand the discrepancies in the
608 results that you observe between both settings (visualization can be
609 precious for that). Then, try to modify your model (of the platform,
610 of the collective operations) to reduce the most preeminent differences.
612 If the discrepancies come from the computing time, try adapting the \c
613 smpi/host-speed: reduce it if your simulation runs faster than in
614 reality. If the error come from the communication, then you need to
615 fiddle with your platform file.
617 Be inventive in your modeling. Don't be afraid if the names given by
618 SimGrid does not match the real names: we got very good results by
619 modeling multicore/GPU machines with a set of separate hosts
620 interconnected with very fast networks (but don't trust your model
621 because it has the right names in the right place either).
623 Finally, you may want to check [this
624 article](https://hal.inria.fr/hal-00907887) on the classical pitfalls
625 in modeling distributed systems.
627 @section SMPI_troubleshooting Troubleshooting with SMPI
629 @subsection SMPI_trouble_buildchain My ./configure refuses to use smpicc
631 Alas, some building infrastructures cannot use smpicc as a project
632 compiler, and your <tt>./configure</tt> may report that the compiler
633 is not functional. If this happens, define the
634 <tt>SMPI_PRETEND_CC</tt> environment variable before running the
638 SMPI_PRETEND_CC=1 ./configure # here come the configure parameters
644 Make sure that SMPI_PRETEND_CC is only set when calling ./configure,
645 not during the actual compilation. With that variable, smpicc does
646 not do anything, to not hurt the ./configure feelings. But you need
647 smpicc do actually do something to get your application compiled.
652 /** @example include/smpi/smpi.h */