example, to set the item ``Item`` to the value ``Value``, simply
type the following on the command-line:
-.. code-block:: shell
+.. code-block:: console
- my_simulator --cfg=Item:Value (other arguments)
+ $ my_simulator --cfg=Item:Value (other arguments)
Several ``--cfg`` command line arguments can naturally be used. If you
need to include spaces in the argument, don't forget to quote the
- **surf/precision:** :ref:`cfg=surf/precision`
- **For collective operations of SMPI,** please refer to Section :ref:`cfg=smpi/coll-selector`
+- **smpi/auto-shared-malloc-thresh:** :ref:`cfg=smpi/auto-shared-malloc-thresh`
- **smpi/async-small-thresh:** :ref:`cfg=smpi/async-small-thresh`
- **smpi/buffering:** :ref:`cfg=smpi/buffering`
- **smpi/bw-factor:** :ref:`cfg=smpi/bw-factor`
- **smpi/coll-selector:** :ref:`cfg=smpi/coll-selector`
- **smpi/comp-adjustment-file:** :ref:`cfg=smpi/comp-adjustment-file`
- **smpi/cpu-threshold:** :ref:`cfg=smpi/cpu-threshold`
+- **smpi/display-allocs:** :ref:`cfg=smpi/display-allocs`
- **smpi/display-timing:** :ref:`cfg=smpi/display-timing`
+- **smpi/errors-are-fatal:** :ref:`cfg=smpi/errors-are-fatal`
+- **smpi/finalization-barrier:** :ref:`cfg=smpi/finalization-barrier`
- **smpi/grow-injected-times:** :ref:`cfg=smpi/grow-injected-times`
- **smpi/host-speed:** :ref:`cfg=smpi/host-speed`
- **smpi/IB-penalty-factors:** :ref:`cfg=smpi/IB-penalty-factors`
- **smpi/or:** :ref:`cfg=smpi/or`
- **smpi/os:** :ref:`cfg=smpi/os`
- **smpi/papi-events:** :ref:`cfg=smpi/papi-events`
+- **smpi/pedantic:** :ref:`cfg=smpi/pedantic`
- **smpi/privatization:** :ref:`cfg=smpi/privatization`
- **smpi/privatize-libs:** :ref:`cfg=smpi/privatize-libs`
- **smpi/send-is-detached-thresh:** :ref:`cfg=smpi/send-is-detached-thresh`
end, you have two host models: The default one allows aggregation of
an existing CPU model with an existing network model, but does not
allow parallel tasks because these beasts need some collaboration
- between the network and CPU model. That is why, ptask_07 is used by
- default when using SimDag.
+ between the network and CPU model.
- **default:** Default host model. Currently, CPU:Cas01 and
network:LV08 (with cross traffic enabled)
- **ptask_L07:** Host model somehow similar to Cas01+CM02 but
allowing "parallel tasks", that are intended to model the moldable
tasks of the grid scheduling literature.
+ - **ptask_BMF:** More realistic model for heterogeneous resource sharing.
+ Implements BMF (Bottleneck max fairness) fairness. To be used with
+ parallel tasks instead of ptask_L07.
- ``storage/model``: specify the used storage model. Only one model is
provided so far.
be retrieved using the following commands. Both give a set of values,
and you should use the last one, which is the maximal size.
-.. code-block:: shell
+.. code-block:: console
- cat /proc/sys/net/ipv4/tcp_rmem # gives the sender window
- cat /proc/sys/net/ipv4/tcp_wmem # gives the receiver window
+ $ cat /proc/sys/net/ipv4/tcp_rmem # gives the sender window
+ $ cat /proc/sys/net/ipv4/tcp_wmem # gives the receiver window
-.. _cfg=smpi/IB-penalty-factors:
.. _cfg=network/bandwidth-factor:
.. _cfg=network/latency-factor:
.. _cfg=network/weight-S:
Study and Improvement of Network Simulation in the SimGrid Framework
<http://mescal.imag.fr/membres/arnaud.legrand/articles/simutools09.pdf>`_.
+- **network/latency-factor**: apply a multiplier to latency.
+ Models the TCP slow-start mechanism.
+- **network/bandwidth-factor**: actual bandwidth perceived by the
+ user.
+- **network/weight-S**: bottleneck sharing constant parameter. Used
+ to calculate RTT.
+
+These parameters are the same for all communications in your simulation,
+independently of message size or source/destination hosts. A more flexible
+mechanism based on callbacks was introduced in SimGrid. It provides the user
+a callback that will be called for each communication, allowing the user
+to set different latency and bandwidth factors, based on the message size, links used
+or zones traversed. To more details of how to use it, please look at the
+`examples/cpp/network-factors/s4u-network-factors.cpp <https://framagit.org/simgrid/simgrid/tree/master/examples/cpp/network-factors/s4u-network-factors.cpp>`_.
+
If you are using the SMPI model, these correction coefficients are
themselves corrected by constant values depending on the size of the
exchange. By default SMPI uses factors computed on the Stampede
Supercomputer at TACC, with optimal deployment of processes on
nodes. Again, only hardcore experts should bother about this fact.
+For more details, see SMPI sections about :ref:`cfg=smpi/bw-factor` and :ref:`cfg=smpi/lat-factor`.
+
+
+.. _cfg=smpi/IB-penalty-factors:
+
+Infiniband model
+^^^^^^^^^^^^^^^^
InfiniBand network behavior can be modeled through 3 parameters
``smpi/IB-penalty-factors:"βe;βs;γs"``, as explained in `this PhD
thesis
-<http://mescal.imag.fr/membres/jean-marc.vincent/index.html/PhD/Vienne.pdf>`_.
+<http://mescal.imag.fr/membres/jean-marc.vincent/index.html/PhD/Vienne.pdf>`_ (in French)
+or more concisely in `this paper <https://hal.inria.fr/hal-00953618/document>`_,
+even if that paper does only describe models for myrinet and ethernet.
+You can see in Fig 2 some results for Infiniband, for example. This model
+may be outdated by now for modern infiniband, anyway, so a new
+validation would be good.
+
+The three paramaters are defined as follows:
+
+- βs: penalty factor for outgoing messages, computed by running a simple send to
+ two nodes and checking slowdown compared to a single send to one node,
+ dividing by 2
+- βe: penalty factor for ingoing messages, same computation method but with one
+ node receiving several messages
+- γr: slowdown factor when communication buffer memory is saturated. It needs a
+ more complicated pattern to run in order to be computed (5.3 in the thesis,
+ page 107), and formula in the end is γr = time(c)/(3×βe×time(ref)), where
+ time(ref) is the time of a single comm with no contention).
+
+Once these values are computed, a penalty is assessed for each message (this is
+the part implemented in the simulator) as shown page 106 of the thesis. Here is
+a simple translation of this text. First, some notations:
-.. todo:: This section should be rewritten, and actually explain the
- options network/bandwidth-factor, network/latency-factor,
- network/weight-S.
+- ∆e(e) which corresponds to the incoming degree of node e, that is to say the number of communications having as destination node e.
+- ∆s (s) which corresponds to the degree outgoing from node s, that is to say the number of communications sent by node s.
+- Φ (e) which corresponds to the number of communications destined for the node e but coming from a different node.
+- Ω (s, e) which corresponds to the number of messages coming from node s to node e. If node e only receives communications from different nodes then Φ (e) = ∆e (e). On the other hand if, for example, there are three messages coming from node s and going from node e then Φ (e) 6 = ∆e (e) and Ω (s, e) = 3
+
+To determine the penalty for a communication, two values need to be calculated. First, the penalty caused by the conflict in transmission, noted ps.
+
+
+- if ∆s (i) = 1 then ps = 1.
+- if ∆s (i) ≥ 2 and ∆e (i) ≥ 3 then ps = ∆s (i) × βs × γr
+- else, ps = ∆s (i) × βs
+
+
+Then, the penalty caused by the conflict in reception (noted pe) should be computed as follows:
+
+- if ∆e (i) = 1 then pe = 1
+- else, pe = Φ (e) × βe × Ω (s, e)
+
+Finally, the penalty associated with the communication is:
+p = max (ps ∈ s, pe)
.. _cfg=network/crosstraffic:
Configuring loopback link
^^^^^^^^^^^^^^^^^^^^^^^^^
-Several network model provide an implicit loopback link to account for local
+Several network model provide an implicit loopback link to account for local
communication on a host. By default it has a 10GBps bandwidth and a null latency.
-This can be changed with ``network/loopback-lat`` and ``network/loopback-bw``
+This can be changed with ``network/loopback-lat`` and ``network/loopback-bw``
items.
.. _cfg=smpi/async-small-thresh:
To enable SimGrid's model-checking support, the program should
be executed using the simgrid-mc wrapper:
-.. code-block:: shell
+.. code-block:: console
- simgrid-mc ./my_program
+ $ simgrid-mc ./my_program
Safety properties are expressed as assertions using the function
:cpp:func:`void MC_assert(int prop)`.
property, as formatted by the `ltl2ba <https://github.com/utwente-fmt/ltl2ba>`_ program.
Note that ltl2ba is not part of SimGrid and must be installed separately.
-.. code-block:: shell
+.. code-block:: console
- simgrid-mc ./my_program --cfg=model-check/property:<filename>
+ $ simgrid-mc ./my_program --cfg=model-check/property:<filename>
.. _cfg=model-check/checkpoint:
Exploration Depth Limit
.......................
-The ``model-checker/max-depth`` can set the maximum depth of the
+The ``model-check/max-depth`` can set the maximum depth of the
exploration graph of the model checker. If this limit is reached, a
logging message is sent and the results might not be exact.
-By default, there is no depth limit.
+By default, the exploration is limited to the depth of 1000.
.. _cfg=model-check/timeout:
execution graph (where a safety or liveness property is violated), it
generates an identifier for this path. Here is an example of the output:
-.. code-block:: shell
+.. code-block:: console
[ 0.000000] (0:@) Check a safety property
[ 0.000000] (0:@) **************************
8192 (in KiB), while our Chord simulation works with stacks as small
as 16 KiB, for example. You can ensure that some actors have a specific
size by simply changing the value of this configuration item before
-creating these actors. The :cpp:func:`simgrid::s4u::Engine::set_config`
+creating these actors. The :cpp:func:`simgrid::s4u::Engine::set_config`
functions are handy for that.
This *setting is ignored* when using the thread factory (because there
Configuring the Tracing
-----------------------
-The :ref:`tracing subsystem <outcomes_vizu>` can be configured in
-several different ways depending on the used interface (S4U, SMPI, SimDag)
+The :ref:`tracing subsystem <outcome_vizu>` can be configured in
+several different ways depending on the used interface (S4U, SMPI)
and the kind of traces that needs to be obtained. See the
:ref:`Tracing Configuration Options subsection
<tracing_tracing_options>` for a full description of each
you never used the tracing API.
-- Any SimGrid-based simulator (MSG, SimDag, SMPI, ...) and raw traces:
+- Any SimGrid-based simulator (MSG, SMPI, ...) and raw traces:
- .. code-block:: shell
+ .. code-block:: none
--cfg=tracing:yes --cfg=tracing/uncategorized:yes
tells it to trace host and link utilization (without any
categorization).
-- MSG or SimDag-based simulator and categorized traces (you need to
- declare categories and classify your tasks according to them)
+- MSG-based simulator and categorized traces (you need to
+ declare categories and classify your tasks according to them)
- .. code-block:: shell
+ .. code-block:: none
--cfg=tracing:yes --cfg=tracing/categorized:yes
- SMPI simulator and traces for a space/time view:
- .. code-block:: shell
+ .. code-block:: console
- smpirun -trace ...
+ $ smpirun -trace ...
The `-trace` parameter for the smpirun script runs the simulation
with ``--cfg=tracing:yes --cfg=tracing/smpi:yes``. Check the
- Add a string on top of the trace file as comment:
- .. code-block:: shell
+ .. code-block:: none
--cfg=tracing/comment:my_simulation_identifier
- Add the contents of a textual file on top of the trace file as comment:
- .. code-block:: shell
+ .. code-block:: none
--cfg=tracing/comment-file:my_file_with_additional_information.txt
first column defines the section that will be subject to a speedup;
the second column is the speedup. For instance:
-.. code-block:: shell
+.. code-block:: none
"start:stop","ratio"
"exchange_1.f:30:exchange_1.f:130",1.18244559422142
simulation ends. If you enable the ``smpi/display-timing`` item,
``smpirun`` will display this information when the simulation
ends.
+SMPI will also display information about the amout of real time spent
+in application code and in SMPI internals, to provide hints about the
+need to use sampling to reduce simulation time.
+
+.. _cfg=smpi/display-allocs:
+
+Reporting memory allocations
+............................
+
+**Option** ``smpi/display-allocs`` **Default:** 0 (false)
+
+SMPI intercepts malloc and calloc calls performed inside the running
+application, if it wasn't compiled with SMPI_NO_OVERRIDE_MALLOC.
+With this option, SMPI will show at the end of execution the amount of
+memory allocated through these calls, and locate the most expensive one.
+This helps finding the targets for manual memory sharing, or the threshold
+to use for smpi/auto-shared-malloc-thresh option (see :ref:`cfg=smpi/auto-shared-malloc-thresh`).
.. _cfg=smpi/keep-temps:
The first draft, however, just implements a "global" (i.e., for all processes) set
of counters, the "default" set.
-.. code-block:: shell
+.. code-block:: none
--cfg=smpi/papi-events:"default:PAPI_L3_LDM:PAPI_L2_LDM"
or full names. Check with ldd the name of the library you want to
use. For example:
-.. code-block:: shell
+.. code-block:: console
- ldd allpairf90
+ $ ldd allpairf90
...
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007fbb4d91b000)
...
.. TODO:: All available collective algorithms will be made available
via the ``smpirun --help-coll`` command.
+.. _cfg=smpi/finalization-barrier:
+
+Add a barrier in MPI_Finalize
+.............................
+
+**Option** ``smpi/finalization-barrier`` **default:** off
+
+By default, SMPI processes are destroyed as soon as soon as their code ends,
+so after a successful MPI_Finalize call returns. In some rare cases, some data
+might have been attached to MPI objects still active in the remaining processes,
+and can be destroyed eagerly by the finished process.
+If your code shows issues at finalization, such as segmentation fault, triggering
+this option will add an explicit MPI_Barrier(MPI_COMM_WORLD) call inside the
+MPI_Finalize, so that all processes will terminate at almost the same point.
+It might affect the total timing by the cost of a barrier.
+
+.. _cfg=smpi/errors-are-fatal:
+
+Disable MPI fatal errors
+........................
+
+**Option** ``smpi/errors-are-fatal`` **default:** on
+
+By default, SMPI processes will crash if a MPI error code is returned. MPI allows
+to explicitely set MPI_ERRORS_RETURN errhandler to avoid this behaviour. This flag
+will turn on this behaviour by default (for all concerned types and errhandlers).
+This can ease debugging by going after the first reported error.
+
+.. _cfg=smpi/pedantic:
+
+Disable pedantic MPI errors
+...........................
+
+**Option** ``smpi/pedantic`` **default:** on
+
+By default, SMPI will report all errors it finds in MPI codes. Some of these errors
+may not be considered as errors by all developers. This flag can be turned off to
+avoid reporting some usually harmless mistakes.
+Concerned errors list (will be expanded in the future):
+
+ - Calling MPI_Win_fence only once in a program, hence just opening an epoch without
+ ever closing it.
+
.. _cfg=smpi/iprobe:
Inject constant times for MPI_Iprobe
assignment). In practice, change the calls for malloc() and free() into
SMPI_SHARED_MALLOC() and SMPI_SHARED_FREE().
-SMPI provides two algorithms for this feature. The first one, called
+SMPI provides two algorithms for this feature. The first one, called
``local``, allocates one block per call to SMPI_SHARED_MALLOC()
(each call site gets its own block) ,and this block is shared
among all MPI ranks. This is implemented with the shm_* functions
To activate this, you must mount a hugetlbfs on your system and allocate
at least one huge page:
-.. code-block:: shell
+.. code-block:: console
- mkdir /home/huge
- sudo mount none /home/huge -t hugetlbfs -o rw,mode=0777
- sudo sh -c 'echo 1 > /proc/sys/vm/nr_hugepages' # echo more if you need more
+ $ mkdir /home/huge
+ $ sudo mount none /home/huge -t hugetlbfs -o rw,mode=0777
+ $ sudo sh -c 'echo 1 > /proc/sys/vm/nr_hugepages' # echo more if you need more
Then, you can pass the option
``--cfg=smpi/shared-malloc-hugepage:/home/huge`` to smpirun to
actually activate the huge page support in shared mallocs.
+.. _cfg=smpi/auto-shared-malloc-thresh:
+
+Automatically share allocations
+...............................
+
+**Option** ``smpi/auto-shared-malloc-thresh:`` **Default:** 0 (false)
+ This value in bytes represents the size above which all allocations
+ will be "shared" by default (as if they were performed through
+ SMPI_SHARED_MALLOC macros). Default = 0 = disabled feature.
+ The value must be carefully chosen to only select data buffers which
+ will not modify execution path or cause crash if their content is false.
+ Option :ref:`cfg=smpi/display-allocs` can be used to locate the largest
+ allocation detected in a run, and provide a good starting threshold.
+ Note : malloc, calloc and free are overridden by smpicc/cxx by default.
+ This can cause some troubles if codes are already overriding these. If this
+ is the case, defining SMPI_NO_OVERRIDE_MALLOC in the compilation flags can
+ help, but will make this feature unusable.
+
.. _cfg=smpi/wtime:
Inject constant times for MPI_Wtime, gettimeofday and clock_gettime
writing in global variable simgrid::simix::breakpoint. For example,
with gdb:
-.. code-block:: shell
+.. code-block:: none
set variable simgrid::simix::breakpoint = 3.1416
If you want to have spaces in your log format, you should protect it. Otherwise, SimGrid will consider that this is a space-separated list of several parameters. But you should
-also protect it from the shell that also splits command line arguments on spaces. At the end, you should use something such as ``--log="'root.fmt:%l: [%p/%c]: %m%n'"``.
-Another option is to use the ``%e`` directive for spaces, as in ``--log=root.fmt:%l:%e[%p/%c]:%e%m%n``.
+also protect it from the shell that also splits command line arguments on spaces. At the end, you should use something such as ``--log="'root.fmt:%l: [%p/%c]: %m%n'"``.
+Another option is to use the ``%e`` directive for spaces, as in ``--log=root.fmt:%l:%e[%p/%c]:%e%m%n``.
Category appender
.................
