- **surf/precision:** :ref:`cfg=surf/precision`
- **For collective operations of SMPI,** please refer to Section :ref:`cfg=smpi/coll-selector`
+- **smpi/auto-shared-malloc-thresh:** :ref:`cfg=smpi/auto-shared-malloc-thresh`
- **smpi/async-small-thresh:** :ref:`cfg=smpi/async-small-thresh`
- **smpi/buffering:** :ref:`cfg=smpi/buffering`
- **smpi/bw-factor:** :ref:`cfg=smpi/bw-factor`
- **smpi/coll-selector:** :ref:`cfg=smpi/coll-selector`
- **smpi/comp-adjustment-file:** :ref:`cfg=smpi/comp-adjustment-file`
- **smpi/cpu-threshold:** :ref:`cfg=smpi/cpu-threshold`
+- **smpi/display-allocs:** :ref:`cfg=smpi/display-allocs`
- **smpi/display-timing:** :ref:`cfg=smpi/display-timing`
+- **smpi/errors-are-fatal:** :ref:`cfg=smpi/errors-are-fatal`
+- **smpi/finalization-barrier:** :ref:`cfg=smpi/finalization-barrier`
- **smpi/grow-injected-times:** :ref:`cfg=smpi/grow-injected-times`
- **smpi/host-speed:** :ref:`cfg=smpi/host-speed`
- **smpi/IB-penalty-factors:** :ref:`cfg=smpi/IB-penalty-factors`
- **smpi/or:** :ref:`cfg=smpi/or`
- **smpi/os:** :ref:`cfg=smpi/os`
- **smpi/papi-events:** :ref:`cfg=smpi/papi-events`
+- **smpi/pedantic:** :ref:`cfg=smpi/pedantic`
- **smpi/privatization:** :ref:`cfg=smpi/privatization`
- **smpi/privatize-libs:** :ref:`cfg=smpi/privatize-libs`
- **smpi/send-is-detached-thresh:** :ref:`cfg=smpi/send-is-detached-thresh`
- **smpi/simulate-computation:** :ref:`cfg=smpi/simulate-computation`
- **smpi/test:** :ref:`cfg=smpi/test`
- **smpi/wtime:** :ref:`cfg=smpi/wtime`
+- **smpi/list-leaks** :ref:`cfg=smpi/list-leaks`
- **Tracing configuration options** can be found in Section :ref:`tracing_tracing_options`
cat /proc/sys/net/ipv4/tcp_rmem # gives the sender window
cat /proc/sys/net/ipv4/tcp_wmem # gives the receiver window
-.. _cfg=smpi/IB-penalty-factors:
.. _cfg=network/bandwidth-factor:
.. _cfg=network/latency-factor:
.. _cfg=network/weight-S:
Supercomputer at TACC, with optimal deployment of processes on
nodes. Again, only hardcore experts should bother about this fact.
-InfiniBand network behavior can be modeled through 3 parameters
-``smpi/IB-penalty-factors:"βe;βs;γs"``, as explained in `this PhD
-thesis
-<http://mescal.imag.fr/membres/jean-marc.vincent/index.html/PhD/Vienne.pdf>`_.
.. todo:: This section should be rewritten, and actually explain the
options network/bandwidth-factor, network/latency-factor,
network/weight-S.
+.. _cfg=smpi/IB-penalty-factors:
+
+Infiniband model
+^^^^^^^^^^^^^^^^
+
+InfiniBand network behavior can be modeled through 3 parameters
+``smpi/IB-penalty-factors:"βe;βs;γs"``, as explained in `this PhD
+thesis
+<http://mescal.imag.fr/membres/jean-marc.vincent/index.html/PhD/Vienne.pdf>`_ (in French)
+or more concisely in `this paper <https://hal.inria.fr/hal-00953618/document>`_,
+even if that paper does only describe models for myrinet and ethernet.
+You can see in Fig 2 some results for Infiniband, for example. This model
+may be outdated by now for modern infiniband, anyway, so a new
+validation would be good.
+
+The three paramaters are defined as follows:
+
+- βs: penalty factor for outgoing messages, computed by running a simple send to
+ two nodes and checking slowdown compared to a single send to one node,
+ dividing by 2
+- βe: penalty factor for ingoing messages, same computation method but with one
+ node receiving several messages
+- γr: slowdown factor when communication buffer memory is saturated. It needs a
+ more complicated pattern to run in order to be computed (5.3 in the thesis,
+ page 107), and formula in the end is γr = time(c)/(3×βe×time(ref)), where
+ time(ref) is the time of a single comm with no contention).
+
+Once these values are computed, a penalty is assessed for each message (this is
+the part implemented in the simulator) as shown page 106 of the thesis. Here is
+a simple translation of this text. First, some notations:
+
+- ∆e(e) which corresponds to the incoming degree of node e, that is to say the number of communications having as destination node e.
+- ∆s (s) which corresponds to the degree outgoing from node s, that is to say the number of communications sent by node s.
+- Φ (e) which corresponds to the number of communications destined for the node e but coming from a different node.
+- Ω (s, e) which corresponds to the number of messages coming from node s to node e. If node e only receives communications from different nodes then Φ (e) = ∆e (e). On the other hand if, for example, there are three messages coming from node s and going from node e then Φ (e) 6 = ∆e (e) and Ω (s, e) = 3
+
+To determine the penalty for a communication, two values need to be calculated. First, the penalty caused by the conflict in transmission, noted ps.
+
+
+- if ∆s (i) = 1 then ps = 1.
+- if ∆s (i) ≥ 2 and ∆e (i) ≥ 3 then ps = ∆s (i) × βs × γr
+- else, ps = ∆s (i) × βs
+
+
+Then, the penalty caused by the conflict in reception (noted pe) should be computed as follows:
+
+- if ∆e (i) = 1 then pe = 1
+- else, pe = Φ (e) × βe × Ω (s, e)
+
+Finally, the penalty associated with the communication is:
+p = max (ps ∈ s, pe)
+
.. _cfg=network/crosstraffic:
Simulating Cross-Traffic
of a file in which to write a dot file of the path leading to the
property violation discovered (safety or liveness violation), as well
as the cycle for liveness properties. This dot file can then be fed to the
-graphviz dot tool to generate an corresponding graphical representation.
+graphviz dot tool to generate a corresponding graphical representation.
.. _cfg=model-check/max-depth:
Exploration Depth Limit
.......................
-The ``model-checker/max-depth`` can set the maximum depth of the
+The ``model-check/max-depth`` can set the maximum depth of the
exploration graph of the model checker. If this limit is reached, a
logging message is sent and the results might not be exact.
-By default, there is no depth limit.
+By default, the exploration is limited to the depth of 1000.
.. _cfg=model-check/timeout:
and b are the return values of their simcalls. In the previous
example, ``1/3;1/4``, you can see from the full output that the actor
1 is doing MC_RANDOM simcalls, so the 3 and 4 simply denote the values
-that these simcall return.
+that these simcall return on the execution branch leading to the
+violation.
Configuring the User Code Virtualization
----------------------------------------
-----------------------
The :ref:`tracing subsystem <outcomes_vizu>` can be configured in
-several different ways depending on the nature of the simulator (MSG,
-SimDag, SMPI) and the kind of traces that need to be obtained. See the
+several different ways depending on the used interface (S4U, SMPI, SimDag)
+and the kind of traces that needs to be obtained. See the
:ref:`Tracing Configuration Options subsection
-<tracing_tracing_options>` to get a detailed description of each
+<tracing_tracing_options>` for a full description of each
configuration option.
We detail here a simple way to get the traces working for you, even if
simulation ends. If you enable the ``smpi/display-timing`` item,
``smpirun`` will display this information when the simulation
ends.
+SMPI will also display information about the amout of real time spent
+in application code and in SMPI internals, to provide hints about the
+need to use sampling to reduce simulation time.
+
+.. _cfg=smpi/display-allocs:
+
+Reporting memory allocations
+............................
+
+**Option** ``smpi/display-allocs`` **Default:** 0 (false)
+
+SMPI intercepts malloc and calloc calls performed inside the running
+application, if it wasn't compiled with SMPI_NO_OVERRIDE_MALLOC.
+With this option, SMPI will show at the end of execution the amount of
+memory allocated through these calls, and locate the most expensive one.
+This helps finding the targets for manual memory sharing, or the threshold
+to use for smpi/auto-shared-malloc-thresh option (see :ref:`cfg=smpi/auto-shared-malloc-thresh`).
.. _cfg=smpi/keep-temps:
.. TODO:: All available collective algorithms will be made available
via the ``smpirun --help-coll`` command.
+Add a barrier in MPI_Finalize
+.............................
+
+.. _cfg=smpi/finalization-barrier:
+
+**Option** ``smpi/finalization-barrier`` **default:** off
+
+By default, SMPI processes are destroyed as soon as soon as their code ends,
+so after a successful MPI_Finalize call returns. In some rare cases, some data
+might have been attached to MPI objects still active in the remaining processes,
+and can be destroyed eagerly by the finished process.
+If your code shows issues at finalization, such as segmentation fault, triggering
+this option will add an explicit MPI_Barrier(MPI_COMM_WORLD) call inside the
+MPI_Finalize, so that all processes will terminate at almost the same point.
+It might affect the total timing by the cost of a barrier.
+
+.. _cfg=smpi/errors-are-fatal:
+
+**Option** ``smpi/errors-are-fatal`` **default:** on
+
+By default, SMPI processes will crash if a MPI error code is returned. MPI allows
+to explicitely set MPI_ERRORS_RETURN errhandler to avoid this behaviour. This flag
+will turn on this behaviour by default (for all concerned types and errhandlers).
+This can ease debugging by going after the first reported error.
+
+.. _cfg=smpi/pedantic:
+
+**Option** ``smpi/pedantic`` **default:** on
+
+By default, SMPI will report all errors it finds in MPI codes. Some of these errors
+may not be considered as errors by all developers. This flag can be turned off to
+avoid reporting some usually harmless mistakes.
+Concerned errors list (will be expanded in the future):
+ - Calling MPI_Win_fence only once in a program, hence just opening an epoch without
+ ever closing it.
+
.. _cfg=smpi/iprobe:
Inject constant times for MPI_Iprobe
``--cfg=smpi/shared-malloc-hugepage:/home/huge`` to smpirun to
actually activate the huge page support in shared mallocs.
+.. _cfg=smpi/auto-shared-malloc-thresh:
+
+Automatically share allocations
+...............................
+
+**Option** ``smpi/auto-shared-malloc-thresh:`` **Default:** 0 (false)
+ This value in bytes represents the size above which all allocations
+ will be "shared" by default (as if they were performed through
+ SMPI_SHARED_MALLOC macros). Default = 0 = disabled feature.
+ The value must be carefully chosen to only select data buffers which
+ will not modify execution path or cause crash if their content is false.
+ Option :ref:`cfg=smpi/display-allocs` can be used to locate the largest
+ allocation detected in a run, and provide a good starting threshold.
+ Note : malloc, calloc and free are overridden by smpicc/cxx by default.
+ This can cause some troubles if codes are already overriding these. If this
+ is the case, defining SMPI_NO_OVERRIDE_MALLOC in the compilation flags can
+ help, but will make this feature unusable.
+
.. _cfg=smpi/wtime:
Inject constant times for MPI_Wtime, gettimeofday and clock_gettime
If the simulation speed really matters to you, you can avoid this
extra delay by setting smpi/wtime to 0.
+.. _cfg=smpi/list-leaks:
+
+Report leaked MPI objects
+.........................
+
+**Option** ``smpi/list-leaks`` **default:** 0
+
+This option controls whether to report leaked MPI objects.
+The parameter is the number of leaks to report.
+
Other Configurations
--------------------
the paths of source files depend of the build settings. That would
break most of the tests since their output is continually compared.
-Logging Configuration
+.. _logging_config:
+
+Logging configuration
---------------------
-This can be done by using XBT. Go to :ref:`XBT_log` for more details.
+As introduced in :ref:`outcome_logs`, the SimGrid logging mechanism allows to configure at runtime the messages that should be displayed and those that should be omitted. Each
+message produced in the code is given a category (denoting its topic) and a priority. Then at runtime, each category is given a threshold (only messages of priority higher than
+that threshold are displayed), a layout (deciding how the messages in this category are formatted), and an appender (deciding what to do with the message: either print on stderr or
+to a file).
+
+This section explains how to configure this logging features. You can also refer to the documentation of the :ref:`programmer's interface <logging_prog>`, that allows to produce
+messages from your code.
+
+Most of the time, the logging mechanism is configured at runtime using the ``--log`` command-line argument, even if you can also use :c:func:`xbt_log_control_set()` to control it from
+your program. To pass configure more than one setting, you can either pass several ``--log`` arguments, or separate your settings with spaces, that must be quoted accordingly. In
+practice, the following is equivalent to the above settings: ``--log=root.thresh:error --log=s4u_host.thresh:debug``.
+
+If you want to specify more than one setting, you can either pass several ``--log`` argument to your program as above, or separate them with spaces. In this case, you want to quote
+your settings, as in ``--log="root.thresh:error s4u_host.thresh:debug"``. The parameters are interpreted in order, from left to right.
+
+
+Threshold configuration
+.......................
+
+The keyword ``threshold`` controls which logging event will get displayed in a given category. For example, ``--log=root.threshold:debug`` displays *every* message produced in the
+``root`` category and its subcategories (i.e., every message produced -- this is *extremely* verbose), while ``--log=root.thres:critical`` turns almost everything off. As you can
+see, ``threshold`` can be abbreviated here.
+
+Existing thresholds:
+
+ - ``trace`` some functions display a message at this level when entering or returning
+ - ``debug`` output that is mostly useful when debugging the corresponding module.
+ - ``verbose`` verbose output that is only mildly interesting and can easily be ignored
+ - ``info`` usual output (this is the default threshold of all categories)
+ - ``warning`` minor issue encountered
+ - ``error`` issue encountered
+ - ``critical`` major issue encountered, such as assertions failures
+
+.. _log/fmt:
+
+Format configuration
+....................
+
+The keyword ``fmt`` controls the layout (the format) of a logging category. For example, ``--log=root.fmt:%m`` reduces the output to the user-message only, removing any decoration such
+as the date, or the actor ID, everything. Existing format directives:
+
+ - %%: the % char
+ - %n: line separator (LOG4J compatible)
+ - %e: plain old space (SimGrid extension)
+
+ - %m: user-provided message
+
+ - %c: Category name (LOG4J compatible)
+ - %p: Priority name (LOG4J compatible)
+
+ - %h: Hostname (SimGrid extension)
+ - %a: Actor name (SimGrid extension -- note that with SMPI this is the integer value of the process rank)
+ - %i: Actor PID (SimGrid extension -- this is a 'i' as in 'i'dea)
+ - %t: Thread "name" (LOG4J compatible -- actually the address of the thread in memory)
+
+ - %F: file name where the log event was raised (LOG4J compatible)
+ - %l: location where the log event was raised (LOG4J compatible, like '%%F:%%L' -- this is a l as in 'l'etter)
+ - %L: line number where the log event was raised (LOG4J compatible)
+ - %M: function name (LOG4J compatible -- called method name here of course).
+
+ - %d: date (UNIX-like epoch)
+ - %r: application age (time elapsed since the beginning of the application)
+
+
+``--log=root.fmt:'[%h:%a:(%i) %r] %l: %m%n'`` gives you the default layout used for info messages while ``--log=root.fmt:'[%h:%a:(%i) %r] %l: [%c/%p] %m%n'`` gives you the default
+layout for the other priorities (it adds the source code location). Also, the actor identification is omitted by the default layout for the messages coming directly from the
+SimGrid kernel, so info messages are formatted with ``[%r] [%c/%p] %m%n`` in this case. When specifying the layout manually, such distinctions are currently impossible, and the
+provided layout is used for every messages.
+
+As with printf, you can specify the precision and width of the fields. For example, ``%.4r`` limits the date precision to four digits while ``%15h`` limits the host name to at most
+15 chars.
+
+
+If you want to have spaces in your log format, you should protect it. Otherwise, SimGrid will consider that this is a space-separated list of several parameters. But you should
+also protect it from the shell that also splits command line arguments on spaces. At the end, you should use something such as ``--log="'root.fmt:%l: [%p/%c]: %m%n'"``.
+Another option is to use the ``%e`` directive for spaces, as in ``--log=root.fmt:%l:%e[%p/%c]:%e%m%n``.
+
+Category appender
+.................
+
+The keyword ``app`` controls the appended of a logging category. For example ``--log=root.app:file:mylogfile`` redirects every output to the file ``mylogfile``.
+
+With the ``splitfile`` appender, a new file is created when the size of the output reaches the specified size. The format is ``--log=root.app:splitfile:<size>:<file name>``. For
+example, ``--log=root.app:splitfile:500:mylog_%`` creates log files of at most 500 bytes, using the names ``mylog_0``, ``mylog_1``, ``mylog_2``, etc.
+
+The ``rollfile`` appender uses one file only, but the file is emptied and recreated when its size reaches the specified maximum. For example, ``--log=root.app:rollfile:500:mylog``
+ensures that the log file ``mylog`` will never overpass 500 bytes in size.
+
+Any appender setup this way have its own layout format, that you may change afterward. When specifying a new appender, its additivity is set to false to prevent log event displayed
+by this appender to "leak" to any other appender higher in the hierarchy. You can naturally change that if you want your messages to be displayed twice.
+
+Category additivity
+...................
+
+The keyword ``add`` controls the additivity of a logging category. By default, the messages are only passed one appender only: the more specific, i.e. the first one found when
+climbing the tree from the category in which they were produced. In Log4J parlance, it is said that the default additivity of appenders is false. If you change this setting to
+``on`` (or ``yes`` or ``1``), the produced messages will also be passed to the upper appender.
+
+Let's consider a more complex example: ``--log="root.app:file:all.log s4u.app:file:iface.log xbt.app:file:xbt.log xbt.add:yes``. Here, the logging of s4u will be sent to the
+``iface.log`` file; the logging of the xbt toolbox will be sent to both the ``xbt.log`` file and the ``all.log`` file (because xbt additivity was enabled); and every other loggings
+will only be sent to ``all.log``.
+
+Other options
+.............
+
+``--help-logs`` displays a complete help message about logging in SimGrid.
+
+``--help-log-categories`` displays the actual hierarchy of log categories for this binary.
+
+``--log=no_loc`` hides the source locations (file names and line numbers) from the messages. This is useful to make tests reproducible.
+
.. |br| raw:: html
