From 43037dfd252cbfffd2afe9867b2ff0a8c84b3463 Mon Sep 17 00:00:00 2001 From: Martin Quinson Date: Mon, 15 Oct 2018 01:08:21 +0200 Subject: [PATCH] start rewriting the platform documentation --- doc/doxygen/deployment.doc | 2 +- .../source/img/zone_hierarchy.png | Bin docs/source/index.rst | 3 + docs/source/intro_concepts.rst | 2 + docs/source/platform.rst | 37 ++- docs/source/platform_examples.rst | 60 ++++ docs/source/platform_reference.rst | 282 ++++++++++++++++++ tools/cmake/DefinePackages.cmake | 2 +- 8 files changed, 385 insertions(+), 3 deletions(-) rename doc/webcruft/AS_hierarchy.png => docs/source/img/zone_hierarchy.png (100%) create mode 100644 docs/source/platform_examples.rst create mode 100644 docs/source/platform_reference.rst diff --git a/doc/doxygen/deployment.doc b/doc/doxygen/deployment.doc index 20dd046db6..beee64f493 100644 --- a/doc/doxygen/deployment.doc +++ b/doc/doxygen/deployment.doc @@ -63,7 +63,7 @@ are named @c ***_d_xml. @verbatim - + diff --git a/doc/webcruft/AS_hierarchy.png b/docs/source/img/zone_hierarchy.png similarity index 100% rename from doc/webcruft/AS_hierarchy.png rename to docs/source/img/zone_hierarchy.png diff --git a/docs/source/index.rst b/docs/source/index.rst index 3418071c97..c175e2b1fd 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -30,6 +30,9 @@ Welcome to SimGrid's documentation!    The MSG Interface    The Java Bindings Describing the Simulated Platform +    Examples +    Modeling Hints +    DTD Reference Describing the Experimental Scenario    Configuring SimGrid The SimGrid Models diff --git a/docs/source/intro_concepts.rst b/docs/source/intro_concepts.rst index a952fc2f71..db93878e8d 100644 --- a/docs/source/intro_concepts.rst +++ b/docs/source/intro_concepts.rst @@ -1,3 +1,5 @@ +.. _intro_concepts: + Main Concepts ============= diff --git a/docs/source/platform.rst b/docs/source/platform.rst index 295d23e831..b6c1cb368e 100644 --- a/docs/source/platform.rst +++ b/docs/source/platform.rst @@ -13,4 +13,39 @@
Describing your Simulated Platform -================================== +################################## + +The goal of SimGrid is to run an application on a simulated platform. +For that, you have to describe **each element of your platform**, such +as computing hosts, clusters, each disks, links, etc. You must also +define the **routing on your platform**, ie which path is taken +between two hosts. Finally, you may also describe an **experimental +scenario**, with qualitative changes (e.g., bandwidth changes +representing an external load) and qualitative changes (representing +how some elements fail and restart over time). + +You should really separate your application from the platform +description, as it will ease your experimental campain afterward. +Mixing them is seen as a really bad experimental practice. The easiest +to enforce this split is to put the platform description in a XML +file. Many example platforms are provided in the archive, and this +page gives all needed details to write such files, as well as some +hints and tricks about describing your platform. + +On the other side, XML is sometimes not expressive enough, in +particular for large platforms exhibiting repetitive patterns that are +not simply expressed in XML. In practice, many users end up +generating their XML platform files from some sort of scripts. It is +probably preferable to rewrite your XML :ref:`platform using the lua +scripting language ` instead. In the future, it should +be possible to describe the platform in python or directly in C++, but +this is not possible yet. + +As usual, SimGrid is a versatile framework, and you should find the +way of describing your platform that best fits your experimental +practice. + + + + +.. LocalWords: SimGrid diff --git a/docs/source/platform_examples.rst b/docs/source/platform_examples.rst new file mode 100644 index 0000000000..259c7c3204 --- /dev/null +++ b/docs/source/platform_examples.rst @@ -0,0 +1,60 @@ +.. raw:: html + + + +
+
+ +Platform Examples +================= + +Here is a very simple platform file, containing 3 resources (two hosts +and one link), and explicitly giving the route between the hosts. + +.. literalinclude:: ../../examples/platforms/two_hosts.xml + :language: xml + +The root tag must be ````, and its ``version`` attribute +specifies the used DTD version. When the DTD evolutions introduce +backward-incompatible changes, this number gets updated. Use the +``simgrid_update_xml`` utility to upgrade your platform files on need. + + +Then, every platform element must be located within a given +**networking zone** introduced with the :ref:`pf_tag_zone` tag. Zones +are in charge of the routing: an host wants to communicate with +another host of the same zone, it is the zone's duty to find the list +of links that are involved in the communication. Here, since we use +``routing="Full"``, all routes must be explicitly given using the +:ref:`pf_tag_route` and :ref:`pf_tag_linkctn` tags (this :ref:`routing +model ` is both simple and inefficient :) It is OK to not +specify each and every route between hosts, as long as you don't start +at runtime any communications on the missing routes. + +Any zone may contain sub-zones itself, leading to a hierarchical +decomposition of the platform. This can be more efficient (as the +inter-zone routing gets factorized with :ref:`pf_tag_zoneroute`), and +allows to have more than one routing model in your platform. For +example, you could have a coordinate-based routing for the WAN parts +of your platforms, a full routing within each datacenter, and a highly +optimized routing within each cluster of the datacenter. In this +case, determining the route between two given hosts gets @ref +routing_basics "somewhat more complex" but SimGrid still computes +these routes for you in a time- and space-efficient manner. +Here is an illustration of these concepts: + +.. image:: img/zone_hierarchy.png + +The zone "AS2" models the core of a national network interconnecting a +small flat cluster (AS4) and a larger hierarchical cluster (AS5), a +subset of a LAN (AS6), and a set of peers scattered around the world +(AS7). + +.. todo:: Add more examples, such as the cloud example descibed in + previous paragraph + diff --git a/docs/source/platform_reference.rst b/docs/source/platform_reference.rst new file mode 100644 index 0000000000..a4bbc57645 --- /dev/null +++ b/docs/source/platform_reference.rst @@ -0,0 +1,282 @@ +.. raw:: html + + + +
+
+ +DTD Reference +************* + +Your platform description should follow the specification presented in +the `simgrid.dtd `_ +DTD file. The same DTD is used for both the platform and deployment +files. + +.. _pf_tag_host: + +------------------------------------------------------------------ + +------------------------------------------------------------------ + +An host is the computing resource on which an actor can execute. See :cpp:class:`simgrid::s4u::Host`. + +**Parent tags:** :ref:`pf_tag_zone` (only leaf zones, i.e. zones containing no inner zones nor clusters) |br| +**Children tags:** :ref:`pf_tag_prop`, :ref:`pf_tag_storage` |br| +**Attributes:** + +:``id``: Host name. + Must be unique over the whole platform. +:``speed``: Computational power (per core, in flop/s). + If you use DVFS, provide a comma-separated list of values for each pstate (see :ref:`howto_dvfs`). +:``core``: Amount of cores (default: 1). + See :ref:`howto_multicore`. +:``availability_file``: + File containing the availability profile. + Almost every lines of such files describe timed events as ``date ratio``. + Example: + + .. code-block:: python + + 1 0.5 + 2 0.2 + 5 1 + LOOPAFTER 8 + + - At time t=1, half of its power is taken by some background + computations, so only 50% of its initial power remains available + (0.5 means 50%). + - At time t=2, the available power drops at 20% of the total. + - At time t=5, the host computes back at full speed. + - At time t=10, the history is reset (because that's 5 seconds after + the last event). So the available speed will drop at t=11. + + If your trace does not contain a LOOPAFTER line, then your profile + is only executed once and not repetitively. + + .. warning:: Don't get fooled: Bandwidth and Latency profiles of a + :ref:`pf_tag_link` are absolute values, but Availability + profiles of :ref:`pf_tag_host` are ratio. +:``state_file``: File containing the state profile. + Almost every lines of such files describe timed events as ``date boolean``. + Example: + + .. code-block:: python + + 1 0 + 2 1 + LOOPAFTER 8 + + - At time t=1, the host is turned off (value 0 means OFF) + - At time t=2, it is turned back on (other values means ON) + - At time t=10, the history is reset (because that's 8 seconds after + the last event). So the host will be turned off again at t=11. + + If your trace does not contain a LOOPAFTER line, then your profile + is only executed once and not repetitively. + +:``coordinates``: Vivaldi coordinates (Vivaldi zones only). + See :ref:`pf_tag_peer`. +:``pstate``: Initial pstate (default: 0, the first one). + See :ref:`howto_dvfs`. + +.. _pf_tag_link: + +------------------------------------------------------------------ + +------------------------------------------------------------------ + +Network links can represent one-hop network connections. See :cpp:class:`simgrid::s4u::Link`. + +**Parent tags:** :ref:`pf_tag_zone` (both leaf zones and inner zones) |br| +**Children tags:** :ref:`pf_tag_prop` |br| +**Attributes:** + +:``id``: Link name. Must be unique over the whole platform. +:``bandwidth``: Maximum bandwidth for this link. You must specify the + unit as follows. + + **Units in bytes and powers of 2** (1 KiBps = 1024 Bps): + Bps, KiBps, MiBps, GiBps, TiBps, PiBps, EiBps |br| + **Units in bits and powers of 2** (1 Bps = 8 bps): + bps, Kibps, Mibps, Gibps, Tibps, Pibps, Eibps |br| + **Units in bytes and powers of 10:** (1 KBps = 1000 Bps) + Bps, KBps, MBps, GBps, TBps, PBps, EBps |br| + **Units in bits and powers of 10:** + 'Ebps', 'Pbps', 'Tbps', 'Gbps', 'Mbps', 'kbps', 'bps' + +:``latency``: Latency for this link (default: 0.0). You must specify + the unit as follows. + + ==== =========== ====================== + Unit Meaning Duration in seconds + ==== =========== ====================== + ps picosecond 10⁻¹² = 0.000000000001 + ns nanosecond 10⁻⁹ = 0.000000001 + us microsecond 10⁻⁶ = 0.000001 + ms millisecond 10⁻³ = 0.001 + s second 1 + m minute 60 + h hour 60 * 60 + d day 60 * 60 * 24 + w week 60 * 60 * 24 * 7 + ==== =========== ====================== + + +:``sharing_policy``: Sharing policy for the link. + Either ``SHARED``, ``FATPIPE`` or ``SPLITDUPLEX`` (default: ``SHARED``). + + If set to ``SHARED``, the available bandwidth is shared fairly + between all flows traversing this link. This tend to model the + sharing behavior of UDP or TCP. + + If set to ``FATPIPE``, the flows have no mutual impact, and each + flow can obtain the full bandwidth of this link. This is intended + to model the internet backbones that cannot get saturated by your + application: you mostly experience their latency. + + If set to ``SPLITDUPLEX``, the link models cross-traffic + effects. Under the ``SHARED`` policy, two flows of reverse + direction share the same resource, and can only get half of the + bandwidth each. But TCP connections are full duplex, meaning that + all both directions can get the full bandwidth. To model this, any + link under the ``SPLITDUPLEX`` policy is split in two links (their + names are suffixed with "_UP" and "_DOWN"). You must then specify + which direction gets actually used when referring to that link in a + :ref:`pf_tag_link_ctn`. + +:``bandwidth_file``: File containing the bandwidth profile. + Almost every lines of such files describe timed events as ``date + bandwidth`` (in bytes per second). + Example: + + .. code-block:: python + + 4.0 40000000 + 8.0 60000000 + PERIODICITY 12.0 + + - At time t=4, the bandwidth is of 40 MBps. + - At time t=8, it raises to 60MBps. + - At time t=24, it drops at 40 MBps again. + + .. warning:: Don't get fooled: Bandwidth and Latency profiles of a + :ref:`pf_tag_link` are absolute values, but Availability + profiles of :ref:`pf_tag_host` are ratio. +:``latency_file``: File containing the latency profile. + Almost every lines of such files describe timed events as ``date + latency`` (in seconds). + Example: + + .. code-block:: python + + 1.0 0.001 + 3.0 0.1 + LOOPAFTER 5.0 + + - At time t=1, the latency is of 1ms (0.001 second) + - At time t=3, the latency is of 100ms (0.1 second) + - At time t=8 (5 seconds after the last event), the profile loops. + - At time t=9 (1 second after the loop reset), the latency is back at 1ms. + + If your trace does not contain a LOOPAFTER line, then your profile + is only executed once and not repetitively. + + .. warning:: Don't get fooled: Bandwidth and Latency profiles of a + :ref:`pf_tag_link` are absolute values, but Availability + profiles of :ref:`pf_tag_host` are ratio. +:``state_file``: File containing the state profile. See :ref:`pf_tag_host`. + +.. _pf_tag_peer: + +------------------------------------------------------------------ + +------------------------------------------------------------------ + +This tag represents a peer, as in Peer-to-Peer (P2P) networks. It is +handy to model situations where hosts have an asymmetric +connectivity. Computers connected through set-to-boxes usually have a +much better download rate than their upload rate. To model this, + creates and connects several elements: an host, an upload link +and a download link. + +**Parent tags:** :ref:`pf_tag_zone` (only with Vivaldi routing) |br| +**Children tags:** none |br| +**Attributes:** + +:``id``: Name of the host. Must be unique on the whole platform. +:``speed``: Computational power (in flop/s). + If you use DVFS, provide a comma-separated list of values for each pstate (see :ref:`howto_dvfs`). +:``bw_in``: Bandwidth of the private downstream link, along with its + unit. See :ref:`pf_tag_link`. +:``bw_out``: Bandwidth of the private upstream link, along with its + unit. See :ref:`pf_tag_link`. +:``lat``: Latency of both private links. See :ref:`pf_tag_link`. +:``coordinates``: Coordinates of the gateway for this peer. + + The communication latency between an host A=(xA,yA,zA) and an host + B=(xB,yB,zB) is computed as follows: + + latency = sqrt( (xA-xB)² + (yA-yB)² ) + zA + zB + + See the documentation of + :cpp:class:`simgrid::kernel::routing::VivaldiZone` for details on + how the latency is computed from the coordinate, and on the the up + and down bandwidth are used. +:``availability_file``: File containing the availability profile. + See the full description in :ref:`pf_tag_host` +:``state_file``: File containing the state profile. + See the full description in :ref:`pf_tag_host` + + + +.. _pf_tag_prop: + +------------------------------------------------------------------ + +------------------------------------------------------------------ + +This tag can be used to attach user-defined properties to some +platform elements. Both the name and the value can be any string of +your wish. You can use this to pass extra parameters to your code and +the plugins. + +From your code, you can interact with these properties using the +following functions: +- Host: :cpp:func:`simgrid::s4u::Host::get_property` or :cpp:func:`MSG_host_get_property_value` + +**Parent tags:** :ref:`pf_tag_actor`, :ref:`pf_tag_config`, :ref:`pf_tag_cluster`, :ref:`pf_tag_host`, +:ref:`pf_tag_link`, :ref:`pf_tag_storage`, :ref:`pf_tag_zone` |br| +**Children tags:** none |br| +**Attributes:** + +:``id``: Name of the defined property. +:``value``: Value of the defined property. + +.. _pf_tag_router: + +------------------------------------------------------------------ + +------------------------------------------------------------------ + +A router is similar to an :ref:`pf_tag_host`, but it cannot contain +any actor. It is only useful to some routing algorithms. In +particular, they are useful when you want to use the NS3 bindings to +break the routes that are longer than 1 hop. + +**Parent tags:** :ref:`pf_tag_zone` (only leaf zones, i.e. zones containing no inner zones nor clusters) |br| +**Children tags:** :ref:`pf_tag_prop`, :ref:`pf_tag_storage` |br| +**Attributes:** + +:``id``: Router name. + No other host or router may have the same name over the whole platform. +:``coordinates``: Vivaldi coordinates. See :ref:`pf_tag_peer`. + +.. |br| raw:: html + +
diff --git a/tools/cmake/DefinePackages.cmake b/tools/cmake/DefinePackages.cmake index e7f3e5bb90..6ab51fd392 100644 --- a/tools/cmake/DefinePackages.cmake +++ b/tools/cmake/DefinePackages.cmake @@ -909,6 +909,7 @@ set(DOC_SOURCES docs/source/img/graphical-toc.svg docs/source/img/smpi_simgrid_alltoall_pair_16.png docs/source/img/smpi_simgrid_alltoall_ring_16.png + docs/source/img/zone_hierarchy.png docs/ignored_symbols docs/source/application.rst @@ -965,7 +966,6 @@ set(DOC_TOOLS # these files get copied automatically to the html documentation set(DOC_IMG - ${CMAKE_HOME_DIRECTORY}/doc/webcruft/AS_hierarchy.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/eclipseScreenShot.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/Paje_MSG_screenshot.jpg ${CMAKE_HOME_DIRECTORY}/doc/webcruft/Paje_MSG_screenshot_thn.jpg -- 2.20.1