From: Martin Quinson Date: Thu, 23 Aug 2018 00:50:10 +0000 (+0200) Subject: user manual: create the page, and fiddle with the graphical TOC X-Git-Tag: v3_21~213 X-Git-Url: http://info.iut-bm.univ-fcomte.fr/pub/gitweb/simgrid.git/commitdiff_plain/57ede4de69c189960532a287ae2f2f7779b11650 user manual: create the page, and fiddle with the graphical TOC --- diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 6a3b5630ac..25f3c2db9c 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -6,7 +6,7 @@ pages: - apt install -y libboost-all-dev libboost-dev python3-pip doxygen fig2dev - pip3 install --requirement docs/requirements.txt - cd docs - - sphinx-build -M html source/ build/ + - ./Build.sh - mv build/html ../public artifacts: paths: diff --git a/docs/Build.sh b/docs/Build.sh new file mode 100755 index 0000000000..74bcbfd3a0 --- /dev/null +++ b/docs/Build.sh @@ -0,0 +1,10 @@ +#! /bin/sh +# +# Simplistic script to rebuild our documentation with sphinx-build + +rm -rf build/doxy/ source/api/ +sphinx-build -M html source build ${SPHINXOPTS} +cat source/img/graphical-toc.svg \ + | perl -pe 's/(xlink:href="http)/target="_top" $1/' \ + | perl -pe 's/(xlink:href=".*?.html)/target="_top" $1/' \ + > build/html/graphical-toc.svg diff --git a/docs/source/app_s4u.rst b/docs/source/app_s4u.rst new file mode 100644 index 0000000000..43d79e8da8 --- /dev/null +++ b/docs/source/app_s4u.rst @@ -0,0 +1,4 @@ +The S4U Interface +================= + + diff --git a/docs/source/application.rst b/docs/source/application.rst new file mode 100644 index 0000000000..f9e50f046a --- /dev/null +++ b/docs/source/application.rst @@ -0,0 +1,28 @@ +.. _application: + +.. raw:: html + + + +
+
+ +Describing your Application +*************************** + +Every SimGrid simulation entails a distributed application, that +virtually executes on the simulated platform. This application can +be either an existing MPI program (if you use the SMPI interface), or +a program specifically written to execute within SimGrid, using one of +the dedicated APIs. + +.. include:: app_s4u.rst + +.. include:: app_smpi.rst + +.. include:: app_legacy.rst diff --git a/docs/source/img/graphical-toc.svg b/docs/source/img/graphical-toc.svg new file mode 100644 index 0000000000..e887737540 --- /dev/null +++ b/docs/source/img/graphical-toc.svg @@ -0,0 +1,7658 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + HPC + Clouds + P2P + + + Scheduling + Grids + + + Application + + ExperimentalSetup + Simulation + Model Checking + + + + + Property + Reduction + + + + (what you test) + + + + Virtual Platform + + ▸ Resources + ▸ Routing + ▸ External Events + ▸ Actors: + ▸ MPI Legacy Code + ▸ Offline Traces + ▸ Centralized Algo + + C/C++/Java + + + + + + ▸ Safety + ▸ Liveness + ▸ Patterns + + + ▸ DPOR + + ▸ State + Equality + + (highly experimental) + + + Models + Plugins + ▸ Raw Perf. + ▸ Contention + ▸ Collective + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + x + + + ← + + + 2 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + x ≮ y + + + y + + + ← + + + 1 + + + + + + + + + + + + + + + + + + + + + send(1) + + + send(2) + + + Your code + ▸ Signals + ▸ Extensions + + deep inside + + $./my_simulator|MSG_visualization/colorize.pl + [0.000] + [Tremblay:master]Got3workersand6taskstoprocess + [0.000] + [Tremblay:master]Sending’Task_0’to’Jupiter’ + [0.148] + [Tremblay:master]Sending’Task_1’to’Fafard’ + [0.148] + [Jupiter:worker]Processing’Task_0’ + [0.347] + [Tremblay:master]Sending’Task_2’to’Ginette’ + [0.347] + [Fafard:worker]Processing’Task_1’ + [0.476] + [Tremblay:master]Sending’Task_3’to’Jupiter’ + [0.476] + [Ginette:worker]Processing’Task_2’ + [0.803] + [Jupiter:worker]’Task_0’done + [0.951] + [Tremblay:master]Sending’Task_4’to’Fafard’ + [0.951] + [Jupiter:worker]Processing’Task_3’ + [1.003] + [Fafard:worker]’Task_1’done + [1.202] + [Tremblay:master]Sending’Task_5’to’Ginette’ + [1.202] + [Fafard:worker]Processing’Task_4’ + [1.507] + [Ginette:worker]’Task_2’done + [1.606] + [Jupiter:worker]’Task_3’done + [1.635] + [Tremblay:master]Alltasksdispatched.Let’sstopworkers. + [1.635] + [Ginette:worker]Processing’Task_5’ + [1.637] + [Jupiter:worker]I’mdone.Seeyou! + [1.857] + [Fafard:worker]’Task_4’done + [1.859] + [Fafard:worker]I’mdone.Seeyou! + [2.666] + [Ginette:worker]’Task_5’done + [2.668] + [Tremblay:master]Goodbyenow! + [2.668] + [Ginette:worker]I’mdone.Seeyou! + [2.668][]Simulationtime2.66766 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1 + + + 3 + + + 45 + 6 + 2 + Root + + + + End + + + Time, Energy + (CPU, Links, Disks) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 3 + + + + + + + + + + + + + + 6 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 4 + + + + + + + + + + + + + + + + + + + + + + 10G + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1 + + + + + + + 13G + + + + + + + + + + + + + + + + + + 1.5 + + + Config + + + Domains + operations + Exhaustive test + Counter example + + R visualizations + + Textual logs + (paths) + + Calibration + + + Workflows + Fog + + + Volunteer + IoT + + + + + + App Deployment + + + + + diff --git a/docs/source/intro_concepts.rst b/docs/source/intro_concepts.rst index f16e53483c..106b57c314 100644 --- a/docs/source/intro_concepts.rst +++ b/docs/source/intro_concepts.rst @@ -36,6 +36,11 @@ clicking on the "Edit on GitLab" button at the top of every page. Typical Study based on SimGrid ------------------------------ +.. raw:: html + + + + Any SimGrid study entails the following components: - The studied **Application**. This can be either a distributed diff --git a/docs/source/main_concepts.rst b/docs/source/main_concepts.rst new file mode 100644 index 0000000000..051dddd81d --- /dev/null +++ b/docs/source/main_concepts.rst @@ -0,0 +1,267 @@ +.. First introduction + +What is SimGrid +=============== + +SimGrid is a framework to simulate distributed computer systems. + +It can be used to either assess abstract algorithms, or to profile and +debug real distributed applications. SimGrid enables studies in the +domains of (data-)Grids, IaaS Clouds, Clusters, High Performance +Computing, Volunteer Computing and Peer-to-Peer systems. + +Technically speaking, SimGrid is a library. It is neither a graphical +interface nor a command-line simulator running user scripts. The +interaction with SimGrid is done by writing programs with the exposed +functions to build your own simulator. + +SimGrid offers many features, many options and many possibilities. The +documentation aims at smoothing the learning curve. But nothing's +perfect, and this documentation is really no exception here. Please +help us improving it by reporting any issue that you see and +proposing the content that is still missing. + +SimGrid is a Free Software distributed under the LGPL licence. You are +thus welcome to use it as you wish, or even to modify and distribute +your version (as long as your version is as free as ours). It also +means that SimGrid is developed by a vivid community of users and +developers. We hope that you will come and join us! + +SimGrid is the result of almost 20 years of research from several +groups, both in France and in the USA. It benefited of many funding +from various research instances, including the ANR, Inria, CNRS, +University of Lorraine, University of Hawai'i at Manoa, ENS Rennes and +many others. Many thanks to our generous sponsors! + +Typical Study based on SimGrid +------------------------------ + +Any SimGrid study entails the following components: + + - The studied **Application**. This can be either a distributed + algorithm described in our simple APIs, or a full featured real + parallel application using for example the MPI interface + @ref application "(more info)". + + - The **Virtual Platform**. This is a description of a given + distributed system (machines, links, disks, clusters, etc). Most of + the platform files are written in XML althrough a Lua interface is + under development. SimGrid makes it easy to augment the Virtual + Platform with a Dynamic Scenario where for example the links are + slowed down (because of external usage), the machines fail. You + have even support to specify the applicative workload that you want + to feed to your application @ref platform "(more info)". + + - The application's **Deployment Description**. In SimGrid + terminology, the application is an inert set of source files and + binaries. To make it run, you have to describe how your application + should be deployed on the virtual platform. You need to specify + which process is mapped on which host, along with their parameters + @ref deployment "(more info)". + + - The **Platform Models**. They describe how the virtual platform + reacts to the actions of the application. For example, they compute + the time taken by a given communication on the virtual platform. + These models are already included in SimGrid, and you only need to + pick one and maybe tweak its configuration to get your results + @ref models "(more info)". + +These components are put together to run a **simulation**, that is an +experiment or a probe. The result of one or many simulation provides +an **outcome** (logs, visualization, statistical analysis) that help +answering the **question** targeted by this study. + +The questions that SimGrid can solve include the following: + + - **Compare an Application to another**. This is the classical use + case for scientists, who use SimGrid to test how the solution that + they contribute compares to the existing solutions from the + literature. + + - **Design the best Virtual Platform for a given Application.** + Tweaking the platform file is much easier than building a new real + platform for testing purpose. SimGrid also allows co-design of the + platform and the application by modifying both of them. + + - **Debug Real Applications**. With real systems, is sometimes + difficult to reproduce the exact run leading to the bug that you + are tracking. SimGrid gives you experimental reproducibility, + clairevoyance (you can explore every part of the system, and your + probe will not change the simulated state). It also makes it easy + to mock some parts of the real system that are not under study. + +Depending on the context, you may see some parts of this process as +less important, but you should pay close attention if you want to be +confident in the results coming out of your simulations. In +particular, you should not trust blindly your results but always +strive to double-check them. Likewise, you should question the realism +of your input configuration, and we even encourage you to doubt (and +check) the provided performance models. + +To ease such questionning, you really should logically separate these +parts in your experimental setup. It is seen as a very bad practice to +merge the application, the platform and the deployment all together. +SimGrid is versatile and your milleage may vary, but you should start +with your Application specified as a C++ or Java program, using one of +the provided XML platform file, and with your deployment in a separate +XML file. + +SimGrid Execution Modes +----------------------- + +Depending on the intended study, SimGrid can be run in several execution modes. + +** **Simulation Mode**. This is the most common execution mode, where you want +to study how your application behaves on the virtual platform under +the experimental scenario. + +In this mode, SimGrid can provide information about the time taken by +your application, the amount of energy dissipated by the platform to +run your application and the detailed usage of each resource. + +** **Model-Checking Mode**. This can be seen as a sort of exhaustive +testing mode, where every possible outcome of your application is +explored. In some sense, this mode tests your application for all +possible platforms that you could imagine (and more). + +You just provide the application and its deployment (amount of +processes and parameters), and the model-checker will litterally +explore all possible outcomes by testing all possible message +interleavings: if at some point a given process can either receive the +message A first or the message B depending on the platform +characteristics, the model-checker will explore the scenario where A +arrives first, and then rewind to the same point to explore the +scenario where B arrives first. + +This is a very powerful mode, where you can evaluate the correction of +your application. It can verify either **safety properties** (asserts) +or **liveless properties** stating for example that if a given event +occures, then another given event will occur in a finite amount of +steps. This mode is not only usable with the abstract algorithms +developed on top of the SimGrid APIs, but also with real MPI +applications (to some extend). + +The main limit of Model Checking lays in the huge amount of scenarios +to explore. SimGrid tries to explore only non-redundent scenarios +thanks to classical reduction techniques (such as DPOR and stateful +exploration) but the exploration may well never finish if you don't +carefully adapt your application to this mode. + +A classical trap is that the Model Checker can only verify whether +your application fits the provided properties, which is useless if you +have a bug in your property. Remember also that one way for your +application to never violate a given assert is to not start at all +because of a stupid bug. + +Another limit of this mode is that it does not use the performance +models of the simulation mode. Time becomes discrete: You can say for +example that the application took 42 steps to run, but there is no way +to know the amount of seconds that it took or the amount of watts that +it dissipated. + +Finally, the model checker only explores the interleavings of +computations and communications. Other factors such as thread +execution interleaving are not considered by the SimGrid model +checker. + +The model checker may well miss existing issues, as it computes the +possible outcomes *from a given initial situation*. There is no way to +prove the correction of your application in all generality with this +tool. + +** **Benchmark Recording Mode**. During debug sessions, continuous +integration testing and other similar use cases, you are often only +interested in the control flow. If your application apply filters to +huge images split in small blocks, the filtered image is probably not +what you are interested in. You are probably looking for a way to run +each computation kernel only once, save on disk the time it takes and +some other metadata. This code block can then be skipped in simulation +and replaced by a synthetic block using the cached information. The +virtual platform will take this block into account without requesting +the real hosting machine to benchmark it. + +SimGrid Limits +-------------- + +This framework is by no means the perfect holly grail able to solve +every problem on earth. + +** **SimGrid scope is limited to distributed systems.** Real-time +multithreaded systems are not in the scope. You could probably tweak +SimGrid for such studies (or the framework could possibily be extended +in this direction), but another framework specifically targeting this +usecase would probably be more suited. + +** **There is currently no support for IoT studies and wireless networks**. +The framework could certainly be improved in this direction, but this +is still to be done. + +** **There is no perfect model, only models adapted to your study.** +The SimGrid models target fast, large studies yet requesting a +realistic results. In particular, our models abstract away parameters +and phenomenon that are often irrelevant to the realism in our +context. + +SimGrid is simply not intended to any study that would mandate the +abstracted phenomenon. Here are some **studies that you should not do +with SimGrid**: + + - Studying the effect of L3 vs L2 cache effects on your application + - Comparing variantes of TCP + - Exploring pathological cases where TCP breaks down, resulting in + abnormal executions. + - Studying security aspects of your application, in presence of + malicious agents. + +SimGrid Success Stories +----------------------- + +SimGrid was cited in over 1,500 scientific papers (according to Google +Scholar). Among them +`over 200 publications `_ +(written by about 300 individuals) use SimGrid as a scientific +instrument to conduct their experimental evaluation. These +numbers do not count the articles contributing to SimGrid. +This instrument was used in many research communities, such as +`High-Performance Computing `_, +`Cloud Computing `_, +`Workflow Scheduling `_, +`Big Data `_ and +`MapReduce `_, +`Data Grid `_, +`Volunteer Computing `_, +`Peer-to-Peer Computing `_, +`Network Architecture `_, +`Fog Computing `_, or +`Batch Scheduling `_ +`(more info) `_. + +If your platform description is accurate enough (see +`here `_ or +`there `_), +SimGrid can provide high-quality performance predictions. For example, +we determined the speedup achieved by the Tibidabo Arm-based +cluster before its construction +(`paper `_). In this case, +some differences between the prediction and the real timings were due to +misconfiguration or other problems with the real platforms. To some extent, +SimGrid could even be used to debug the real platform :) + +SimGrid is also used to debug, improve and tune several large +applications. +`BigDFT `_ (a massively parallel code +computing the electronic structure of chemical elements developped by +the CEA), `StarPU `_ (a +Unified Runtime System for Heterogeneous Multicore Architectures +developped by Inria Bordeaux) and +`TomP2P `_ (a high performance +key-value pair storage library developped at University of Zurich). +Some of these applications enjoy large user communities themselves. + +Where to proceed next? +---------------------- + +Now that you know about the basic concepts of SimGrid, you can give it +a try. If it's not done yet, first :ref:`install it `. Then, +proceed to the section on @ref application "describing the application" that +you want to study. diff --git a/docs/source/models.rst b/docs/source/models.rst new file mode 100644 index 0000000000..a180643175 --- /dev/null +++ b/docs/source/models.rst @@ -0,0 +1,4 @@ +.. _models: + +The SimGrid Models +=================== diff --git a/docs/source/platform.rst b/docs/source/platform.rst new file mode 100644 index 0000000000..b2209b082f --- /dev/null +++ b/docs/source/platform.rst @@ -0,0 +1,16 @@ +.. _platform: + +.. raw:: html + + + +
+
+ +Describing your Virtual Platform +================================ diff --git a/docs/source/scenar_config.rst b/docs/source/scenar_config.rst new file mode 100644 index 0000000000..671a6b0dc2 --- /dev/null +++ b/docs/source/scenar_config.rst @@ -0,0 +1,16 @@ +.. _options: + +Configuring SimGrid +=================== + +.. raw:: html + + + +
+
diff --git a/docs/source/scenario.rst b/docs/source/scenario.rst new file mode 100644 index 0000000000..defdec29cc --- /dev/null +++ b/docs/source/scenario.rst @@ -0,0 +1,6 @@ +.. _scenario: + +Describing the Experimental Scenario +************************************ + +.. include:: scenar_config.rst