6 SimGrid was conceived as a tool to study distributed algorithms. Its
7 modern S4U interface makes it easy to assess Cloud, P2P, HPC, IoT and
10 A typical SimGrid simulation is composed of several **Actors**
11 |api_s4u_Actor|_ , that execute user-provided functions. The actors
12 have to explicitly use the S4U interface to express their computation,
13 communication, disk usage and other **Activities** |api_s4u_Activity|_
14 , so that they get reflected within the simulator. These activities
15 take place on **Resources** (CPUs, links, disks). SimGrid predicts the
16 time taken by each activity and orchestrates accordingly the actors
17 waiting for the completion of these activities.
19 .. |api_s4u_Actor| image:: /images/extlink.png
22 .. _api_s4u_Actor: api/classsimgrid_1_1s4u_1_1Actor.html#class-documentation
24 .. |api_s4u_Activity| image:: /images/extlink.png
27 .. _api_s4u_Activity: api/classsimgrid_1_1s4u_1_1Activity.html#class-documentation
30 Each actor executes a user-provided function on a simulated **Host**
31 |api_s4u_Host|_ with which it can interact. Communications are not
32 directly sent to actors, but posted onto **Mailboxes**
33 |api_s4u_Mailbox|_ that serve as rendez-vous points between
34 communicating processes.
36 .. |api_s4u_Host| image:: /images/extlink.png
39 .. _api_s4u_Host: api/classsimgrid_1_1s4u_1_1Host.html#class-documentation
41 .. |api_s4u_Mailbox| image:: /images/extlink.png
44 .. _api_s4u_Mailbox: api/classsimgrid_1_1s4u_1_1Mailbox.html#class-documentation
46 **In the remainder of this tutorial**, you will discover a simple yet
47 fully functioning example of SimGrid simulation: the Master/Workers
48 application. We will detail each part of the code and necessary
49 configuration to make it working. After this tour, several exercises
50 are proposed to let you discover some of the SimGrid features, hands
51 on the keyboard. This practical session will be given in C++, that you
52 are supposed to know beforehand.
55 Discover the Master/Workers
56 ---------------------------
58 This section introduces a first example of SimGrid simulation. This
59 simple application is composed of two kind of actors: the **master**
60 is in charge of distributing some computational tasks to a set of
61 **workers** that execute them.
63 .. image:: /tuto_s4u/img/intro.svg
66 We first present a round-robin version of this application, where the
67 master dispatches the tasks to the workers, one after the other, until
68 all tasks are dispatched. Later in this tutorial, you will be given
69 the opportunity to improve this scheme.
74 Let's start with the code of the worker. It is represented by the
75 *master* function below. This simple function takes at least 3
76 parameters (the amount of tasks to dispatch, their computational size
77 in flops to compute and their communication size in bytes to
78 exchange). Every parameter after the third one must be the name of an
79 host on which a worker is waiting for something to compute.
81 Then, the tasks are sent one after the other, each on a mailbox named
82 after the worker's hosts. On the other side, a given worker (which
83 code is given below) wait for incoming tasks on its own
88 At the end, once all tasks are dispatched, the master dispatches
89 another task per worker, but this time with a negative amount of flops
90 to compute. Indeed, this application decided by convention, that the
91 workers should stop when encountering such a negative compute_size.
93 At the end of the day, the only SimGrid specific functions used in
94 this example are :cpp:func:`simgrid::s4u::Mailbox::by_name` and
95 :cpp:func:`simgrid::s4u::Mailbox::put`. Also, :c:macro:`XBT_INFO` is used
96 as a replacement to printf() or to cout to ensure that the messages
97 are nicely logged along with the simulated time and actor name.
100 .. literalinclude:: ../../examples/s4u/app-masterworkers/s4u-app-masterworkers-fun.cpp
102 :start-after: master-begin
103 :end-before: master-end
105 Here comes the code of the worker actors. This function expects no
106 parameter from its vector of strings. Its code is very simple: it
107 expects messages on the mailbox that is named after its own host. As long as it gets valid
108 computation requests (whose compute_amount is positive), it compute
109 this task and waits for the next one.
111 The worker retrieves its own host with
112 :cpp:func:`simgrid::s4u::this_actor::get_host`. The
113 :ref:`simgrid::s4u::this_actor <namespace_simgrid__s4u__this_actor>`
114 namespace contains many such helping functions.
116 .. literalinclude:: ../../examples/s4u/app-masterworkers/s4u-app-masterworkers-fun.cpp
118 :start-after: worker-begin
119 :end-before: worker-end
121 Starting the Simulation
122 .......................
124 And this is it. In only a few lines, we defined the algorithm of our
125 master/workers examples.
127 That being said, an algorithm alone is not enough to define a
128 simulation: SimGrid is a library, not a program. So you need to define
129 your own ``main()`` function as follows. This function is in charge of
130 creating a SimGrid simulation engine (on line 3), register the actor
131 functions to the engine (on lines 7 and 8), load the virtual platform
132 from its description file (on line 11), map actors onto that platform
133 (on line 12) and run the simulation until its completion on line 15.
135 .. literalinclude:: ../../examples/s4u/app-masterworkers/s4u-app-masterworkers-fun.cpp
137 :start-after: main-begin
138 :end-before: main-end
141 As you can see, this also requires a platform file and a deployment
147 Platform files define the virtual platform on which the provided
148 application will take place. In contains one or several **Network
149 Zone** |api_s4u_NetZone|_ that contain both **Host-** |api_s4u_Host|_
150 and **Link-** |api_s4u_Link|_ Resources, as well as routing
153 Such files can get rather long and boring, so the example below is
154 only an excerpts of the full ``examples/platforms/small_platform.xml``
155 file. For example, most routing information are missing, and only the
156 route between the hosts Tremblay and Fafard is given. This path
157 traverses 6 links (named 4, 3, 2, 0, 1 and 8). There are several
158 examples of platforms in the archive under ``examples/platforms``.
160 .. |api_s4u_NetZone| image:: /images/extlink.png
163 .. _api_s4u_NetZone: api/classsimgrid_1_1s4u_1_1NetZone.html#class-documentation
165 .. |api_s4u_Link| image:: /images/extlink.png
168 .. _api_s4u_Link: api/classsimgrid_1_1s4u_1_1Link.html#class-documentation
170 .. literalinclude:: ../../examples/platforms/small_platform.xml
172 :lines: 1-10,12-20,56-62,192-
173 :caption: (excerpts of the small_platform.xml file)
178 Deployment files specify the execution scenario: it lists the actors
179 that should be started, along with their parameter. In the following
180 example, we start 6 actors: one master and 5 workers.
182 .. literalinclude:: ../../examples/s4u/app-masterworkers/s4u-app-masterworkers_d.xml
188 This time, we have all parts: once the program is compiled, we can
189 execute it as follows. Note how the XBT_INFO() requests turned into
190 informative messages.
192 .. literalinclude:: ../../examples/s4u/app-masterworkers/s4u-app-masterworkers.tesh
194 :start-after: s4u-app-masterworkers-fun
195 :prepend: $$$ ./masterworkers platform.xml deploy.xml
203 In this section, you will modify the example presented earlier to
204 explore the quality of the proposed algorithm. For now, it works and
205 the simulation prints things, but the truth is that we have no idea of
206 whether this is a good algorithm to dispatch tasks to the workers.
207 This very simple setting raises many interesting questions:
209 .. image:: /tuto_s4u/img/question.svg
212 - Which algorithm should the master use? Or should the worker decide
215 Round Robin is not an efficient algorithm when all tasks are not
216 processed at the same speed. It would probably be more efficient
217 if the workers were asking for tasks when ready.
219 - Should tasks be grouped in batches or sent separately?
221 The workers will starve if they don't get the tasks fast
222 enough. One possibility to reduce latency would be to send tasks
223 in pools instead of one by one. But if the pools are too big, the
224 load balancing will likely get uneven, in particular when
225 distributing the last tasks.
227 - How does the quality of such algorithm dependent on the platform
228 characteristics and on the task characteristics?
230 Whenever the input communication time is very small compared to
231 processing time and workers are homogeneous, it is likely that the
232 round-robin algorithm performs very well. Would it still hold true
233 when transfer time is not negligible? What if some tasks are
234 performed faster on some specific nodes?
236 - The network topology interconnecting the master and the workers
237 may be quite complicated. How does such a topology impact the
240 When data transfers are the bottleneck, it is likely that a good
241 modeling of the platform becomes essential. The SimGrid platform
242 models are particularly handy to account for complex platform
245 - What is the best applicative topology?
247 Is a flat master worker deployment sufficient? Should we go for a
248 hierarchical algorithm, with some forwarders taking large pools of
249 tasks from the master, each of them distributing their tasks to a
250 sub-pool of workers? Or should we introduce super-peers,
251 dupplicating the master's role in a peer-to-peer manner? Do the
252 algorithms require a perfect knowledge of the network?
254 - How is such an algorithm sensitive to external workload variation?
256 What if bandwidth, latency and computing speed can vary with no
257 warning? Shouldn't you study whether your algorithm is sensitive
258 to such load variations?
260 - Although an algorithm may be more efficient than another, how does
261 it interfere with unrelated applications executing on the same
264 **SimGrid was invented to answer such questions.** Do not believe the
265 fools saying that all you need to study such settings is a simple
266 discrete event simulator. Do you really want to reinvent the wheel,
267 debug and optimize your own tool, and validate its models against real
268 settings for ages, or do you prefer to sit on the shoulders of a
269 giant? With SimGrid, you can focus on your algorithm. The whole
270 simulation mechanism is already working.
272 Here is the visualization of a SimGrid simulation of two master worker
273 applications (one in light gray and the other in dark gray) running in
274 concurrence and showing resource usage over a long period of time. It
275 was obtained with the Triva software.
277 .. image:: /tuto_s4u/img/result.png
283 The easiest way to take the tutorial is to use the dedicated Docker
284 image. Once you `installed Docker itself
285 <https://docs.docker.com/install/>`_, simply do the following:
287 .. code-block:: shell
289 docker pull simgrid/tuto-s4u
290 docker run -it --rm --name simgrid --volume ~/simgrid-tutorial:/src/tutorial simgrid/tuto-s4u bash
292 This will start a new container with all you need to take this
293 tutorial, and create a ``simgrid-tutorial`` directory in your home on
294 your host machine that will be visible as ``/src/tutorial`` within the
295 container. You can then edit the files you want with your favorite
296 editor in ``~/simgrid-tutorial``, and compile them within the
297 container to enjoy the provided dependencies.
301 Any change to the container out of ``/src/tutorial`` will be lost
302 when you log out of the container, so don't edit the other files!
304 All needed dependencies are already installed in this container
305 (SimGrid, a C++ compiler, cmake, pajeng and R). Vite being only
306 optional in this tutorial, it is not installed to reduce the image
309 The code template is available under ``/src/simgrid-template-s4u`` in
310 the image. You should copy it to your working directory when you first
313 .. code-block:: shell
315 cp -r /src/simgrid-template-s4u/* /src/tutorial
318 Using your Computer Natively
319 ............................
321 To take the tutorial on your machine, you first need to :ref:`install
322 SimGrid <install>`, a C++ compiler and also ``pajeng`` to visualize
323 the traces. You may want to install `Vite
324 <http://vite.gforge.inria.fr/>`_ to get a first glance at the
325 traces. The provided code template requires cmake to compile. On
326 Debian and Ubuntu for example, you can get them as follows:
328 .. code-block:: shell
330 sudo apt install simgrid pajeng cmake g++ vite
332 An initial version of the source code is provided on framagit. This
333 template compiles with cmake. If SimGrid is correctly installed, you
334 should be able to clone the `repository
335 <https://framagit.org/simgrid/simgrid-template-s4u>`_ and recompile
336 everything as follows:
338 .. code-block:: shell
340 git clone git@framagit.org:simgrid/simgrid-template-s4u.git
341 cd simgrid-template-s4u/
345 If you struggle with the compilation, then you should double check
346 your :ref:`SimGrid installation <install>`. On need, please refer to
347 the :ref:`Troubleshooting your Project Setup
348 <install_yours_troubleshooting>` section.
350 Discovering the Provided Code
351 .............................
353 Please compile and execute the provided simulator as follows:
356 .. code-block:: shell
359 ./master-workers small_platform.xml master-workers_d.xml
361 For a more "fancy" output, you can use simgrid-colorizer.
363 .. code-block:: shell
365 ./master-workers small_platform.xml master-workers_d.xml 2>&1 | simgrid-colorizer
367 If you installed SimGrid to a non-standard path, you may have to
368 specify the full path to simgrid-colorizer on the above line, such as
369 ``/opt/simgrid/bin/simgrid-colorizer``. If you did not install it at all,
370 you can find it in <simgrid_root_directory>/bin/colorize.
372 For a classical Gantt-Chart vizualisation, you can use `Vite
373 <http://vite.gforge.inria.fr/>`_ if you have it installed, as
374 follows. But do not spend too much time installing Vite, because there
375 is a better way to visualize SimGrid traces (see below).
377 .. code-block:: shell
379 ./master-workers small_platform.xml master-workers_d.xml --cfg=tracing:yes --cfg=tracing/msg/process:yes
382 .. image:: /tuto_s4u/img/vite-screenshot.png
385 If you want the full power to visualize SimGrid traces, you need
386 to use R. As a start, you can download this `starter script
387 <https://framagit.org/simgrid/simgrid/raw/master/docs/source/tuto_s4u/draw_gantt.R>`_
388 and use it as follows:
390 .. code-block:: shell
392 ./master-workers small_platform.xml master-workers_d.xml --cfg=tracing:yes --cfg=tracing/msg/process:yes
393 pj_dump --ignore-incomplete-links simgrid.trace | grep STATE > gantt.csv
394 Rscript draw_gantt.R gantt.csv
396 It produces a ``Rplots.pdf`` with the following content:
398 .. image:: /tuto_s4u/img/Rscript-screenshot.png
402 Lab 1: Simpler Deployments
403 --------------------------
405 In the provided example, adding more workers quickly becomes a pain:
406 You need to start them (at the bottom of the file), and to inform the
407 master of its availability with an extra parameter. This is mandatory
408 if you want to inform the master of where the workers are running. But
409 actually, the master does not need to have this information.
411 We could leverage the mailbox mechanism flexibility, and use a sort of
412 yellow page system: Instead of sending data to the worker running on
413 Fafard, the master could send data to the third worker. Ie, instead of
414 using the worker location (which should be filled in two locations),
415 we could use their ID (which should be filled in one location
418 This could be done with the following deployment file. It's clearly
419 not shorter than the previous one, but it's still simpler because the
420 information is only written once. It thus follows the `DRY
421 <https://en.wikipedia.org/wiki/Don't_repeat_yourself>`_ `SPOT
422 <http://wiki.c2.com/?SinglePointOfTruth>`_ design principle.
424 .. literalinclude:: tuto_s4u/deployment1.xml
428 Copy your ``master-workers.cpp`` into ``master-workers-lab1.cpp`` and
429 add a new executable into ``CMakeLists.txt``. Then modify your worker
430 function so that it gets its mailbox name not from the name of its
431 host, but from the string passed as ``args[1]``. The master will send
432 messages to all workers based on their number, for example as follows:
436 for (int i = 0; i < tasks_count; i++) {
437 std::string worker_rank = std::to_string(i % workers_count);
438 std::string mailbox_name = std::string("worker-") + worker_rank;
439 simgrid::s4u::MailboxPtr mailbox = simgrid::s4u::Mailbox::by_name(mailbox_name);
450 The mailboxes are a very powerful mechanism in SimGrid, allowing many
451 interesting application settings. They may feel surprising if you are
452 used to BSD sockets or other classical systems, but you will soon
453 appreciate their power. They are only used to match the
454 communications, but have no impact on the communication
455 timing. ``put()`` and ``get()`` are matched regardless of their
456 initiators' location and then the real communication occures between
457 the involved parties.
459 Please refer to the full `API of Mailboxes
460 <api/classsimgrid_1_1s4u_1_1Mailbox.html#class-documentation>`_
461 |api_s4u_Mailbox|_ for more details.
464 Lab 2: Using the Whole Platform
465 -------------------------------
467 It is now easier to add a new worker, but you still has to do it
468 manually. It would be much easier if the master could start the
469 workers on its own, one per available host in the platform. The new
470 deployment file should be as simple as:
472 .. literalinclude:: tuto_s4u/deployment2.xml
476 Creating the workers from the master
477 ....................................
479 For that, the master needs to retrieve the list of hosts declared in
480 the platform with :cpp:func:`simgrid::s4u::Engine::get_all_hosts`.
481 Then, the master should start the worker processes with
482 :cpp:func:`simgrid::s4u::Actor::create`.
484 ``Actor::create(name, host, func, params...)`` is a very flexible
485 function. Its third parameter is the function that the actor should
486 execute. This function can take any kind of parameter, provided that
487 you pass similar parameters to ``Actor::create()``. For example, you
488 could have something like this:
492 void my_actor(int param1, double param2, std::string param3) {
495 int main(int argc, char argv**) {
497 simgrid::s4u::ActorPtr actor;
498 actor = simgrid::s4u::Actor::create("name", simgrid::s4u::Host::by_name("the_host"),
499 &my_actor, 42, 3.14, "thevalue");
504 Master-Workers Communication
505 ............................
507 Previously, the workers got from their parameter the name of the
508 mailbox they should use. We can still do so: the master should build
509 such a parameter before using it in the ``Actor::create()`` call. The
510 master could even pass directly the mailbox as a parameter to the
513 Since we want later to study concurrent applications, it is advised to
514 use a mailbox name that is unique over the simulation even if there is
515 more than one master.
517 One possibility for that is to use the actor ID (aid) of each worker
518 as a mailbox name. The master can retrieve the aid of the newly
519 created actor with ``actor->get_pid()`` while the actor itself can
520 retrieve its own aid with ``simgrid::s4u::this_actor::get_pid()``.
521 The retrieved value is an ``aid_t``, which is an alias for ``long``.
523 Instead of having one mailbox per worker, you could also reorganize
524 completely your application to have only one mailbox per master. All
525 the workers of a given master would pull their work from the same
526 mailbox, which should be passed as parameter to the workers. This
527 reduces the amount of mailboxes, but prevents the master from taking
528 any scheduling decision. It really depends on how you want to organize
529 your application and what you want to study with your simulator. In
530 this tutorial, that's probably not a good idea.
535 In this exercise, we reduced the amount of configuration that our
536 simulator requests. This is both a good idea, and a dangerous
537 trend. This simplification is another application of the good old DRY/SPOT
538 programming principle (`Don't Repeat Yourself / Single Point Of Truth
539 <https://en.wikipedia.org/wiki/Don%27t_repeat_yourself>`_), and you
540 really want your programming artefacts to follow these software
541 engineering principles.
543 But at the same time, you should be careful in separating your
544 scientific contribution (the master/workers algorithm) and the
545 artefacts used to test it (platform, deployment and workload). This is
546 why SimGrid forces you to express your platform and deployment files
547 in XML instead of using a programming interface: it forces a clear
548 separation of concerns between things of very different nature.
550 Lab 3: Fixed Experiment Duration
551 --------------------------------
553 In the current version, the number of tasks is defined through the
554 worker arguments. Hence, tasks are created at the very beginning of
555 the simulation. Instead, have the master dispatching tasks for a
556 predetermined amount of time. The tasks must now be created on demand
557 instead of beforehand.
559 Of course, usual time functions like ``gettimeofday`` will give you the
560 time on your real machine, which is prety useless in the
561 simulation. Instead, retrieve the time in the simulated world with
562 :cpp:func:`simgrid::s4u::Engine::get_clock`.
564 You can still stop your workers with a specific task as previously,
565 or you may kill them forcefully with
566 :cpp:func:`simgrid::s4u::Actor::kill` (if you already have a reference
567 to the actor you want to kill) or
568 :cpp:func:`void simgrid::s4u::Actor::kill(aid_t)` (if you only have its ID).
571 Anyway, the new deployment `deployment3.xml` file should thus look
574 .. literalinclude:: tuto_s4u/deployment3.xml
577 Controlling the message verbosity
578 .................................
580 Not all messages are equally informative, so you probably want to
581 change some of the ``XBT_INFO`` into ``XBT_DEBUG`` so that they are
582 hidden by default. For example, you may want to use ``XBT_INFO`` once
583 every 100 tasks and ``XBT_DEBUG`` when sending all the other tasks. Or
584 you could show only the total number of tasks processed by
585 default. You can still see the debug messages as follows:
587 .. code-block:: shell
589 ./master-workers-lab3 small_platform.xml deployment3.xml --log=msg_test.thres:debug
592 Lab 4: Competing Applications
593 -----------------------------
595 It is now time to start several applications at once, with the following ``deployment4.xml`` file.
597 .. literalinclude:: tuto_s4u/deployment4.xml
600 Things happen when you do so, but it remains utterly difficult to
601 understand what's happening exactely. Even Gantt visualizations
602 contain too much information to be useful: it is impossible to
603 understand which task belong to which application. To fix this, we
604 will categorize the tasks.
606 Instead of starting the execution in one function call only with
607 ``this_actor::execute(cost)``, you need to
608 create the execution activity, set its tracing category, and then start
609 it and wait for its completion, as follows:
613 simgrid::s4u::ExecPtr exec = simgrid::s4u::this_actor::exec_init(compute_cost);
614 exec->set_tracing_category(category);
615 // exec->start() is optional here as wait() starts the activity on need
618 You can make the same code shorter as follows:
622 simgrid::s4u::this_actor::exec_init(compute_cost)->set_tracing_category(category)->wait();
624 Visualizing the result
625 .......................
627 vite is not enough to understand the situation, because it does not
628 deal with categorization. This time, you absolutely must switch to R,
629 as explained on `this page
630 <http://simgrid.gforge.inria.fr/contrib/R_visualization.php>`_.
634 Include here the minimal setting to view something in R.
637 Lab 5: Better Scheduling
638 ------------------------
640 You don't need a very advanced visualization solution to notice that
641 round-robin is completely suboptimal: most of the workers keep waiting
642 for more work. We will move to a First-Come First-Served mechanism
645 For that, your workers should explicitely request for work with a
646 message sent to a channel that is specific to their master. The name
647 of that private channel can be the one used to categorize the
648 executions, as it is already specific to each master.
650 The master should serve in a round-robin manner the requests it
651 receives, until the time is up. Changing the communication schema can
652 be a bit hairy, but once it works, you will see that such as simple
653 FCFS schema allows to double the amount of tasks handled over time
654 here. Things may be different with another platform file.
659 From this, many things can easily be added. For example, you could:
661 - Allow workers to have several pending requests so as to overlap
662 communication and computations as much as possible. Non-blocking
663 communication will probably become handy here.
664 - Add a performance measurement mechanism, enabling the master to make smart scheduling choices.
665 - Test your code on other platforms, from the ``examples/platforms``
666 directory in your archive.
668 What is the largest number of tasks requiring 50e6 flops and 1e5
669 bytes that you manage to distribute and process in one hour on
671 - Optimize not only for the amount of tasks handled, but also for the
672 total energy dissipated.
673 - And so on. If you come up with a really nice extension, please share
674 it with us so that we can extend this tutorial.
679 This tutorial is now terminated. You could keep reading the [online documentation][fn:4] or
680 [tutorials][fn:7], or you could head up to the example section to read some code.
684 TODO: Points to improve for the next time
686 - Propose equivalent exercises and skeleton in java.
687 - Propose a virtualbox image with everything (simgrid, pajeng, ...) already set up.
688 - Ease the installation on mac OS X (binary installer) and windows.
690 .. LocalWords: SimGrid