From: Martin Quinson Date: Fri, 24 Aug 2018 07:37:27 +0000 (+0200) Subject: start removing from doc/ what's converted in docs/ X-Git-Tag: v3_21~200 X-Git-Url: http://info.iut-bm.univ-fcomte.fr/pub/gitweb/simgrid.git/commitdiff_plain/a451588d036aa6addbf51c71bbb670f81acfe319 start removing from doc/ what's converted in docs/ --- diff --git a/doc/doxygen/application.doc b/doc/doxygen/application.doc index 1bfb58a6c3..af5fe1d399 100644 --- a/doc/doxygen/application.doc +++ b/doc/doxygen/application.doc @@ -1,28 +1,6 @@ /** @page application Describing your application -TBD - -* Main concepts - - - *Actor* (or process in legacy interfaces of SimGrid) - - - *Activities*: resource usage that applications do. This is what - takes time. - - Any given actor can do at most one foreground activity, while it - can also do several non-blocking activities in the background. - - - These activities take place on *Resources* (links, compute - machine, disks). Resources should be described (created) in the - Virtual Platform (link), but your application have many other ways - to interact with the resource. - - - The resources are arranged in a hierarchy of *Networking Zones*, - with which the application can also interact. The netzone knows the - networking path between one resource to another. - -Speak of mailboxes here? Where if not? * Interfaces to describe applications diff --git a/doc/doxygen/getting_started.doc b/doc/doxygen/getting_started.doc deleted file mode 100644 index f7a049a532..0000000000 --- a/doc/doxygen/getting_started.doc +++ /dev/null @@ -1,263 +0,0 @@ -/*! @page getting_started Getting Started: SimGrid Main Concepts - -@tableofcontents - -SimGrid is a framework to simulate distributed computer systems. - -It can be used to either assess abstract algorithms, or to profile and -debug real distributed applications. SimGrid enables studies in the -domains of (data-)Grids, IaaS Clouds, Clusters, High Performance -Computing, Volunteer Computing and Peer-to-Peer systems. - -Technically speaking, SimGrid is a library. It is neither a graphical -interface nor a command-line simulator running user scripts. The -interaction with SimGrid is done by writing programs with the exposed -functions to build your own simulator. - -SimGrid offers many features, many options and many possibilities. The -documentation aims at smoothing the learning curve. But nothing's -perfect, and this documentation is really no exception here. Please -help us improving it by reporting any issue that you see and -proposing the content that is still missing. - -SimGrid is a Free Software distributed under the LGPL licence. You are -thus welcome to use it as you wish, or even to modify and distribute -your version (as long as your version is as free as ours). It also -means that SimGrid is developed by a vivid community of users and -developers. We hope that you will come and join us! - -SimGrid is the result of almost 20 years of research from several -groups, both in France and in the USA. It benefited of many funding -from various research instances, including the ANR, Inria, CNRS, -University of Lorraine, University of Hawai'i at Manoa, ENS Rennes and -many others. Many thanks to our generous sponsors! - -@section starting_components Typical Study based on SimGrid - -Any SimGrid study entails the following components: - - - The studied **Application**. This can be either a distributed - algorithm described in our simple APIs, or a full featured real - parallel application using for example the MPI interface - @ref application "(more info)". - - - The **Virtual Platform**. This is a description of a given - distributed system (machines, links, disks, clusters, etc). Most of - the platform files are written in XML althrough a Lua interface is - under development. SimGrid makes it easy to augment the Virtual - Platform with a Dynamic Scenario where for example the links are - slowed down (because of external usage), the machines fail. You - have even support to specify the applicative workload that you want - to feed to your application @ref platform "(more info)". - - - The application's **Deployment Description**. In SimGrid - terminology, the application is an inert set of source files and - binaries. To make it run, you have to describe how your application - should be deployed on the virtual platform. You need to specify - which process is mapped on which host, along with their parameters - @ref deployment "(more info)". - - - The **Platform Models**. They describe how the virtual platform - reacts to the actions of the application. For example, they compute - the time taken by a given communication on the virtual platform. - These models are already included in SimGrid, and you only need to - pick one and maybe tweak its configuration to get your results - @ref models "(more info)". - -These components are put together to run a **simulation**, that is an -experiment or a probe. The result of one or many simulation provides -an **outcome** (logs, visualization, statistical analysis) that help -answering the **question** targeted by this study. - -The questions that SimGrid can solve include the following: - - - **Compare an Application to another**. This is the classical use - case for scientists, who use SimGrid to test how the solution that - they contribute compares to the existing solutions from the - literature. - - - **Design the best Virtual Platform for a given Application.** - Tweaking the platform file is much easier than building a new real - platform for testing purpose. SimGrid also allows co-design of the - platform and the application by modifying both of them. - - - **Debug Real Applications**. With real systems, is sometimes - difficult to reproduce the exact run leading to the bug that you - are tracking. SimGrid gives you experimental reproducibility, - clairevoyance (you can explore every part of the system, and your - probe will not change the simulated state). It also makes it easy - to mock some parts of the real system that are not under study. - -Depending on the context, you may see some parts of this process as -less important, but you should pay close attention if you want to be -confident in the results coming out of your simulations. In -particular, you should not trust blindly your results but always -strive to double-check them. Likewise, you should question the realism -of your input configuration, and we even encourage you to doubt (and -check) the provided performance models. - -To ease such questionning, you really should logically separate these -parts in your experimental setup. It is seen as a very bad practice to -merge the application, the platform and the deployment all together. -SimGrid is versatile and your milleage may vary, but you should start -with your Application specified as a C++ or Java program, using one of -the provided XML platform file, and with your deployment in a separate -XML file. - -@section starting_gears SimGrid Execution Modes - -Depending on the intended study, SimGrid can be run in several execution modes. - -** **Simulation Mode**. This is the most common execution mode, where you want -to study how your application behaves on the virtual platform under -the experimental scenario. - -In this mode, SimGrid can provide information about the time taken by -your application, the amount of energy dissipated by the platform to -run your application and the detailed usage of each resource. - -** **Model-Checking Mode**. This can be seen as a sort of exhaustive -testing mode, where every possible outcome of your application is -explored. In some sense, this mode tests your application for all -possible platforms that you could imagine (and more). - -You just provide the application and its deployment (amount of -processes and parameters), and the model-checker will litterally -explore all possible outcomes by testing all possible message -interleavings: if at some point a given process can either receive the -message A first or the message B depending on the platform -characteristics, the model-checker will explore the scenario where A -arrives first, and then rewind to the same point to explore the -scenario where B arrives first. - -This is a very powerful mode, where you can evaluate the correction of -your application. It can verify either **safety properties** (asserts) -or **liveless properties** stating for example that if a given event -occures, then another given event will occur in a finite amount of -steps. This mode is not only usable with the abstract algorithms -developed on top of the SimGrid APIs, but also with real MPI -applications (to some extend). - -The main limit of Model Checking lays in the huge amount of scenarios -to explore. SimGrid tries to explore only non-redundent scenarios -thanks to classical reduction techniques (such as DPOR and stateful -exploration) but the exploration may well never finish if you don't -carefully adapt your application to this mode. - -A classical trap is that the Model Checker can only verify whether -your application fits the provided properties, which is useless if you -have a bug in your property. Remember also that one way for your -application to never violate a given assert is to not start at all -because of a stupid bug. - -Another limit of this mode is that it does not use the performance -models of the simulation mode. Time becomes discrete: You can say for -example that the application took 42 steps to run, but there is no way -to know the amount of seconds that it took or the amount of watts that -it dissipated. - -Finally, the model checker only explores the interleavings of -computations and communications. Other factors such as thread -execution interleaving are not considered by the SimGrid model -checker. - -The model checker may well miss existing issues, as it computes the -possible outcomes *from a given initial situation*. There is no way to -prove the correction of your application in all generality with this -tool. - -** **Benchmark Recording Mode**. During debug sessions, continuous -integration testing and other similar use cases, you are often only -interested in the control flow. If your application apply filters to -huge images split in small blocks, the filtered image is probably not -what you are interested in. You are probably looking for a way to run -each computation kernel only once, save on disk the time it takes and -some other metadata. This code block can then be skipped in simulation -and replaced by a synthetic block using the cached information. The -virtual platform will take this block into account without requesting -the real hosting machine to benchmark it. - -@section starting_limits SimGrid Limits - -This framework is by no means the perfect holly grail able to solve -every problem on earth. - -** **SimGrid scope is limited to distributed systems.** Real-time -multithreaded systems are not in the scope. You could probably tweak -SimGrid for such studies (or the framework could possibily be extended -in this direction), but another framework specifically targeting this -usecase would probably be more suited. - -** **There is currently no support for IoT studies and wireless networks**. -The framework could certainly be improved in this direction, but this -is still to be done. - -** **There is no perfect model, only models adapted to your study.** -The SimGrid models target fast, large studies yet requesting a -realistic results. In particular, our models abstract away parameters -and phenomenon that are often irrelevant to the realism in our -context. - -SimGrid is simply not intended to any study that would mandate the -abstracted phenomenon. Here are some **studies that you should not do -with SimGrid**: - - - Studying the effect of L3 vs L2 cache effects on your application - - Comparing variantes of TCP - - Exploring pathological cases where TCP breaks down, resulting in - abnormal executions. - - Studying security aspects of your application, in presence of - malicious agents. - -@section starting_successes SimGrid Success Stories - -SimGrid was cited in over 1,500 scientific papers (according to Google -Scholar). Among them -over 200 -publications (written by about 300 individuals) use SimGrid as a -scientific instrument to conduct their experimental evaluation. These -numbers do not count the articles contributing to SimGrid. -This instrument was used in many research communities, such as -High-Performance Computing, -Cloud Computing, -Workflow Scheduling, -Big Data and -MapReduce, -Data Grid, -Volunteer Computing, -Peer-to-Peer Computing, -Network Architecture, -Fog Computing, or -Batch Scheduling -(more info). - -If your platform description is accurate enough (see -here or -there), -SimGrid can provide high-quality performance predictions. For example, -we determined the speedup achieved by the Tibidabo Arm-based -cluster before its construction -(paper). In this case, -some differences between the prediction and the real timings were due to -misconfiguration or other problems with the real platforms. To some extent, -SimGrid could even be used to debug the real platform :) - -SimGrid is also used to debug, improve and tune several large -applications. -BigDFT (a massively parallel code -computing the electronic structure of chemical elements developped by -the CEA), StarPU (a -Unified Runtime System for Heterogeneous Multicore Architectures -developped by Inria Bordeaux) and -TomP2P (a high performance -key-value pair storage library developped at University of Zurich). -Some of these applications enjoy large user communities themselves. - -@section starting_next Where to proceed next? - -Now that you know about the basic concepts of SimGrid, you can give it -a try. If it's not done yet, first @ref install "install it". Then, -proceed to the section on @ref application "describing the application" that -you want to study. - -*/ \ No newline at end of file diff --git a/doc/doxygen/models.doc b/doc/doxygen/models.doc deleted file mode 100644 index caed4c6b40..0000000000 --- a/doc/doxygen/models.doc +++ /dev/null @@ -1,10 +0,0 @@ -/** -@page models SimGrid Models - -TBD - - - Main existing models (contention, cste, LM07) - - Main concepts (Routing, LMM) + link to the papers - - How to switch on the command line - -*/ \ No newline at end of file diff --git a/doc/doxygen/scenario.doc b/doc/doxygen/scenario.doc deleted file mode 100644 index 7fd0d5e6ae..0000000000 --- a/doc/doxygen/scenario.doc +++ /dev/null @@ -1,13 +0,0 @@ -/** -@page scenario Describing the experimental scenario - -TBD - -Main concepts: - - - Deployment file - - Availability and state profiles - - Reproducible random number generation - - Command line options, in particular on the model switching - -*/ \ No newline at end of file diff --git a/doc/tuto-msg/Makefile b/doc/tuto-msg/Makefile deleted file mode 100644 index fbac5c35c4..0000000000 --- a/doc/tuto-msg/Makefile +++ /dev/null @@ -1,37 +0,0 @@ -# This Makefile is specifically tailored for the binaries of this tutorial. - -# For your own project, you should use the one provided at -# http://simgrid.gforge.inria.fr/simgrid/latest/doc/install_yours.html - -# Some configuration -SIMGRID_INSTALL_PATH ?= ../.. # Where you installed simgrid -CC ?= gcc # Your compiler (on Mac, use clang instead) - -# No change needed bellow for this tutorial. -############################################################################ - -all: masterworker - -WARNINGS = -Wall -Wextra -Wshadow -Wcast-align -Waggregate-return -Wmissing-prototypes \ - -Wmissing-declarations -Wmissing-prototypes -Wpointer-arith -Wwrite-strings \ - -Wmissing-noreturn -Wredundant-decls \ - -Wno-nested-externs -Wno-strict-prototypes -Wno-unused-parameter -WARNINGS += -Werror # Comment that line to not be in paranoid mode - -# CFLAGS = -std=gnu99 -g -O0 $(WARNINGS) $(EXTRA_CFLAGS) # Use this line to make debugging easier -CFLAGS = -std=gnu99 -g -O2 $(WARNINGS) $(EXTRA_CFLAGS) # Use this line to get better performance - -# No change should be mandated past that line -############################################# -# The following are implicit rules, used by default to actually build -# the targets for which you listed the dependencies above. - -.SUFFIXES: .c -# The blanks before the $(CC) must be a Tab char, not spaces -.c: - $(CC) -I$(strip $(SIMGRID_INSTALL_PATH))/include -L$(strip $(SIMGRID_INSTALL_PATH))/lib/ $(CFLAGS) $< -lsimgrid -o $@ - -clean: - rm -f *.o *~ masterworker masterworker-sol1 masterworker-sol2 masterworker-sol3 masterworker-sol4 -.PHONY: clean - diff --git a/doc/tuto-msg/deployment0.xml b/doc/tuto-msg/deployment0.xml deleted file mode 100644 index a9fe0365de..0000000000 --- a/doc/tuto-msg/deployment0.xml +++ /dev/null @@ -1,21 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - diff --git a/doc/tuto-msg/deployment1.xml b/doc/tuto-msg/deployment1.xml deleted file mode 100644 index 2286d864b9..0000000000 --- a/doc/tuto-msg/deployment1.xml +++ /dev/null @@ -1,10 +0,0 @@ - - - - - - - - - - diff --git a/doc/tuto-msg/deployment2.xml b/doc/tuto-msg/deployment2.xml deleted file mode 100644 index 6cddb86c47..0000000000 --- a/doc/tuto-msg/deployment2.xml +++ /dev/null @@ -1,9 +0,0 @@ - - - - - - - - - diff --git a/doc/tuto-msg/deployment3.xml b/doc/tuto-msg/deployment3.xml deleted file mode 100644 index e5dcfe11d5..0000000000 --- a/doc/tuto-msg/deployment3.xml +++ /dev/null @@ -1,19 +0,0 @@ - - - - - - - - - - - - - - - - - - - diff --git a/doc/tuto-msg/deployment_general.xml b/doc/tuto-msg/deployment_general.xml deleted file mode 100644 index adbb7cb3e5..0000000000 --- a/doc/tuto-msg/deployment_general.xml +++ /dev/null @@ -1,9 +0,0 @@ - - - - - - - - - diff --git a/doc/tuto-msg/masterworker-sol1.c b/doc/tuto-msg/masterworker-sol1.c deleted file mode 100644 index 180c61e1b0..0000000000 --- a/doc/tuto-msg/masterworker-sol1.c +++ /dev/null @@ -1,133 +0,0 @@ -/* Copyright (c) 2007-2018. The SimGrid Team. All rights reserved. */ - -/* This program is free software; you can redistribute it and/or modify it - * under the terms of the license (GNU LGPL) which comes with this package. */ - -#include "simgrid/msg.h" - -#include /* sprintf */ - -XBT_LOG_NEW_DEFAULT_CATEGORY(msg_test, "Messages specific for this msg example"); - -#define FINALIZE ((void*)221297) /* a magic number to tell people to stop working */ - -static char* build_channel_name(char* buffer, const char* sender, const char* receiver) -{ - strcpy(buffer, sender); - strcat(buffer, ":"); - strcat(buffer, receiver); - return buffer; -} - -/* forward definitions */ -static int master(int argc, char* argv[]); -static int worker(int argc, char* argv[]); - -static int master(int argc, char* argv[]) -{ - msg_host_t host_self = MSG_host_self(); - const char* master_name = MSG_host_get_name(host_self); - char channel[1024]; - - long number_of_tasks = xbt_str_parse_int(argv[1], "Invalid amount of tasks: %s"); /** - Number of tasks */ - double comp_size = xbt_str_parse_double(argv[2], "Invalid computational size: %s"); /** - Task compute cost */ - double comm_size = xbt_str_parse_double(argv[3], "Invalid communication size: %s"); /** - Task communication size */ - - /* Create the tasks in advance */ - msg_task_t* todo = xbt_new0(msg_task_t, number_of_tasks); - - for (int i = 0; i < number_of_tasks; i++) { - char sprintf_buffer[64]; - sprintf(sprintf_buffer, "Task_%d", i); - todo[i] = MSG_task_create(sprintf_buffer, comp_size, comm_size, NULL); - } - - /* Get the info about the worker processes (directly from SimGrid) */ - int worker_count = MSG_get_host_number(); - msg_host_t* workers = xbt_dynar_to_array(MSG_hosts_as_dynar()); - - for (int i = 0; i < worker_count; i++) - if (host_self == workers[i]) { - workers[i] = workers[worker_count - 1]; - worker_count--; - break; - } - - for (int i = 0; i < worker_count; i++) - MSG_process_create("worker", worker, (void*)master_name, workers[i]); - XBT_INFO("Got %d workers and %ld tasks to process", worker_count, number_of_tasks); - - /* Dispatch the tasks */ - for (int i = 0; i < number_of_tasks; i++) { - build_channel_name(channel, master_name, MSG_host_get_name(workers[i % worker_count])); - - XBT_INFO("Sending '%s' to channel '%s'", todo[i]->name, channel); - - MSG_task_send(todo[i], channel); - XBT_INFO("Sent"); - } - - XBT_INFO("All tasks have been dispatched. Let's tell everybody the computation is over."); - for (int i = 0; i < worker_count; i++) { - msg_task_t finalize = MSG_task_create("finalize", 0, 0, FINALIZE); - MSG_task_send(finalize, build_channel_name(channel, master_name, MSG_host_get_name(workers[i % worker_count]))); - } - - XBT_INFO("Goodbye now!"); - free(workers); - free(todo); - return 0; -} /* end_of_master */ - -/** Receiver function */ -static int worker(int argc, char* argv[]) -{ - char channel[1024]; - - build_channel_name(channel, MSG_process_get_data(MSG_process_self()), MSG_host_get_name(MSG_host_self())); - - XBT_INFO("Receiving on channel '%s'", channel); - - while (1) { - msg_task_t task = NULL; - int res = MSG_task_receive(&(task), channel); - xbt_assert(res == MSG_OK, "MSG_task_receive failed"); - - XBT_INFO("Received '%s'", MSG_task_get_name(task)); - if (!strcmp(MSG_task_get_name(task), "finalize")) { - MSG_task_destroy(task); - break; - } - - XBT_INFO("Processing '%s'", MSG_task_get_name(task)); - MSG_task_execute(task); - XBT_INFO("'%s' done", MSG_task_get_name(task)); - MSG_task_destroy(task); - } - XBT_INFO("I'm done. See you!"); - return 0; -} /* end_of_worker */ - -/** Main function */ -int main(int argc, char* argv[]) -{ - MSG_init(&argc, argv); - xbt_assert(argc > 2, - "Usage: %s platform_file deployment_file\n" - "\tExample: %s msg_platform.xml msg_deployment.xml\n", - argv[0], argv[0]); - - /* Create a simulated platform */ - MSG_create_environment(argv[1]); - - /* Application deployment */ - MSG_function_register("master", master); - MSG_function_register("worker", worker); - MSG_launch_application(argv[2]); - - /* Run the simulation */ - msg_error_t res = MSG_main(); - - XBT_INFO("Simulation time %g", MSG_get_clock()); - return (res != MSG_OK); -} diff --git a/doc/tuto-msg/masterworker-sol2.c b/doc/tuto-msg/masterworker-sol2.c deleted file mode 100644 index 16d44b0446..0000000000 --- a/doc/tuto-msg/masterworker-sol2.c +++ /dev/null @@ -1,131 +0,0 @@ -/* Copyright (c) 2007-2018. The SimGrid Team. All rights reserved. */ - -/* This program is free software; you can redistribute it and/or modify it - * under the terms of the license (GNU LGPL) which comes with this package. */ - -#include "simgrid/msg.h" - -#include /* sprintf */ - -XBT_LOG_NEW_DEFAULT_CATEGORY(msg_test, "Messages specific for this msg example"); - -#define FINALIZE ((void*)221297) /* a magic number to tell people to stop working */ - -static char* build_channel_name(char* buffer, const char* sender, const char* receiver) -{ - strcpy(buffer, sender); - strcat(buffer, ":"); - strcat(buffer, receiver); - return buffer; -} - -/* forward definitions */ -static int master(int argc, char* argv[]); -static int worker(int argc, char* argv[]); - -static int master(int argc, char* argv[]) -{ - msg_host_t host_self = MSG_host_self(); - char* master_name = (char*)MSG_host_get_name(host_self); - char channel[1024]; - - double timeout = xbt_str_parse_double(argv[1], "Invalid timeout: %s"); /** - timeout */ - double comp_size = xbt_str_parse_double(argv[2], "Invalid computational size: %s"); /** - Task compute cost */ - double comm_size = xbt_str_parse_double(argv[3], "Invalid communication size: %s"); /** - Task communication size */ - - /* Get the info about the worker processes (directly from SimGrid) */ - int worker_count = MSG_get_host_number(); - msg_host_t* workers = xbt_dynar_to_array(MSG_hosts_as_dynar()); - - for (int i = 0; i < worker_count; i++) // Remove my host from the list - if (host_self == workers[i]) { - workers[i] = workers[worker_count - 1]; - worker_count--; - break; - } - - for (int i = 0; i < worker_count; i++) - MSG_process_create("worker", worker, (void*)master_name, workers[i]); - XBT_INFO("Got %d workers and will send tasks for %g seconds", worker_count, timeout); - - /* Dispatch the tasks */ - int task_num = 0; - while (1) { - if (MSG_get_clock() > timeout) - break; - - char sprintf_buffer[64]; - sprintf(sprintf_buffer, "Task_%d", task_num); - msg_task_t task = MSG_task_create(sprintf_buffer, comp_size, comm_size, NULL); - - build_channel_name(channel, master_name, MSG_host_get_name(workers[task_num % worker_count])); - - XBT_DEBUG("Sending '%s' to channel '%s'", task->name, channel); - MSG_task_send(task, channel); - XBT_DEBUG("Sent"); - task_num++; - } - - XBT_DEBUG("All tasks have been dispatched. Let's tell everybody the computation is over."); - for (int i = 0; i < worker_count; i++) { - msg_task_t finalize = MSG_task_create("finalize", 0, 0, FINALIZE); - MSG_task_send(finalize, build_channel_name(channel, master_name, MSG_host_get_name(workers[i % worker_count]))); - } - - XBT_DEBUG("Sent %d tasks in total!", task_num); - free(workers); - return 0; -} - -/** Worker function */ -static int worker(int argc, char* argv[]) -{ - char channel[1024]; - - build_channel_name(channel, MSG_process_get_data(MSG_process_self()), MSG_host_get_name(MSG_host_self())); - - XBT_DEBUG("Receiving on channel '%s'", channel); - - while (1) { - msg_task_t task = NULL; - int res = MSG_task_receive(&(task), channel); - xbt_assert(res == MSG_OK, "MSG_task_receive failed"); - - XBT_DEBUG("Received '%s'", MSG_task_get_name(task)); - if (!strcmp(MSG_task_get_name(task), "finalize")) { - MSG_task_destroy(task); - break; - } - - XBT_DEBUG("Processing '%s'", MSG_task_get_name(task)); - MSG_task_execute(task); - XBT_DEBUG("'%s' done", MSG_task_get_name(task)); - MSG_task_destroy(task); - } - XBT_DEBUG("I'm done. See you!"); - return 0; -} - -/** Main function */ -int main(int argc, char* argv[]) -{ - MSG_init(&argc, argv); - xbt_assert(argc > 2, - "Usage: %s platform_file deployment_file\n" - "\tExample: %s msg_platform.xml msg_deployment.xml\n", - argv[0], argv[0]); - - /* Create a simulated platform */ - MSG_create_environment(argv[1]); - - /* Application deployment */ - MSG_function_register("master", master); - MSG_function_register("worker", worker); - MSG_launch_application(argv[2]); - - /* Run the simulation */ - msg_error_t res = MSG_main(); - - XBT_INFO("Simulation time %g", MSG_get_clock()); - return (res != MSG_OK); -} diff --git a/doc/tuto-msg/masterworker-sol3.c b/doc/tuto-msg/masterworker-sol3.c deleted file mode 100644 index 2c187ec62f..0000000000 --- a/doc/tuto-msg/masterworker-sol3.c +++ /dev/null @@ -1,134 +0,0 @@ -/* Copyright (c) 2007-2018. The SimGrid Team. All rights reserved. */ - -/* This program is free software; you can redistribute it and/or modify it - * under the terms of the license (GNU LGPL) which comes with this package. */ - -#include "simgrid/msg.h" - -#include /* sprintf */ - -XBT_LOG_NEW_DEFAULT_CATEGORY(msg_test, "Messages specific for this msg example"); - -#define FINALIZE ((void*)221297) /* a magic number to tell people to stop working */ - -static char* build_channel_name(char* buffer, const char* sender, const char* receiver) -{ - strcpy(buffer, sender); - strcat(buffer, ":"); - strcat(buffer, receiver); - return buffer; -} - -/* forward definitions */ -static int master(int argc, char* argv[]); -static int worker(int argc, char* argv[]); - -static int master(int argc, char* argv[]) -{ - msg_host_t host_self = MSG_host_self(); - char* master_name = (char*)MSG_host_get_name(host_self); - char channel[1024]; - - TRACE_category(master_name); - - double timeout = xbt_str_parse_double(argv[1], "Invalid timeout: %s"); /** - timeout */ - double comp_size = xbt_str_parse_double(argv[2], "Invalid computational size: %s"); /** - Task compute cost */ - double comm_size = xbt_str_parse_double(argv[3], "Invalid communication size: %s"); /** - Task communication size */ - - /* Get the info about the worker processes */ - int worker_count = MSG_get_host_number(); - msg_host_t* workers = xbt_dynar_to_array(MSG_hosts_as_dynar()); - - for (int i = 0; i < worker_count; i++) // Remove my host from the list - if (host_self == workers[i]) { - workers[i] = workers[worker_count - 1]; - worker_count--; - break; - } - - for (int i = 0; i < worker_count; i++) - MSG_process_create("worker", worker, (void*)master_name, workers[i]); - XBT_INFO("Got %d workers and will send tasks for %g seconds", worker_count, timeout); - - /* Dispatch the tasks */ - int task_num = 0; - while (1) { - if (MSG_get_clock() > timeout) - break; - - char sprintf_buffer[64]; - sprintf(sprintf_buffer, "Task_%d", task_num); - - msg_task_t task = MSG_task_create(sprintf_buffer, comp_size, comm_size, NULL); - MSG_task_set_category(task, master_name); - - build_channel_name(channel, master_name, MSG_host_get_name(workers[task_num % worker_count])); - - XBT_DEBUG("Sending '%s' to channel '%s'", task->name, channel); - MSG_task_send(task, channel); - XBT_DEBUG("Sent"); - task_num++; - } - - XBT_DEBUG("All tasks have been dispatched. Let's tell everybody the computation is over."); - for (int i = 0; i < worker_count; i++) { - msg_task_t finalize = MSG_task_create("finalize", 0, 0, FINALIZE); - MSG_task_send(finalize, build_channel_name(channel, master_name, MSG_host_get_name(workers[i % worker_count]))); - } - - XBT_INFO("Sent %d tasks in total!", task_num); - free(workers); - return 0; -} - -/** Worker function */ -static int worker(int argc, char* argv[]) -{ - char channel[1024]; - build_channel_name(channel, MSG_process_get_data(MSG_process_self()), MSG_host_get_name(MSG_host_self())); - - XBT_DEBUG("Receiving on channel '%s'", channel); - - while (1) { - msg_task_t task = NULL; - int res = MSG_task_receive(&(task), channel); - xbt_assert(res == MSG_OK, "MSG_task_get failed"); - - XBT_DEBUG("Received '%s'", MSG_task_get_name(task)); - if (!strcmp(MSG_task_get_name(task), "finalize")) { - MSG_task_destroy(task); - break; - } - - XBT_DEBUG("Processing '%s'", MSG_task_get_name(task)); - MSG_task_execute(task); - XBT_DEBUG("'%s' done", MSG_task_get_name(task)); - MSG_task_destroy(task); - } - XBT_DEBUG("I'm done. See you!"); - return 0; -} - -/** Main function */ -int main(int argc, char* argv[]) -{ - MSG_init(&argc, argv); - xbt_assert(argc > 2, - "Usage: %s platform_file deployment_file\n" - "\tExample: %s msg_platform.xml msg_deployment.xml\n", - argv[0], argv[0]); - - /* Create a simulated platform */ - MSG_create_environment(argv[1]); - - /* Application deployment */ - MSG_function_register("master", master); - MSG_function_register("worker", worker); - MSG_launch_application(argv[2]); - - /* Run the simulation */ - msg_error_t res = MSG_main(); - - XBT_INFO("Simulation time %g", MSG_get_clock()); - return (res != MSG_OK); -} diff --git a/doc/tuto-msg/masterworker-sol4.c b/doc/tuto-msg/masterworker-sol4.c deleted file mode 100644 index 6235ed639f..0000000000 --- a/doc/tuto-msg/masterworker-sol4.c +++ /dev/null @@ -1,158 +0,0 @@ -/* Copyright (c) 2007-2018. The SimGrid Team. All rights reserved. */ - -/* This program is free software; you can redistribute it and/or modify it - * under the terms of the license (GNU LGPL) which comes with this package. */ - -#include "simgrid/msg.h" - -#include /* sprintf */ - -XBT_LOG_NEW_DEFAULT_CATEGORY(msg_test, "Messages specific for this msg example"); - -#define FINALIZE ((void*)221297) /* a magic number to tell people to stop working */ - -static char* build_channel_name(char* buffer, const char* sender, const char* receiver) -{ - strcpy(buffer, sender); - strcat(buffer, ":"); - strcat(buffer, receiver); - return buffer; -} - -/* forward definitions */ -static int master(int argc, char* argv[]); -static int worker(int argc, char* argv[]); - -static int master(int argc, char* argv[]) -{ - msg_host_t host_self = MSG_host_self(); - char* master_name = (char*)MSG_host_get_name(host_self); - char channel[1024]; - - TRACE_category(master_name); - - double timeout = xbt_str_parse_double(argv[1], "Invalid timeout: %s"); /** - timeout */ - double comp_size = xbt_str_parse_double(argv[2], "Invalid computational size: %s"); /** - Task compute cost */ - double comm_size = xbt_str_parse_double(argv[3], "Invalid communication size: %s"); /** - Task communication size */ - - /* Get the info about the worker processes */ - int worker_count = MSG_get_host_number(); - msg_host_t* workers = xbt_dynar_to_array(MSG_hosts_as_dynar()); - - for (int i = 0; i < worker_count; i++) // Remove my host from the list - if (host_self == workers[i]) { - workers[i] = workers[worker_count - 1]; - worker_count--; - break; - } - - for (int i = 0; i < worker_count; i++) - MSG_process_create("worker", worker, (void*)master_name, workers[i]); - XBT_INFO("Got %d workers and will send tasks for %g seconds", worker_count, timeout); - - /* Dispatch the tasks */ - int task_num = 0; - while (MSG_get_clock() < timeout) { - - /* Retrieve the next incomming request */ - XBT_DEBUG("Retrieve the next incomming request on %s", master_name); - msg_task_t request = NULL; - int res = MSG_task_receive(&(request), master_name); - xbt_assert(res == MSG_OK, "MSG_task_receive failed"); - msg_host_t requester = MSG_task_get_data(request); - MSG_task_destroy(request); - - /* Prepare the task to be sent */ - char sprintf_buffer[64]; - sprintf(sprintf_buffer, "Task_%d", task_num); - msg_task_t task = MSG_task_create(sprintf_buffer, comp_size, comm_size, NULL); - MSG_task_set_category(task, master_name); - - /* Send this out */ - build_channel_name(channel, master_name, MSG_host_get_name(requester)); - - XBT_DEBUG("Sending '%s' to channel '%s'", task->name, channel); - MSG_task_send(task, channel); - XBT_DEBUG("Sent"); - task_num++; - } - - XBT_DEBUG("Time is up. Let's tell everybody the computation is over."); - for (int i = 0; i < worker_count; i++) { /* We don't write in order, but the total amount is right */ - - /* Don't write to a worker that did not request for work, or it will deadlock: both would be sending something */ - msg_task_t request = NULL; - int res = MSG_task_receive(&(request), master_name); - xbt_assert(res == MSG_OK, "MSG_task_receive failed"); - msg_host_t requester = MSG_task_get_data(request); - MSG_task_destroy(request); - - XBT_DEBUG("Stop worker %s", MSG_host_get_name(requester)); - msg_task_t finalize = MSG_task_create("finalize", 0, 0, FINALIZE); - MSG_task_send(finalize, build_channel_name(channel, master_name, MSG_host_get_name(requester))); - } - - XBT_INFO("Sent %d tasks in total!", task_num); - free(workers); - return 0; -} - -/** Worker function */ -static int worker(int argc, char* argv[]) -{ - char channel[1024]; - - const char* my_master = MSG_process_get_data(MSG_process_self()); - build_channel_name(channel, my_master, MSG_host_get_name(MSG_host_self())); - - XBT_DEBUG("Receiving on channel \"%s\"", channel); - - while (1) { - /* Send a request */ - XBT_DEBUG("Sent a request to my master on %s", my_master); - msg_task_t request = MSG_task_create("request", 0, 0, MSG_host_self()); - MSG_task_send(request, my_master); - - /* Wait for the answer */ - msg_task_t task = NULL; - int res = MSG_task_receive(&(task), channel); - xbt_assert(res == MSG_OK, "MSG_task_receive failed"); - - XBT_DEBUG("Received '%s'", MSG_task_get_name(task)); - if (!strcmp(MSG_task_get_name(task), "finalize")) { - MSG_task_destroy(task); - break; - } - - XBT_DEBUG("Processing '%s'", MSG_task_get_name(task)); - MSG_task_execute(task); - XBT_DEBUG("'%s' done", MSG_task_get_name(task)); - MSG_task_destroy(task); - } - XBT_DEBUG("I'm done. See you!"); - return 0; -} - -/** Main function */ -int main(int argc, char* argv[]) -{ - MSG_init(&argc, argv); - xbt_assert(argc > 2, - "Usage: %s platform_file deployment_file\n" - "\tExample: %s msg_platform.xml msg_deployment.xml\n", - argv[0], argv[0]); - - /* Create a simulated platform */ - MSG_create_environment(argv[1]); - - /* Application deployment */ - MSG_function_register("master", master); - MSG_function_register("worker", worker); - MSG_launch_application(argv[2]); - - /* Run the simulation */ - msg_error_t res = MSG_main(); - - XBT_INFO("Simulation time %g", MSG_get_clock()); - return (res != MSG_OK); -} diff --git a/doc/tuto-msg/masterworker.c b/doc/tuto-msg/masterworker.c deleted file mode 100644 index 2a2b906354..0000000000 --- a/doc/tuto-msg/masterworker.c +++ /dev/null @@ -1,109 +0,0 @@ -/* Copyright (c) 2007-2018. The SimGrid Team. All rights reserved. */ - -/* This program is free software; you can redistribute it and/or modify it - * under the terms of the license (GNU LGPL) which comes with this package. */ - -#include "simgrid/msg.h" - -#include /* sprintf */ - -XBT_LOG_NEW_DEFAULT_CATEGORY(msg_test, "Messages specific for this msg example"); - -#define FINALIZE ((void*)221297) /* a magic number to tell people to stop working */ - -/** Master expects 3+ arguments given in the XML deployment file: */ -static int master(int argc, char* argv[]) -{ - long number_of_tasks = xbt_str_parse_int(argv[1], "Invalid amount of tasks: %s"); /** - Number of tasks */ - double comp_size = xbt_str_parse_double(argv[2], "Invalid computational size: %s"); /** - Task compute cost */ - double comm_size = xbt_str_parse_double(argv[3], "Invalid communication size: %s"); /** - Task communication size */ - - /* Create the tasks in advance */ - msg_task_t* todo = xbt_new0(msg_task_t, number_of_tasks); - - for (int i = 0; i < number_of_tasks; i++) { - char sprintf_buffer[64]; - sprintf(sprintf_buffer, "Task_%d", i); - todo[i] = MSG_task_create(sprintf_buffer, comp_size, comm_size, NULL); - } - - /* Get the info about the worker processes from my parameters */ - int worker_count = argc - 4; - msg_host_t* workers = xbt_new0(msg_host_t, worker_count); - - for (int i = 4; i < argc; i++) { - workers[i - 4] = MSG_get_host_by_name(argv[i]); - xbt_assert(workers[i - 4] != NULL, "Unknown host %s. Stopping Now! ", argv[i]); - } - XBT_INFO("Got %d workers and %ld tasks to process", worker_count, number_of_tasks); - - /* Dispatch the tasks */ - for (int i = 0; i < number_of_tasks; i++) { - XBT_INFO("Sending '%s' to '%s'", todo[i]->name, MSG_host_get_name(workers[i % worker_count])); - if (MSG_host_self() == workers[i % worker_count]) { - XBT_INFO("Hey ! It's me ! :)"); - } - - MSG_task_send(todo[i], MSG_host_get_name(workers[i % worker_count])); - XBT_INFO("Sent"); - } - - XBT_INFO("All tasks have been dispatched. Let's tell everybody the computation is over."); - for (int i = 0; i < worker_count; i++) { - msg_task_t finalize = MSG_task_create("finalize", 0, 0, FINALIZE); - MSG_task_send(finalize, MSG_host_get_name(workers[i])); - } - - XBT_INFO("Goodbye now!"); - free(workers); - free(todo); - return 0; -} - -/** Worker does not expect any argument from XML deployment file. */ -static int worker(int argc, char* argv[]) -{ - while (1) { - msg_task_t task = NULL; - int res = MSG_task_receive(&(task), MSG_host_get_name(MSG_host_self())); - xbt_assert(res == MSG_OK, "MSG_task_receive failed"); - - XBT_INFO("Received '%s'", MSG_task_get_name(task)); - if (!strcmp(MSG_task_get_name(task), "finalize")) { - MSG_task_destroy(task); - break; - } - - XBT_INFO("Processing \"%s\"", MSG_task_get_name(task)); - MSG_task_execute(task); - XBT_INFO("'%s' done", MSG_task_get_name(task)); - MSG_task_destroy(task); - } - XBT_INFO("I'm done. See you!"); - return 0; -} /* end_of_worker */ - -/** Main function */ -int main(int argc, char* argv[]) -{ - - MSG_init(&argc, argv); - xbt_assert(argc > 2, - "Usage: %s platform_file deployment_file\n" - "\tExample: %s msg_platform.xml msg_deployment.xml\n", - argv[0], argv[0]); - - /* Create a simulated platform */ - MSG_create_environment(argv[1]); - - /* Application deployment */ - MSG_function_register("master", master); - MSG_function_register("worker", worker); - MSG_launch_application(argv[2]); - - /* Run the simulation */ - msg_error_t res = MSG_main(); - - XBT_INFO("Simulation time %g", MSG_get_clock()); - return (res != MSG_OK); -} /* end_of_main */ diff --git a/doc/tuto-msg/overview.svg b/doc/tuto-msg/overview.svg deleted file mode 100644 index c3188764f9..0000000000 --- a/doc/tuto-msg/overview.svg +++ /dev/null @@ -1,1240 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - image/svg+xml - - - - - - - Master - - - - T - - - - T - - - - Worker - - - - Worker - - - - Worker - - - - Worker - - - - Worker - - - - - - - - - How should the masterdistribute the tasks? - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - - T - - - ? - - diff --git a/doc/tuto-msg/tuto-msg.doc b/doc/tuto-msg/tuto-msg.doc deleted file mode 100644 index c0ebbb3e60..0000000000 --- a/doc/tuto-msg/tuto-msg.doc +++ /dev/null @@ -1,428 +0,0 @@ -/*! @page tutorial_msg SimGrid Tutorial with MSG - -@tableofcontents - -@section tuto-msg-intro Introduction - -@subsection tuto-msg-intro-settings Settings - -@warning Before you take this tutorial, you should remember that the -MSG interface is currently deprecated. It means that it will remain as -is, inchanged, for a few years, but that new developments should use -the new S4U interface instead. Unfortunately, the S4U tutorial is not -written yet. Sorry about that. - -This tutorial will guide your create and run your first SimGrid -simulator. Let's consider the following scenario: - -> Assume we have a (possibly large) bunch of (possibly large) data to -> process and which originally reside on a server (a.k.a. master). For -> sake of simplicity, we assume all input file require the same amount -> of computation. We assume the server can be helped by a (possibly -> large) set of worker machines. What is the best way to organize the -> computations ? - -@htmlonly -
-@endhtmlonly -@htmlinclude tuto-msg/overview.svg -@htmlonly -
-@endhtmlonly - -@subsection tuto-msg-intro-questions Raised Questions - -Although this looks like a very simple setting it raises several -interesting questions: - -- Which algorithm should the master use to send workload? - - The provided code sends the tasks to the workers with a trivial - round-robin algorithm. It would probably be more efficient if the - workers were asking for tasks, to let the master distribute the - tasks in a more cleaver way. - -- Should the worker specify how many tasks they want? Or should the - master decide everything? - - The workers will starve if they don't get the tasks fast - enough. One possibility to reduce latency would be to send tasks - in pools instead of one by one. But if the pools are too big, the - load balancing will likely get uneven, in particular when - distributing the last tasks. - -- How does the quality of such algorithm dependent on the platform - characteristics and on the task characteristics? - - Whenever the input communication time is very small compared to - processing time and workers are homogeneous, it is likely that the - round-robin algorithm performs very well. Would it still hold true - when transfer time is not negligible and the platform is, say, a - volunteer computing system ? What if some tasks are performed - faster on some specific nodes? - -- The network topology interconnecting the master and the workers - may be quite complicated. How does such a topology impact the - previous result? - - When data transfers are the bottleneck, it is likely that a good - modeling of the platform becomes essential. The SimGrid platform - models are particularly handy to account for complex platform - topologies. - -- What topology to use for the application? - - Is a flat master worker deployment sufficient? Should we go for a - hierarchical algorithm, with some forwarders taking large pools of - tasks from the master, each of them distributing their tasks to a - sub-pool of workers? Or should we introduce super-peers, - dupplicating the master's role in a peer-to-peer manner? Do the - algorithms require a perfect knowledge of the network? - -- How is such an algorithm sensitive to external workload variation? - - What if bandwidth, latency and computing speed can vary with no - warning? Shouldn't you study whether your algorithm is sensitive - to such load variations? - -- Although an algorithm may be more efficient than another, how - does it interfere with other applications? - - -- Etc, etc. - -As you can see, this very simple setting may need to evolve way beyond -what you initially imagined. And this is a good news. - -But don't believe the fools saying that all you need to study such -settings is a simple discrete event simulator. Do you really want to -reinvent the wheel, write your own tool, debug it, optimize it and -validate its models against real settings for ages, or do you prefer -to sit on the shoulders of a giant?
-With SimGrid, you can forget about most technical details (but not -all), and focus on your algorithm. The whole simulation mechanism is -already working. - -@subsection tuto-msg-intro-goal Envisionned Study - - -The following figure is a screenshot of [triva][fn:1] visualizing a [SimGrid -simulation][fn:2] of two master worker applications (one in light gray and -the other in dark gray) running in concurrence and showing resource -usage over a long period of time. - -![Test](./sc3-description.png) - -@section tuto-msg-starting Getting Started - -@subsection tuto-msg-prerequesite Prerequisite - -In this example, we use Pajeng and Vite to visualize the result of -SimGrid simulations. These external tools are usually very easy to -install. On Debian and Ubuntu for example, you can get them as follows: - -~~~~{.sh} -sudo apt-get install pajeng vite -~~~~ - -@subsection tuto-msg-setup Setting up and Compiling - -The corresponding source files can be obtained -[online on GitLab](https://gitlab.inria.fr/simgrid/simgrid/tree/master/doc/tuto-msg/src). -There is a button on the top right to download the whole -directory in one archive file. If you wish, other platform files are available from -[this GitLab directory](https://gitlab.inria.fr/simgrid/simgrid/tree/master/examples/platforms). - -As you can see, there is already a little Makefile that compiles -everything for you. If you struggle with the compilation, then you should double check -your @ref install "SimGrid installation". -On need, please refer to the @ref install_yours_trouble section. - -@section tuto-msg-ex0 Discovering the provided simulator - -Please compile and execute the provided simulator as follows: - -~~~~{.sh} -make masterworker -./masterworker examples/platforms/small_platform.xml deployment0.xml -~~~~ - -For a more "fancy" output, you can use simgrid-colorizer. - -~~~~{.sh} -./masterworker examples/platforms/small_platform.xml deployment0.xml 2>&1 | simgrid-colorizer -~~~~ - -If you installed SimGrid to a non-standard path, you may have to -specify the full path to simgrid-colorizer on the above line, such as -@c /opt/simgrid/bin/simgrid-colorizer. If you did not install it at all, -you can find it in @/bin/colorize. - -For a classical Gantt-Chart visualization, you can produce a [Paje][fn:5] trace: - -~~~~{.sh} -./masterworker platforms/platform.xml deployment0.xml --cfg=tracing:yes \ - --cfg=tracing/msg/process:yes -pajeng simgrid.trace -~~~~ - -Alternatively, you can use [vite][fn:6]. - -~~~~{.sh} -./masterworker platforms/platform.xml deployment0.xml --cfg=tracing:yes \ - --cfg=tracing/msg/process:yes --cfg=tracing/basic:yes -vite simgrid.trace -~~~~ - -@subsection tuto-msg-exo0-source Understanding this source code - -Explore the @ref doc/tuto-msg/masterworker.c source file. It contains 3 functions: - - @c master: that's the code executed by the master process.
- It creates a large array containing all tasks, - dispatches all tasks to the workers and then dispatch - specific tasks which name is "finalize". - - @c worker: each workers will execute this function.
- That's an infinite loop waiting for incomming tasks. - We exit the loop if the name of the received task is "finalize", or process the task otherwise. - - @c main: this setups the simulation. - -How does SimGrid know that we need one master and several workers? -Because it's written in the deployment file (called @c -deployment0.xml), that we pass to MSG_create_environment() during the setup. - -@include doc/tuto-msg/deployment0.xml - -@section tuto-msg-exo1 Exercise 1: Simplifying the deployment file - -In the provided example, the deployment file `deployment0.xml` is -tightly connected to the platform file `small_platform.xml` and adding -more workers quickly becomes a pain: You need to start them (at the -bottom of the file), add to inform the master that they are available -(in the master parameters list). - -Instead, modify the simulator `masterworker.c` into `masterworker-exo1.c` -so that the master launches a worker process on all the other machines -at startup. The new deployment file `deployment1.xml` should be as -simple as: - -@include doc/tuto-msg/deployment1.xml - -For that, the master needs to retrieve the list of hosts declared in -the platform, with the following functions (follow the links for their -documentation): - -~~~~{.c} -int MSG_get_host_number(void); -xbt_dynar_t MSG_hosts_as_dynar(void); -void * xbt_dynar_to_array (xbt_dynar_t dynar); -~~~~ - -Then, the master should start the worker processes with the following function: - -~~~~{.c} -msg_process_t MSG_process_create(const char *name, xbt_main_func_t code, void *data, msg_host_t host); -~~~~ - -@subsection tuto-msg-exo1-config Increasing configurability - -The worker processes wait for incomming messages on a channel which -name they need to know beforehand. In the provided code, each worker -uses the name of its host as a channel name. You can see this in the -receiver source code: - -~~~~{.c} - int res = MSG_task_receive(&(task), MSG_host_get_name(MSG_host_self())); - xbt_assert(res == MSG_OK, "MSG_task_receive failed"); -~~~~ - -This way, you can have at most one worker per host. To later study the -behavior of concurrent applications on the platform, we need to -alleviate this. Several solutions exist: - -Now that the the master creates the workers, it knows their PID -(process ID -- given by @ref MSG_process_get_PID()), so you could use -it in the channel name. - -Another possibility for the master is to determine a channel name -before the process creation, and give that name as a parameter to the -starting process. This is what the `data` parameter of @ref -MSG_process_create is meant for. You can pass any arbitrary pointer, -and the created process can retrieve this value later with the @ref -MSG_process_get_data and @ref MSG_process_self functions. Since we -want later to study concurrent applications, it is advised to use a -channel name such as `master_name:worker_name`. - -A third possibility would be to inverse the communication architecture -and have the workers pulling work from the master. This require to -pass the master's channel to the workers. - -@subsection tuto-msg-exo1-wrapup Wrap up - -In this exercise, we reduced the amount of configuration that our -simulator requests. This is both a good idea, and a dangerous -trend. This simplification is an application of the good old DRY/SPOT -programming principle (Don't Repeat Yourself / Single Point Of Truth --- more on wikipedia), -and you really want your programming artefacts to follow these software engineering principles. - -But at the same time, you should be careful in separating your -scientific contribution (the master/wokers algorithm) and the -artefacts used to test it (platform, deployment and workload). This is -why SimGrid forces you to expres your platform and deployment files in -XML instead of using a programming interface: it forces a clear -separation of concerns between things that are of very different -nature. - -If you struggle with this exercise, have a look at -our solution in @ref doc/tuto-msg/masterworker-sol1.c -This is not perfect at all, and many other solutions would have been possible, of course. - -@section tuto-msg-exo2 Exercise 2: Infinite amount of work, fixed experiment duration - -In the current version, the number of tasks is defined through the -worker arguments. Hence, tasks are created at the very beginning of -the simulation. Instead, have the master dispatching tasks for a -predetermined amount of time. The tasks must now be created on demand -instead of beforehand. - -Of course, usual time functions like `gettimeofday` will give you the -time on your real machine, which is prety useless in the -simulation. Instead, retrieve the time in the simulated world with -@ref MSG_get_clock. - -You can still stop your workers with a specific task as previously, -but other methods exist. You can forcefully stop processes with the -following functions, but be warned that SimGrid traditionnally had -issues with forcefully stopping procsses involved in computations or -communications. We hope that it's better now, but YMMV. - -~~~~{.c} -void MSG_process_kill(msg_process_t process); -int MSG_process_killall(int reset_PIDs); -~~~~ - -Anyway, the new deployment `deployment2.xml` file should thus look -like this: - -@include doc/tuto-msg/deployment2.xml - -@subsection tuto-msg-exo2-verbosity Controlling the message verbosity - -Not all messages are equally informative, so you probably want to -change most of the `XBT_INFO` into `XBT_DEBUG` so that they are hidden -by default. You could for example show only the total number of tasks -processed by default. You can still see the debug messages as follows: - -~~~~{.sh} -./masterworker examples/platforms/small_platform.xml deployment2.xml --log=msg_test.thres:debug -~~~~ - -@subsection tuto-msg-exo2-wrapup Wrap up - -Our imperfect solution to this exercise is available as @ref doc/tuto-msg/masterworker-sol2.c -But there is still much to improve in that code. - -@section tuto-msg-exo3 Exercise 3: Understanding how competing applications behave - -It is now time to start several applications at once, with the following `deployment3.xml` file. - -@include doc/tuto-msg/deployment3.xml - -Things happen when you do so, but it remains utterly difficult to -understand what's happening exactely. Even visualizations with pajeng -and Vite contain too much information to be useful: it is impossible -to understand which task belong to which application. To fix this, we -will categorize the tasks. - -For that, first let each master create its own category of tasks with -@ref TRACE_category(), and then assign this category to each task using -@ref MSG_task_set_category(). - -The outcome can then be visualized as a Gantt-chart as follows: - -~~~~{.sh} -./masterworker examples/platforms/small_platform.xml deployment3.xml --cfg=tracing:yes --cfg=tracing/msg/process:yes -vite simgrid.trace -~~~~ - -@subsection tuto-msg-exo3-further Going further - -vite is not enough to understand the situation, because it does not -deal with categorization. That is why you should switch to R to -visualize your outcomes, as explained on this -page. - -As usual, you can explore our imperfect solution, in @ref doc/tuto-msg/masterworker-sol3.c. - -@section tuto-msg-exo4 Exercise 4: Better scheduling: FCFS - -You don't need a very advanced visualization solution to notice that -round-robin is completely suboptimal: most of the workers keep waiting -for more work. We will move to a First-Come First-Served mechanism -instead. - -For that, your workers should explicitely request for work with a -message sent to a channel that is specific to their master. The name -of their private channel name should be attached (using the last -parameter of @ref MSG_task_create()) to the message sent, so that -their master can answer. - -The master should serve requests in a round-robin manner, until the -time is up. Things get a bit more complex to stop the workers -afterward: the master cannot simply send a terminating task, as the -workers are blocked until their request for work is accepted. So -instead, the master should wait for incomming requests even once the -time is up, and answer with a terminating task. - -Once it works, you will see that such as simple FCFS schema allows to -double the amount of tasks handled over time in this case. - -@subsection tuto-msg-exo4-further Going further - -From this, many things can easily be added. For example, you could: -- Allow workers to have several pending requests so as to overlap - communication and computations as much as possible. Non-blocking communication will probably become handy here. -- Add a performance measurement mechanism, enabling the master to make smart scheduling choices. -- Test your code on other platforms, from the `examples/platforms` directory in your archive.
- What is the largest number of tasks requiring 50e6 flops and 1e5 - bytes that you manage to distribute and process in one hour on - `g5k.xml` (you should use `deployment_general.xml`)? -- Optimize not only for the amount of tasks handled, but also for the total energy dissipated. -- And so on. If you come up with a really nice extension, please share it with us so that we can extend this tutorial. - -@section tuto-msg-further Where to go from here? - -This tutorial is now terminated. You could keep reading the [online documentation][fn:4] or -[tutorials][fn:7], or you could head up to the example section to read some code. - -@subsection tuto-msg-further-todo TODO: Points to improve for the next time - -- Propose equivalent exercises and skeleton in java. -- Propose a virtualbox image with everything (simgrid, pajeng, ...) already set - up. -- Ease the installation on mac OS X (binary installer) and - windows. -- Explain that programming in C or java and having a working - development environment is a prerequisite. - - -[fn:1]: http://triva.gforge.inria.fr/index.html -[fn:2]: http://hal.inria.fr/inria-00529569 -[fn:3]: http://hal.inria.fr/hal-00738321 -[fn:4]: http://simgrid.gforge.inria.fr/simgrid/latest/doc/ -[fn:5]: https://github.com/schnorr/pajeng/ -[fn:6]: http://vite.gforge.inria.fr/ -[fn:7]: http://simgrid.org/tutorials/ - - -*/ - - -/** - * @example doc/tuto-msg/masterworker.c - * @example doc/tuto-msg/masterworker-sol1.c - * @example doc/tuto-msg/masterworker-sol2.c - * @example doc/tuto-msg/masterworker-sol3.c - * @example doc/tuto-msg/masterworker-sol4.c - */ diff --git a/docs/source/application.rst b/docs/source/application.rst index fc1eb63136..7b8a9b7271 100644 --- a/docs/source/application.rst +++ b/docs/source/application.rst @@ -21,6 +21,10 @@ be either an existing MPI program (if you use the SMPI interface), or a program specifically written to execute within SimGrid, using one of the dedicated APIs. +.. raw:: html + +
+ .. _S4U_doc: The S4U Interface @@ -57,9 +61,9 @@ synchronization mechanisms** such as |Barrier|_, |Semaphore|_, |Mutex|_ and |ConditionVariable|_. Each actor is located on a simulated |Host|_. Each host is located -itself in a |NetZone|_, that route communications through the -links. Each NetZone is included in another one, forming a tree of -NetZones which root zone contains the whole platform. +itself in a |NetZone|_, that knows the networking path between one +resource to another. Each NetZone is included in another one, forming +a tree of NetZones which root zone contains the whole platform. The :ref:`simgrid::s4u::this_actor ` namespace provides many helper @@ -95,7 +99,7 @@ functions to simplify the code of actors. - :ref:`class s4u::Exec ` Computation activity, started on Host and consuming CPU resources. - :ref:`class s4u::Io ` - I/O activities, started on and consumming Storages. + I/O activity, started on and consumming Storages. - **Synchronization Mechanisms**: Classical IPC that actors can use diff --git a/docs/source/models.rst b/docs/source/models.rst index a180643175..ef590e673d 100644 --- a/docs/source/models.rst +++ b/docs/source/models.rst @@ -1,4 +1,11 @@ .. _models: The SimGrid Models -=================== +================== + +.. todo:: + + - Main existing models (contention, cste, LM07) + - Main concepts (Routing, LMM) + link to the papers + - How to switch on the command line + diff --git a/docs/source/scenario.rst b/docs/source/scenario.rst index defdec29cc..43d927a970 100644 --- a/docs/source/scenario.rst +++ b/docs/source/scenario.rst @@ -3,4 +3,16 @@ Describing the Experimental Scenario ************************************ +.. todo:: + + Main concepts: + + - Deployment file + - Availability and state profiles + - Reproducible random number generation + - Command line options, in particular on the model switching + .. include:: scenar_config.rst + + + diff --git a/tools/cmake/DefinePackages.cmake b/tools/cmake/DefinePackages.cmake index bd9de442b3..68ecbc594e 100644 --- a/tools/cmake/DefinePackages.cmake +++ b/tools/cmake/DefinePackages.cmake @@ -878,7 +878,6 @@ set(DOC_SOURCES doc/doxygen/community.doc doc/doxygen/deployment.doc doc/doxygen/footer.html - doc/doxygen/getting_started.doc doc/doxygen/header.html doc/doxygen/howtos.doc doc/doxygen/index.doc @@ -891,10 +890,7 @@ set(DOC_SOURCES doc/doxygen/install.doc doc/doxygen/install_yours.doc doc/doxygen/java.doc - doc/tuto-msg/tuto-msg.doc - doc/tuto-msg/overview.svg doc/doxygen/tutorial_smpi.doc - doc/doxygen/models.doc doc/doxygen/module-msg.doc doc/doxygen/module-s4u.doc doc/doxygen/module-sd.doc @@ -912,7 +908,6 @@ set(DOC_SOURCES doc/doxygen/outcomes_vizu.doc doc/doxygen/platform.doc doc/doxygen/platform_lua.doc - doc/doxygen/scenario.doc doc/doxygen/stylesheet.css doc/doxygen/uhood.doc doc/doxygen/uhood_switch.doc @@ -925,18 +920,6 @@ set(DOC_SOURCES doc/manpage/smpirun.1 doc/manpage/tesh.pod - doc/tuto-msg/deployment0.xml - doc/tuto-msg/deployment1.xml - doc/tuto-msg/deployment2.xml - doc/tuto-msg/deployment3.xml - doc/tuto-msg/deployment_general.xml - doc/tuto-msg/Makefile - doc/tuto-msg/masterworker.c - doc/tuto-msg/masterworker-sol1.c - doc/tuto-msg/masterworker-sol2.c - doc/tuto-msg/masterworker-sol3.c - doc/tuto-msg/masterworker-sol4.c - CITATION.bib )