X-Git-Url: http://info.iut-bm.univ-fcomte.fr/pub/gitweb/simgrid.git/blobdiff_plain/be748c7d49c80a290da4d155e492e3f04f79fbd9..8de7a90b044d2c8cf008e498f533cd7131c9f106:/doc/doxygen/outcomes_vizu.doc
diff --git a/doc/doxygen/outcomes_vizu.doc b/doc/doxygen/outcomes_vizu.doc
new file mode 100644
index 0000000000..2673e2f546
--- /dev/null
+++ b/doc/doxygen/outcomes_vizu.doc
@@ -0,0 +1,602 @@
+/*! \page outcomes_vizu Visualization and Statistical Analysis
+
+SimGrid comes with an extensive support to trace and register what
+happens during the simulation, so that it can be either visualized or
+statistically analysed after the simulation.
+
+This tracing is widely used to observe and understand the behavior of
+parallel applications and distributed algorithms. Usually, this is
+done in a two-step fashion: the user instruments the application and
+the traces are analyzed after the end of the execution. The analysis
+can highlights unexpected behaviors, bottlenecks and sometimes can be
+used to correct distributed algorithms. The SimGrid team has
+instrumented the library in order to let users trace their simulations
+and analyze them. This part of the user manual explains how the
+tracing-related features can be enabled and used during the
+development of simulators using the SimGrid library.
+
+\section instr_category_functions Tracing categories functions
+
+The SimGrid library is instrumented so users can trace the platform
+utilization using MSG, SimDAG and SMPI interfaces. It registers how
+much power is used for each host and how much bandwidth is used for
+each link of the platform. The idea with this type of tracing is to
+observe the overall view of resources utilization in the first place,
+especially the identification of bottlenecks, load-balancing among
+hosts, and so on.
+
+Another possibility is to trace resource utilization by
+categories. Categorized resource utilization tracing gives SimGrid
+users to possibility to classify MSG and SimDAG tasks by category,
+tracing resource utilization for each of the categories. The functions
+below let the user declare a category and apply it to tasks. The
+tasks that are not classified according to a category are not
+traced. Even if the user does not specify any category, the
+simulations can still be traced in terms of resource utilization by
+using a special parameter that is detailed below (see section \ref
+tracing_tracing_options).
+
+\li \c TRACE_category(const char *category)
+\li \c TRACE_category_with_color(const char *category, const char *color)
+\li \c MSG_task_set_category(msg_task_t task, const char *category)
+\li \c MSG_task_get_category(msg_task_t task)
+\li \c SD_task_set_category(SD_task_t task, const char *category)
+\li \c SD_task_get_category(SD_task_t task)
+
+\section instr_mark_functions Tracing marks functions
+\li \c TRACE_declare_mark(const char *mark_type)
+\li \c TRACE_mark(const char *mark_type, const char *mark_value)
+
+\section instr_uservariables_functions Tracing user variables functions
+
+For hosts:
+
+\li \c TRACE_host_variable_declare(const char *variable)
+\li \c TRACE_host_variable_declare_with_color(const char *variable, const char *color)
+\li \c TRACE_host_variable_set(const char *host, const char *variable, double value)
+\li \c TRACE_host_variable_add(const char *host, const char *variable, double value)
+\li \c TRACE_host_variable_sub(const char *host, const char *variable, double value)
+\li \c TRACE_host_variable_set_with_time(double time, const char *host, const char *variable, double value)
+\li \c TRACE_host_variable_add_with_time(double time, const char *host, const char *variable, double value)
+\li \c TRACE_host_variable_sub_with_time(double time, const char *host, const char *variable, double value)
+
+For links:
+
+\li \c TRACE_link_variable_declare(const char *variable)
+\li \c TRACE_link_variable_declare_with_color(const char *variable, const char *color)
+\li \c TRACE_link_variable_set(const char *link, const char *variable, double value)
+\li \c TRACE_link_variable_add(const char *link, const char *variable, double value)
+\li \c TRACE_link_variable_sub(const char *link, const char *variable, double value)
+\li \c TRACE_link_variable_set_with_time(double time, const char *link, const char *variable, double value)
+\li \c TRACE_link_variable_add_with_time(double time, const char *link, const char *variable, double value)
+\li \c TRACE_link_variable_sub_with_time(double time, const char *link, const char *variable, double value)
+
+For links, but use source and destination to get route:
+
+\li \c TRACE_link_srcdst_variable_set(const char *src, const char *dst, const char *variable, double value)
+\li \c TRACE_link_srcdst_variable_add(const char *src, const char *dst, const char *variable, double value)
+\li \c TRACE_link_srcdst_variable_sub(const char *src, const char *dst, const char *variable, double value)
+\li \c TRACE_link_srcdst_variable_set_with_time(double time, const char *src, const char *dst, const char *variable, double value)
+\li \c TRACE_link_srcdst_variable_add_with_time(double time, const char *src, const char *dst, const char *variable, double value)
+\li \c TRACE_link_srcdst_variable_sub_with_time(double time, const char *src, const char *dst, const char *variable, double value)
+
+\section tracing_tracing_options Tracing configuration Options
+
+To check which tracing options are available for your simulator, you
+can just run it with the option \verbatim --help-tracing \endverbatim
+to get a very detailed and updated explanation of each tracing
+parameter. These are some of the options accepted by the tracing
+system of SimGrid, you can use them by running your simulator with the
+--cfg= switch:
+
+\li \c
+tracing
+:
+ Safe switch. It activates (or deactivates) the tracing system.
+ No other tracing options take effect if this one is not activated.
+\verbatim
+--cfg=tracing:yes
+\endverbatim
+
+\li \c
+tracing/categorized
+:
+ It activates the categorized resource utilization tracing. It should
+ be enabled if tracing categories are used by this simulator.
+\verbatim
+--cfg=tracing/categorized:yes
+\endverbatim
+
+\li \c
+tracing/uncategorized
+:
+ It activates the uncategorized resource utilization tracing. Use it if
+ this simulator do not use tracing categories and resource use have to be
+ traced.
+\verbatim
+--cfg=tracing/uncategorized:yes
+\endverbatim
+
+\li \c
+tracing/filename
+:
+ A file with this name will be created to register the simulation. The file
+ is in the Paje format and can be analyzed using Viva or Paje visualization
+ tools. More information can be found in these webpages:
+ http://github.com/schnorr/viva/
+ http://github.com/schnorr/pajeng/
+\verbatim
+--cfg=tracing/filename:mytracefile.trace
+\endverbatim
+ If you do not provide this parameter, the trace file will be named simgrid.trace.
+
+\li \c
+tracing/smpi
+:
+ This option only has effect if this simulator is SMPI-based. Traces the MPI
+ interface and generates a trace that can be analyzed using Gantt-like
+ visualizations. Every MPI function (implemented by SMPI) is transformed in a
+ state, and point-to-point communications can be analyzed with arrows.
+\verbatim
+--cfg=tracing/smpi:yes
+\endverbatim
+
+\li \c
+tracing/smpi/group
+:
+ This option only has effect if this simulator is SMPI-based. The processes
+ are grouped by the hosts where they were executed.
+\verbatim
+--cfg=tracing/smpi/group:yes
+\endverbatim
+
+\li \c
+tracing/smpi/computing
+:
+ This option only has effect if this simulator is SMPI-based. The parts external
+to SMPI are also outputted to the trace. Provides better way to analyze the data automatically.
+\verbatim
+--cfg=tracing/smpi/computing:yes
+\endverbatim
+
+\li \c
+tracing/smpi/internals
+:
+ This option only has effect if this simulator is SMPI-based. Display internal communications
+happening during a collective MPI call.
+\verbatim
+--cfg=tracing/smpi/internals:yes
+\endverbatim
+
+\li \c
+tracing/smpi/display-sizes
+:
+ This option only has effect if this simulator is SMPI-based. Display the sizes of the messages
+exchanged in the trace, both in the links and on the states. For collective, size means the global size of data sent by the process in general.
+\verbatim
+--cfg=tracing/smpi/display-sizes:yes
+\endverbatim
+
+\li \c
+tracing/smpi/sleeping
+:
+TODO
+\verbatim
+TODO
+\endverbatim
+
+\li \c
+tracing/smpi/format
+:
+TODO
+\verbatim
+TODO
+\endverbatim
+
+\li \c
+tracing/smpi/format/ti-one-file
+:
+TODO
+\verbatim
+TODO
+\endverbatim
+
+\li \c
+tracing/msg/vm
+:
+TODO
+\verbatim
+TODO
+\endverbatim
+
+\li \c
+tracing/msg/process
+:
+ This option only has effect if this simulator is MSG-based. It traces the
+ behavior of all categorized MSG processes, grouping them by hosts. This option
+ can be used to track process location if this simulator has process migration.
+\verbatim
+--cfg=tracing/msg/process:yes
+\endverbatim
+
+\li \c
+tracing/buffer
+:
+ This option put some events in a time-ordered buffer using the
+ insertion sort algorithm. The process of acquiring and releasing
+ locks to access this buffer and the cost of the sorting algorithm
+ make this process slow. The simulator performance can be severely
+ impacted if this option is activated, but you are sure to get a trace
+ file with events sorted.
+\verbatim
+--cfg=tracing/buffer:yes
+\endverbatim
+
+\li \c
+tracing/onelink-only
+:
+This option changes the way SimGrid register its platform on the trace
+file. Normally, the tracing considers all routes (no matter their
+size) on the platform file to re-create the resource topology. If this
+option is activated, only the routes with one link are used to
+register the topology within an AS. Routes among AS continue to be
+traced as usual.
+\verbatim
+--cfg=tracing/onelink-only:yes
+\endverbatim
+
+\li \c
+tracing/disable-link
+:
+TODO
+\verbatim
+TODO
+\endverbatim
+
+\li \c
+tracing/disable-power
+:
+TODO
+\verbatim
+TODO
+\endverbatim
+
+\li \c
+tracing/disable-destroy
+:
+Disable the destruction of containers at the end of simulation. This
+can be used with simulators that have a different notion of time
+(different from the simulated time).
+\verbatim
+--cfg=tracing/disable-destroy:yes
+\endverbatim
+
+\li \c
+tracing/basic
+:
+Some visualization tools are not able to parse correctly the Paje file format.
+Use this option if you are using one of these tools to visualize the simulation
+trace. Keep in mind that the trace might be incomplete, without all the
+information that would be registered otherwise.
+\verbatim
+--cfg=tracing/basic:yes
+\endverbatim
+
+\li \c
+tracing/comment
+:
+Use this to add a comment line to the top of the trace file.
+\verbatim
+--cfg=tracing/comment:my_string
+\endverbatim
+
+\li \c
+tracing/comment-file
+:
+Use this to add the contents of a file to the top of the trace file as comment.
+\verbatim
+--cfg=tracing/comment-file:textual_file.txt
+\endverbatim
+
+\li \c
+tracing/precision
+:
+TODO
+\verbatim
+TODO
+\endverbatim
+
+\li \c
+tracing/platform
+:
+TODO
+\verbatim
+TODO
+\endverbatim
+
+\li \c
+tracing/platform/topology
+:
+TODO
+\verbatim
+TODO
+\endverbatim
+
+\li \c
+viva/categorized
+:
+ This option generates a graph configuration file for Viva considering
+ categorized resource utilization.
+\verbatim
+--cfg=viva/categorized:graph_categorized.plist
+\endverbatim
+
+\li \c
+viva/uncategorized
+:
+ This option generates a graph configuration file for Viva considering
+ uncategorized resource utilization.
+\verbatim
+--cfg=viva/uncategorized:graph_uncategorized.plist
+\endverbatim
+
+Please pass \verbatim --help-tracing \endverbatim to your simulator
+for the updated list of tracing options.
+
+\section tracing_tracing_example_parameters Case studies
+
+Some scenarios that might help you decide which tracing options
+you should use to analyze your simulator.
+
+\li I want to trace the resource utilization of all hosts
+and links of the platform, and my simulator does not use
+the tracing API. For that, you can run a uncategorized trace
+with the following parameters (it will work with any Simgrid
+simulator):
+\verbatim
+./your_simulator \
+ --cfg=tracing:yes \
+ --cfg=tracing/uncategorized:yes \
+ --cfg=tracing/filename:mytracefile.trace \
+ --cfg=viva/uncategorized:uncat.plist
+\endverbatim
+
+\li I want to trace only a subset of my MSG (or SimDAG) tasks.
+For that, you will need to create tracing categories using the
+TRACE_category (...) function (as explained above),
+and then classify your tasks to a previously declared category
+using the MSG_task_set_category (...)
+(or SD_task_set_category (...) for SimDAG tasks). After
+recompiling, run your simulator with the following parameters:
+\verbatim
+./your_simulator \
+ --cfg=tracing:yes \
+ --cfg=tracing/categorized:yes \
+ --cfg=tracing/filename:mytracefile.trace \
+ --cfg=viva/categorized:cat.plist
+\endverbatim
+
+
+\section tracing_tracing_example Example of Instrumentation
+
+A simplified example using the tracing mandatory functions.
+
+\verbatim
+int main (int argc, char **argv)
+{
+ MSG_init (&argc, &argv);
+
+ //(... after deployment ...)
+
+ //note that category declaration must be called after MSG_create_environment
+ TRACE_category_with_color ("request", "1 0 0");
+ TRACE_category_with_color ("computation", "0.3 1 0.4");
+ TRACE_category ("finalize");
+
+ msg_task_t req1 = MSG_task_create("1st_request_task", 10, 10, NULL);
+ msg_task_t req2 = MSG_task_create("2nd_request_task", 10, 10, NULL);
+ msg_task_t req3 = MSG_task_create("3rd_request_task", 10, 10, NULL);
+ msg_task_t req4 = MSG_task_create("4th_request_task", 10, 10, NULL);
+ MSG_task_set_category (req1, "request");
+ MSG_task_set_category (req2, "request");
+ MSG_task_set_category (req3, "request");
+ MSG_task_set_category (req4, "request");
+
+ msg_task_t comp = MSG_task_create ("comp_task", 100, 100, NULL);
+ MSG_task_set_category (comp, "computation");
+
+ msg_task_t finalize = MSG_task_create ("finalize", 0, 0, NULL);
+ MSG_task_set_category (finalize, "finalize");
+
+ //(...)
+
+ MSG_clean();
+ return 0;
+}
+\endverbatim
+
+\section tracing_tracing_analyzing Analyzing SimGrid Simulation Traces
+
+A SimGrid-based simulator, when executed with the correct parameters
+(see above) creates a trace file in the Paje file format holding the
+simulated behavior of the application or the platform. You have
+several options to analyze this trace file:
+
+- Dump its contents to a CSV-like format using `pj_dump` (see PajeNG's wiki
+ on pj_dump and more generally the PajeNG suite) and use
+ gnuplot to plot resource usage, time spent on blocking/executing
+ functions, and so on. Filtering capabilities are at your hand by
+ doing `grep`, with the best regular expression you can provide, to
+ get only parts of the trace (for instance, only a subset of
+ resources or processes).
+
+- Derive statistics from trace metrics (the ones built-in with any
+ SimGrid simulation, but also those metrics you injected in the trace
+ using the TRACE module) using the R project and all its
+ modules. You can also combine R with ggplot2 to get a number of high
+ quality plots from your simulation metrics. You need to `pj_dump`
+ the contents of the SimGrid trace file to use R.
+
+- Visualize the behavior of your simulation using classic space/time
+ views (gantt-charts) provided by the PajeNG suite and any
+ other tool that supports the Paje
+ file format. Consider this option if you need to understand the
+ causality of your distributed simulation.
+
+- Visualize the behavior of your simulation with treemaps (specially
+ if your simulation has a platform with several thousand resources),
+ provided by the Viva
+ visualization tool. See Viva's wiki for
+ further details on what is a treemap and how to use it.
+
+- Correlate the behavior of your simulator with the platform topology
+ with an interactive, force-directed, and hierarchical graph
+ visualization, provided by Viva. Check Viva's wiki for
+ further details. This research report,
+ published at ISPASS 2013, has a detailed description of this
+ visualization technique.
+
+- You can also check our online tutorial
+ section that contains a dedicated tutorial with several
+ suggestions on how to use the tracing infrastructure. Look for the
+ SimGrid User::Visualization 101 tutorial.
+
+- Ask for help on the simgrid-user@lists.gforge.inria.fr
+ mailing list, giving us a detailed explanation on what your
+ simulator does and what kind of information you want to trace. You
+ can also check the mailing
+ list archive for old messages regarding tracing and analysis.
+
+\subsection tracing_viva_analysis Viva Visualization Tool
+
+This subsection describe some of the concepts regarding the Viva Visualization Tool and
+its relation with SimGrid traces. You should refer to Viva's website
+for further details on all its visualization techniques.
+
+\subsubsection tracing_viva_time_slice Time Slice
+
+The analysis of a trace file using the tool always takes into account
+the concept of the time-slice. This concept means that what
+is being visualized in the screen is always calculated considering a
+specific time frame, with its beggining and end timestamp. The
+time-slice is configured by the user and can be changed dynamically
+through the window called Time Interval that is opened
+whenever a trace file is being analyzed. Users are capable to select
+the beggining and size of the time slice.
+
+\subsubsection tracing_viva_graph Hierarchical Graph View
+
+%As stated above (see section \ref tracing_tracing_analyzing), one
+possibility to analyze SimGrid traces is to use Viva's graph view with
+a graph configuration to customize the graph according to the
+traces. A valid graph configuration (we are using the non-XML Property List
+Format to describe the configuration) can be created for any
+SimGrid-based simulator using the
+--cfg=viva/uncategorized:graph_uncategorized.plist or
+--cfg=viva/categorized:graph_categorized.plist (if the
+simulator defines resource utilization categories) when executing the
+simulation.
+
+\subsubsection basic_conf Basic Graph Configuration
+
+The basic description of the configuration is as follows:
+\verbatim
+{
+ node = (LINK, HOST, );
+ edge = (HOST-LINK, LINK-HOST, LINK-LINK, );
+\endverbatim
+
+The nodes of the graph will be created based on the node
+parameter, which in this case is the different "HOST"s and
+"LINK"s of the platform used to simulate. The edge
+parameter indicates that the edges of the graph will be created based
+on the "HOST-LINK"s, "LINK-HOST"s, and
+"LINK-LINK"s of the platform. After the definition of these
+two parameters, the configuration must detail how the nodes
+(HOSTs and LINKs) should be drawn.
+
+For that, the configuration must have an entry for each of
+the types used. For HOST, as basic configuration, we have:
+
+\verbatim
+ HOST = {
+ type = square;
+ size = power;
+ values = (power_used);
+ };
+\endverbatim
+
+The parameter size indicates which variable from the trace
+file will be used to define the size of the node HOST in the
+visualization. If the simulation was executed with availability
+traces, the size of the nodes will be changed according to these
+traces. The parameter type indicates which geometrical shape
+will be used to represent HOST, and the values parameter
+indicates which values from the trace will be used to fill the shape.
+
+For LINK we have:
+
+\verbatim
+ LINK = {
+ type = rhombus;
+ size = bandwidth;
+ values = (bandwidth_used);
+ };
+}
+\endverbatim
+
+The same configuration parameters are used here: type (with a
+rhombus), the size (whose value is from trace's bandwidth
+variable) and the values.
+
+\subsubsection custom_graph Customizing the Graph Representation
+
+Viva is capable to handle a customized graph representation based on
+the variables present in the trace file. In the case of SimGrid, every
+time a category is created for tasks, two variables in the trace file
+are defined: one to indicate node utilization (how much power was used
+by that task category), and another to indicate link utilization (how
+much bandwidth was used by that category). For instance, if the user
+declares a category named request, there will be variables
+named prequest and a brequest (p
+for power and b for bandwidth). It is important to notice that
+the variable prequest in this case is only available for HOST,
+and brequest is only available for LINK. Example:
+suppose there are two categories for tasks: request and compute. To
+create a customized graph representation with a proportional
+separation of host and link utilization, use as configuration for HOST
+and LINK this:
+
+\verbatim
+ HOST = {
+ type = square;
+ size = power;
+ values = (prequest, pcomputation);
+ };
+ LINK = {
+ type = rhombus;
+ size = bandwidth;
+ values = (brequest, bcomputation);
+ };
+\endverbatim
+
+This configuration enables the analysis of resource utilization by MSG
+tasks through the identification of load-balancing issues and network
+bottlenecks, for instance.
+
+*/