Update and reorganization.

git-svn-id: svn+ssh://scm.gforge.inria.fr/svn/simgrid/simgrid/trunk@1858 48e7efb5-ca39-0410-a469-dd3cf9ba447f

diff --git a/doc/FAQ.doc b/doc/FAQ.doc
index 9acea65..c18f236 100644 (file)
--- a/doc/FAQ.doc
+++ b/doc/FAQ.doc
@@ -26,7 +26,7 @@ make install
 
 If at some point, something fails, you can report me this problem but,
 please, avoid sending a laconic mail like "There is a problem. Is it
 
 If at some point, something fails, you can report me this problem but,
 please, avoid sending a laconic mail like "There is a problem. Is it

-normal ?". Send me the config.log file which is automatically
+normal?". Send me the config.log file which is automatically

 generated by configure. Try to capture both the standard output and
 the error output of the <tt>make</tt> command. There is no way for me
 to help you if you do not give me a little bit of information.
 generated by configure. Try to capture both the standard output and
 the error output of the <tt>make</tt> command. There is no way for me
 to help you if you do not give me a little bit of information.


@@ -89,9 +89,9 @@ feel free to organize it as you want).
           MSG_launch_application()) and the call to
           MSG_main()).
 
           MSG_launch_application()) and the call to
           MSG_main()).
 

-To compile such a program, I suggest to use the following Makefile. It
-is a generic Makefile that I generally use with my students when I
-teach the C language.
+To compile such a program, we suggest to use the following
+Makefile. It is a generic Makefile that we have used many times with
+our students when we teach the C language.

 
 \verbatim
 all: masterslave 
 
 \verbatim
 all: masterslave 


@@ -141,7 +141,7 @@ in a terminal : <tt>info make</tt> and read the introduction. The
 previous example should be enough for a first try but you may want to
 perform some more complex compilations...
 
 previous example should be enough for a first try but you may want to
 perform some more complex compilations...
 

-\section faq_simgrid I'm new to SimGrid. I have some questions. Where should I start ?
+\section faq_simgrid I'm new to SimGrid. I have some questions. Where should I start?

 
 You are at the right place... Having a look to these
 <a href="http://graal.ens-lyon.fr/~alegrand/articles/Simgrid-Introduction.pdf">slides</a>
 
 You are at the right place... Having a look to these
 <a href="http://graal.ens-lyon.fr/~alegrand/articles/Simgrid-Introduction.pdf">slides</a>


@@ -157,38 +157,6 @@ tasks and distributes them) , some slaves (that process tasks whenever
 they receive one) and some forwarder agents (that simply pass the
 tasks they receive to some slaves).
 
 they receive one) and some forwarder agents (that simply pass the
 tasks they receive to some slaves).
 

-\subsection faq_examples I want some more complex examples !
-
-Many people have come to ask me a more complex example and each time,
-they have realized afterward that the basics were in the previous three
-examples. 
-
-Of course they have often been needing more complex functions like
-MSG_process_suspend(), MSG_process_resume() and
-MSG_process_isSuspended() (to perform synchronization), or
-MSG_task_Iprobe() and MSG_process_sleep() (to avoid blocking
-receptions), or even MSG_process_create() (to design asynchronous
-communications or computations). But the examples are sufficient to
-start.
-
-I know I should add some more examples, but not some more complex
-ones... I should add some examples that illustrate some other
-functionalities (like how to simply encode asynchronous
-communications, RPC, process migrations, thread synchronization, ...)
-and I will do it when I will have a little bit more time. I have tried
-to document the examples so that they are understandable. I know it is
-not really satisfying but it is the best I have managed to do yet.
-
-\subsection faq_platform Building a realistic platform
-
-I can speak more than an hour on this subject and I still do not have
-the right answer, just some ideas. You can read the following
-<a href="http://graal.ens-lyon.fr/~alegrand/articles/Simgrid-Introduction.pdf">slides</a>.
-It may give you some hints. You can also have a look at the
-<tt>tools/platform_generation/</tt> directory. There is a perl-script
-I use to annotate a Tiers generated platform (may not be up-to-date
-though).
-

 \subsection faq_visualization Visualizing the schedule
 
 It is sometime convenient to "see" how the agents are behaving. If you
 \subsection faq_visualization Visualizing the schedule
 
 It is sometime convenient to "see" how the agents are behaving. If you


@@ -208,68 +176,120 @@ generates an input to <a href="http://www-id.imag.fr/Logiciels/paje/">Paje</a>.
 \endhtmlonly
 </center>
 
 \endhtmlonly
 </center>
 

-\subsection faq_context_1000 I want thousands of simulated processes
-
-SimGrid can use either pthreads library or the UNIX98 contextes. On most
-systems, the number of pthreads is limited and then your simulation may be
-limited for a stupid reason. This is especially true with the current linux
-pthreads, and I cannot get more than 2000 simulated processes with pthreads
-on my box. The UNIX98 contextes allow me to raise the limit to 25,000
-simulated processes on my laptop.
-
-The <tt>--with-context</tt> option of the <tt>./configure</tt> script allows
-you to choose between UNIX98 contextes (<tt>--with-context=ucontext</tt>)
-and the pthread version ( (<tt>--with-context=pthread</tt>). The default
-value is ucontext when the script detect a working UNIX98 context
-implementation. On Windows boxes, the provided value is discarded and an
-adapted version is picked up.
-
-We experienced some issues with contextes on some rare systems (solaris 8
-and lower comes to mind). The main problem is that the configure script
-detect the contextes as being functional when it's not true. If you happen
-to use such a system, switch manually to the pthread version, and provide us
-with a good patch for the configure script so that it is done automatically ;)
-
-\subsection faq_context_10000 I want hundred thousands of simulated processes
-
-As explained above, SimGrid can use UNIX98 contextes to represent and handle
-the simulated processes. Thanks to this, the main limitation to the number
-of simulated processes becomes the available memory. 
-
-Here are some tricks I had to use in order to run a token ring between
-25,000 processes on my laptop (1Gb memory, 1.5Gb swap).
-
- - First of all, make sure your code runs for a few hundreds processes
-   before trying to push the limit. Make sure it's valgrind-clean, ie that
-   valgrind does not report neither memory error nor memory leaks. Indeed,
-   numerous simulated processes result in *fat* simulation hindering debugging.
-
- - It was really boring to write 25,000 entries in the deployment file, so I wrote 
-   a little script <tt>examples/gras/tokenS/make_deployment.pl</tt>, which you may
-   want to adapt to your case.
-
- - The deployment file became quite big, so I had to do what is in the FAQ
-   entry \ref faq_flexml_limit
+\subsection faq_postmortem_analysis Online/postmortem analysis
+
+Vizualization with Paje can be seen as a kind of postmortem
+analysis. However, as soon as you start playing with big simulations,
+you'll realize that processing such output is kind of tricky. There is
+so much generic informations that it is hard to find the information
+you are looking for.
+
+As a matter of fact, loging really depends on simulations (e.g. what
+kind of events is important...). That is why we do not propose a big
+dump of your whole simulation (it would slow everything down) but give
+you neat tools to structure you logs. Have a look at \ref XBT_log. In
+fact, rather than a post-mortem analysis, you may want to do it on the
+fly. The process you are running can do whatever you want. Have you
+thought about adding a global structure where you directly compute the
+informations that are really important rather than writing everything
+down and then processing huge files ?
+
+\section faq_questions How to ....? Is there a function in the API to simply ....?
+
+Here is the deal. The whole SimGrid project (MSG, SURF, GRAS, ...) is
+meant to be kept as simple and generic as possible. We cannot add
+functions for everybody's need when these functions can easily be
+built from the ones already in the API. Most of the time, it is
+possible and when it was not possible we always have upgraded the API
+accordingly. When somebody asks us a question like "How to do that ?
+Is there a function in the API to simply do this ?", we're always glad
+to answer and help. However if we don't need this code for our own
+need, there is no chance we're going to write it... it's your job! :)
+The counterpart to our answers is that once you come up with a neat
+implementation of this feature (task duplication, RPC, thread
+synchronization, ...), you should send it to us and we will be glad to
+add it to the distribution. Thus, other people will take advantage of
+it (and we don't have to answer this question again and again ;).
+
+You'll find in this section a few "Missing In Action" features. Many
+people have asked about it and we have given hints on how to simply do
+it with MSG. Feel free to contribute...
+
+\subsection faq_examples I want some more complex examples!

 
 

- - Each UNIX98 context has its own stack entry. As debugging this is quite 
-   hairly, the default value is a bit overestimated so that user don't get 
-   into trouble about this. You want to tune this size to increse the number 
-   of processes. This is the <tt>STACK_SIZE</tt> define in 
-   <tt>src/xbt/context_private.h</tt>, which is 128kb by default.
-   Reduce this as much as you can, but be warned that if this value is too 
-   low, you'll get a segfault. The token ring example, which is quite simple, 
-   runs with 40kb stacks.
+Many people have come to ask me a more complex example and each time,
+they have realized afterward that the basics were in the previous three
+examples. 

 
 

+Of course they have often been needing more complex functions like
+MSG_process_suspend(), MSG_process_resume() and
+MSG_process_isSuspended() (to perform synchronization), or
+MSG_task_Iprobe() and MSG_process_sleep() (to avoid blocking
+receptions), or even MSG_process_create() (to design asynchronous
+communications or computations). But the examples are sufficient to
+start.

 
 

-\section faq_SG Where has SG disappeared ?!?
+We know. We should add some more examples, but not really some more
+complex ones... We should add some examples that illustrate some other
+functionalities (like how to simply encode asynchronous
+communications, RPC, process migrations, thread synchronization, ...)
+and we will do it when we will have a little bit more time. We have
+tried to document the examples so that they are understandable. Tell
+us if something is not clear and once again feel free to participate!
+:)
+
+\subsection faq_examples_MIA_taskdup Missing in action: Task duplication/replication
+
+There is no task duplication in MSG. When you create a task, you can
+process it or send it somewhere else. As soon as a process has sent
+this task, he doesn't have this task anymore. It's gone. The receiver
+process has got the task. However, you could decide upon receiving to
+create a "copy" of a task but you have to handle by yourself the
+semantic associated to this "duplication".
+
+As we already told, we prefer keeping the API as simple as
+possible. This kind of feature is rather easy to implement by users
+and the semantic you associate really depends on people. Having a
+*generic* task duplication mechanism is not that trivial (in
+particular because of the data field). That is why I would recommand
+that you write it by yourself even if I can give you advice on how to
+do it.
+
+You have the following functions to get informations about a task:
+MSG_task_get_name(), MSG_task_get_compute_duration(),
+MSG_task_get_remaining_computation(), MSG_task_get_data_size(),
+and MSG_task_get_data().
+
+You could use a dictionnary (#xbt_dict_t) of dynars (#xbt_dict_t). If
+you still don't see how to do it, please come back to us...
+
+\subsection faq_examples_MIA_asynchronous I want to do asynchronous communications.
+
+Up until now, there is no asynchronous communications in MSG. However,
+you can create as many process as you want so you should be able to do
+whatever you want... I've written a queue module to help implementing
+some asynchronous communications at low cost (creating thousands of
+process only to handle communications may be problematic in term of
+performance at some point). I'll add it in the distribution asap.
+
+\subsection faq_examples_MIA_thread_synchronization I need to synchronize my process
+
+You obviously cannot use pthread_mutexes of pthread_conds. The best
+thing would be to propose similar structures. Unfortunately, we
+haven't found time to do it yet. However you can try to play with
+MSG_process_suspend() and MSG_process_resume(). You can even do some
+synchronization with fake communications (using MSG_task_get(),
+MSG_task_put() and MSG_task_Iprobe()).
+
+\section faq_SG Where has SG disappeared?!?

 
 OK, it's time to explain what's happening to the SimGrid project. Let's
 start with a little bit of history.
 
 * Historically, SimGrid was a low-level toolkit for scheduling with
 
 OK, it's time to explain what's happening to the SimGrid project. Let's
 start with a little bit of history.
 
 * Historically, SimGrid was a low-level toolkit for scheduling with

-classical models such as DAGs. That was SimGrid v.1.* aka SG, written by
-Henri Casanova. I had been using it in its earliest versions during an
-internship at UCSD.
+classical models such as DAGs. That was SimGrid v.1.* aka SG, written
+by Henri Casanova. I (Arnaud) had been using it in its earliest
+versions during an internship at UCSD.

 
 Then we have realized that encoding distributed algorithm in SG was a
 real pain.
 
 Then we have realized that encoding distributed algorithm in SG was a
 real pain.


@@ -327,7 +347,7 @@ speed provided by SURF.
 Those two things are working, but we want to make everything as clean as
 possible before releasing SimGrid v.3.
 
 Those two things are working, but we want to make everything as clean as
 possible before releasing SimGrid v.3.
 

-So what about those nice DAGs we used to have in SimGrid v.1. ? They're not
+So what about those nice DAGs we used to have in SimGrid v.1.? They're not

 anymore in SimGrid v.3. Let me recall you the way SimGrid 3 is organized:
 
 \verbatim
 anymore in SimGrid v.3. Let me recall you the way SimGrid 3 is organized:
 
 \verbatim


@@ -368,7 +388,7 @@ parallel task model, and ... Anyway, we finally have migrated our CVS
 to gforge so people that are interested by helping on this part will
 have the possibility to do it.
 
 to gforge so people that are interested by helping on this part will
 have the possibility to do it.
 

-\subsection faq_SG_DAG But I wanted to implement a distributed dynamic scheduler of DAGs... How can I do that it SG is not available anymore in the next versions ?
+\subsection faq_SG_DAG But I wanted to implement a distributed dynamic scheduler of DAGs... How can I do that if SG is not available anymore in the next versions?

 
 Distributed is somehow "contagious". If you start making distributed
 decisions, there is no way to handle DAGs directly anymore (unless I am
 
 Distributed is somehow "contagious". If you start making distributed
 decisions, there is no way to handle DAGs directly anymore (unless I am


@@ -384,8 +404,18 @@ keep using the 2.18.5 versions until somebody has ported SG on top of SURF.
 Note however that SURF will be slower than the old SG to handle traces with
 a lots of variations (there is no trace integration anymore).
 
 Note however that SURF will be slower than the old SG to handle traces with
 a lots of variations (there is no trace integration anymore).
 

+\section faq_dynamic Dynamic resources and platform building

 
 

-\subsection faq_SURF_dynami How can I have variable resource availability?
+\subsection faq_platform Building a realistic platform
+
+We can speak more than an hour on this subject and we still do not have
+the right answer, just some ideas. You can read the following
+<a href="http://graal.ens-lyon.fr/~alegrand/articles/Simgrid-Introduction.pdf">slides</a>.
+It may give you some hints. You can also have a look at the
+<tt>tools/platform_generation/</tt> directory. There is a perl-script
+we use to annotate a Tiers generated platform.
+
+\subsection faq_SURF_dynamic How can I have variable resource availability?

 
 A nice feature of SimGrid is that it enables you to seamlessly have
 resources whose availability change over time. When you build a
 
 A nice feature of SimGrid is that it enables you to seamlessly have
 resources whose availability change over time. When you build a


@@ -441,7 +471,42 @@ latency_file and state_file. The only difference with CPUs is that
 bandwidth_file and latency_file do not express fraction of available
 power but are expressed directly in Mb/s and seconds.
 
 bandwidth_file and latency_file do not express fraction of available
 power but are expressed directly in Mb/s and seconds.
 

-\section faq_flexml_bypassing How could I have some C functions do what the platform and deployment files do?
+\subsection faq_host_load Where is the get_host_load function hidden in MSG?
+
+There is no such thing because its semantic wouldn't be really clear. Of
+course, it is something about the amount of host throughput, but there is as
+many definition of "host load" as people asking for this function.
+
+It may be instantaneous value or an average one. Moreover it may be only the
+power of the computer, or may take the background load into account, or may
+even take the currently running tasks into account. In some SURF models,
+communications have an influence on computational power. Should it be taken
+into account too?
+
+So, we decided not to include such a function into MSG and let people do it
+thereselves so that they get the value matching exactly what they mean. One
+possibility is to run active measurement as in next code snippet. It is very
+close from what you would have to do out of the simulator, and thus gives
+you information that you could also get in real settings to not hinder the
+realism of your simulation. 
+
+\verbatim
+double get_host_load() {
+   m_task_t task = MSG_task_create("test", 0.001, 0, NULL);
+   double date = MSG_get_clock();
+
+   MSG_task_execute(task);
+   date = MSG_get_clock() - date;
+   MSG_task_destroy(task);
+   return (0.001/date);
+}
+\endverbatim
+
+Of course, it may not match your personal definition of "host load". In this
+case, please detail what you mean on the mailing list, and we will extend
+this FAQ section to fit your taste if possible.
+
+\subsection faq_flexml_bypassing How could I have some C functions do what the platform and deployment files do?

 
 So you want to bypass the XML files parser, uh? Maybe doin some parameter
 sweep experiments on your simulations or so? This is possible, but it's not
 
 So you want to bypass the XML files parser, uh? Maybe doin some parameter
 sweep experiments on your simulations or so? This is possible, but it's not


@@ -511,7 +576,61 @@ Then, tell SimGrid that you want to use your own "parser" instead of the stock o
 
 An example of this trick is distributed in the file examples/msg/msg_test_surfxml_bypassed.c
 
 
 An example of this trick is distributed in the file examples/msg/msg_test_surfxml_bypassed.c
 

-\section faq_flexml_limit I get the message "surf_parse_lex: Assertion `next<limit' failed."
+\section faq_troubleshooting Troubleshooting
+
+\subsection faq_context_1000 I want thousands of simulated processes
+
+SimGrid can use either pthreads library or the UNIX98 contextes. On most
+systems, the number of pthreads is limited and then your simulation may be
+limited for a stupid reason. This is especially true with the current linux
+pthreads, and I cannot get more than 2000 simulated processes with pthreads
+on my box. The UNIX98 contexts allow me to raise the limit to 25,000
+simulated processes on my laptop.
+
+The <tt>--with-context</tt> option of the <tt>./configure</tt> script allows
+you to choose between UNIX98 contextes (<tt>--with-context=ucontext</tt>)
+and the pthread version ( (<tt>--with-context=pthread</tt>). The default
+value is ucontext when the script detect a working UNIX98 context
+implementation. On Windows boxes, the provided value is discarded and an
+adapted version is picked up.
+
+We experienced some issues with contextes on some rare systems (solaris 8
+and lower comes to mind). The main problem is that the configure script
+detect the contextes as being functional when it's not true. If you happen
+to use such a system, switch manually to the pthread version, and provide us
+with a good patch for the configure script so that it is done automatically ;)
+
+\subsection faq_context_10000 I want hundred thousands of simulated processes
+
+As explained above, SimGrid can use UNIX98 contextes to represent and handle
+the simulated processes. Thanks to this, the main limitation to the number
+of simulated processes becomes the available memory. 
+
+Here are some tricks I had to use in order to run a token ring between
+25,000 processes on my laptop (1Gb memory, 1.5Gb swap).
+
+ - First of all, make sure your code runs for a few hundreds processes
+   before trying to push the limit. Make sure it's valgrind-clean, ie that
+   valgrind does not report neither memory error nor memory leaks. Indeed,
+   numerous simulated processes result in *fat* simulation hindering debugging.
+
+ - It was really boring to write 25,000 entries in the deployment file, so I wrote 
+   a little script <tt>examples/gras/tokenS/make_deployment.pl</tt>, which you may
+   want to adapt to your case.
+
+ - The deployment file became quite big, so I had to do what is in the FAQ
+   entry \ref faq_flexml_limit
+
+ - Each UNIX98 context has its own stack entry. As debugging this is quite 
+   hairly, the default value is a bit overestimated so that user don't get 
+   into trouble about this. You want to tune this size to increse the number 
+   of processes. This is the <tt>STACK_SIZE</tt> define in 
+   <tt>src/xbt/context_private.h</tt>, which is 128kb by default.
+   Reduce this as much as you can, but be warned that if this value is too 
+   low, you'll get a segfault. The token ring example, which is quite simple, 
+   runs with 40kb stacks.
+
+\subsection faq_flexml_limit I get the message "surf_parse_lex: Assertion `next&lt;limit' failed."

 
 This is because your platform file is too big for the parser. 
 
 
 This is because your platform file is too big for the parser. 
 


@@ -547,42 +666,6 @@ These are changes to FleXML itself, not SimGrid. But since we kinda hijacked
 the development of FleXML, I can grant you that any patches would be really
 welcome and quickly integrated.
 
 the development of FleXML, I can grant you that any patches would be really
 welcome and quickly integrated.
 

-\section faq_host_load Where is the get_host_load function hidden in MSG?
-
-There is no such thing because its semantic wouldn't be really clear. Of
-course, it is something about the amount of host throughput, but there is as
-many definition of "host load" as people asking for this function.
-
-It may be instantaneous value or an average one. Moreover it may be only the
-power of the computer, or may take the background load into account, or may
-even take the currently running tasks into account. In some SURF models,
-communications have an influence on computational power. Should it be taken
-into account too?
-
-So, we decided not to include such a function into MSG and let people do it
-thereselves so that they get the value matching exactly what they mean. One
-possibility is to run active measurement as in next code snippet. It is very
-close from what you would have to do out of the simulator, and thus gives
-you information that you could also get in real settings to not hinder the
-realism of your simulation. 
-
-\verbatim
-double get_host_load() {
-   m_task_t task = MSG_task_create("test", 0.001, 0, NULL);
-   double date = MSG_get_clock();
-
-   MSG_task_execute(task);
-   date = MSG_get_clock() - date;
-   MSG_task_destroy(task);
-   return (0.001/date);
-}
-\endverbatim
-
-Of course, it may not match your personal definition of "host load". In this
-case, please detail what you mean on the mailing list, and we will extend
-this FAQ section to fit your taste if possible.
-
-

 \author Arnaud Legrand (arnaud.legran::imag.fr)
 \author Martin Quinson (martin.quinson::loria.fr)
 
 \author Arnaud Legrand (arnaud.legran::imag.fr)
 \author Martin Quinson (martin.quinson::loria.fr)