From: Navarrop Date: Mon, 9 Jan 2012 13:57:57 +0000 (+0100) Subject: Add the faq to documentation because it is under version control X-Git-Tag: exp_20120216~180 X-Git-Url: http://info.iut-bm.univ-fcomte.fr/pub/gitweb/simgrid.git/commitdiff_plain/272c65e3b433ad56cb27ecfe229dc888bcbdc6e2 Add the faq to documentation because it is under version control --- diff --git a/buildtools/Cmake/GenerateDoc.cmake b/buildtools/Cmake/GenerateDoc.cmake index 56acede8dd..11441e5d8c 100644 --- a/buildtools/Cmake/GenerateDoc.cmake +++ b/buildtools/Cmake/GenerateDoc.cmake @@ -91,7 +91,7 @@ if(DOXYGEN_PATH AND FIG2DEV_PATH AND BIBTEX2HTML_PATH AND GOOD_BIBTEX2HTML_VERSI COMMAND ${CMAKE_COMMAND} -E echo "XX First Doxygen pass" COMMAND ${DOXYGEN_PATH}/doxygen Doxyfile COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/index_create.pl simgrid.tag index-API.doc - COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/toc_create.pl pls.doc index.doc gtut-introduction.doc installSimgrid.doc bindings.doc options.doc tracing.doc + COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/toc_create.pl pls.doc index.doc FAQ.doc gtut-introduction.doc installSimgrid.doc bindings.doc options.doc tracing.doc COMMAND ${CMAKE_COMMAND} -E echo "XX Second Doxygen pass" COMMAND ${DOXYGEN_PATH}/doxygen Doxyfile diff --git a/doc/Doxyfile.in b/doc/Doxyfile.in index c1dd9e6f7f..a99a83b132 100644 --- a/doc/Doxyfile.in +++ b/doc/Doxyfile.in @@ -614,6 +614,7 @@ WARN_LOGFILE = # with spaces. INPUT = index.doc \ + FAQ.doc \ installSimgrid.doc \ bindings.doc \ options.doc \ diff --git a/doc/FAQ.doc b/doc/FAQ.doc new file mode 100644 index 0000000000..583e80dfdd --- /dev/null +++ b/doc/FAQ.doc @@ -0,0 +1,1073 @@ +/*! \page FAQ Frequently Asked Questions + +\htmlinclude .FAQ.doc.toc + +\section faq_simgrid I'm new to SimGrid. I have some questions. Where should I start? + +You are at the right place... Having a look to these +the slides of the HPCS'10 tutorial +(or to these ancient +slides, or to these +"obsolete" slides) +may give you some insights on what SimGrid can help you to do and what +are its limitations. Then you definitely should read the \ref +MSG_examples. The \ref GRAS_tut can also help you. + +If you are stuck at any point and if this FAQ cannot help you, please drop us a +mail to the user mailing list: . + +\subsection faq_interfaces What is the difference between MSG, SimDag, and GRAS? Do they serve the same purpose? + +It depend on how you define "purpose", I guess ;) + +They all allow you to build a prototype of application which you can run +within the simulator afterward. They all share the same simulation kernel, +which is the core of the SimGrid project. They differ by the way you express +your application. + +With SimDag, you express your code as a collection of interdependent +parallel tasks. So, in this model, applications can be seen as a DAG of +tasks. This is the interface of choice for people wanting to port old +code designed for SimGrid v1 or v2 to the framework current version. + +With both GRAS and MSG, your application is seen as a set of communicating +processes, exchanging data by the way of messages and performing computation +on their own. + +The difference between both is that MSG is somehow easier to use, but GRAS +is not limited to the simulator. Once you're done writing your GRAS code, +you can run your code both in the simulator and on a real platform. For this, +there are two implementations of the GRAS interface, one for simulation, and one +for real execution. So, you just have to relink your code to choose one of +both worlds. + +\subsection faq_visualization Visualizing and analyzing the results + +It is sometime convenient to "see" how the agents are behaving. If you +like colors, you can use tools/MSG_visualization/colorize.pl +as a filter to your MSG outputs. It works directly with INFO. Beware, +INFO() prints on stderr. Do not forget to redirect if you want to +filter (e.g. with bash): +\verbatim +./msg_test small_platform.xml small_deployment.xml 2>&1 | ../../tools/MSG_visualization/colorize.pl +\endverbatim + +We also have a more graphical output. Have a look at section \ref options_tracing. + +\subsection faq_C Argh! Do I really have to code in C? + +Currently bindings on top of MSG are supported for Java, Ruby and Lua. You can find a few +documentation about them on the doc page. Note that bindings are released separately from the main dist +and so have their own version numbers. + +Moreover If you use C++, +you should be able to use the SimGrid library as a standard C library +and everything should work fine (simply link against this +library; recompiling SimGrid with a C++ compiler won't work and it +wouldn't help if you could). + +For now, +we do not feel a real demand for any other language. But if you think there is one, + please speak up! + +\section faq_howto Feature related questions + +\subsection faq_MIA "Could you please add (your favorite feature here) to SimGrid?" + +Here is the deal. The whole SimGrid project (MSG, SURF, GRAS, ...) is +meant to be kept as simple and generic as possible. We cannot add +functions for everybody's needs when these functions can easily be +built from the ones already in the API. Most of the time, it is +possible and when it was not possible we always have upgraded the API +accordingly. When somebody asks us a question like "How to do that? +Is there a function in the API to simply do this?", we're always glad +to answer and help. However if we don't need this code for our own +need, there is no chance we're going to write it... it's your job! :) +The counterpart to our answers is that once you come up with a neat +implementation of this feature (task duplication, RPC, thread +synchronization, ...), you should send it to us and we will be glad to +add it to the distribution. Thus, other people will take advantage of +it (and we don't have to answer this question again and again ;). + +You'll find in this section a few "Missing In Action" features. Many +people have asked about it and we have given hints on how to simply do +it with MSG. Feel free to contribute... + +\subsection faq_MIA_MSG MSG features + +\subsubsection faq_MIA_examples I want some more complex MSG examples! + +Many people have come to ask me a more complex example and each time, +they have realized afterward that the basics were in the previous three +examples. + +Of course they have often been needing more complex functions like +MSG_process_suspend(), MSG_process_resume() and +MSG_process_isSuspended() (to perform synchronization), or +MSG_task_Iprobe() and MSG_process_sleep() (to avoid blocking +receptions), or even MSG_process_create() (to design asynchronous +communications or computations). But the examples are sufficient to +start. + +We know. We should add some more examples, but not really some more +complex ones... We should add some examples that illustrate some other +functionalists (like how to simply encode asynchronous +communications, RPC, process migrations, thread synchronization, ...) +and we will do it when we will have a little bit more time. We have +tried to document the examples so that they are understandable. Tell +us if something is not clear and once again feel free to participate! +:) + +\subsubsection faq_MIA_taskdup Missing in action: MSG Task duplication/replication + +There is no task duplication in MSG. When you create a task, you can +process it or send it somewhere else. As soon as a process has sent +this task, he doesn't have this task anymore. It's gone. The receiver +process has got the task. However, you could decide upon receiving to +create a "copy" of a task but you have to handle by yourself the +semantic associated to this "duplication". + +As we already told, we prefer keeping the API as simple as +possible. This kind of feature is rather easy to implement by users +and the semantic you associate really depends on people. Having a +*generic* task duplication mechanism is not that trivial (in +particular because of the data field). That is why I would recommend +that you write it by yourself even if I can give you advice on how to +do it. + +You have the following functions to get information about a task: +MSG_task_get_name(), MSG_task_get_compute_duration(), +MSG_task_get_remaining_computation(), MSG_task_get_data_size(), +and MSG_task_get_data(). + +You could use a dictionary (#xbt_dict_t) of dynars (#xbt_dynar_t). If +you still don't see how to do it, please come back to us... + +\subsubsection faq_MIA_asynchronous I want to do asynchronous communications in MSG + +In the past (version <= 3.4), there was no function to perform asynchronous communications. +It could easily be implemented by creating new process when needed though. Since version 3.5, +we have introduced the following functions: + - MSG_task_isend() + - MSG_task_irecv() + - MSG_comm_test() + - MSG_comm_wait() + - MSG_comm_waitall() + - MSG_comm_waitany() + - MSG_comm_destroy() + +We refer you to the description of these functions for more details on their usage as well +as to the example section on \ref MSG_ex_asynchronous_communications. + +\subsubsection faq_MIA_thread_synchronization I need to synchronize my MSG processes + +You obviously cannot use pthread_mutexes of pthread_conds since we handle every +scheduling related decision within SimGrid. + +In the past (version <=3.3.4) you could do it by playing with +MSG_process_suspend() and MSG_process_resume() or with fake communications (using MSG_task_get(), +MSG_task_put() and MSG_task_Iprobe()). + +Since version 3.4, you can use classical synchronization structures. See page \ref XBT_synchro or simply check in +include/xbt/synchro_core.h. + +\subsubsection faq_MIA_host_load Where is the get_host_load function hidden in MSG? + +There is no such thing because its semantic wouldn't be really +clear. Of course, it is something about the amount of host throughput, +but there is as many definition of "host load" as people asking for +this function. First, you have to remember that resource availability +may vary over time, which make any load notion harder to define. + +It may be instantaneous value or an average one. Moreover it may be only the +power of the computer, or may take the background load into account, or may +even take the currently running tasks into account. In some SURF models, +communications have an influence on computational power. Should it be taken +into account too? + +First of all, it's near to impossible to predict the load beforehand in the +simulator since it depends on too much parameters (background load +variation, bandwidth sharing algorithmic complexity) some of them even being +not known beforehand (other task starting at the same time). So, getting +this information is really hard (just like in real life). It's not just that +we want MSG to be as painful as real life. But as it is in some way +realistic, we face some of the same problems as we would face in real life. + +How would you do it for real? The most common option is to use something +like NWS that performs active probes. The best solution is probably to do +the same within MSG, as in next code snippet. It is very close from what you +would have to do out of the simulator, and thus gives you information that +you could also get in real settings to not hinder the realism of your +simulation. + +\verbatim +double get_host_load() { + m_task_t task = MSG_task_create("test", 0.001, 0, NULL); + double date = MSG_get_clock(); + + MSG_task_execute(task); + date = MSG_get_clock() - date; + MSG_task_destroy(task); + return (0.001/date); +} +\endverbatim + +Of course, it may not match your personal definition of "host load". In this +case, please detail what you mean on the mailing list, and we will extend +this FAQ section to fit your taste if possible. + +\subsubsection faq_MIA_communication_time How can I get the *real* communication time? + +Communications are synchronous and thus if you simply get the time +before and after a communication, you'll only get the transmission +time and the time spent to really communicate (it will also take into +account the time spent waiting for the other party to be +ready). However, getting the *real* communication time is not really +hard either. The following solution is a good starting point. + +\verbatim +int sender() +{ + m_task_t task = MSG_task_create("Task", task_comp_size, task_comm_size, + calloc(1,sizeof(double))); + *((double*) task->data) = MSG_get_clock(); + MSG_task_put(task, slaves[i % slaves_count], PORT_22); + XBT_INFO("Send completed"); + return 0; +} +int receiver() +{ + m_task_t task = NULL; + double time1,time2; + + time1 = MSG_get_clock(); + a = MSG_task_get(&(task), PORT_22); + time2 = MSG_get_clock(); + if(time1<*((double *)task->data)) + time1 = *((double *) task->data); + XBT_INFO("Communication time : \"%f\" ", time2-time1); + free(task->data); + MSG_task_destroy(task); + return 0; +} +\endverbatim + +\subsection faq_MIA_SimDag SimDag related questions + +\subsubsection faq_SG_comm Implementing communication delays between tasks. + +A classic question of SimDag newcomers is about how to express a +communication delay between tasks. The thing is that in SimDag, both +computation and communication are seen as tasks. So, if you want to +model a data dependency between two DAG tasks t1 and t2, you have to +create 3 SD_tasks: t1, t2 and c and add dependencies in the following +way: + +\verbatim +SD_task_dependency_add(NULL, NULL, t1, c); +SD_task_dependency_add(NULL, NULL, c, t2); +\endverbatim + +This way task t2 cannot start before the termination of communication c +which in turn cannot start before t1 ends. + +When creating task c, you have to associate an amount of data (in bytes) +corresponding to what has to be sent by t1 to t2. + +Finally to schedule the communication task c, you have to build a list +comprising the workstations on which t1 and t2 are scheduled (w1 and w2 +for example) and build a communication matrix that should look like +[0;amount ; 0; 0]. + +\subsubsection faq_SG_DAG How to implement a distributed dynamic scheduler of DAGs. + +Distributed is somehow "contagious". If you start making distributed +decisions, there is no way to handle DAGs directly anymore (unless I +am missing something). You have to encode your DAGs in term of +communicating process to make the whole scheduling process +distributed. Here is an example of how you could do that. Assume T1 +has to be done before T2. + +\verbatim + int your_agent(int argc, char *argv[] { + ... + T1 = MSG_task_create(...); + T2 = MSG_task_create(...); + ... + while(1) { + ... + if(cond) MSG_task_execute(T1); + ... + if((MSG_task_get_remaining_computation(T1)=0.0) && (you_re_in_a_good_mood)) + MSG_task_execute(T2) + else { + /* do something else */ + } + } + } +\endverbatim + +If you decide that the distributed part is not that much important and that +DAG is really the level of abstraction you want to work with, then you should +give a try to \ref SD_API. + +\subsection faq_MIA_generic Generic features + +\subsubsection faq_more_processes Increasing the amount of simulated processes + +Here are a few tricks you can apply if you want to increase the amount +of processes in your simulations. + + - A few thousands of simulated processes (soft tricks)\n + SimGrid can use either pthreads library or the UNIX98 contexts. On + most systems, the number of pthreads is limited and then your + simulation may be limited for a stupid reason. This is especially + true with the current linux pthreads, and I cannot get more than + 2000 simulated processes with pthreads on my box. The UNIX98 + contexts allow me to raise the limit to 25,000 simulated processes + on my laptop.\n\n + The --with-context option of the ./configure + script allows you to choose between UNIX98 contexts + (--with-context=ucontext) and the pthread version + (--with-context=pthread). The default value is ucontext + when the script detect a working UNIX98 context implementation. On + Windows boxes, the provided value is discarded and an adapted + version is picked up.\n\n + We experienced some issues with contexts on some rare systems + (solaris 8 and lower or old alpha linuxes comes to mind). The main + problem is that the configure script detect the contexts as being + functional when it's not true. If you happen to use such a system, + switch manually to the pthread version, and provide us with a good + patch for the configure script so that it is done automatically ;) + + - Hundred thousands of simulated processes (hard-core tricks)\n + As explained above, SimGrid can use UNIX98 contexts to represent + and handle the simulated processes. Thanks to this, the main + limitation to the number of simulated processes becomes the + available memory.\n\n + Here are some tricks I had to use in order to run a token ring + between 25,000 processes on my laptop (1Gb memory, 1.5Gb swap).\n + - First of all, make sure your code runs for a few hundreds + processes before trying to push the limit. Make sure it's + valgrind-clean, i.e. that valgrind does not report neither memory + error nor memory leaks. Indeed, numerous simulated processes + result in *fat* simulation hindering debugging. + - It was really boring to write 25,000 entries in the deployment + file, so I wrote a little script + examples/gras/mutual_exclusion/simple_token/make_deployment.pl, which you may + want to adapt to your case. You could also think about hijacking + the SURFXML parser (have look at \ref faq_flexml_bypassing). + - The deployment file became quite big, so I had to do what is in + the FAQ entry \ref faq_flexml_limit + - Each UNIX98 context has its own stack entry. As debugging this is + quite hairy, the default value is a bit overestimated so that + user doesn't get into trouble about this. You want to tune this + size to increase the number of processes. This is the + STACK_SIZE define in + src/xbt/xbt_context_sysv.c, which is 128kb by default. + Reduce this as much as you can, but be warned that if this value + is too low, you'll get a segfault. The token ring example, which + is quite simple, runs with 40kb stacks. + - You may tweak the logs to reduce the stack size further. When + logging something, we try to build the string to display in a + char array on the stack. The size of this array is constant (and + equal to XBT_LOG_BUFF_SIZE, defined in include/xbt/log/h). If the + string is too large to fit this buffer, we move to a dynamically + sized buffer. In which case, we have to traverse one time the log + event arguments to compute the size we need for the buffer, + malloc it, and traverse the argument list again to do the actual + job.\n + The idea here is to move XBT_LOG_BUFF_SIZE to 1, forcing the logs + to use a dynamic array each time. This allows us to lower further + the stack size at the price of some performance loss...\n + This allowed me to run the reduce the stack size to ... 4k. Ie, + on my 1Gb laptop, I can run more than 250,000 processes! + +\subsubsection faq_MIA_batch_scheduler Is there a native support for batch schedulers in SimGrid? + +No, there is no native support for batch schedulers and none is +planned because this is a very specific need (and doing it in a +generic way is thus very hard). However some people have implemented +their own batch schedulers. Vincent Garonne wrote one during his PhD +and put his code in the contrib directory of our SVN so that other can +keep working on it. You may find inspiring ideas in it. + +\subsubsection faq_MIA_checkpointing I need a checkpointing thing + +Actually, it depends on whether you want to checkpoint the simulation, or to +simulate checkpoints. + +The first one could help if your simulation is a long standing process you +want to keep running even on hardware issues. It could also help to +rewind the simulation by jumping sometimes on an old checkpoint to +cancel recent calculations.\n +Unfortunately, such thing will probably never exist in SG. One would have to +duplicate all data structures because doing a rewind at the simulator level +is very very hard (not talking about the malloc free operations that might +have been done in between). Instead, you may be interested in the Libckpt +library (http://www.cs.utk.edu/~plank/plank/www/libckpt.html). This is the +checkpointing solution used in the condor project, for example. It makes it +easy to create checkpoints (at the OS level, creating something like core +files), and rerunning them on need. + +If you want to simulate checkpoints instead, it means that you want the +state of an executing task (in particular, the progress made towards +completion) to be saved somewhere. So if a host (and the task executing on +it) fails (cf. #MSG_HOST_FAILURE), then the task can be restarted +from the last checkpoint.\n + +Actually, such a thing does not exist in SimGrid either, but it's just +because we don't think it is fundamental and it may be done in the user code +at relatively low cost. You could for example use a watcher that +periodically get the remaining amount of things to do (using +MSG_task_get_remaining_computation()), or fragment the task in smaller +subtasks. + +\subsection faq_platform Platform building and Dynamic resources + +\subsubsection faq_platform_example Where can I find SimGrid platform files? + +There are several little examples in the archive, in the examples/msg +directory. From time to time, we are asked for other files, but we +don't have much at hand right now. + +You should refer to the Platform Description Archive +(http://pda.gforge.inria.fr) project to see the other platform file we +have available, as well as the Simulacrum simulator, meant to generate +SimGrid platforms using all classical generation algorithms. + +\subsubsection faq_platform_alnem How can I automatically map an existing platform? + +We are working on a project called ALNeM (Application-Level Network +Mapper) which goal is to automatically discover the topology of an +existing network. Its output will be a platform description file +following the SimGrid syntax, so everybody will get the ability to map +their own lab network (and contribute them to the catalog project). +This tool is not ready yet, but it move quite fast forward. Just stay +tuned. + +\subsubsection faq_platform_synthetic Generating synthetic but realistic platforms + +The third possibility to get a platform file (after manual or +automatic mapping of real platforms) is to generate synthetic +platforms. Getting a realistic result is not a trivial task, and +moreover, nobody is really able to define what "realistic" means when +speaking of topology files. You can find some more thoughts on this +topic in these +slides. + +If you are looking for an actual tool, there we have a little tool to +annotate Tiers-generated topologies. This perl-script is in +tools/platform_generation/ directory of the SVN. Dinda et Al. +released a very comparable tool, and called it GridG. + +\subsubsection faq_SURF_multicore Modeling multi-core resources + +Since version 3.6 of simgrid we can specify the core number of a resource. +To use this feature use tag 'host' with 'core' parameter. +\verbatim + + + + + + + +\endverbatim + +\subsubsection faq_SURF_dynamic Modeling dynamic resource availability + +A nice feature of SimGrid is that it enables you to seamlessly have +resources whose availability change over time. When you build a +platform, you generally declare hosts like that: + +\verbatim + +\endverbatim + +If you want the availability of "host A" to change over time, the only +thing you have to do is change this definition like that: + +\verbatim + +\endverbatim + +For hosts, availability files are expressed in fraction of available +power. Let's have a look at what "trace_A.txt" may look like: + +\verbatim +PERIODICITY 1.0 +0.0 1.0 +11.0 0.5 +20.0 0.9 +\endverbatim + +At time 0, our host will deliver 100 flop/s. At time 11.0, it will +deliver only 50 flop/s until time 20.0 where it will start +delivering 90 flop/s. Last at time 21.0 (20.0 plus the periodicity +1.0), we'll be back to the beginning and it will deliver 100 flop/s. + +Now let's look at the state file: +\verbatim +PERIODICITY 10.0 +1.0 -1.0 +2.0 1.0 +\endverbatim + +A negative value means "off" while a positive one means "on". At time +1.0, the host is on. At time 1.0, it is turned off and at time 2.0, it +is turned on again until time 12 (2.0 plus the periodicity 10.0). It +will be turned on again at time 13.0 until time 23.0, and so on. + +Now, let's look how the same kind of thing can be done for network +links. A usual declaration looks like: + +\verbatim + +\endverbatim + +You have at your disposal the following options: bandwidth_file, +latency_file and state_file. The only difference with hosts is that +bandwidth_file and latency_file do not express fraction of available +power but are expressed directly in bytes per seconds and seconds. + +\subsubsection faq_platform_multipath How to express multipath routing in platform files? + +It is unfortunately impossible to express the fact that there is more +than one routing path between two given hosts. Let's consider the +following platform file: + +\verbatim + + + + + + + + + +\endverbatim + +Although it is perfectly valid, it does not mean that data traveling +from A to C can either go directly (using link 3) or through B (using +links 1 and 2). It simply means that the routing on the graph is not +trivial, and that data do not following the shortest path in number of +hops on this graph. Another way to say it is that there is no implicit +in these routing descriptions. The system will only use the routes you +declare (such as <route src="A" dst="C"><link_ctn +id="3"/></route>), without trying to build new routes by aggregating +the provided ones. + +You are also free to declare platform where the routing is not +symmetric. For example, add the following to the previous file: + +\verbatim + + + + +\endverbatim + +This makes sure that data from C to A go through B where data from A +to C go directly. Don't worry about realism of such settings since +we've seen ways more weird situation in real settings (in fact, that's +the realism of very regular platforms which is questionable, but +that's another story). + +\subsubsection faq_flexml_bypassing Bypassing the XML parser with your own C functions + +So you want to bypass the XML files parser, uh? Maybe doing some parameter +sweep experiments on your simulations or so? This is possible, and +it's not even really difficult (well. Such a brutal idea could be +harder to implement). Here is how it goes. + +For this, you have to first remember that the XML parsing in SimGrid is done +using a tool called FleXML. Given a DTD, this gives a flex-based parser. If +you want to bypass the parser, you need to provide some code mimicking what +it does and replacing it in its interactions with the SURF code. So, let's +have a look at these interactions. + +FleXML parser are close to classical SAX parsers. It means that a +well-formed SimGrid platform XML file might result in the following +"events": + + - start "platform_description" with attribute version="2" + - start "host" with attributes id="host1" power="1.0" + - end "host" + - start "host" with attributes id="host2" power="2.0" + - end "host" + - start "link" with ... + - end "link" + - start "route" with ... + - start "link_ctn" with ... + - end "link_ctn" + - end "route" + - end "platform_description" + +The communication from the parser to the SURF code uses two means: +Attributes get copied into some global variables, and a surf-provided +function gets called by the parser for each event. For example, the event + - start "host" with attributes id="host1" power="1.0" + +let the parser do something roughly equivalent to: +\verbatim + strcpy(A_host_id,"host1"); + A_host_power = 1.0; + STag_host(); +\endverbatim + +In SURF, we attach callbacks to the different events by initializing the +pointer functions to some the right surf functions. Since there can be +more than one callback attached to the same event (if more than one +model is in use, for example), they are stored in a dynar. Example in +workstation_ptask_L07.c: +\verbatim + /* Adding callback functions */ + surf_parse_reset_parser(); + surfxml_add_callback(STag_surfxml_host_cb_list, &parse_cpu_init); + surfxml_add_callback(STag_surfxml_prop_cb_list, &parse_properties); + surfxml_add_callback(STag_surfxml_link_cb_list, &parse_link_init); + surfxml_add_callback(STag_surfxml_route_cb_list, &parse_route_set_endpoints); + surfxml_add_callback(ETag_surfxml_link_c_ctn_cb_list, &parse_route_elem); + surfxml_add_callback(ETag_surfxml_route_cb_list, &parse_route_set_route); + + /* Parse the file */ + surf_parse_open(file); + xbt_assert(!surf_parse(), "Parse error in %s", file); + surf_parse_close(); +\endverbatim + +So, to bypass the FleXML parser, you need to write your own version of the +surf_parse function, which should do the following: + - Fill the A__ variables with the wanted values + - Call the corresponding STag__fun function to simulate tag start + - Call the corresponding ETag__fun function to simulate tag end + - (do the same for the next set of values, and loop) + +Then, tell SimGrid that you want to use your own "parser" instead of the stock one: +\verbatim + surf_parse = surf_parse_bypass_environment; + MSG_create_environment(NULL); + surf_parse = surf_parse_bypass_application; + MSG_launch_application(NULL); +\endverbatim + +A set of macros are provided at the end of +include/surf/surfxml_parse.h to ease the writing of the bypass +functions. An example of this trick is distributed in the file +examples/msg/masterslave/masterslave_bypass.c + +\section faq_troubleshooting Troubleshooting + +\subsection faq_trouble_lib_compil SimGrid compilation and installation problems + +\subsubsection faq_trouble_lib_config cmake fails! + +We know only one reason for the configure to fail: + + - You are using a broken build environment\n + If symptom is that the configury magic complains about gcc not being able to build + executables, you are probably missing the libc6-dev package. Damn Ubuntu. + +If you experience other kind of issue, please get in touch with us. We are +always interested in improving our portability to new systems. + +\subsubsection faq_trouble_distcheck Dude! "ctest" fails on my machine! + +Don't assume we never run this target, because we do. Check +http://cdash.inria.fr/CDash/index.php?project=Simgrid (click on +previous if there is no result for today: results are produced only by +11am, French time) and +https://buildd.debian.org/status/logs.php?pkg=simgrid if you don't believe us. + +If it's failing on your machine in a way not experienced by the +autobuilders above, please drop us a mail on the mailing list so that +we can check it out. Make sure to read \ref faq_bugrepport before you +do so. + +\subsection faq_trouble_compil User code compilation problems + +\subsubsection faq_trouble_err_logcat "gcc: _simgrid_this_log_category_does_not_exist__??? undeclared (first use in this function)" + +This is because you are using the log mecanism, but you didn't created +any default category in this file. You should refer to \ref XBT_log +for all the details, but you simply forgot to call one of +XBT_LOG_NEW_DEFAULT_CATEGORY() or XBT_LOG_NEW_DEFAULT_SUBCATEGORY(). + +\subsubsection faq_trouble_pthreadstatic "gcc: undefined reference to pthread_key_create" + +This indicates that one of the library SimGrid depends on (libpthread +here) was missing on the linking command line. Dependencies of +libsimgrid are expressed directly in the dynamic library, so it's +quite impossible that you see this message when doing dynamic linking. + +If you compile your code statically (and if you use a pthread version +of SimGrid -- see \ref faq_more_processes), you must absolutely +specify -lpthread on the linker command line. As usual, this should +come after -lsimgrid on this command line. + +\subsection faq_trouble_errors Runtime error messages + +\subsubsection faq_flexml_limit "surf_parse_lex: Assertion `next limit' failed." + +This is because your platform file is too big for the parser. + +Actually, the message comes directly from FleXML, the technology on top of +which the parser is built. FleXML has the bad idea of fetching the whole +document in memory before parsing it. And moreover, the memory buffer size +must be determined at compilation time. + +We use a value which seems big enough for our need without bloating the +simulators footprints. But of course your mileage may vary. In this case, +just edit src/surf/surfxml.l modify the definition of +FLEXML_BUFFERSTACKSIZE. E.g. + +\verbatim +#define FLEXML_BUFFERSTACKSIZE 1000000000 +\endverbatim + +Then recompile and everything should be fine, provided that your version of +Flex is recent enough (>= 2.5.31). If not the compilation process should +warn you. + +A while ago, we worked on FleXML to reduce a bit its memory consumption, but +these issues remain. There is two things we should do: + + - use a dynamic buffer instead of a static one so that the only limit + becomes your memory, not a stupid constant fixed at compilation time + (maybe not so difficult). + - change the parser so that it does not need to get the whole file in + memory before parsing + (seems quite difficult, but I'm a complete newbe wrt flex stuff). + +These are changes to FleXML itself, not SimGrid. But since we kinda hijacked +the development of FleXML, I can grant you that any patches would be really +welcome and quickly integrated. + +Update: A new version of FleXML (1.7) was released. Most of the work +was done by William Dowling, who use it in his own work. The good point is +that it now use a dynamic buffer, and that the memory usage was greatly +improved. The downside is that William also changed some things internally, +and it breaks the hack we devised to bypass the parser, as explained in +\ref faq_flexml_bypassing. Indeed, this is not a classical usage of the +parser, and Will didn't imagine that we may have used (and even documented) +such a crude usage of FleXML. So, we now have to repair the bypassing +functionality to use the lastest FleXML version and fix the memory usage in +SimGrid. + +\subsubsection faq_trouble_gras_transport GRAS spits networking error messages + +Gras, on real platforms, naturally use regular sockets to communicate. They +are deeply hidden in the gras abstraction, but when things go wrong, you may +get some weird error messages. Here are some example, with the probable +reason: + + - Transport endpoint is not connected: several processes try to open + a server socket on the same port number of the same machine. This is + naturally bad and each process should pick its own port number for this.\n + Maybe, you just have some processes remaining from a previous experiment + on your machine.\n + Killing them may help, but again if you kill -KILL them, you'll have to + wait for a while: they didn't close there sockets properly and the system + needs a while to notice that this port is free again. + + - Socket closed by remote side: if the remote process is not + supposed to close the socket at this point, it may be dead. + + - Connection reset by peer: I found this on Internet about this + error. I think it's what's happening here, too:\n + This basically means that a network error occurred while the client was + receiving data from the server. But what is really happening is that the + server actually accepts the connection, processes the request, and sends + a reply to the client. However, when the server closes the socket, the + client believes that the connection has been terminated abnormally + because the socket implementation sends a TCP reset segment telling the + client to throw away the data and report an error.\n + Sometimes, this problem is caused by not properly closing the + input/output streams and the socket connection. Make sure you close the + input/output streams and socket connection properly. If everything is + closed properly, however, and the problem persists, you can work around + it by adding a one-second sleep before closing the streams and the + socket. This technique, however, is not reliable and may not work on all + systems.\n + Since GRAS sockets are closed properly (repeat after me: there is no bug + in GRAS), it is either that you are closing your sockets on server side + before the client get a chance to read them (use gras_os_sleep() to delay + the server), or the server died awfully before the client got the data. + +\subsubsection faq_trouble_errors_big_fat_warning I'm told that my XML files are too old. + +The format of the XML platform description files is sometimes +improved. For example, we decided to change the units used in SimGrid +from MBytes, MFlops and seconds to Bytes, Flops and seconds to ease +people exchanging small messages. We also reworked the route +descriptions to allow more compact descriptions. + +That is why the XML files are versionned using the 'version' attribute +of the root tag. Currently, it should read: +\verbatim + +\endverbatim + +If your files are too old, you can use the simgrid_update_xml.pl +script which can be found in the tools directory of the archive. + +\subsection faq_trouble_valgrind Valgrind-related and other debugger issues + +If you don't, you really should use valgrind to debug your code, it's +almost magic. + +\subsubsection faq_trouble_vg_longjmp longjmp madness in valgrind + +This is when valgrind starts complaining about longjmp things, just like: + +\verbatim ==21434== Conditional jump or move depends on uninitialised value(s) +==21434== at 0x420DBE5: longjmp (longjmp.c:33) +==21434== +==21434== Use of uninitialised value of size 4 +==21434== at 0x420DC3A: __longjmp (__longjmp.S:48) +\endverbatim + +This is the sign that you didn't used the exception mecanism well. Most +probably, you have a return; somewhere within a TRY{} +block. This is evil, and you must not do this. Did you read the section +about \ref XBT_ex?? + +\subsubsection faq_trouble_vg_libc Valgrind spits tons of errors about backtraces! + +It may happen that valgrind, the memory debugger beloved by any decent C +programmer, spits tons of warnings like the following : +\verbatim ==8414== Conditional jump or move depends on uninitialised value(s) +==8414== at 0x400882D: (within /lib/ld-2.3.6.so) +==8414== by 0x414EDE9: (within /lib/tls/i686/cmov/libc-2.3.6.so) +==8414== by 0x400B105: (within /lib/ld-2.3.6.so) +==8414== by 0x414F937: _dl_open (in /lib/tls/i686/cmov/libc-2.3.6.so) +==8414== by 0x4150F4C: (within /lib/tls/i686/cmov/libc-2.3.6.so) +==8414== by 0x400B105: (within /lib/ld-2.3.6.so) +==8414== by 0x415102D: __libc_dlopen_mode (in /lib/tls/i686/cmov/libc-2.3.6.so) +==8414== by 0x412D6B9: backtrace (in /lib/tls/i686/cmov/libc-2.3.6.so) +==8414== by 0x8076446: xbt_dictelm_get_ext (dict_elm.c:714) +==8414== by 0x80764C1: xbt_dictelm_get (dict_elm.c:732) +==8414== by 0x8079010: xbt_cfg_register (config.c:208) +==8414== by 0x806821B: MSG_config (msg_config.c:42) +\endverbatim + +This problem is somewhere in the libc when using the backtraces and there is +very few things we can do ourselves to fix it. Instead, here is how to tell +valgrind to ignore the error. Add the following to your ~/.valgrind.supp (or +create this file on need). Make sure to change the obj line according to +your personnal mileage (change 2.3.6 to the actual version you are using, +which you can retrieve with a simple "ls /lib/ld*.so"). + +\verbatim { + name: Backtrace madness + Memcheck:Cond + obj:/lib/ld-2.3.6.so + fun:dl_open_worker + fun:_dl_open + fun:do_dlopen + fun:dlerror_run + fun:__libc_dlopen_mode +}\endverbatim + +Then, you have to specify valgrind to use this suppression file by passing +the --suppressions=$HOME/.valgrind.supp option on the command line. +You can also add the following to your ~/.bashrc so that it gets passed +automatically. Actually, it passes a bit more options to valgrind, and this +happen to be my personnal settings. Check the valgrind documentation for +more information. + +\verbatim export VALGRIND_OPTS="--leak-check=yes --leak-resolution=high --num-callers=40 --tool=memcheck --suppressions=$HOME/.valgrind.supp" \endverbatim + +\subsubsection faq_trouble_backtraces Truncated backtraces + +When debugging SimGrid, it's easier to pass the +--disable-compiler-optimization flag to the configure if valgrind or +gdb get fooled by the optimization done by the compiler. But you +should remove these flag when everything works before going in +production (before launching your 1252135 experiments), or everything +will run only one half of the true SimGrid potential. + +\subsection faq_deadlock There is a deadlock in my code!!! + +Unfortunately, we cannot debug every code written in SimGrid. We +furthermore believe that the framework provides ways enough +information to debug such informations yourself. If the textual output +is not enough, Make sure to check the \ref faq_visualization FAQ entry to see +how to get a graphical one. + +Now, if you come up with a really simple example that deadlocks and +you're absolutely convinced that it should not, you can ask on the +list. Just be aware that you'll be severely punished if the mistake is +on your side... We have plenty of FAQ entries to redact and new +features to implement for the impenitents! ;) + +\subsection faq_surf_network_latency I get weird timings when I play with the latencies. + +OK, first of all, remember that units should be Bytes, Flops and +Seconds. If you don't use such units, some SimGrid constants (e.g. the +SG_TCP_CTE_GAMMA constant used in most network models) won't have the +right unit and you'll end up with weird results. + +Here is what happens with a single transfer of size L on a link +(bw,lat) when nothing else happens. + +\verbatim +0-----lat--------------------------------------------------t +|-----|**** real_bw =min(bw,SG_TCP_CTE_GAMMA/(2*lat)) *****| +\endverbatim + +In more complex situations, this min is the solution of a complex +max-min linear system. Have a look +here +and read the two threads "Bug in SURF?" and "Surf bug not +fixed?". You'll have a few other examples of such computations. You +can also read "A Network Model for Simulation of Grid Application" by +Henri Casanova and Loris Marchal to have all the details. The fact +that the real_bw is smaller than bw is easy to understand. The fact +that real_bw is smaller than SG_TCP_CTE_GAMMA/(2*lat) is due to the +window-based congestion mechanism of TCP. With TCP, you can't exploit +your huge network capacity if you don't have a good round-trip-time +because of the acks... + +Anyway, what you get is t=lat + L/min(bw,SG_TCP_CTE_GAMMA/(2*lat)). + + * if I you set (bw,lat)=(100 000 000, 0.00001), you get t = 1.00001 (you fully +use your link) + * if I you set (bw,lat)=(100 000 000, 0.0001), you get t = 1.0001 (you're on the +limit) + * if I you set (bw,lat)=(100 000 000, 0.001), you get t = 10.001 (ouch!) + +This bound on the effective bandwidth of a flow is not the only thing +that may make your result be unexpected. For example, two flows +competing on a saturated link receive an amount of bandwidth inversely +proportional to their round trip time. + +\subsection faq_bugrepport So I've found a bug in SimGrid. How to report it? + +We do our best to make sure to hammer away any bugs of SimGrid, but this is +still an academic project so please be patient if/when you find bugs in it. +If you do, the best solution is to drop an email either on the simgrid-user +or the simgrid-devel mailing list and explain us about the issue. You can +also decide to open a formal bug report using the +relevant +interface. You need to login on the server to get the ability to submit +bugs. + +We will do our best to solve any problem repported, but you need to help us +finding the issue. Just telling "it segfault" isn't enough. Telling "It +segfaults when running the attached simulator" doesn't really help either. +You may find the following article interesting to see how to repport +informative bug repports: +http://www.chiark.greenend.org.uk/~sgtatham/bugs.html (it is not SimGrid +specific at all, but it's full of good advices). + +\author Arnaud Legrand (arnaud.legrand::imag.fr) +\author Martin Quinson (martin.quinson::loria.fr) + + +*/ + +****************************************************************** +* OLD CRUFT NOT USED ANYMORE * +****************************************************************** + + +subsection faq_crosscompile Cross-compiling a Windows DLL of SimGrid from linux + +At the moment, we do not distribute Windows pre-compiled version of SimGrid +because the support for this platform is still experimental. We know that +some parts of the GRAS environment do not work, and we think that the others +environments (MSG and SD) have good chances to work, but we didn't test +ourselves. This section explains how we generate the SimGrid DLL so that you +can build it for yourself. First of all, you need to have a version more +recent than 3.1 (ie, a SVN version as time of writting). + +In order to cross-compile the package to windows from linux, you need to +install mingw32 (minimalist gnu win32). On Debian, you can do so by +installing the packages mingw32 (compiler), mingw32-binutils (linker and +so), mingw32-runtime. + +You can use the VPATH support of configure to compile at the same time for +linux and windows without dupplicating the source nor cleaning the tree +between each. Just run bootstrap (if you use the SVN) to run the autotools. +Then, create a linux and a win directories. Then, type: +\verbatim cd linux; ../configure --srcdir=.. ; make; cd .. +cd win; ../configure --srcdir=.. --host=i586-mingw32msvc ; make; cd .. +\endverbatim +The trick to VPATH builds is to call configure from another directory, +passing it an extra --srcdir argument to tell it where all the sources are. +It will understand you want to use VPATH. Then, the trick to cross-compile +is simply to add a --host argument specifying the target you want to build +for. The i586-mingw32msvc string is what you have to pass to use the mingw32 +environment as distributed in Debian. + +After that, you can run all make targets from both directories, and test +easily that what you change for one arch does not break the other one. + +It is possible that this VPATH build thing breaks from time to time in the +SVN since it's quite fragile, but it's granted to work in any released +version. If you experience problems, drop us a mail. + +Another possible source of issue is that at the moment, building the +examples request to use the gras_stub_generator tool, which is a compiled +program, not a script. In cross-compilation, you need to cross-execute with +wine for example, which is not really pleasant. We are working on this, but +in the meanwhile, simply don't build the examples in cross-compilation +(cd src before running make). + +Program (cross-)compiled with mingw32 do request an extra DLL at run-time to be +usable. For example, if you want to test your build with wine, you should do +the following to put this library where wine looks for DLLs. +\verbatim +cp /usr/share/doc/mingw32-runtime/mingwm10.dll.gz ~/.wine/c/windows/system/ +gunzip ~/.wine/c/windows/system/mingwm10.dll.gz +\endverbatim + +The DLL is built in src/.libs, and installed in the prefix/bin directory +when you run make install. + +If you want to use it in a native project on windows, you need to use +simgrid.dll and mingwm10.dll. For each DLL, you need to build .def file +under linux (listing the defined symbols), and convert it into a .lib file +under windows (specifying this in a way that windows compilers like). To +generate the def files, run (under linux): +\verbatim echo "LIBRARY libsimgrid-0.dll" > simgrid.def +echo EXPORTS >> simgrid.def +nm libsimgrid-0.dll | grep ' T _' | sed 's/.* T _//' >> simgrid.def +nm libsimgrid-0.dll | grep ' D _' | sed 's/.* D _//' | sed 's/$/ DATA/' >> simgrid.def + +echo "LIBRARY mingwm10.dll" > mingwm10.def +echo EXPORTS >> mingwm10.def +nm mingwm10.dll | grep ' T _' | sed 's/.* T _//' >> mingwm10.def +nm mingwm10.dll | grep ' D _' | sed 's/.* D _//' | sed 's/$/ DATA/' >> mingwm10.def +\endverbatim + +To create the import .lib files, use the lib windows tool (from +MSVC) the following way to produce simgrid.lib and mingwm10.lib +\verbatim lib /def:simgrid.def +lib /def:mingwm10.def +\endverbatim + +If you happen to use Borland C Builder, the right command line is the +following (note that you don't need any file.def to get this working). +\verbatim implib simgrid.lib libsimgrid-0.dll +implib mingwm10.lib mingwm10.dll +\endverbatim + +Then, set the following parameters in Visual C++ 2005: +Linker -> Input -> Additional dependencies = simgrid.lib mingwm10.lib + +Just in case you wonder how to generate a DLL from libtool in another +project, we added -no-undefined to any lib*_la_LDFLAGS variables so that +libtool accepts to generate a dynamic library under windows. Then, to make +it true, we pass any dependencies (such as -lws2 under windows or -lpthread +on need) on the linking line. Passing such deps is a good idea anyway so +that they get noted in the library itself, avoiding the users to know about +our dependencies and put them manually on their compilation line. Then we +added the AC_LIBTOOL_WIN32_DLL macro just before AC_PROG_LIBTOOL in the +configure.ac. It means that we exported any symbols which need to be. +Nowadays, functions get automatically exported, so we don't need to load our +header files with tons of __declspec(dllexport) cruft. We only need to do so +for data, but there is no public data in SimGrid so we are good. diff --git a/tools/doxygen/doxygen_postprocesser.pl b/tools/doxygen/doxygen_postprocesser.pl index bd15975168..378b4cd057 100755 --- a/tools/doxygen/doxygen_postprocesser.pl +++ b/tools/doxygen/doxygen_postprocesser.pl @@ -6,7 +6,7 @@ use strict; my @extra_files = qw(html/index.html html/pages.html html/modules.html html/annotated.html html/functions.html html/functions_vars.html index.php html/GRAS_tut.html html/tracing.html html/installSimgrid.html html/bindings.html - html/options.html html/use.html html/pls.html); + html/options.html html/use.html html/pls.html html/FAQ.html); # GRAS tutorial map {push @extra_files, "html/GRAS_tut_$_.html"} qw (intro @@ -385,6 +385,7 @@ foreach my $file (@allfiles) { $tmp_buff .= '
  • Forge
    • '."\n"; $tmp_buff .= '
  • Website
    • '."\n"; $tmp_buff .= '
  • Documentation index
    • '."\n"; + $tmp_buff .= '
  • FAQ
    • '."\n"; $tmp_buff .= $_; $tabs = 0; @@ -425,7 +426,8 @@ foreach my $file (@allfiles) { # Rework the navbar # Fix the current "button" of buggy Doxygen tabs - if($file =~ /^html\/pages.*/) + if($file =~ /^html\/pages.*/ + || $file =~ /^html\/FAQ.*/) { my $filename = $file; $filename =~ s/html\///g;