New function: gras_msg_wait_ext (for a finer control of accepted messages); introduce...

[simgrid.git] / doc / module-gras.doc

#####################################################################
###########################  CORE ###################################
#####################################################################

/** \addtogroup GRAS_API
  
    \section GRAS_funct Offered functionnalities
     - <b>Communication facilities</b>: Exchanging messages between peers
       - \ref GRAS_dd : any data which may transit on the network must be
         described beforehand so that GRAS can handle the platform
         heterogeneity and convert them if needed.
       - \ref GRAS_sock : this is how to open a communication channel to
         other processes, and retrive information about them.
       - \ref GRAS_msg : communications are message oriented. You have to
         describe all possible messages and their payload beforehand, and
         can then attach callbacks to the arrival of a given kind of message. 
       - \ref GRAS_timer : this is how to program repetitive and delayed
         tasks, not unlike cron(8) and at(1). This cannot be used to timeout
         a function (like setitimer(2) or signal(2) games could do).
     - <b>Virtualization</b>: Running both on top of the simulator and on
       top of real platforms, and portability support.
       - \ref GRAS_virtu : You naturally don't want to call the
          gettimeofday(2) function in simulation mode since it would give
          you the time on the host running the simulation, not the time in
          the simulated world (you are belonging to).\n
          This a system call virtualization layer, which also acts as a
          portability layer.
       - \ref GRAS_globals : The use of globals is forbidden since the
         "processes" are threads in simulation mode. \n
         This is how to let GRAS handle your globals properly.
       - \ref GRAS_emul : Support to emulate code excution (ie, reporting
         execution time into the simulator and having code sections specific
         to simulation or to real mode).
     - <b>Project management tools</b>: Here are some tools which may help
          you setting up a GRAS project.\n
          Setting up and building a GRAS application is complicated by the
          library schizoid. The code to setup the environment differs
          depending on whether you run on the simulator on a real platform.
          And then, you'll have to deal with the usual distributed
          application development difficulties.
       - \ref GRAS_main_generation : Since processes are threads in
          simulation mode and regular processes in the real world, GRAS does
          generate your main functions for you.
       - \ref GRAS_compile
     
          
    \section GRAS_example Examples
      
    There is for now rather few examples of GRAS, but it's better than
    nothing, isn't it?
    
       - \ref GRAS_ex_ping
       - \ref GRAS_ex_mmrpc
       - \ref GRAS_ex_timer
               
    @{ */     
       /** @defgroup GRAS_dd      Data description      */       
       /** @defgroup GRAS_sock    Sockets               */           
       /** @defgroup GRAS_msg     Messages              */               
       /** @defgroup GRAS_timer   Timers                */               
         
       /** @defgroup GRAS_globals Globals               */ 
       /** @defgroup GRAS_emul    Emulation support */ 
       /** @defgroup GRAS_virtu   Syscalls              */ 

/** @} */

#####################################################################
#########################  EXTRA PAGES ##############################
#####################################################################

---------------------------------------------------------------------
--------------------- main() generation -----------------------------
---------------------------------------------------------------------

/** \page GRAS_main_generation main() and GRAS

  <center><table><tr><td><b>Top</b>    <td> [\ref index]::[\ref GRAS_API]
                 <tr><td><b>Prev</b>   <td> [\ref GRAS_emul]
                 <tr><td><b>Next</b>   <td> [\ref GRAS_compile]            </table></center>


    \section GRAS_maingen_toc Table of content
     
     - \ref GRAS_maingen_intro
     - \ref GRAS_maingen_script
     - \ref GRAS_maingen_make
    
    <hr>

    \section GRAS_maingen_intro What's the matter with main() functions in GRAS?

    In simulation mode, all processes are run as thread of the same process
    while they are real processes in the real life. Unfortunately, the main
    function of a real process must be called <tt>main</tt> while this
    function must not use this name for threads.
    
    To deal with this, you should call the main function of your processes
    with another name (usually, the process function such as client, server,
    or such). Then GRAS can generate the wrapper functions adapted to the
    real and simulated modes.

    \section GRAS_maingen_script Generating the main()s automatically
    
    This is done by the gras_stub_generator program, which gets installed on
    <tt>make install</tt> (the source resides in the tools/gras/ directory).
    Here is the calling syntax: 
    \verbatim gras_stub_generator <project_name> <deployment_file.xml>\endverbatim
    
    It parses the deployment file, searching for all the kind of processes
    you have in your project. It then generates the following C files:
     - a <tt>_<project_name>_<process_kind>.c</tt> file for each process kind you
       have\n
       They are used to launch your project in real life. They
       contain a main() in charge of initializing the GRAS infrastructure and
       launching your code afterward.
     - a <tt>_<project_name>_simulator.c</tt> file.\n
       This file is suited to the simulation mode. It contains a main()
       function initializing the simulator and launching your project within.
    
    For this to work, the name of process described in your deployment file
    should match the name of a function in your code, which prototype is for
    example: \verbatim int client(int argc,char *argv[]);\endverbatim
    
    Unfortunately, all this is still partially documented. I guess I ought
    to improve this situation somehow. In the meanwhile, check the generated 
    code and maybe also the GRAS \ref GRAS_example, sorry. 
        
    \section GRAS_maingen_make Integration within an hand-made Makefile 
    
    The easiest to set it up is to add the following chunk at the end of
    your Makefile (or Makefile.am), putting the right values into NAME and
    PROCESSES.
\verbatim NAME=your_project_name
 PROCESSES=list of processes type in your project

 $(foreach proc, $(PROCESSES), _$(NAME)_$(proc).c) _$(NAME)_simulator.c: $(NAME).c $(NAME)_deployment.xml
        path/to/gras_stub_generator $(NAME) $(NAME)_deployment.xml >/dev/null
\endverbatim

    Of course, your personal millage may vary. For the \ref GRAS_ex_ping, may read:
\verbatim _ping_client.c _ping_server.c _ping_simulator.c: ping.c ping_deployment.xml 
        $(top_srcdir)/tools/gras/gras_stub_generator ping ping_deployment.xml >/dev/null
\endverbatim

   \warning 
   Actually, gras_stub_generator also generates some makefiles both for
   local compilation and remote code distribution and installation. See the
   section \ref GRAS_compile for more details.

*/

---------------------------------------------------------------------
------------------------- Compiling ---------------------------------
---------------------------------------------------------------------

/** \page GRAS_compile Compiling your GRAS project

    <center><table><tr><td><b>Top</b>    <td> [\ref index]::[\ref GRAS_API]
                   <tr><td><b>Prev</b>   <td> [\ref GRAS_main_generation]
                   <tr><td><b>Next</b>   <td> [\ref GRAS_ex_ping]            </table></center>

    As explained in section \ref GRAS_main_generation, the
    gras_stub_generator tool can be used to generate the system
    initialization code in your projet. While we were at this, this tool
    also generates the makefiles you will need to compile your project
    properly.
    
    Code source deployment and remote compilation also constitutes a
    challenging area in distributed applications development. The GRASPE
    (GRAS Platform Expender) tool was designed to make this less painful.

    \section GRAS_compile_toc Table of content
    
      - \ref GRAS_compile_local
        - \ref GRAS_compile_local_install
        - \ref GRAS_compile_local_helpfiles
        - \ref GRAS_compile_local_makefile
      - \ref GRAS_compile_remote
      
    <hr>
    
    \section GRAS_compile_local Local compilation of GRAS projects
    
    \subsection GRAS_compile_local_install Installing SimGrid and GRAS
    
    To compile locally a GRAS project, you first need to install SimGrid on
    your machine. Use the --prefix flag to the configure script to specify
    where you want to install the toolkit (refere to section \ref
    faq_compiling for more information)
    
    \subsection GRAS_compile_local_helpfiles Simulation description files
    
    Then, you will probably need to write a platform description file and
    application deployment description file to feed the simulator with. This
    part is unfortunatelly not documented enough. Files examples can be
    found along with the MSG \ref MSG_ex_master_slave example. 

    \note yes, both platform and application description files are portable
    between MSG and GRAS. Actually, there depend on the SURF, not on the
    programming environment you use.
    
    For the first try, you could probably reuse the provided platform file
    as is while you will need to adapt the application file to fit your
    needs. 
    
    To generate new platform files, we usually use the Tiers Topology
    Generator (ask google about it) and annotate the generated graph with
    home-made scripts to let them fit the SURF. Those scripts live in the
    tools/platform_generation/ directory of the distribution.
    
    \subsection GRAS_compile_local_makefile Generating a Makefile usable for your project
    
    From the information contained in the application description file, the
    gras_stub_generator tool can create a Makefile which can be used to
    seamlessly compile your project. Just go to the directory containing all
    your project files, and type:
    
\verbatim path/to/gras_stub_generator [project_name] [application_deployment.file] >/dev/null
\endverbatim

    The first argument is the name of your project, such as
    "MyLovelyApplication" while the second one is the application deployment
    file. 
    
    Several files get generated by this command. One C file per kind of
    process in your project (such as "master" and "slave") plus one C file
    for simulating your project. All those files are (or should ;) described
    in section \ref GRAS_main_generation.
    
    The most intersting file in this context is
    [project_name].Makefile.local (you can safely ignore the others for
    now). To use it, simply type (from your project main directory):
    
\verbatim GRAS_ROOT=/path/to/simgrid/installation make -f [project_name].Makefile.local
\endverbatim
    
    And that's it, all the binaries are built and linked against the correct
    libraries.
    
    \section GRAS_compile_remote Distribution and remote compilation of GRAS projects
    
    Actually, there is two somehow parallel ways to do so since both Arnaud
    and Martin gave it a try. Merging both approaches is underway. As usual,
    if you want to help, you're welcome ;)
    
*/

#####################################################################
#########################  EXAMPLES #################################
#####################################################################

---------------------------------------------------------------------
------------------------- Ping Pong ---------------------------------
---------------------------------------------------------------------

/** \page GRAS_ex_ping The classical Ping-Pong in GRAS

    <center><table><tr><td><b>Top</b>    <td> [\ref index]::[\ref GRAS_API]
                   <tr><td><b>Prev</b>   <td> [\ref GRAS_compile]
                   <tr><td><b>Next</b>   <td> [\ref GRAS_ex_mmrpc]            </table></center>

    This example implements the very classical ping-pong in GRAS. It
    involves a client (initiating the ping-pong) and a server (answering to 
    client's requests).

    It works the following way:
     - Both the client and the server register all needed messages
     - The server registers a callback to the ping message, which sends pong
       to the expeditor
     - The client sends the ping message to the server, and waits for the
       pong message as an answer.
 
    This example resides in the <b>examples/gras/ping/ping.c</b> file. Yes, both
    the code of the client and of the server is placed in the same file. See
    the \ref GRAS_main_generation section if wondering.

    \section GRAS_ex_ping_toc Table of contents of the ping example
      - \ref GRAS_ex_ping_common
        - \ref GRAS_ex_ping_initial
        - \ref GRAS_ex_ping_register
      - \ref GRAS_ex_ping_server
        - \ref GRAS_ex_ping_serdata
        - \ref GRAS_ex_ping_sercb
        - \ref GRAS_ex_ping_sermain
      - \ref GRAS_ex_ping_client
        - \ref GRAS_ex_ping_climain
        
    <hr>

    \dontinclude gras/ping/ping.c
    
    \section GRAS_ex_ping_common 1) Common code to the client and the server 
    
    \subsection GRAS_ex_ping_initial 1.a) Initial settings
    
    Let's first load the gras header and declare a logging category (see
    \ref XBT_log for more info on logging).
    
    \skip include
    \until XBT_LOG

    \subsection GRAS_ex_ping_register 1.b) Register the messages
    
    This function, called by both the client and the server is in charge of
    declaring the existing messages to GRAS. Since the payload does not
    involve any newly created types but only int, this is quite easy. 
    (to exchange more complicated types, see \ref GRAS_dd or 
    \ref GRAS_ex_mmrpc for an example).
    
    \skip register_messages
    \until }

    [Back to \ref GRAS_ex_ping_toc]

    \section GRAS_ex_ping_server 2) Server's code
    
    \subsection GRAS_ex_ping_serdata 2.a) The server's globals

    In order to ensure the communication between the "main" and the callback
    of the server, we need to declare some globals. We have to put them in a
    struct definition so that they can be handled properly in GRAS (see the
    \ref GRAS_globals for more info).

    \skip typedef struct
    \until }
    
    \subsection GRAS_ex_ping_sercb 2.b) The callback to the ping message

    Here is the callback run when the server receives any ping message (this
    will be registered later by the server).
    
    \skip server_cb_ping_handler
    \until end_of_server_cb_ping_handler

    \subsection GRAS_ex_ping_sermain 2.c) The "main" of the server
    
    This is the "main" of the server. As explained in the \ref
    GRAS_main_generation, you must not write any main()
    function yourself. Instead, you just have to write a regular function
    like this one which will act as a main.
    
    \skip server
    \until end_of_server

    [Back to \ref GRAS_ex_ping_toc]
    
    \section GRAS_ex_ping_client 3) Client's code
    
    \subsection GRAS_ex_ping_climain 3.a) Client's "main" function
    
    This function is quite straightforward, and the inlined comments should
    be enough to understand it.

    \skip client
    \until end_of_client

    [Back to \ref GRAS_ex_ping_toc]
 */

---------------------------------------------------------------------
-------------------------- MM RPC -----------------------------------
---------------------------------------------------------------------

/** \page GRAS_ex_mmrpc A simple RPC for matrix multiplication

    <center><table><tr><td><b>Top</b>    <td> [\ref index]::[\ref GRAS_API]
                   <tr><td><b>Prev</b>   <td> [\ref GRAS_ex_ping]
                   <tr><td><b>Next</b>   <td> [\ref GRAS_ex_timer]            </table></center>

    This example implements a remote matrix multiplication. It involves a client 
    (creating the matrices and sending the multiplications requests) and a server 
    (computing the multiplication on client's behalf).

    This example also constitutes a more advanced example of data description 
    mechanisms, since the message payload type is a bit more complicated than in 
    other examples such as the ping one (\ref GRAS_ex_ping).

    It works the following way (not very different from the ping example):
     - Both the client and the server register all needed messages and datatypes
     - The server registers a callback to the "request" message, which computes
       what needs to be and returns the result to the expeditor.
     - The client creates two matrices, ask for their multiplication and check 
       the server's answer.
 
    This example resides in the <b>examples/gras/mmrpc/mmrpc.c</b> file. (See
    the \ref GRAS_main_generation section if wondering why both the server
    and the client live in the same source file)

    \section GRAS_ex_mmrpc_toc Table of contents of the mmrpc example
      - \ref GRAS_ex_mmrpc_common
        - \ref GRAS_ex_mmrpc_initial
        - \ref GRAS_ex_mmrpc_dataregister
        - \ref GRAS_ex_mmrpc_msgregister
      - \ref GRAS_ex_mmrpc_server
        - \ref GRAS_ex_mmrpc_sercb
        - \ref GRAS_ex_mmrpc_sermain
      - \ref GRAS_ex_mmrpc_client
        - \ref GRAS_ex_mmrpc_climain
        
    <hr>

    \dontinclude gras/mmrpc/mmrpc.c
    
    \section GRAS_ex_mmrpc_common 1) Common code to the client and the server 
    
    \subsection GRAS_ex_mmrpc_initial 1.a) Initial settings
    
    Let's first load the gras header, specify the matrix size and declare a 
    logging category (see \ref XBT_log for more info on logging).
    
    \skip include
    \until XBT_LOG

    \subsection GRAS_ex_mmrpc_dataregister 1.b) Register the data types

    The messages involved in this example do use structures as payload, 
    so we have to declare it to GRAS. Hopefully, this can be done easily by enclosing 
    the structure declaration within a \ref GRAS_DEFINE_TYPE macro call. It will then copy this 
    declaration into an hidden string variable, which can be automatically parsed at 
    run time. Of course, the declaration is also copied unmodified by this macro, so that it
    gets parsed by the compiler also. 

    There is some semantic that GRAS cannot guess alone and you need to  <i>annotate</i>
    your declaration to add some. For example, the ctn pointer can be a reference to an 
    object or a whole array (in which case you also has to specify its size). This is done 
    with the GRAS_ANNOTE call. It is removed from the text passed to the compiler, but it helps
    GRAS getting some information about the semantic of your data. Here, it says that \a ctn is an 
    array, which size is the result of the operation \a rows * \a cols (with \a rows and \a cols 
    being the other fields of the structure). 

    Please note that this annotation mechanism is not as robust and cool as this example seems to 
    imply. If you want to use it yourself, you'd better use the exact right syntax, which is 
    detailed in the \ref GRAS_dd section.

    \skip GRAS_DEFINE_TYPE
    \until matrix_t

    \subsection GRAS_ex_mmrpc_msgregister 1.c) Register the messages
    
    This function, called by both the client and the server is in charge of
    declaring the existing messages to GRAS. Note the use of the \ref gras_datadesc_by_symbol 
    function to parse and retrieve the structure declaration which were passed to \ref GRAS_DEFINE_TYPE 
    above. 

    The datatype description builded that way can then be used to build an array datatype or 
    to declare messages.
    
    \skip register_messages
    \until }

    [Back to \ref GRAS_ex_mmrpc_toc]

    \section GRAS_ex_mmrpc_server 2) Server's code
    
    \subsection GRAS_ex_mmrpc_sercb 2.a) The callback to the mmrpc message

    Here is the callback run when the server receives any mmrpc message (this
    will be registered later by the server). Note the way we get the message 
    payload. In the ping example, there was one additional level of pointer 
    indirection (see \ref GRAS_ex_ping_sercb). This is because the payload is
    an array here (ie a pointer) whereas it is a scalar in the ping example.
    
    \skip server_cb_request_handler
    \until end_of_server_cb_request_handler

    \subsection GRAS_ex_mmrpc_sermain 2.b) The "main" of the server
    
    This is the "main" of the server. As explained in the \ref
    GRAS_main_generation, you must not write any main()
    function yourself. Instead, you just have to write a regular function
    like this one which will act as a main.
    
    \skip server
    \until end_of_server
    
    [Back to \ref GRAS_ex_mmrpc_toc]

    \section GRAS_ex_mmrpc_client 3) Client's code
    
    \subsection GRAS_ex_mmrpc_climain 3.a) Client's "main" function
    
    This function is quite straightforward, and the inlined comments should
    be enough to understand it.

    \skip client
    \until end_of_client

    [Back to \ref GRAS_ex_mmrpc_toc]
  */

---------------------------------------------------------------------
---------------------------- Timers ---------------------------------
---------------------------------------------------------------------

/** \page GRAS_ex_timer Some timer games

    <center><table><tr><td><b>Top</b>    <td> [\ref index]::[\ref GRAS_API]
                   <tr><td><b>Prev</b>   <td> [\ref GRAS_ex_mmrpc]
                   <tr><td>   Next       <td>            </table></center>

    This example fools around with the GRAS timers (\ref GRAS_timer). It is
    mainly a regression test, since it uses almost all timer features.
    
    The main program registers a repetititive task and a delayed one, and
    then loops until the <tt>still_to_do</tt> variables of its globals reach
    0. The delayed task set it to 5, and the repetititive one decrease it
    each time. Here is an example of output:
\verbatim Initialize GRAS
 Initialize XBT
 [1108335471] Programming the repetitive_action with a frequency of 1.000000 sec
 [1108335471] Programming the delayed_action for after 2.000000 sec
 [1108335471] Have a rest
 [1108335472] Canceling the delayed_action.
 [1108335472] Re-programming the delayed_action for after 2.000000 sec
 [1108335472] Repetitive_action has nothing to do yet
 [1108335473] Repetitive_action has nothing to do yet
 [1108335473] delayed_action setting globals->still_to_do to 5
 [1108335474] repetitive_action decrementing globals->still_to_do. New value: 4
 [1108335475] repetitive_action decrementing globals->still_to_do. New value: 3
 [1108335476] repetitive_action decrementing globals->still_to_do. New value: 2
 [1108335477] repetitive_action decrementing globals->still_to_do. New value: 1
 [1108335478] repetitive_action decrementing globals->still_to_do. New value: 0
 Exiting GRAS\endverbatim

    Source code:
     - \ref GRAS_ex_timer_decl
     - \ref GRAS_ex_timer_delay
     - \ref GRAS_ex_timer_repeat
     - \ref GRAS_ex_timer_main

    \dontinclude timer.c
    
    \section GRAS_ex_timer_decl   1. Declarations and headers
    \skip include
    \until my_globals
    
    \section GRAS_ex_timer_delay  2. Source code of the delayed action
    \skip repetitive_action
    \until end_of_repetitive_action
    
    \section GRAS_ex_timer_repeat 3. Source code of the repetitive action
    \skip delayed_action
    \until end_of_delayed_action
    
    \section GRAS_ex_timer_main   4. Source code of main function
    \skip client
    \until end_of_client
*/