Integrate the tutorial on modeling I/O to the flow of existing pages

diff --git a/docs/source/Platform_howtos.rst b/docs/source/Platform_howtos.rst
index 4ff8021..0bf8ac9 100644 (file)
--- a/docs/source/Platform_howtos.rst
+++ b/docs/source/Platform_howtos.rst
@@ -101,7 +101,7 @@ https://hal.inria.fr/hal-01523608
 
 .. _howto_churn:
 
 
 .. _howto_churn:
 

-Modeling Churn (e.g., in P2P)
+Modeling churn (e.g., in P2P)

 *****************************
 
 One of the biggest challenges in P2P settings is to cope with the
 *****************************
 
 One of the biggest challenges in P2P settings is to cope with the


@@ -137,10 +137,10 @@ listing the events, or something else if you wish.
 
 .. _howto_multicore:
 
 
 .. _howto_multicore:
 

-Modeling Multicore Machines
+Modeling multicore machines

 ***************************
 
 ***************************
 

-Default Model
+Default model

 =============
 
 Multicore machines are very complex, and there are many ways to model
 =============
 
 Multicore machines are very complex, and there are many ways to model


@@ -186,7 +186,7 @@ and unconvincing, but some users achieved very realistic simulations
 of multicore and GPU machines this way.
 
 Modeling machine boot and shutdown periods
 of multicore and GPU machines this way.
 
 Modeling machine boot and shutdown periods

-********************************************
+******************************************

 
 When a physical host boots up, a lot of things happen. It takes time
 during which the machine is not usable but dissipates energy, and
 
 When a physical host boots up, a lot of things happen. It takes time
 during which the machine is not usable but dissipates energy, and


@@ -288,8 +288,4 @@ So the final platform for our example becomes :
         <link_ctn id="link2"/>
     </route>
 
         <link_ctn id="link2"/>
     </route>
 

-
-:ref:`Modeling I/O: the realistic way<howto_disk>`
-*****************
-
-Tutorial available at: :ref:`howto_disk`.
+.. include:: tuto_disk/analysis.irst

diff --git a/docs/source/tuto_disk/analysis.rst b/docs/source/tuto_disk/analysis.irst
similarity index 96%
rename from docs/source/tuto_disk/analysis.rst
rename to docs/source/tuto_disk/analysis.irst
index e07daf0..dfc8912 100644 (file)
--- a/docs/source/tuto_disk/analysis.rst
+++ b/docs/source/tuto_disk/analysis.irst
@@ -1,13 +1,11 @@
-
-
-
+.. This file has "irst" as an extension to ensure that it's not parsed by sphinx as is. Instead, it's included in another file that is parsed.

 .. _howto_disk:
 
 Modeling I/O: the realistic way
 .. _howto_disk:
 
 Modeling I/O: the realistic way

--------------------------------
+*******************************

 
 Introduction
 
 Introduction

-~~~~~~~~~~~~
+============

 
 This tutorial presents how to perform faithful IO experiments in
 SimGrid. It is based on the paper "Adding Storage Simulation
 
 This tutorial presents how to perform faithful IO experiments in
 SimGrid. It is based on the paper "Adding Storage Simulation


@@ -38,7 +36,7 @@ in IO operations (Fig. 5 to 7).
   evolved and may not be available anymore.
 
 Running this tutorial
   evolved and may not be available anymore.
 
 Running this tutorial

-^^^^^^^^^^^^^^^^^^^^^
+---------------------

 
 A Dockerfile is available in ``docs/source/tuto_disk``. It allows you to
 re-run this tutorial. For that, build the image and run the container:
 
 A Dockerfile is available in ``docs/source/tuto_disk``. It allows you to
 re-run this tutorial. For that, build the image and run the container:


@@ -48,12 +46,12 @@ re-run this tutorial. For that, build the image and run the container:
 - ``docker run -it tuto_disk``
 
 Analyzing the experimental data
 - ``docker run -it tuto_disk``
 
 Analyzing the experimental data

-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+===============================

 
 We start by analyzing and extracting the real data available.
 
 Scripts
 
 We start by analyzing and extracting the real data available.
 
 Scripts

-^^^^^^^
+-------

 
 We use a special method to create non-uniform histograms to represent
 the noise in IO operations. 
 
 We use a special method to create non-uniform histograms to represent
 the noise in IO operations. 


@@ -64,7 +62,7 @@ here.
 Copied from: `https://rdrr.io/github/dlebauer/pecan-priors/src/R/plots.R <https://rdrr.io/github/dlebauer/pecan-priors/src/R/plots.R>`_
 
 Data preparation
 Copied from: `https://rdrr.io/github/dlebauer/pecan-priors/src/R/plots.R <https://rdrr.io/github/dlebauer/pecan-priors/src/R/plots.R>`_
 
 Data preparation

-^^^^^^^^^^^^^^^^
+----------------

 
 Some initial configurations/list of packages.
 
 
 Some initial configurations/list of packages.
 


@@ -227,10 +225,10 @@ Preparing data for concurrent analysis.
     dfrange$Jobs=16
 
 Griffon (SATA)
     dfrange$Jobs=16
 
 Griffon (SATA)

-^^^^^^^^^^^^^^
+--------------

 
 Modeling resource sharing w/ concurrent access
 
 Modeling resource sharing w/ concurrent access

-::::::::::::::::::::::::::::::::::::::::::::::
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

 
 This figure presents the overall performance of IO operation with
 concurrent access to the disk. Note that the image is different
 
 This figure presents the overall performance of IO operation with
 concurrent access to the disk. Note that the image is different


@@ -249,10 +247,10 @@ available data to obtain exaclty the same results.
       geom_errorbar(data=dfd, aes(x=Jobs, ymin=BW-ci, ymax=BW+ci),color="black",width=.6) +
       xlab("Number of concurrent operations") + ylab("Aggregated Bandwidth (MiB/s)")  + guides(color=FALSE)  + xlim(0,NA) + ylim(0,NA)
 
       geom_errorbar(data=dfd, aes(x=Jobs, ymin=BW-ci, ymax=BW+ci),color="black",width=.6) +
       xlab("Number of concurrent operations") + ylab("Aggregated Bandwidth (MiB/s)")  + guides(color=FALSE)  + xlim(0,NA) + ylim(0,NA)
 

-.. image:: fig/griffon_deg.png
+.. image:: tuto_disk/fig/griffon_deg.png

 
 Read
 
 Read

-''''
+""""

 
 Getting read data for Griffon from 1 to 15 concurrent reads.
 
 
 Getting read data for Griffon from 1 to 15 concurrent reads.
 


@@ -276,7 +274,7 @@ Getting read data for Griffon from 1 to 15 concurrent reads.
     }
 
 Write
     }
 
 Write

-'''''
+"""""

 
 Same for write operations.
 
 
 Same for write operations.
 


@@ -301,7 +299,7 @@ Same for write operations.
     }
 
 Modeling read/write bandwidth variability
     }
 
 Modeling read/write bandwidth variability

-:::::::::::::::::::::::::::::::::::::::::
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

 
 Fig.5 in the paper presents the noise in the read/write operations in
 the Griffon SATA disk.
 
 Fig.5 in the paper presents the noise in the read/write operations in
 the Griffon SATA disk.


@@ -312,7 +310,7 @@ effective bandwidth. However, in this tutorial, we use dhist
 more precise information over the highly dense areas around the mean.
 
 Read
 more precise information over the highly dense areas around the mean.
 
 Read

-''''
+""""

 
 First, we present the histogram for read operations.
 
 
 First, we present the histogram for read operations.
 


@@ -321,7 +319,7 @@ First, we present the histogram for read operations.
     griffon_read = df %>% filter(grepl("^Griffon", Cluster)) %>% filter(Operation == "Read") %>% select(Bwi)
     dhist(1/griffon_read$Bwi)
 
     griffon_read = df %>% filter(grepl("^Griffon", Cluster)) %>% filter(Operation == "Read") %>% select(Bwi)
     dhist(1/griffon_read$Bwi)
 

-.. image:: fig/griffon_read_dhist.png
+.. image:: tuto_disk/fig/griffon_read_dhist.png

 
 Saving it to be exported in json format.
 
 
 Saving it to be exported in json format.
 


@@ -355,7 +353,7 @@ Saving it to be exported in json format.
     }
 
 Write
     }
 
 Write

-'''''
+"""""

 
 Same analysis for write operations.
 
 
 Same analysis for write operations.
 


@@ -364,7 +362,7 @@ Same analysis for write operations.
     griffon_write = df %>% filter(grepl("^Griffon", Cluster)) %>% filter(Operation == "Write") %>% select(Bwi)
     dhist(1/griffon_write$Bwi)
 
     griffon_write = df %>% filter(grepl("^Griffon", Cluster)) %>% filter(Operation == "Write") %>% select(Bwi)
     dhist(1/griffon_write$Bwi)
 

-.. image:: fig/griffon_write_dhist.png
+.. image:: tuto_disk/fig/griffon_write_dhist.png

 
 .. code:: R
 
 
 .. code:: R
 


@@ -401,15 +399,15 @@ Same analysis for write operations.
     }
 
 Edel (SSD)
     }
 
 Edel (SSD)

-^^^^^^^^^^
+----------

 
 This section presents the exactly same analysis for the Edel SSDs.
 
 Modeling resource sharing w/ concurrent access
 
 This section presents the exactly same analysis for the Edel SSDs.
 
 Modeling resource sharing w/ concurrent access

-::::::::::::::::::::::::::::::::::::::::::::::
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

 
 Read
 
 Read

-''''
+""""

 
 Getting read data for Edel from 1 to 15 concurrent operations.
 
 
 Getting read data for Edel from 1 to 15 concurrent operations.
 


@@ -450,7 +448,7 @@ Getting read data for Edel from 1 to 15 concurrent operations.
     }
 
 Write
     }
 
 Write

-'''''
+"""""

 
 Same for write operations.
 
 
 Same for write operations.
 


@@ -493,17 +491,17 @@ Same for write operations.
     }
 
 Modeling read/write bandwidth variability
     }
 
 Modeling read/write bandwidth variability

-:::::::::::::::::::::::::::::::::::::::::
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

 
 Read
 
 Read

-''''
+""""

 
 .. code:: R
 
     edel_read = df %>% filter(grepl("^Edel", Cluster)) %>% filter(Operation == "Read") %>% select(Bwi)
     dhist(1/edel_read$Bwi)
 
 
 .. code:: R
 
     edel_read = df %>% filter(grepl("^Edel", Cluster)) %>% filter(Operation == "Read") %>% select(Bwi)
     dhist(1/edel_read$Bwi)
 

-.. image:: fig/edel_read_dhist.png
+.. image:: tuto_disk/fig/edel_read_dhist.png

 
 Saving it to be exported in json format.
 
 
 Saving it to be exported in json format.
 


@@ -555,7 +553,7 @@ Saving it to be exported in json format.
     }
 
 Write
     }
 
 Write

-'''''
+"""""

 
 .. code:: R
 
 
 .. code:: R
 


@@ -563,7 +561,7 @@ Write
     edel_write = df %>% filter(grepl("^Edel", Cluster)) %>% filter(Operation == "Write") %>% select(Bwi)
     dhist(1/edel_write$Bwi)
 
     edel_write = df %>% filter(grepl("^Edel", Cluster)) %>% filter(Operation == "Write") %>% select(Bwi)
     dhist(1/edel_write$Bwi)
 

-.. image:: fig/edel_write_dhist.png
+.. image:: tuto_disk/fig/edel_write_dhist.png

 
 Saving it to be exported later.
 
 
 Saving it to be exported later.
 


@@ -620,7 +618,7 @@ Saving it to be exported later.
     }
 
 Exporting to JSON
     }
 
 Exporting to JSON

-~~~~~~~~~~~~~~~~~
+=================

 
 Finally, let's save it to a file to be opened by our simulator.
 
 
 Finally, let's save it to a file to be opened by our simulator.
 


@@ -630,14 +628,14 @@ Finally, let's save it to a file to be opened by our simulator.
     cat(json, file="IO_noise.json")
 
 Injecting this data in SimGrid
     cat(json, file="IO_noise.json")
 
 Injecting this data in SimGrid

-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+==============================

 
 To mimic this behavior in SimGrid, we use two features in the platform
 description: non-linear sharing policy and bandwidth factors. For more
 details, please see the source code in ``tuto_disk.cpp``.
 
 Modeling resource sharing w/ concurrent access
 
 To mimic this behavior in SimGrid, we use two features in the platform
 description: non-linear sharing policy and bandwidth factors. For more
 details, please see the source code in ``tuto_disk.cpp``.
 
 Modeling resource sharing w/ concurrent access

-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+----------------------------------------------

 
 The ``set_sharing_policy`` method allows the user to set a callback to
 dynamically change the disk capacity. The callback is called each time
 
 The ``set_sharing_policy`` method allows the user to set a callback to
 dynamically change the disk capacity. The callback is called each time


@@ -658,7 +656,7 @@ capacity.
     disk->set_sharing_policy(sg4::Disk::Operation::READ, sg4::Disk::SharingPolicy::NONLINEAR, &disk_dynamic_sharing);
 
 Modeling read/write bandwidth variability
     disk->set_sharing_policy(sg4::Disk::Operation::READ, sg4::Disk::SharingPolicy::NONLINEAR, &disk_dynamic_sharing);
 
 Modeling read/write bandwidth variability

-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+-----------------------------------------

 
 The noise in I/O operations can be obtained by applying a factor to
 the I/O bandwidth of the disk. This factor is applied when we update
 
 The noise in I/O operations can be obtained by applying a factor to
 the I/O bandwidth of the disk. This factor is applied when we update


@@ -681,7 +679,7 @@ write). It must return a multiply factor (e.g. 1.0 for doing nothing).
     disk->set_factor_cb(&disk_variability);
 
 Running our simulation
     disk->set_factor_cb(&disk_variability);
 
 Running our simulation

-^^^^^^^^^^^^^^^^^^^^^^
+----------------------

 
 The binary was compiled in the provided docker container.
 
 
 The binary was compiled in the provided docker container.
 


@@ -690,7 +688,7 @@ The binary was compiled in the provided docker container.
     ./tuto_disk > ./simgrid_disk.csv
 
 Analyzing the SimGrid results
     ./tuto_disk > ./simgrid_disk.csv
 
 Analyzing the SimGrid results

-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+=============================

 
 The figure below presents the results obtained by SimGrid.
 
 
 The figure below presents the results obtained by SimGrid.
 


@@ -717,7 +715,7 @@ the real platform.
         facet_wrap(disk~op,ncol=2,scale="free_y")+ # ) + #
         xlab("Number of concurrent operations") + ylab("Aggregated Bandwidth (MiB/s)")  + guides(color=FALSE)  + xlim(0,NA) + ylim(0,NA)
 
         facet_wrap(disk~op,ncol=2,scale="free_y")+ # ) + #
         xlab("Number of concurrent operations") + ylab("Aggregated Bandwidth (MiB/s)")  + guides(color=FALSE)  + xlim(0,NA) + ylim(0,NA)
 

-.. image:: fig/simgrid_results.png
+.. image:: tuto_disk/fig/simgrid_results.png

 
 Note: The variability in griffon read operation seems to decrease when
 we have more concurrent operations. This is a particularity of the
 
 Note: The variability in griffon read operation seems to decrease when
 we have more concurrent operations. This is a particularity of the