SimGrid (3.28.1) NOT RELEASED YET (v3.29 expected September 22. 2021, 19:21 UTC)
-New features:
- - Non-linear resource sharing: allows to dynamically change the resource
- capacity. Users can change this capacity by setting a callback which will
- be called when SimGrid is sharing the resource between the active
- activities. An activity can be a task running, a read/write IO operation or a
- communication flow.
- - Callback signature: double (double capacity, int n_activities): given the
- current resource's capacity and number of activities running on it,
- returns the new capacity.
- - Note that this callback is in the critical path of the system solve and
- should not take long to execute.
- - Examples:
- - Disk: examples/cpp/io-degradation
- - Link: examples/cpp/network-nonlinear
- - CPU: examples/cpp/exec-cpu-nonlinear
+New modeling features:
+ - Non-linear resource sharing for decay models:
+ - The total capacity may depend on the number of concurrent usages
+ - For that, resources can take a callback that computes the capacity
+ depending on the idle capacity and the number of concurrent usages
+ - Examples (both cpp and python): io-degradation, network-nonlinear, exec-cpu-nonlinear
+
- Dynamic factors for CPU and disk: similarly to dynamic network factors,
allows the user to set a callback which can affect the progress of activities
(multiplicative factor applied when updating the amount of work remaining).
Python:
- Added support to programmatic platform creation in Python.
- - Several new methods were added to Python interface. For more details, see
- the corresponding documentation.
- - New python examples:
- - Clusters: examples/python/clusters-multicpu
- - Disk: examples/python/io-degradation
- - CPU: examples/python/exec-cpu-nonlienar
- - Link: examples/python/network-nonlinear
+ Example: examples/python/clusters-multicpu
S4U:
- - New: s4u::Disk::set_sharing_policy() and s4u::Host::set_sharing_policy().
- Allows the configuration of non-linear resource sharing for hosts and
- disks.
+ - Disk and Host now have a set_sharing_policy() too, for non-linear sharing.
+ This can only be set through the API, not through XML files.
SMPI:
- TI Tracing/Replay:
- - Multiple fixes to ensure reproducibility of tracing
- - scan/excan can now be replayed
- - wait action now uses ranks and not pid, as the other ones.
- - smpi/init and smpi/finalization-barrier are now valid for replays.
+ - Multiple fixes to ensure reproducibility of tracing
+ - scan/excan can now be replayed
+ - wait action now uses ranks and not pid, as the other ones.
+ - smpi/init and smpi/finalization-barrier are now valid for replays.
Documentation:
* New section "Release Notes" documenting recent and current developments.
- * New section "Modeling I/O: the realistic way" presenting how to properly
- model disks in SimGrid.
+ * New section "Modeling I/O: the realistic way" presenting how to properly model disks in SimGrid.
* Improvements in API Reference for C++ and Python interfaces.
ns-3 model:
include docs/source/intl.rst
include docs/source/tuto_disk/CMakeLists.txt
include docs/source/tuto_disk/Dockerfile
+include docs/source/tuto_disk/analysis.irst
include docs/source/tuto_disk/analysis.org
-include docs/source/tuto_disk/analysis.rst
include docs/source/tuto_disk/fig/edel_read_dhist.png
include docs/source/tuto_disk/fig/edel_write_dhist.png
include docs/source/tuto_disk/fig/griffon_deg.png
\ V / __/ | \__ \ | (_) | | | | ___) | / __/ \__, |
\_/ \___|_| |___/_|\___/|_| |_| |____(_)_____| /_/
(unreleased)
+
+The "Give peace a chance" release (International Day of Peace)
+
+ * Python bindings for the platform creation API
+ * Introduce non-linear ressource sharing, allowing decay models
+ * New documentation section on realistic I/O modeling
_ _____ ____ ___
__ _____ _ __ ___(_) ___ _ __ |___ / |___ \( _ )
--- /dev/null
+# Nix package for SimGrid's sphinx documentation.
+# Example usage: nix-build ./default.nix -A simgrid-doc && firefox result/index.html
+{ pkgs ? import (fetchTarball {
+ url = "https://github.com/NixOS/nixpkgs/archive/21.05.tar.gz";
+ sha256 = "1ckzhh24mgz6jd1xhfgx0i9mijk6xjqxwsshnvq789xsavrmsc36";
+ }) {}
+}:
+
+let
+ pythonPackages = pkgs.python3Packages;
+ buildPythonPackage = pythonPackages.buildPythonPackage;
+
+ self = rec {
+ # the desired package
+ simgrid-doc = pkgs.stdenv.mkDerivation rec {
+ name = "simgrid-doc";
+ src = ./..;
+ buildInputs = [
+ pkgs.doxygen
+ pythonPackages.sphinx
+ pythonPackages.sphinx_rtd_theme
+ pythonPackages.breathe
+ javasphinx
+ sphinx-tabs
+ ];
+ phases = [ "unpackPhase" "buildPhase" "installPhase" ];
+ buildPhase = ''
+ cd docs
+ rm -rf build # this is not done in your directory, this is on the copy made by the Nix build daemon
+ sphinx-build -b html source build
+ '';
+ installPhase = ''
+ mkdir -p $out
+ mv build/* $out/
+ '';
+ };
+
+ # sphinx extensions (and their dependencies) shenanigans
+ javalang = buildPythonPackage rec {
+ pname = "javalang";
+ version = "0.13.0";
+ src = pythonPackages.fetchPypi {
+ inherit pname version;
+ sha256 = "0k22ldm4xn0sb26qclp5zz0fc9dy589zvvby5ba1d0d5h2jab08n";
+ };
+ buildInputs = with pythonPackages; [ six ];
+ doCheck = false;
+ };
+
+ javasphinx = buildPythonPackage rec {
+ pname = "javasphinx";
+ version = "0.9.15+simgrid";
+ src = pkgs.fetchFromGitHub {
+ owner = "simgrid";
+ repo = pname;
+ rev = "659209069603a5f221596dd039e724cb89b31b82";
+ sha256 = "1nbz822zd2ikzzmpyqsrrpqvlpng72qvl86wcqfn89szbp85c20a";
+ };
+ propagatedBuildInputs = with pythonPackages; [
+ beautifulsoup4
+ docutils
+ javalang
+ future
+ lxml
+ sphinx
+ ];
+ };
+
+ sphinx-tabs = pythonPackages.buildPythonPackage rec {
+ pname = "sphinx-tabs";
+ version = "3.1.0";
+ name = "${pname}-${version}";
+
+ src = pythonPackages.fetchPypi {
+ inherit pname version;
+ sha256 = "0kv935qhml40mly33rk5am128g2ygqkfvizh33vf29hjkf32mvjy";
+ };
+
+ propagatedBuildInputs = with pythonPackages; [
+ docutils
+ pygments
+ sphinx
+ ];
+ };
+
+ };
+in
+ self
INPUT += ../../include/simgrid/semaphore.h
INPUT += ../../include/simgrid/vm.h
INPUT += ../../include/simgrid/zone.h
+INPUT += ../../include/xbt/dynar.h
+INPUT += ../../src/xbt/dynar.cpp
INPUT += ../../include/xbt/signal.hpp
INPUT += ../../src/msg/
INPUT += ../../src/plugins/
$ apt install simgrid-java # if you want to develop in Java
$ apt install python3-simgrid # if you want to develop in Python
+If you use the Nix_ package manager, the latest SimGrid release is packaged as ``simgrid`` in Nixpkgs_.
+Previous SimGrid versions are maintained in `NUR-Kapack`_ and are available
+pre-compiled in release and debug modes on the `capack cachix binary cache`_
+— refer to `NUR-Kapack's documentation`_ for usage instructions.
+
+If you use a pacman-based system (*e.g.*, Arch Linux and derived distributions),
+the latest SimGrid is available in the `simgrid AUR package`_
+— refer to `AUR official documentation`_ for installation instructions.
+
If you build pre-compiled packages for other distributions, drop us an
email.
+.. _Nix: https://nixos.org/
+.. _Nixpkgs: https://github.com/NixOS/nixpkgs
+.. _NUR-Kapack: https://github.com/oar-team/nur-kapack
+.. _capack cachix binary cache: https://app.cachix.org/cache/capack
+.. _NUR-Kapack's documentation: https://github.com/oar-team/nur-kapack
+.. _simgrid AUR package: https://aur.archlinux.org/packages/simgrid/
+.. _AUR official documentation: https://wiki.archlinux.org/title/Arch_User_Repository
+
.. _install_java_precompiled:
Stable Java Package
You should obviously install the ``boost-context`` library on your
machine, for example with ``apt``.
+.. _deprecation_policy:
+
Version numbering and deprecation
---------------------------------
.. _howto_churn:
-Modeling Churn (e.g., in P2P)
+Modeling churn (e.g., in P2P)
*****************************
One of the biggest challenges in P2P settings is to cope with the
.. _howto_multicore:
-Modeling Multicore Machines
+Modeling multicore machines
***************************
-Default Model
+Default model
=============
Multicore machines are very complex, and there are many ways to model
of multicore and GPU machines this way.
Modeling machine boot and shutdown periods
-********************************************
+******************************************
When a physical host boots up, a lot of things happen. It takes time
during which the machine is not usable but dissipates energy, and
<link_ctn id="link2"/>
</route>
-
-:ref:`Modeling I/O: the realistic way<howto_disk>`
-*****************
-
-Tutorial available at: :ref:`howto_disk`.
+.. include:: tuto_disk/analysis.irst
You can even specify a compile-time threshold that will completely remove every logging below the specified priority. Passing ``-DNDEBUG`` to cmake disables every logging of
priority below INFO while ``-DNLOG`` removes any logging at compile time. Note that using this feature may hinder the stability of SimGrid, as we consider the logs to be fast
enough to not thoughtfully test the case where they are removed at compile time.
+
+Dynamic arrays
+**************
+
+As SimGrid used to be written in pure C, it used to rely on custom data containers such as dynamic arrays and dictionnaries. Nowadays, the standard library of
+C++ is used internally, but some part of the interface still rely on the old containers, that are thus still available.
+
+.. warning::
+
+ You should probably not start a new project using these data structures, as we will :ref:`deprecate them from SimGrid <deprecation_policy>`
+ as soon as possible. Better implementations exist out there anyway, in particular if you're not writting pure C code.
+
+.. doxygentypedef:: xbt_dynar_t
+.. doxygentypedef:: const_xbt_dynar_t
+
+Creation and destruction
+========================
+
+.. doxygenfunction:: xbt_dynar_new
+.. doxygenfunction:: xbt_dynar_free
+.. doxygenfunction:: xbt_dynar_free_container
+.. doxygenfunction:: xbt_dynar_shrink
+
+Dynars as regular arrays
+========================
+
+.. doxygenfunction:: xbt_dynar_get_cpy
+.. doxygenfunction:: xbt_dynar_insert_at
+.. doxygenfunction:: xbt_dynar_remove_at
+.. doxygenfunction:: xbt_dynar_member
+.. doxygenfunction:: xbt_dynar_sort
+
+Dynar size
+==========
+
+.. doxygenfunction:: xbt_dynar_is_empty
+.. doxygenfunction:: xbt_dynar_length
+.. doxygenfunction:: xbt_dynar_reset
+
+Perl-like interface
+===================
+
+.. doxygenfunction:: xbt_dynar_push
+.. doxygenfunction:: xbt_dynar_pop
+.. doxygenfunction:: xbt_dynar_unshift
+.. doxygenfunction:: xbt_dynar_shift
+.. doxygenfunction:: xbt_dynar_map
+
+Direct content manipulation
+===========================
+
+Those functions do not retrieve the content, but only their address.
+
+.. doxygenfunction:: xbt_dynar_set_at_ptr
+.. doxygenfunction:: xbt_dynar_get_ptr
+.. doxygenfunction:: xbt_dynar_insert_at_ptr
+.. doxygenfunction:: xbt_dynar_push_ptr
+.. doxygenfunction:: xbt_dynar_pop_ptr
+
+Dynars of scalars
+=================
+
+While the other functions use a memcpy to retrieve the content into the user provided area, those ones use a
+regular affectation. It only works for scalar values, but should be a little faster.
+
+.. doxygendefine:: xbt_dynar_get_as
+.. doxygendefine:: xbt_dynar_set_as
+
+.. doxygendefine:: xbt_dynar_getlast_as
+.. doxygendefine:: xbt_dynar_getfirst_as
+.. doxygendefine:: xbt_dynar_push_as
+.. doxygendefine:: xbt_dynar_pop_as
+
+Iterating over Dynars
+=====================
+
+.. doxygendefine:: xbt_dynar_foreach
+
+Example with scalar values
+==========================
+
+.. code-block:: c
+
+ xbt_dynar_t d = xbt_dynar_new(sizeof(int), nullptr);
+
+ /* 1. Populate the dynar */
+ xbt_dynar_t d = xbt_dynar_new(sizeof(int), nullptr);
+ for (int cpt = 0; cpt < NB_ELEM; cpt++) {
+ xbt_dynar_push_as(d, int, cpt); /* This is faster (and possible only with scalars) */
+ /* xbt_dynar_push(d,&cpt); This would also work */
+ xbt_test_log("Push %d, length=%lu", cpt, xbt_dynar_length(d));
+ }
+
+ /* 2. Traverse manually the dynar */
+ for (cursor = 0; cursor < NB_ELEM; cursor++) {
+ int* iptr = (int*)xbt_dynar_get_ptr(d, cursor);
+ /* 1. Populate further the dynar */
+ for (int cpt = 0; cpt < NB_ELEM; cpt++) {
+ xbt_dynar_insert_at(d, cpt, &cpt);
+ xbt_test_log("Push %d, length=%lu", cpt, xbt_dynar_length(d));
+ }
+
+ /* 3. Traverse the dynar */
+ int cpt;
+ xbt_dynar_foreach(d, cursor, cpt) {
+ xbt_test_assert(cursor == (unsigned int) cpt, "The retrieved value is not the same than the injected one (%u!=%d)", cursor, cpt);
+ }
+
+ /* 4. Reset the values */
+ for (int i = 0; i < NB_ELEM; i++)
+ *(int*)xbt_dynar_get_ptr(d, i) = i;
+
+ /* 5. Shift all the values */
+ for (int i = 0; i < NB_ELEM; i++) {
+ int val;
+ xbt_dynar_shift(d, &val);
+ }
+ // the dynar is empty after shifting all values
+
+ /* 5. Free the resources */
+ xbt_dynar_free(&d);
+
+Example with pointed values
+===========================
+
+.. code-block:: c
+
+ xbt_dynar_t d = xbt_dynar_new(sizeof(char*), &xbt_free_ref);
+
+ /// Push/shift example
+ for (int i = 0; i < NB_ELEM; i++) {
+ char * val = xbt_strdup("hello");
+ xbt_dynar_push(d, &val);
+ }
+ for (int i = 0; i < NB_ELEM; i++) {
+ char *val;
+ xbt_dynar_shift(d, &val);
+ REQUIRE("hello" == val);
+ xbt_free(val);
+ }
+ xbt_dynar_free(&d);
+
+ /// Unshift, traverse and pop example
+ d = xbt_dynar_new(sizeof(char**), &xbt_free_ref);
+ for (int i = 0; i < NB_ELEM; i++) {
+ std::string val = std::to_string(i);
+ s1 = xbt_strdup(val.c_str());
+ xbt_dynar_unshift(d, &s1);
+ }
+ /* 2. Traverse the dynar with the macro */
+ xbt_dynar_foreach (d, iter, s1) {
+ std::string val = std::to_string(NB_ELEM - iter - 1);
+ REQUIRE(s1 == val); // The retrieved value is not the same than the injected one
+ }
+ /* 3. Traverse the dynar with the macro */
+ for (int i = 0; i < NB_ELEM; i++) {
+ std::string val = std::to_string(i);
+ xbt_dynar_pop(d, &s2);
+ REQUIRE(s2 == val); // The retrieved value is not the same than the injected one
+ xbt_free(s2);
+ }
+ /* 4. Free the resources */
+ xbt_dynar_free(&d);
.. doxygenfunction:: simgrid::s4u::Host::get_pstate() const
.. doxygenfunction:: simgrid::s4u::Host::get_pstate_count() const
- .. doxygenfunction:: simgrid::s4u::Host::get_pstate_speed(int pstate_index) const
- .. doxygenfunction:: simgrid::s4u::Host::set_pstate(int pstate_index)
+ .. doxygenfunction:: simgrid::s4u::Host::get_pstate_speed(unsigned long pstate_index) const
+ .. doxygenfunction:: simgrid::s4u::Host::set_pstate(unsigned long pstate_index)
.. doxygenfunction:: simgrid::s4u::Host::set_speed_profile(kernel::profile::Profile *p)
.. doxygenfunction:: simgrid::s4u::Host::set_state_profile(kernel::profile::Profile *p)
.. doxygenfunction:: sg_host_get_available_speed(const_sg_host_t host)
.. doxygenfunction:: sg_host_get_nb_pstates(const_sg_host_t host)
.. doxygenfunction:: sg_host_get_pstate(const_sg_host_t host)
- .. doxygenfunction:: sg_host_get_pstate_speed(const_sg_host_t host, int pstate_index)
- .. doxygenfunction:: sg_host_set_pstate(sg_host_t host, int pstate)
+ .. doxygenfunction:: sg_host_get_pstate_speed(const_sg_host_t host, unsigned long pstate_index)
+ .. doxygenfunction:: sg_host_set_pstate(sg_host_t host, unsigned long pstate)
Execution
---------
idiom on network zones because SimGrid does not allow (yet) to create nor
destroy resources once the simulation is started.
- .. automethod:: simgrid.NetZone.seal
+ .. doxygenfunction:: simgrid::s4u::NetZone::seal
.. group-tab:: Python
.. doxygenfunction:: simgrid::s4u::create_floyd_zone(const std::string& name)
.. doxygenfunction:: simgrid::s4u::create_vivaldi_zone(const std::string& name)
.. doxygenfunction:: simgrid::s4u::create_wifi_zone(const std::string& name)
- .. doxygenfunction:: simgrid::s4u::create_torus_zone(const std::string& name, const NetZone* parent, const std::vector<unsigned int>& dimensions, const ClusterCallbacks& set_callbacks, double bandwidth, double latency, Link::SharingPolicy sharing_policy)
+ .. doxygenfunction:: simgrid::s4u::create_torus_zone
.. doxygenfunction:: simgrid::s4u::create_fatTree_zone(const std::string& name, const NetZone* parent, const FatTreeParams& parameters, const ClusterCallbacks& set_callbacks, double bandwidth, double latency, Link::SharingPolicy sharing_policy)
.. doxygenfunction:: simgrid::s4u::create_dragonfly_zone(const std::string& name, const NetZone* parent, const DragonflyParams& parameters, const ClusterCallbacks& set_callbacks, double bandwidth, double latency, Link::SharingPolicy sharing_policy)
.. doxygenfunction:: simgrid::s4u::Comm::set_dst_data(void **buff, size_t size)
.. doxygenfunction:: simgrid::s4u::Comm::detach()
.. doxygenfunction:: simgrid::s4u::Comm::detach(void(*clean_function)(void *))
- .. doxygenfunction:: simgrid::s4u::Comm::set_payload_size(double bytes)
+ .. doxygenfunction:: simgrid::s4u::Comm::set_payload_size(uint64_t bytes)
.. doxygenfunction:: simgrid::s4u::Comm::set_rate(double rate)
.. doxygenfunction:: simgrid::s4u::Comm::set_src_data(void *buff)
.. doxygenfunction:: simgrid::s4u::Comm::set_src_data(void *buff, size_t size)
('cpp:identifier', 'this_actor'),
('cpp:identifier', 'uint64_t'),
('cpp:identifier', 'xbt'),
+ ('cpp:identifier', 'xbt_dynar_s'),
('cpp:identifier', 'xbt::Extendable<Actor>'),
('cpp:identifier', 'xbt::Extendable<Disk>'),
('cpp:identifier', 'xbt::Extendable<File>'),
-
-
-
+.. This file has "irst" as an extension to ensure that it's not parsed by sphinx as is. Instead, it's included in another file that is parsed.
.. _howto_disk:
Modeling I/O: the realistic way
--------------------------------
+*******************************
Introduction
-~~~~~~~~~~~~
+============
This tutorial presents how to perform faithful IO experiments in
SimGrid. It is based on the paper "Adding Storage Simulation
evolved and may not be available anymore.
Running this tutorial
-^^^^^^^^^^^^^^^^^^^^^
+---------------------
A Dockerfile is available in ``docs/source/tuto_disk``. It allows you to
re-run this tutorial. For that, build the image and run the container:
- ``docker run -it tuto_disk``
Analyzing the experimental data
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+===============================
We start by analyzing and extracting the real data available.
Scripts
-^^^^^^^
+-------
We use a special method to create non-uniform histograms to represent
the noise in IO operations.
Copied from: `https://rdrr.io/github/dlebauer/pecan-priors/src/R/plots.R <https://rdrr.io/github/dlebauer/pecan-priors/src/R/plots.R>`_
Data preparation
-^^^^^^^^^^^^^^^^
+----------------
Some initial configurations/list of packages.
dfrange$Jobs=16
Griffon (SATA)
-^^^^^^^^^^^^^^
+--------------
Modeling resource sharing w/ concurrent access
-::::::::::::::::::::::::::::::::::::::::::::::
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This figure presents the overall performance of IO operation with
concurrent access to the disk. Note that the image is different
geom_errorbar(data=dfd, aes(x=Jobs, ymin=BW-ci, ymax=BW+ci),color="black",width=.6) +
xlab("Number of concurrent operations") + ylab("Aggregated Bandwidth (MiB/s)") + guides(color=FALSE) + xlim(0,NA) + ylim(0,NA)
-.. image:: fig/griffon_deg.png
+.. image:: tuto_disk/fig/griffon_deg.png
Read
-''''
+""""
Getting read data for Griffon from 1 to 15 concurrent reads.
}
Write
-'''''
+"""""
Same for write operations.
}
Modeling read/write bandwidth variability
-:::::::::::::::::::::::::::::::::::::::::
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Fig.5 in the paper presents the noise in the read/write operations in
the Griffon SATA disk.
more precise information over the highly dense areas around the mean.
Read
-''''
+""""
First, we present the histogram for read operations.
griffon_read = df %>% filter(grepl("^Griffon", Cluster)) %>% filter(Operation == "Read") %>% select(Bwi)
dhist(1/griffon_read$Bwi)
-.. image:: fig/griffon_read_dhist.png
+.. image:: tuto_disk/fig/griffon_read_dhist.png
Saving it to be exported in json format.
}
Write
-'''''
+"""""
Same analysis for write operations.
griffon_write = df %>% filter(grepl("^Griffon", Cluster)) %>% filter(Operation == "Write") %>% select(Bwi)
dhist(1/griffon_write$Bwi)
-.. image:: fig/griffon_write_dhist.png
+.. image:: tuto_disk/fig/griffon_write_dhist.png
.. code:: R
}
Edel (SSD)
-^^^^^^^^^^
+----------
This section presents the exactly same analysis for the Edel SSDs.
Modeling resource sharing w/ concurrent access
-::::::::::::::::::::::::::::::::::::::::::::::
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Read
-''''
+""""
Getting read data for Edel from 1 to 15 concurrent operations.
}
Write
-'''''
+"""""
Same for write operations.
}
Modeling read/write bandwidth variability
-:::::::::::::::::::::::::::::::::::::::::
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Read
-''''
+""""
.. code:: R
edel_read = df %>% filter(grepl("^Edel", Cluster)) %>% filter(Operation == "Read") %>% select(Bwi)
dhist(1/edel_read$Bwi)
-.. image:: fig/edel_read_dhist.png
+.. image:: tuto_disk/fig/edel_read_dhist.png
Saving it to be exported in json format.
}
Write
-'''''
+"""""
.. code:: R
edel_write = df %>% filter(grepl("^Edel", Cluster)) %>% filter(Operation == "Write") %>% select(Bwi)
dhist(1/edel_write$Bwi)
-.. image:: fig/edel_write_dhist.png
+.. image:: tuto_disk/fig/edel_write_dhist.png
Saving it to be exported later.
}
Exporting to JSON
-~~~~~~~~~~~~~~~~~
+=================
Finally, let's save it to a file to be opened by our simulator.
cat(json, file="IO_noise.json")
Injecting this data in SimGrid
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+==============================
To mimic this behavior in SimGrid, we use two features in the platform
description: non-linear sharing policy and bandwidth factors. For more
details, please see the source code in ``tuto_disk.cpp``.
Modeling resource sharing w/ concurrent access
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+----------------------------------------------
The ``set_sharing_policy`` method allows the user to set a callback to
dynamically change the disk capacity. The callback is called each time
disk->set_sharing_policy(sg4::Disk::Operation::READ, sg4::Disk::SharingPolicy::NONLINEAR, &disk_dynamic_sharing);
Modeling read/write bandwidth variability
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+-----------------------------------------
The noise in I/O operations can be obtained by applying a factor to
the I/O bandwidth of the disk. This factor is applied when we update
disk->set_factor_cb(&disk_variability);
Running our simulation
-^^^^^^^^^^^^^^^^^^^^^^
+----------------------
The binary was compiled in the provided docker container.
./tuto_disk > ./simgrid_disk.csv
Analyzing the SimGrid results
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+=============================
The figure below presents the results obtained by SimGrid.
facet_wrap(disk~op,ncol=2,scale="free_y")+ # ) + #
xlab("Number of concurrent operations") + ylab("Aggregated Bandwidth (MiB/s)") + guides(color=FALSE) + xlim(0,NA) + ylim(0,NA)
-.. image:: fig/simgrid_results.png
+.. image:: tuto_disk/fig/simgrid_results.png
Note: The variability in griffon read operation seems to decrease when
we have more concurrent operations. This is a particularity of the
/**
* @brief Describes how the disk is shared between activities for each operation
*
- * Disks have different bandwidths for read and write operations. This method
- * allows you to set different sharing policies for each operation:
+ * Disks have different bandwidths for read and write operations, that can have different policies:
* - Read: resource sharing for read operation
* - Write: resource sharing for write
* - ReadWrite: global sharing for read and write operations
*
+ * Note that the NONLINEAR callback is in the critical path of the solver, so it should be fast.
+ *
* @param op Operation type
* @param policy Sharing policy
* @param cb Callback for NONLINEAR policies
/**
* @brief Describes how the CPU is shared between concurrent tasks
*
+ * Note that the NONLINEAR callback is in the critical path of the solver, so it should be fast.
+ *
* @param policy Sharing policy
* @param cb Callback for NONLINEAR policies
*/
*/
Link* set_latency(const std::string& value);
- /** @brief Describes how the link is shared between flows */
+ /** @brief Describes how the link is shared between flows
+ *
+ * Note that the NONLINEAR callback is in the critical path of the solver, so it should be fast.
+ */
Link* set_sharing_policy(SharingPolicy policy, const NonLinearResourceCb& cb = {});
SharingPolicy get_sharing_policy() const;
SG_BEGIN_DECL
-/** @addtogroup XBT_dynar
- * @brief DynArr are dynamically sized vector which may contain any type of variables.
+/** @brief Dynar data type (opaque type)
*
- * These are the SimGrid version of the dynamically size arrays, which all C programmer recode one day or another.
- *
- * For performance concerns, the content of DynArr must be homogeneous (in contrary to dictionaries -- see the
- * @ref XBT_dict section). You thus have to provide the function which will be used to free the content at
+ * These are the SimGrid version of the DYNamically sized ARays, which all C programmer recode one day or another.
+ * For performance concerns, the content of dynars must be homogeneous (in contrary to @ref xbt_dict_t).
+ * You thus have to provide the function which will be used to free the content at
* structure creation (of type void_f_pvoid_t).
- *
- * @deprecated If you are using C++, you might want to use `std::vector` instead.
- *
- * @section XBT_dynar_exscal Example with scalar
- * @dontinclude dynar.cpp
- *
- * @skip Vars_decl
- * @skip dyn
- * @until iptr
- * @skip Populate_ints
- * @skip dyn
- * @until end_of_traversal
- * @skip shifting
- * @skip val
- * @until xbt_dynar_free
- *
- * @section XBT_dynar_exptr Example with pointed data
- *
- * @skip test_dynar_string
- * @skip dynar_t
- * @until s2
- * @skip Populate_str
- * @skip dyn
- * @until }
- * @skip macro
- * @until dynar_free
- * @skip end_of_doxygen
- * @until }
- *
- * Note that if you use dynars to store pointed data, xbt_dynar_member() won't be for you. Instead of comparing your
- * pointed elements, it compares the pointer to them.
*/
-/** @defgroup XBT_dynar_cons Dynar constructor and destructor
- * @ingroup XBT_dynar
- *
- * @{
- */
-/** @brief Dynar data type (opaque type) */
typedef struct xbt_dynar_s *xbt_dynar_t;
+/** constant dynar */
typedef const struct xbt_dynar_s* const_xbt_dynar_t;
+/* @warning DO NOT USE THIS STRUCTURE DIRECTLY! Use the public interface, even if it's public for sake of inlining */
+typedef struct xbt_dynar_s {
+ unsigned long size;
+ unsigned long used;
+ unsigned long elmsize;
+ void* data;
+ void_f_pvoid_t free_f;
+} s_xbt_dynar_t;
XBT_PUBLIC xbt_dynar_t xbt_dynar_new(const unsigned long elm_size, void_f_pvoid_t free_f);
XBT_PUBLIC void xbt_dynar_free(xbt_dynar_t* dynar);
XBT_PUBLIC void xbt_dynar_free_container(xbt_dynar_t* dynar);
XBT_PUBLIC void xbt_dynar_shrink(xbt_dynar_t dynar, int empty_slots);
-/** @} */
-/** @defgroup XBT_dynar_array Dynar as a regular array
- * @ingroup XBT_dynar
- *
- * @{
- */
-
+/* Dynar as a regular array */
XBT_PUBLIC void xbt_dynar_get_cpy(const_xbt_dynar_t dynar, unsigned long idx, void* dst);
XBT_PUBLIC void xbt_dynar_insert_at(xbt_dynar_t dynar, int idx, const void* src);
XBT_PUBLIC void xbt_dynar_sort(const_xbt_dynar_t dynar, int_f_cpvoid_cpvoid_t compar_fn);
XBT_ATTRIB_DEPRECATED_v331("This function will be removed") XBT_PUBLIC void* xbt_dynar_to_array(xbt_dynar_t dynar);
-/** @} */
-/** @defgroup XBT_dynar_misc Dynar miscellaneous functions
- * @ingroup XBT_dynar
- *
- * @{
- */
-
+/* Dynar size */
+XBT_PUBLIC void xbt_dynar_reset(xbt_dynar_t dynar);
XBT_PUBLIC unsigned long xbt_dynar_length(const_xbt_dynar_t dynar);
XBT_PUBLIC int xbt_dynar_is_empty(const_xbt_dynar_t dynar);
-XBT_PUBLIC void xbt_dynar_reset(xbt_dynar_t dynar);
-
-/** @} */
-/** @defgroup XBT_dynar_perl Perl-like use of dynars
- * @ingroup XBT_dynar
- *
- * @{
- */
+/* Perl-like interface */
XBT_PUBLIC void xbt_dynar_push(xbt_dynar_t dynar, const void* src);
XBT_PUBLIC void xbt_dynar_pop(xbt_dynar_t dynar, void* dst);
XBT_PUBLIC void xbt_dynar_unshift(xbt_dynar_t dynar, const void* src);
XBT_PUBLIC void xbt_dynar_shift(xbt_dynar_t dynar, void* dst);
XBT_PUBLIC void xbt_dynar_map(const_xbt_dynar_t dynar, void_f_pvoid_t op);
-/** @} */
-/** @defgroup XBT_dynar_ctn Direct manipulation to the dynars content
- * @ingroup XBT_dynar
- *
- * Those functions do not retrieve the content, but only their address.
- *
- * @{
- */
-
+/* Direct content manipulation*/
XBT_PUBLIC void* xbt_dynar_set_at_ptr(const xbt_dynar_t dynar, unsigned long idx);
XBT_PUBLIC void* xbt_dynar_get_ptr(const_xbt_dynar_t dynar, unsigned long idx);
XBT_PUBLIC void* xbt_dynar_insert_at_ptr(xbt_dynar_t dynar, int idx);
XBT_PUBLIC void* xbt_dynar_push_ptr(xbt_dynar_t dynar);
XBT_PUBLIC void* xbt_dynar_pop_ptr(xbt_dynar_t dynar);
-/** @} */
-/** @defgroup XBT_dynar_speed Speed optimized access to dynars of scalars
- * @ingroup XBT_dynar
- *
- * While the other functions use a memcpy to retrieve the content into the user provided area, those ones use a
- * regular affectation. It only works for scalar values, but should be a little faster.
- *
- * @{
- */
-
- /** @brief Quick retrieval of scalar content
- * @hideinitializer */
+/* Dynars of scalar */
+/** @brief Quick retrieval of scalar content
+ * @hideinitializer */
# define xbt_dynar_get_as(dynar,idx,type) \
(*(type*)xbt_dynar_get_ptr((dynar),(idx)))
/** @brief Quick setting of scalar content
# define xbt_dynar_pop_as(dynar,type) \
(*(type*)xbt_dynar_pop_ptr(dynar))
-/** @} */
-/** @defgroup XBT_dynar_cursor Cursors on dynar
- * @ingroup XBT_dynar
- *
- * Cursors are used to iterate over the structure. Never add elements to the DynArr during the traversal.
- *
- * @{
- */
-
-/*
- * @warning DO NOT USE THIS STRUCTURE DIRECTLY! Instead, use the public interface:
- * This was made public to allow:
- * - the inlining of the foreach elements
- * - sending such beasts over the network
- *
- * @see xbt_dynar_length()
- */
-typedef struct xbt_dynar_s {
- unsigned long size;
- unsigned long used;
- unsigned long elmsize;
- void *data;
- void_f_pvoid_t free_f;
-} s_xbt_dynar_t;
-
+/* Iterating over dynars */
static inline int _xbt_dynar_cursor_get(const_xbt_dynar_t dynar, unsigned int idx, void* dst)
{
if (!dynar) /* iterating over a NULL dynar is a no-op */
/** @brief Iterates over the whole dynar.
*
* @param _dynar what to iterate over
- * @param _cursor an integer used as cursor
- * @param _data
+ * @param _cursor an unsigned integer used as cursor
* @hideinitializer
*
* Here is an example of usage:
* @code
xbt_dynar_t dyn;
unsigned int cpt;
-string *str;
+char* str;
xbt_dynar_foreach (dyn,cpt,str) {
printf("Seen %s\n",str);
}
@endcode
*
* Note that underneath, that's a simple for loop with no real black magic involved. It's perfectly safe to interrupt
- * a foreach with a break or a return statement.
+ * a foreach with a break or a return statement, but inserting or removing elements during the traversal is probably a
+ * bad idea (even if you can fix side effects by manually changing the counter).
*/
#define xbt_dynar_foreach(_dynar, _cursor, _data) \
for ((_cursor) = 0; _xbt_dynar_cursor_get((_dynar), (_cursor), &(_data)); (_cursor)++)
-
-/** @} */
SG_END_DECL
#endif /* XBT_DYNAR_H */
void simgrid_set_maestro(void (*code)(void*), void* data)
{
#ifdef _WIN32
- XBT_INFO("WARNING, SIMIX_set_maestro is believed to not work on windows. Please help us investigating this issue if "
+ XBT_WARN("simgrid_set_maestro is believed to not work on windows. Please help us investigating this issue if "
"you need that feature");
#endif
maestro_code = std::bind(code, data);
memcpy(dst, elm, dynar->elmsize);
}
-/** @brief Constructor
+/**
+ * Creates a new dynar. If a @c free_f is provided, the elements have to be pointer of pointer. That is to say that
+ * dynars can contain either base types (int, char, double, etc) or pointer of pointers (struct **).
*
* @param elmsize size of each element in the dynar
* @param free_f function to call each time we want to get rid of an element (or nullptr if nothing to do).
- *
- * Creates a new dynar. If a free_func is provided, the elements have to be pointer of pointer. That is to say that
- * dynars can contain either base types (int, char, double, etc) or pointer of pointers (struct **).
*/
xbt_dynar_t xbt_dynar_new(const unsigned long elmsize, void_f_pvoid_t free_f)
{
return dynar;
}
-/** @brief Destructor of the structure not touching to the content
+/** Destructor of the structure leaving the content unmodified. Ie, the array is freed, but the content is not touched
+ * (the @a free_f function is not used).
*
* @param dynar poor victim
- *
- * kilkil a dynar BUT NOT its content. Ie, the array is freed, but the content is not touched (the @a free_f function
- * is not used)
*/
void xbt_dynar_free_container(xbt_dynar_t* dynar)
{
}
}
-/** @brief Frees the content and set the size to 0
- *
- * @param dynar who to squeeze
- */
+/** @brief Frees the content and set the size to 0 */
void xbt_dynar_reset(xbt_dynar_t dynar)
{
_sanity_check_dynar(dynar);
}
/**
- * @brief Shrink the dynar by removing empty slots at the end of the internal array
- * @param dynar a dynar
- * @param empty_slots_wanted number of empty slots you want to keep at the end of the internal array for further
- * insertions
+ * Shrinks (reduces) the dynar by removing empty slots in the internal storage to save memory.
+ * If @c empty_slots_wanted is not zero, this operation preserves that amount of empty slot, for fast future additions.
+ * Note that if @c empty_slots_wanted is large enough, the internal array is expanded instead of shrunk.
*
- * Reduces the internal array size of the dynar to the number of elements plus @a empty_slots_wanted.
- * After removing elements from the dynar, you can call this function to make the dynar use less memory.
- * Set @a empty_slots_wanted to zero to reduce the dynar internal array as much as possible.
- * Note that if @a empty_slots_wanted is greater than the array size, the internal array is expanded instead of shrunk.
+ * @param dynar a dynar
+ * @param empty_slots_wanted number of empty slots elements that can be inserted the internal storage without resizing it
*/
void xbt_dynar_shrink(xbt_dynar_t dynar, int empty_slots_wanted)
{
_xbt_dynar_resize(dynar, dynar->used + empty_slots_wanted);
}
-/** @brief Destructor
- *
- * @param dynar poor victim
- *
- * kilkil a dynar and its content
- */
+/** @brief Destructor: kilkil a dynar and its content. */
void xbt_dynar_free(xbt_dynar_t* dynar)
{
if (dynar && *dynar) {
}
}
-/** @brief Count of dynar's elements
- *
- * @param dynar the dynar we want to measure
- */
+/** @brief Count of dynar's elements */
unsigned long xbt_dynar_length(const_xbt_dynar_t dynar)
{
return (dynar ? dynar->used : 0UL);
}
-/**@brief check if a dynar is empty
- *
- *@param dynar the dynat we want to check
- */
+/**@brief check if a dynar is empty */
int xbt_dynar_is_empty(const_xbt_dynar_t dynar)
{
return (xbt_dynar_length(dynar) == 0);
/** @brief Retrieve a pointer to the Nth element of a dynar.
*
- * @param dynar information dealer
- * @param idx index of the slot we want to retrieve
- * @return the @a idx-th element of @a dynar.
- *
- * @warning The returned value is the actual content of the dynar.
+ * Note that the returned value is the actual content of the dynar.
* Make a copy before fooling with it.
*/
void* xbt_dynar_get_ptr(const_xbt_dynar_t dynar, unsigned long idx)
memcpy(xbt_dynar_insert_at_ptr(dynar, idx), src, dynar->elmsize);
}
-/** @brief Remove the Nth dynar's element, sliding the previous values to the left
+/** @brief Remove the Nth element, sliding other values to the left
*
* Get the Nth element of a dynar, removing it from the dynar and moving all subsequent values to one position left in
* the dynar.
/** @brief Returns a boolean indicating whether the element is part of the dynar
*
* Beware that if your dynar contains pointed values (such as strings) instead of scalar, this function is probably not
- * what you want.
+ * what you want. It would compare the pointer values, not the pointed elements.
*/
int xbt_dynar_member(const_xbt_dynar_t dynar, const void* elem)
{
* a comparison function. Here is a quick example if you have integers in your dynar:
*
* @verbatim
- * int cmpfunc (const void * a, const void * b) {
- * int intA = *(int*)a;
- * int intB = *(int*)b;
- * return intA - intB;
- * }
- * @endverbatim
+ int cmpfunc (const void * a, const void * b) {
+ int intA = *(int*)a;
+ int intB = *(int*)b;
+ return intA - intB;
+ }
+ @endverbatim
*
- * and now to sort a dynar of MSG hosts depending on their speed:
+ * And now, a function to sort a dynar of MSG hosts depending on their speed:
* @verbatim
- * int cmpfunc(const MSG_host_t a, const MSG_host_t b) {
- * MSG_host_t hostA = *(MSG_host_t*)a;
- * MSG_host_t hostB = *(MSG_host_t*)b;
- * return MSG_host_get_speed(hostA) - MSG_host_get_speed(hostB);
- * }
- * @endverbatim
+ int cmpfunc(const MSG_host_t a, const MSG_host_t b) {
+ MSG_host_t hostA = *(MSG_host_t*)a;
+ MSG_host_t hostB = *(MSG_host_t*)b;
+ return MSG_host_get_speed(hostA) - MSG_host_get_speed(hostB);
+ }
+ @endverbatim
*
* @param dynar the dynar to sort
* @param compar_fn comparison function of type (int (compar_fn*) (const void*) (const void*)).
{
SECTION("Dynars of integers")
{
- /* Vars_decl [doxygen cruft] */
int cpt;
unsigned int cursor;
/* in your code is naturally the way to go outside a regression test */
INFO("==== Push " << NB_ELEM << " int, set them again 3 times, traverse them, shift them");
- /* Populate_ints [doxygen cruft] */
/* 1. Populate the dynar */
d = xbt_dynar_new(sizeof(int), nullptr);
for (int i = 0; i < NB_ELEM; i++) {
docs/source/tuto_smpi/roundtrip.c
docs/source/tuto_disk/analysis.org
- docs/source/tuto_disk/analysis.rst
+ docs/source/tuto_disk/analysis.irst
docs/source/tuto_disk/CMakeLists.txt
docs/source/tuto_disk/Dockerfile
docs/source/tuto_disk/fig/edel_read_dhist.png
+ \.github/.*
++ docs/default.nix
+
+ tools/appveyor-irc-notify\.py
+ tools/docker/.*
+ tools/git-hooks/.*
