From 19b374fc81dd79c284ff091080d686e84962aa66 Mon Sep 17 00:00:00 2001 From: Christian Heinrich Date: Thu, 16 Jul 2015 16:25:01 +0200 Subject: [PATCH] [Doc] Significant update to the storage documentation --- doc/doxygen/platform.doc | 281 ++++++++++++++++++++++++++++++++------- 1 file changed, 233 insertions(+), 48 deletions(-) diff --git a/doc/doxygen/platform.doc b/doc/doxygen/platform.doc index 71309501f3..07f50cb469 100644 --- a/doc/doxygen/platform.doc +++ b/doc/doxygen/platform.doc @@ -656,82 +656,267 @@ id | yes | string | Name of the link that is supposed to act interface. \subsubsection pf_sto_conc Storage Main concepts -Basically there is 3 different entities to know : -\li the storage_type: here you define some kind of storage that - you will instantiate many type on your platform. Think of it like - a definition of throughput of a specific disk. -\li the storage: instance of a storage_type. Defines a - new storage of storage_type -\li the mount: says that the storage is located into this - specific resource. - -the content of a storage has to be defined in a content file that -contains the content. The path to this file has to be passed within -the content attribute . Here is a way to generate it: +Basically there are 3 different entities available in SimGrid that +can be used to model storages: + +Entity name | Description +--------------- | ----------- +\ref pf_storage_entity_storage_type "storage_type" | Defines a template for a particular kind of storage (such as a hard-drive) and specifies important features of the storage, such as capacity, performance (read/write), content, ... Different models of hard-drives use different storage_types (because the difference between an SSD and an HDD does matter), as they differ in some specifications (e.g., different size or read/write performance). +\ref pf_storage_entity_storage "storage" | Defines an actual instance of a storage (disk, RAM, ...); uses a ``storage_type`` template (see line above) so that you don't need to re-specify the same details over and over again. +\ref pf_storage_entity_mount "mount" | Must be wrapped by a \ref pf_host tag; declares which storage(s) this host has mounted and where (i.e., the mountpoint). + + +\anchor pf_storage_content_file +### %Storage Content File ### + +In order to assess exactly how much time is spent reading from the storage, +SimGrid needs to know what is stored in the storage device (identified by distinct (file-)name, like in a file system) +and what size this content has. + +\note + The content file is never changed by the simulation; it is parsed once + per simulation and kept in memory afterwards. When the content of the + storage changes, only the internal SimGrid data structures change. + +\anchor pf_storage_content_file_structure +#### Structure of a %Storage File #### + +Here is an excerpt from two storage file; if you want to see the whole file, check +the file ``examples/platforms/content/storage_content.txt`` that comes with the +SimGrid source code. + +SimGrid essentially supports two different formats: UNIX-style filepaths should +follow the well known format: + +\verbatim +/lib/libsimgrid.so.3.6.2 12710497 +/bin/smpicc 918 +/bin/smpirun 7292 +/bin/smpif2c 1990 +/bin/simgrid_update_xml 5018 +/bin/graphicator 66986 +/bin/simgrid-colorizer 2993 +/bin/smpiff 820 +/bin/tesh 356434 +\endverbatim + +Windows filepaths, unsurprisingly, use the windows style: + +\verbatim +\Windows\avastSS.scr 41664 +\Windows\bfsvc.exe 75264 +\Windows\bootstat.dat 67584 +\Windows\CoreSingleLanguage.xml 31497 +\Windows\csup.txt 12 +\Windows\dchcfg64.exe 335464 +\Windows\dcmdev64.exe 93288 +\endverbatim + +\note + The different file formats come at a cost; in version 3.12 (and most likely + in later versions, too), copying files from windows-style storages to unix-style + storages (and vice versa) is not supported. + +\anchor pf_storage_content_file_create +#### Generate a %Storage Content File #### + +If you want to generate a storage content file based on your own filesystem (or at least a filesystem you have access to), +try running this command (works only on unix systems): \verbatim find /path/you/want -type f -exec ls -l {} \; 2>/dev/null > ./content.txt \endverbatim -\subsubsection pf_sto_sttp storage_type +\subsubsection pf_storage_entities The Storage Entities + +These are the entities that you can use in your platform files to include +storage in your model. See also the list of our \ref pf_storage_example_files "example files"; +these might also help you to get started. +\anchor pf_storage_entity_storage_type +#### \ #### -storage_type attributes : -\li id (mandatory): the identifier of the storage_type to be - used when referring to it. -\li model (mandatory): Unused for now by the simulator (but - mandatory, ok) -\li content: default value 0. The file containing the disk - content. (may be moved soon or later to storage tag. +Attribute name | Mandatory | Values | Description +--------------- | --------- | ------ | ----------- +id | yes | string | Identifier of this storage_type; used when referring to it +model | yes | string | For reasons of future backwards compatibility only; specifies the name of the model for the storage that should be used +size | yes | string | Specifies the amount of available storage space; you can specify storage like "500GiB" or "500GB" if you want. (TODO add a link to all the available abbreviations) +content | yes | string | Path to a \ref pf_storage_content_file "Storage Content File" on your system. This file must exist. +content_type | no | ("txt_unix"\|"txt_win") | Determines which kind of filesystem you're using; make sure the filenames (stored in that file, see \ref pf_storage_content_file_structure "Storage Content File Structure"!) -The tag must contains some predefined model prop, as may do some other -resources tags. -storage_type mandatory model_prop : -\li Bwrite: value in B/s. Write throughput -\li Bread: value in B/s. Read throughput -\li Bconnexion: value in B/s. Connection throughput (i.e. the - throughput of the storage connector). +This tag must contain some predefined model properties, specified via the <model_prop> tag. Here is a list, +see below for an example: + +Property id | Mandatory | Values | Description +--------------- | --------- | ------ | ----------- +Bwrite | yes | string | Bandwidth for write access; in B/s (but you can also specify e.g. "30MBps") +Bread | yes | string | Bandwidth for read access; in B/s (but you can also specify e.g. "30MBps") +Bconnexion | yes | string | Throughput (of the storage connector) in B/s. -A storage_type can also contain the prop tag. The prop tag allows you -to define additional information on this storage_type following the -attribute/value schema. You may want to use it to give information to -the tool you use for rendering your simulation, for example. +\note + A storage_type can also contain the <prop> tag. The <prop> tag allows you + to associate additional information to this <storage_type> and follows the + attribute/value schema; see the example below. You may want to use it to give information to + the tool you use for rendering your simulation, for example. +Here is a complete example for the ``storage_type`` tag: \verbatim - + \endverbatim -\subsubsection pf_sto_st storage +\anchor pf_storage_entity_storage +#### <storage> #### -storage_type attributes : -\li id (mandatory): the identifier of the storage to be used - when referring to it. -\li typeId (mandatory): the identifier of the storage_type that - this storage belongs to. -\li attach (mandatory): the host (name) to which the storage is - attached to. +``storage`` attributes: -\subsubsection pf_sto_mo mount +Attribute name | Mandatory | Values | Description +-------------- | --------- | ------ | ----------- +id | yes | string | Identifier of this ``storage``; used when referring to it +typeId | yes | string | Here you need to refer to an already existing \ref pf_storage_entity_storage_type "\"; the storage entity defined by this tag will then inherit the properties defined there. +attach | yes | string | Name of a host (see Section \ref pf_host) that this storage is physically attached to (e.g., a harddrive in a computer) +content | no | string | When specified, overwrites the content attribute of \ref pf_storage_entity_storage_type "\" +content_type | no | string | When specified, overwrites the content_type attribute of \ref pf_storage_entity_storage_type "\" -mount attributes : -\li id (mandatory): the id of the storage that must be - mounted on that computer. -\li name (mandatory): the name that will be the logical - reference to this disk (the mount point). +Here are two examples: + +\verbatim + + + +\endverbatim + +The first example is straightforward: A disk is defined and called "Disk1"; it is +of type "single_HDD" (shown as an example of \ref pf_storage_entity_storage_type "\" above) and attached +to a host called "bob" (the definition of this host is omitted here). + +The second storage is called "Disk2", is still of the same type as Disk1 but +now specifies a new content file (so the content will be different from Disk1) +and the filesystem uses the windows style; finally, it is attached to a second host, +called alice (which is again not defined here). + +\anchor pf_storage_entity_mount +#### <mount> #### + +Attributes: +| Attribute name | Mandatory | Values | Description | +| ---------------- | ----------- | -------- | ------------- | +| id | yes | string | Refers to a \ref pf_storage_entity_storage "<storage>" entity that will be mounted on that computer | +| name | yes | string | Path/location to/of the logical reference (mount point) of this disk + +This tag must be enclosed by a \ref pf_host tag. It then specifies where the mountpoint of a given storage device (defined by the ``id`` attribute) +is; this location is specified by the ``name`` attribute. + +Here is a simple example, taken from the file ``examples/platform/storage.xml``: + +\verbatim + + + + + + + + + + + + + + + + + +\endverbatim + +This example is quite interesting, as the same device, called "Disk2", is mounted by +two hosts at the same time! Note, however, that the host called ``alice`` is actually +attached to this storage, as can be seen in the \ref pf_storage_entity_storage "<storage>" +tag. This means that ``denise`` must access this storage via network, but SimGrid automatically takes +care of that for you. + +Furthermore, this example shows that ``denise`` has mounted two storages with different +filesystem types (unix and windows). In general, a host can mount as many storage devices as +required. + +\note + Again, the difference between ``attach`` and ``mount`` is simply that + an attached storage is always physically inside (or connected to) that machine; + for instance, a USB stick is attached to one and only one machine (where it's plugged-in) + but it can only be mounted on others, as mounted storage can also be a remote location. + +\anchor pf_storage_entity_mstorage +#### <mstorage> #### +\note + This is currently unused. -\subsubsection pf_sto_mst mstorage -Note : unused for now mstorage attributes : \li typeId (mandatory): the id of the storage that must be mounted on that computer. \li name (mandatory): the name that will be the logical reference to this disk (the mount point). +\subsubsection pf_storage_example_files Example files + +Several examples were already discussed above; if you're interested in full examples, +check the the following platforms: + +1. ``examples/platforms/storage.xml`` +2. ``examples/platforms/remote_io.xml`` + +If you're looking for some examplary C code, you may find the source code +available in the directory ``examples/msg/io/`` useful. + +\subsubsection pf_storage_examples_modelling Modelling different situations + +The storage functionality of SimGrid is type-agnostic, that is, the implementation +does not presume any type of storagei, such as HDDs/SSDs, RAM, +CD/DVD devices, USB sticks etc. + +This allows the user to apply the simulator for a wide variety of scenarios; one +common scenario would be the access of remote RAM. + +#### Modelling the access of remote RAM #### + +How can this be achieved in SimGrid? Let's assume we have a setup where three hosts +(HostA, HostB, HostC) need to access remote RAM: + +\verbatim + Host A + / +RAM -- Host B + \ + Host C +\endverbatim + +An easy way to model this scenario is to setup and define the RAM via the +\ref pf_storage_entity_storage "storage" and \ref pf_storage_entity_storage_type "storage type" +entities and attach it to a remote dummy host; then, every host can have their own links +to this host (modelling for instance certain scenarios, such as PCIe ...) + +\verbatim + Host A + / +RAM - Dummy -- Host B + \ + Host C +\endverbatim + +Now, if read from this storage, the host that mounts this storage +communicates to the dummy host which reads from RAM and +sends the information back. + + \section pf_routing Routing To achieve high performance, the routing tables used within SimGrid are -- 2.20.1