#!/usr/bin/env tesh p Testing a simple master/worker example application handling failures TCP crosstraffic DISABLED ! output sort 19 $ $SG_TEST_EXENV ${bindir:=.}/platform-failures$EXEEXT --log=xbt_cfg.thres:critical --log=no_loc ${platfdir}/small_platform_with_failures.xml ${srcdir}/../app-masterworker/app-masterworker_d.xml --cfg=path:${srcdir} --cfg=network/crosstraffic:0 "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" --log=surf_cpu.t:verbose > [ 0.000000] (0:maestro@) Cannot launch process 'worker' on failed host 'Fafard' > [ 0.000000] (1:master@Tremblay) Got 5 workers and 20 tasks to process > [ 0.000000] (1:master@Tremblay) Send a message to worker-0 > [ 0.010309] (1:master@Tremblay) Send to worker-0 completed > [ 0.010309] (2:worker@Tremblay) Start execution... > [ 0.000000] (2:worker@Tremblay) Waiting a message on worker-0 > [ 0.000000] (3:worker@Jupiter) Waiting a message on worker-1 > [ 0.000000] (4:worker@Ginette) Waiting a message on worker-3 > [ 0.000000] (5:worker@Bourassa) Waiting a message on worker-4 > [ 0.010309] (1:master@Tremblay) Send a message to worker-1 > [ 1.000000] (0:maestro@) Restart processes on host Fafard > [ 1.000000] (6:worker@Fafard) Waiting a message on worker-2 > [ 1.000000] (1:master@Tremblay) Mmh. Something went wrong with 'worker-1'. Nevermind. Let's keep going! > [ 1.000000] (1:master@Tremblay) Send a message to worker-2 > [ 1.000000] (3:worker@Jupiter) Gloups. The cpu on which I'm running just turned off!. See you! > [ 2.000000] (1:master@Tremblay) Mmh. Something went wrong with 'worker-2'. Nevermind. Let's keep going! > [ 2.000000] (6:worker@Fafard) Gloups. The cpu on which I'm running just turned off!. See you! > [ 2.000000] (0:maestro@) Restart processes on host Jupiter > [ 2.000000] (1:master@Tremblay) Send a message to worker-3 > [ 2.000000] (7:worker@Jupiter) Waiting a message on worker-1 > [ 2.010309] (2:worker@Tremblay) Execution complete. > [ 2.010309] (2:worker@Tremblay) Waiting a message on worker-0 > [ 3.030928] (1:master@Tremblay) Send to worker-3 completed > [ 3.030928] (1:master@Tremblay) Send a message to worker-4 > [ 3.030928] (4:worker@Ginette) Start execution... > [ 4.061856] (1:master@Tremblay) Send to worker-4 completed > [ 4.061856] (1:master@Tremblay) Send a message to worker-0 > [ 4.061856] (5:worker@Bourassa) Start execution... > [ 4.072165] (1:master@Tremblay) Send to worker-0 completed > [ 4.072165] (1:master@Tremblay) Send a message to worker-1 > [ 4.072165] (2:worker@Tremblay) Start execution... > [ 5.030928] (4:worker@Ginette) Execution complete. > [ 5.030928] (4:worker@Ginette) Waiting a message on worker-3 > [ 5.103093] (1:master@Tremblay) Send to worker-1 completed > [ 5.103093] (1:master@Tremblay) Send a message to worker-2 > [ 5.103093] (7:worker@Jupiter) Start execution... > [ 6.061856] (5:worker@Bourassa) Execution complete. > [ 6.061856] (5:worker@Bourassa) Waiting a message on worker-4 > [ 6.072165] (2:worker@Tremblay) Execution complete. > [ 6.072165] (2:worker@Tremblay) Waiting a message on worker-0 > [ 7.103093] (7:worker@Jupiter) Execution complete. > [ 7.103093] (7:worker@Jupiter) Waiting a message on worker-1 > [ 15.103093] (1:master@Tremblay) Mmh. Got timeouted while speaking to 'worker-2'. Nevermind. Let's keep going! > [ 15.103093] (1:master@Tremblay) Send a message to worker-3 > [ 15.103093] (1:master@Tremblay) Mmh. Something went wrong with 'worker-3'. Nevermind. Let's keep going! > [ 15.103093] (1:master@Tremblay) Send a message to worker-4 > [ 15.103093] (4:worker@Ginette) Mmh. Something went wrong. Nevermind. Let's keep going! > [ 15.103093] (4:worker@Ginette) Waiting a message on worker-3 > [ 16.134021] (1:master@Tremblay) Send to worker-4 completed > [ 16.134021] (1:master@Tremblay) Send a message to worker-0 > [ 16.134021] (5:worker@Bourassa) Start execution... > [ 16.144330] (1:master@Tremblay) Send to worker-0 completed > [ 16.144330] (1:master@Tremblay) Send a message to worker-1 > [ 16.144330] (2:worker@Tremblay) Start execution... > [ 17.175258] (1:master@Tremblay) Send to worker-1 completed > [ 17.175258] (1:master@Tremblay) Send a message to worker-2 > [ 17.175258] (7:worker@Jupiter) Start execution... > [ 18.134021] (5:worker@Bourassa) Execution complete. > [ 18.134021] (5:worker@Bourassa) Waiting a message on worker-4 > [ 18.144330] (2:worker@Tremblay) Execution complete. > [ 18.144330] (2:worker@Tremblay) Waiting a message on worker-0 > [ 19.175258] (7:worker@Jupiter) Execution complete. > [ 19.175258] (7:worker@Jupiter) Waiting a message on worker-1 > [ 27.175258] (1:master@Tremblay) Mmh. Got timeouted while speaking to 'worker-2'. Nevermind. Let's keep going! > [ 27.175258] (1:master@Tremblay) Send a message to worker-3 > [ 28.206186] (1:master@Tremblay) Send to worker-3 completed > [ 28.206186] (1:master@Tremblay) Send a message to worker-4 > [ 28.206186] (1:master@Tremblay) Mmh. Something went wrong with 'worker-4'. Nevermind. Let's keep going! > [ 28.206186] (1:master@Tremblay) Send a message to worker-0 > [ 28.206186] (4:worker@Ginette) Start execution... > [ 28.206186] (5:worker@Bourassa) Mmh. Something went wrong. Nevermind. Let's keep going! > [ 28.206186] (5:worker@Bourassa) Waiting a message on worker-4 > [ 28.216495] (1:master@Tremblay) Send to worker-0 completed > [ 28.216495] (1:master@Tremblay) Send a message to worker-1 > [ 28.216495] (2:worker@Tremblay) Start execution... > [ 29.247423] (1:master@Tremblay) Send to worker-1 completed > [ 29.247423] (1:master@Tremblay) Send a message to worker-2 > [ 29.247423] (7:worker@Jupiter) Start execution... > [ 30.206186] (4:worker@Ginette) Execution complete. > [ 30.206186] (4:worker@Ginette) Waiting a message on worker-3 > [ 30.216495] (2:worker@Tremblay) Execution complete. > [ 30.216495] (2:worker@Tremblay) Waiting a message on worker-0 > [ 31.247423] (7:worker@Jupiter) Execution complete. > [ 31.247423] (7:worker@Jupiter) Waiting a message on worker-1 > [ 39.247423] (1:master@Tremblay) Mmh. Got timeouted while speaking to 'worker-2'. Nevermind. Let's keep going! > [ 39.247423] (1:master@Tremblay) Send a message to worker-3 > [ 40.278351] (1:master@Tremblay) Send to worker-3 completed > [ 40.278351] (1:master@Tremblay) Send a message to worker-4 > [ 40.278351] (4:worker@Ginette) Start execution... > [ 41.000000] (4:worker@Ginette) Gloups. The cpu on which I'm running just turned off!. See you! > [ 41.309278] (1:master@Tremblay) Send to worker-4 completed > [ 41.309278] (1:master@Tremblay) All tasks have been dispatched. Let's tell everybody the computation is over. > [ 41.309278] (2:worker@Tremblay) I'm done. See you! > [ 41.309278] (5:worker@Bourassa) Start execution... > [ 41.309278] (7:worker@Jupiter) I'm done. See you! > [ 42.309278] (1:master@Tremblay) Mmh. Got timeouted while speaking to 'worker-2'. Nevermind. Let's keep going! > [ 43.309278] (0:maestro@) Simulation time 43.3093 > [ 43.309278] (1:master@Tremblay) Mmh. Got timeouted while speaking to 'worker-3'. Nevermind. Let's keep going! > [ 43.309278] (1:master@Tremblay) Goodbye now! > [ 43.309278] (5:worker@Bourassa) Execution complete. > [ 43.309278] (5:worker@Bourassa) Waiting a message on worker-4 > [ 43.309278] (5:worker@Bourassa) I'm done. See you! p Testing a simple master/worker example application handling failures. TCP crosstraffic ENABLED ! output sort 19 $ $SG_TEST_EXENV ${bindir:=.}/platform-failures$EXEEXT --log=xbt_cfg.thres:critical --log=no_loc ${platfdir}/small_platform_with_failures.xml ${srcdir}/../app-masterworker/app-masterworker_d.xml --cfg=path:${srcdir} "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" --log=surf_cpu.t:verbose > [ 0.000000] (0:maestro@) Cannot launch process 'worker' on failed host 'Fafard' > [ 0.000000] (1:master@Tremblay) Got 5 workers and 20 tasks to process > [ 0.000000] (1:master@Tremblay) Send a message to worker-0 > [ 0.000000] (2:worker@Tremblay) Waiting a message on worker-0 > [ 0.000000] (3:worker@Jupiter) Waiting a message on worker-1 > [ 0.000000] (4:worker@Ginette) Waiting a message on worker-3 > [ 0.000000] (5:worker@Bourassa) Waiting a message on worker-4 > [ 0.010825] (2:worker@Tremblay) Start execution... > [ 0.010825] (1:master@Tremblay) Send to worker-0 completed > [ 0.010825] (1:master@Tremblay) Send a message to worker-1 > [ 1.000000] (0:maestro@) Restart processes on host Fafard > [ 1.000000] (6:worker@Fafard) Waiting a message on worker-2 > [ 1.000000] (1:master@Tremblay) Mmh. Something went wrong with 'worker-1'. Nevermind. Let's keep going! > [ 1.000000] (1:master@Tremblay) Send a message to worker-2 > [ 1.000000] (3:worker@Jupiter) Gloups. The cpu on which I'm running just turned off!. See you! > [ 2.000000] (0:maestro@) Restart processes on host Jupiter > [ 2.000000] (7:worker@Jupiter) Waiting a message on worker-1 > [ 2.000000] (1:master@Tremblay) Mmh. Something went wrong with 'worker-2'. Nevermind. Let's keep going! > [ 2.000000] (1:master@Tremblay) Send a message to worker-3 > [ 2.000000] (6:worker@Fafard) Gloups. The cpu on which I'm running just turned off!. See you! > [ 2.010825] (2:worker@Tremblay) Execution complete. > [ 2.010825] (2:worker@Tremblay) Waiting a message on worker-0 > [ 3.082474] (4:worker@Ginette) Start execution... > [ 3.082474] (1:master@Tremblay) Send to worker-3 completed > [ 3.082474] (1:master@Tremblay) Send a message to worker-4 > [ 4.164948] (5:worker@Bourassa) Start execution... > [ 4.164948] (1:master@Tremblay) Send to worker-4 completed > [ 4.164948] (1:master@Tremblay) Send a message to worker-0 > [ 4.175773] (2:worker@Tremblay) Start execution... > [ 4.175773] (1:master@Tremblay) Send to worker-0 completed > [ 4.175773] (1:master@Tremblay) Send a message to worker-1 > [ 5.082474] (4:worker@Ginette) Execution complete. > [ 5.082474] (4:worker@Ginette) Waiting a message on worker-3 > [ 5.258247] (7:worker@Jupiter) Start execution... > [ 5.258247] (1:master@Tremblay) Send to worker-1 completed > [ 5.258247] (1:master@Tremblay) Send a message to worker-2 > [ 6.164948] (5:worker@Bourassa) Execution complete. > [ 6.164948] (5:worker@Bourassa) Waiting a message on worker-4 > [ 6.175773] (2:worker@Tremblay) Execution complete. > [ 6.175773] (2:worker@Tremblay) Waiting a message on worker-0 > [ 7.258247] (7:worker@Jupiter) Execution complete. > [ 7.258247] (7:worker@Jupiter) Waiting a message on worker-1 > [ 15.258247] (1:master@Tremblay) Mmh. Got timeouted while speaking to 'worker-2'. Nevermind. Let's keep going! > [ 15.258247] (1:master@Tremblay) Send a message to worker-3 > [ 15.258247] (4:worker@Ginette) Mmh. Something went wrong. Nevermind. Let's keep going! > [ 15.258247] (4:worker@Ginette) Waiting a message on worker-3 > [ 15.258247] (1:master@Tremblay) Mmh. Something went wrong with 'worker-3'. Nevermind. Let's keep going! > [ 15.258247] (1:master@Tremblay) Send a message to worker-4 > [ 16.340722] (5:worker@Bourassa) Start execution... > [ 16.340722] (1:master@Tremblay) Send to worker-4 completed > [ 16.340722] (1:master@Tremblay) Send a message to worker-0 > [ 16.351546] (2:worker@Tremblay) Start execution... > [ 16.351546] (1:master@Tremblay) Send to worker-0 completed > [ 16.351546] (1:master@Tremblay) Send a message to worker-1 > [ 17.434021] (7:worker@Jupiter) Start execution... > [ 17.434021] (1:master@Tremblay) Send to worker-1 completed > [ 17.434021] (1:master@Tremblay) Send a message to worker-2 > [ 18.340722] (5:worker@Bourassa) Execution complete. > [ 18.340722] (5:worker@Bourassa) Waiting a message on worker-4 > [ 18.351546] (2:worker@Tremblay) Execution complete. > [ 18.351546] (2:worker@Tremblay) Waiting a message on worker-0 > [ 19.434021] (7:worker@Jupiter) Execution complete. > [ 19.434021] (7:worker@Jupiter) Waiting a message on worker-1 > [ 27.434021] (1:master@Tremblay) Mmh. Got timeouted while speaking to 'worker-2'. Nevermind. Let's keep going! > [ 27.434021] (1:master@Tremblay) Send a message to worker-3 > [ 28.516495] (4:worker@Ginette) Start execution... > [ 28.516495] (1:master@Tremblay) Send to worker-3 completed > [ 28.516495] (1:master@Tremblay) Send a message to worker-4 > [ 28.516495] (5:worker@Bourassa) Mmh. Something went wrong. Nevermind. Let's keep going! > [ 28.516495] (5:worker@Bourassa) Waiting a message on worker-4 > [ 28.516495] (1:master@Tremblay) Mmh. Something went wrong with 'worker-4'. Nevermind. Let's keep going! > [ 28.516495] (1:master@Tremblay) Send a message to worker-0 > [ 28.527320] (2:worker@Tremblay) Start execution... > [ 28.527320] (1:master@Tremblay) Send to worker-0 completed > [ 28.527320] (1:master@Tremblay) Send a message to worker-1 > [ 29.609794] (7:worker@Jupiter) Start execution... > [ 29.609794] (1:master@Tremblay) Send to worker-1 completed > [ 29.609794] (1:master@Tremblay) Send a message to worker-2 > [ 30.516495] (4:worker@Ginette) Execution complete. > [ 30.516495] (4:worker@Ginette) Waiting a message on worker-3 > [ 30.527320] (2:worker@Tremblay) Execution complete. > [ 30.527320] (2:worker@Tremblay) Waiting a message on worker-0 > [ 31.609794] (7:worker@Jupiter) Execution complete. > [ 31.609794] (7:worker@Jupiter) Waiting a message on worker-1 > [ 39.609794] (1:master@Tremblay) Mmh. Got timeouted while speaking to 'worker-2'. Nevermind. Let's keep going! > [ 39.609794] (1:master@Tremblay) Send a message to worker-3 > [ 40.692268] (4:worker@Ginette) Start execution... > [ 40.692268] (1:master@Tremblay) Send to worker-3 completed > [ 40.692268] (1:master@Tremblay) Send a message to worker-4 > [ 41.000000] (4:worker@Ginette) Gloups. The cpu on which I'm running just turned off!. See you! > [ 41.774742] (5:worker@Bourassa) Start execution... > [ 41.774742] (1:master@Tremblay) Send to worker-4 completed > [ 41.774742] (1:master@Tremblay) All tasks have been dispatched. Let's tell everybody the computation is over. > [ 41.774742] (2:worker@Tremblay) I'm done. See you! > [ 41.774742] (7:worker@Jupiter) I'm done. See you! > [ 42.774742] (1:master@Tremblay) Mmh. Got timeouted while speaking to 'worker-2'. Nevermind. Let's keep going! > [ 43.774742] (5:worker@Bourassa) Execution complete. > [ 43.774742] (5:worker@Bourassa) Waiting a message on worker-4 > [ 43.774742] (1:master@Tremblay) Mmh. Got timeouted while speaking to 'worker-3'. Nevermind. Let's keep going! > [ 43.774742] (5:worker@Bourassa) I'm done. See you! > [ 43.774742] (1:master@Tremblay) Goodbye now! > [ 43.774742] (0:maestro@) Simulation time 43.7747 p NOT testing the mixure of failures and CpuTI: p This test leads to a deadlock because of a bug somewhere in surf_solve. p We should debug this instead of ignoring the issue, but it's utterly p complex with such an integration test. One day, we will setup a set of p unit tests for the surf solver, and such issues will be addressable again. p For the time being, I just give up, sorry. p $ $SG_TEST_EXENV ${bindir:=.}/platform-failures$EXEEXT --log=xbt_cfg.thres:critical --log=no_loc ${platfdir}/small_platform_with_failures.xml ${srcdir}/../app-masterworker/app-masterworker_d.xml --cfg=path:${srcdir} --cfg=cpu/optim:TI "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" --log=surf_cpu.t:verbose