3 # Copyright (c) 2012-2015. The SimGrid Team.
6 # This program is free software; you can redistribute it and/or modify it
7 # under the terms of the license (GNU LGPL) which comes with this package.
8 eval 'exec perl -S $0 ${1+"$@"}'
9 if $running_under_some_shell;
11 # If you change this file, please stick to the formatting you got with:
12 # perltidy --backup-and-modify-in-place --maximum-line-length=180 --output-line-ending=unix --cuddled-else
22 B<tesh> [I<options>]... I<testsuite>
26 Tesh is the testing shell, a specialized shell for running tests. It
27 provides the specified input to the tested commands, and check that
28 they produce the expected output and return the expected value.
32 --cd some/directory : ask tesh to switch the working directory before
34 --setenv var=value : set a specific environment variable
35 --cfg arg : add parameter --cfg=arg to each command line
36 --enable-coverage : ignore output lines starting with "profiling:"
38 =head1 TEST SUITE FILE SYTAX
40 A test suite is composed of one or several I<command blocks> separated
41 by empty lines, each of them being composed of a command to run, its
42 input text and the expected output.
44 The first char of each line specifies the type of line according to
45 the following list. The second char of each line is ignored.
47 `$' command to run in foreground
48 `&' command to run in background
50 `<' input to pass to the command
51 `>' output expected from the command
53 `!' metacommand, which can be one of:
54 `timeout' <integer>|no
55 `expect signal' <signal name>
56 `expect return' <integer>
57 `output' <ignore|display>
60 `p' an informative message to print
62 If the expected output do not match the produced output, or if the
63 command did not end as expected, Tesh provides an error message (see
64 the OUTPUT section below) and stops.
66 =head2 Command blocks examples
68 In a given command block, you can declare the command, its input and
69 its expected output in the order that you see fit.
83 You can group several commands together, provided that they don't have
89 =head2 Enforcing the command return code
91 By default, Tesh enforces that the tested command returns 0. If not,
92 it fails with an appropriate message and returns I<code+40> itself.
94 You specify that a given command block is expected to return another
97 # This command MUST return 42
101 The I<expect return> construct applies only to the next command block.
103 =head2 Commands that are expected to raise signals
105 By default, Tesh detects when the command is killed by a signal (such
106 as SEGV on segfaults). This is usually unexpected and unfortunate. But
107 if not, you can specify that a given command block is expected to fail
108 with a signal as follows:
110 # This command MUST raise a segfault
111 ! expect signal SIGSEGV
112 $ ./some_failing_code
114 The I<expect signal> construct applies only to the next command block.
118 By default, no command is allowed to run more than 5 seconds. You can
119 change this value as follows:
121 # Allow some more time to the command
123 $ ./some_longer_command
125 You can also disable the timeout completely by passing "no" as a value:
127 # This command will never timeout
129 $ ./some_very_long_but_safe_command
131 =head2 Setting environment variables
133 You can modify the environment of the tested commands as follows:
138 =head2 Not enforcing the expected output
140 By default, the commands output is matched against the one expected,
141 and an error is raised on discrepancy. Metacommands to change this:
147 The output is completely discarded.
151 The output is displayed, but no error is issued if it differs from the
156 The output is sorted before comparison (see next section).
160 =head2 Sorting output
162 If the order of the command output changes between runs, you want to
163 sort it before enforcing that it is exactly what you expect. In
164 SimGrid for example, this happens when parallel execution is
165 activated: User processes are run in parallel at each timestamp, and
166 the output is not reproducible anymore. Until you sort the lines.
168 You can sort the command output as follows:
171 $ ./some_multithreaded_command
173 Sorting lines this ways often makes the tesh output very intricate,
174 complicating the error analysis: the process logical order is defeated
175 by the lexicographical sort.
177 The solution is to prefix each line of your output with temporal
178 information so that lines can be grouped by timestamps. The
179 lexicographical sort then only applies to lines that occured at the
180 same timestamp. Here is a SimGrid example:
182 # Sort only lines depending on the first 19 chars
184 $ ./some_simgrid_simulator --log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n
186 This approach may seem surprizing at the first glance but it does its job:
190 =item Every timestamps remain separated, as it should;
192 =item In each timestamp, the output order of processes become
193 reproducible: that's the lexicographical order of their name;
195 =item For each process, the order of its execution is preserved: its
196 messages within a given timestamp are not reordered.
200 That way, tesh can do its job (no false positive, no false negative)
201 despite the unpredictable order of executions of processes within a
202 timestamp, and reported errors remain easy to analyze (execution of a
203 given process preserved).
205 This example is very SimGrid oriented, but the feature could even be
206 usable by others, who knows?
209 =head1 BUILTIN COMMANDS
211 =head2 mkfile: creating a file
213 This command creates a file of the name provided as argument, and adds
214 the content it gets as input.
220 It is not possible to use the cat command, as one would expect,
221 because stream redirections are currently not implemented in Tesh.
223 =head1 BUGS, LIMITATIONS AND POSSIBLE IMPROVEMENTS
225 The main limitation is the lack of stream redirections in the commands
226 (">", "<" and "|" shell constructs and friends). The B<mkfile> builtin
227 command makes this situation bearable.
229 It would be nice if we could replace the tesh file completely with
230 command line flags when the output is not to be verified.
235 # Disabling IPC::Run::Debug saves tons of useless calls.
236 $ENV{'IPCRUNDEBUG'} = 'none'
237 unless exists $ENV{'IPCRUNDEBUG'};
240 my $enable_coverage = 0;
242 my $diff_tool_tmp_fh = 0;
243 my $diff_tool_tmp_filename = 0;
244 my $sort_prefix = -1;
251 $SIG{'PIPE'} = 'IGNORE';
257 use lib "@CMAKE_BINARY_DIR@/bin";
259 use Diff qw(diff); # postpone a bit to have time to change INC
261 use Getopt::Long qw(GetOptions);
263 use Text::ParseWords;
264 use IPC::Run qw(start run timeout finish);
269 #### Portability bits for windows
272 use constant RUNNING_ON_WINDOWS => ( $OSNAME =~ /^(?:mswin|dos|os2)/oi );
273 use POSIX qw(:sys_wait_h WIFEXITED WIFSIGNALED WIFSTOPPED WEXITSTATUS WTERMSIG WSTOPSIG
274 :signal_h SIGINT SIGTERM SIGKILL SIGABRT SIGSEGV);
277 if (RUNNING_ON_WINDOWS) { # Missing on windows
278 *WIFEXITED = sub { not $_[0] & 127 };
279 *WEXITSTATUS = sub { $_[0] >> 8 };
280 *WIFSIGNALED = sub { ( $_[0] & 127 ) && ( $_[0] & 127 != 127 ) };
281 *WTERMSIG = sub { $_[0] & 127 };
287 #### Command line option handling
290 my %opts = ( "debug" => 0,
291 "timeout" => 5, # No command should run any longer than 5 seconds by default
294 Getopt::Long::config( 'bundling', 'no_getopt_compat', 'no_auto_abbrev' );
296 'debug|d' => \$opts{"debug"},
298 'difftool=s' => \$diff_tool,
300 'cd=s' => sub { cd_cmd( $_[1] ) },
301 'timeout=s' => \$opts{'timeout'},
302 'setenv=s' => sub { setenv_cmd( $_[1] ) },
303 'cfg=s' => sub { $opts{'cfg'} .= " --cfg=$_[1]" },
304 'enable-coverage+' => \$enable_coverage,
307 $tesh_file = pop @ARGV;
309 print "Enable coverage\n" if ($enable_coverage);
312 use File::Temp qw/ tempfile /;
313 ( $diff_tool_tmp_fh, $diff_tool_tmp_filename ) = tempfile();
314 print "New tesh: $diff_tool_tmp_filename\n";
317 if ( $tesh_file =~ m/(.*)\.tesh/ ) {
319 print "Test suite `$tesh_name'\n";
321 $tesh_file = "(stdin)";
322 $tesh_name = "(stdin)";
323 print "Test suite from stdin\n";
326 ###########################################################################
330 if ( WIFEXITED($status) ) {
331 $exitcode = WEXITSTATUS($status) + 40;
332 return "returned code " . WEXITSTATUS($status);
333 } elsif ( WIFSIGNALED($status) ) {
335 if ( WTERMSIG($status) == SIGINT ) { $code = "SIGINT"; }
336 elsif ( WTERMSIG($status) == SIGTERM ) { $code = "SIGTERM"; }
337 elsif ( WTERMSIG($status) == SIGKILL ) { $code = "SIGKILL"; }
338 elsif ( WTERMSIG($status) == SIGABRT ) { $code = "SIGABRT"; }
339 elsif ( WTERMSIG($status) == SIGSEGV ) { $code = "SIGSEGV"; }
340 $exitcode = WTERMSIG($status) + 4;
341 return "got signal $code";
343 return "Unparsable status. Is the process stopped?";
347 my %cmd = %{ $_[0] };
348 if ( $opts{'debug'} ) {
349 map { print "IN: $_\n" } @{ $cmd{'in'} };
350 map { print "OUT: $_\n" } @{ $cmd{'out'} };
351 print "CMD: $cmd{'cmd'}\n";
354 # cleanup the command line
355 if (RUNNING_ON_WINDOWS) {
356 var_subst( $cmd{'cmd'}, "EXEEXT", ".exe" );
358 var_subst( $cmd{'cmd'}, "EXEEXT", "" );
361 # substitute environ variables
362 foreach my $key ( keys %environ ) {
363 $cmd{'cmd'} = var_subst( $cmd{'cmd'}, $key, $environ{$key} );
366 # substitute remaining variables, if any
367 while ( $cmd{'cmd'} =~ /\${(\w+)(?::[=-][^}]*)?}/ ) {
368 $cmd{'cmd'} = var_subst( $cmd{'cmd'}, $1, "" );
370 while ( $cmd{'cmd'} =~ /\$(\w+)/ ) {
371 $cmd{'cmd'} = var_subst( $cmd{'cmd'}, $1, "" );
375 $cmd{'cmd'} .= " $opts{'cfg'}"
376 if ( defined( $opts{'cfg'} ) && length( $opts{'cfg'} ) );
378 # finally trim any remaining space chars
379 $cmd{'cmd'} =~ s/^\s+//;
380 $cmd{'cmd'} =~ s/\s+$//;
382 print "[$tesh_name:$cmd{'line'}] $cmd{'cmd'}\n";
384 $cmd{'return'} ||= 0;
385 $cmd{'timeout'} ||= $opts{'timeout'};
389 # exec the command line
391 my @cmdline = quotewords( '\s+', 0, $cmd{'cmd'} );
392 my $input = defined($cmd{'in'})? join("\n",@{$cmd{'in'}}) : "";
393 my $output = " " x 10240; $output = ""; # Preallocate 10kB, and reset length to 0
394 $cmd{'got'} = \$output;
395 $cmd{'job'} = start \@cmdline, '<', \$input, '>&', \$output,
396 ($cmd{'timeout'} eq 'no' ? () : timeout($cmd{'timeout'}));
398 if ( $cmd{'background'} ) {
399 # Just enqueue the job. It will be dealed with at the end
400 push @bg_cmds, \%cmd;
402 # Deal with its ending conditions right away
403 analyze_result( \%cmd );
408 my %cmd = %{ $_[0] };
411 finish( $cmd{'job'} );
414 if ($@ =~ /timeout/) {
415 $cmd{'job'}->kill_kill;
416 $cmd{'timeouted'} = 1;
417 } elsif ($@ =~ /^ack / and $@ =~ /pipe/) {
418 print STDERR "Tesh: Broken pipe (ignored).\n";
420 die $@; # Don't know what it is, so let it go.
423 $cmd{'timeouted'} ||= 0;
425 my $gotret = $cmd{'gotret'} = exit_status($?);
429 if ( defined( $cmd{'expect'} ) and ( $cmd{'expect'} ne "" ) ) {
430 $wantret = "got signal $cmd{'expect'}";
432 $wantret = "returned code " . ( defined( $cmd{'return'} ) ? $cmd{'return'} : 0 );
435 # pop all output from executing child
437 map { print "GOT: $_\n" } ${$cmd{'got'}} if $opts{'debug'};
438 foreach my $got ( split("\n", ${$cmd{'got'}}) ) {
441 print $diff_tool_tmp_fh "> $got\n" if ($diff_tool);
443 unless ( $enable_coverage and $got =~ /^profiling:/ ) {
448 if ( $cmd{'sort'} ) {
450 # Save the unsorted observed output to report it on error.
451 map { push @{ $cmd{'unsorted got'} }, $_ } @got;
454 substr( $a, 0, $sort_prefix ) cmp substr( $b, 0, $sort_prefix );
457 if ( $sort_prefix > 0 ) {
458 @got = sort mysort @got;
462 while ( @got and $got[0] eq "" ) {
466 # Sort the expected output to make it easier to write for humans
467 if ( defined( $cmd{'out'} ) ) {
468 if ( $sort_prefix > 0 ) {
469 @{ $cmd{'out'} } = sort mysort @{ $cmd{'out'} };
471 @{ $cmd{'out'} } = sort @{ $cmd{'out'} };
473 while ( @{ $cmd{'out'} } and ${ $cmd{'out'} }[0] eq "" ) {
474 shift @{ $cmd{'out'} };
481 if ( $cmd{'timeouted'} ) {
482 $gotret = "timeout after $cmd{'timeout'} sec";
485 print STDERR "<$cmd{'file'}:$cmd{'line'}> timeouted. Kill the process.\n";
487 if ( $gotret ne $wantret ) {
489 my $msg = "Test suite `$cmd{'file'}': NOK (<$cmd{'file'}:$cmd{'line'}> $gotret)\n";
491 $msg = $msg . "Output of <$cmd{'file'}:$cmd{'line'}> so far:\n";
492 map { $msg .= "|| $_\n" } @got;
494 $msg .= "<$cmd{'file'}:$cmd{'line'}> No output so far.\n";
499 # Does the output match?
501 if ( defined( $cmd{'output display'} ) ) {
502 print "[Tesh/INFO] Here is the (ignored) command output:\n";
503 map { print "||$_\n" } @got;
504 } elsif ( defined( $cmd{'output ignore'} ) ) {
505 print "(ignoring the output of <$cmd{'file'}:$cmd{'line'}> as requested)\n";
507 $diff = build_diff( \@{ $cmd{'out'} }, \@got );
509 if ( length $diff ) {
510 print "Output of <$cmd{'file'}:$cmd{'line'}> mismatch" . ( $cmd{'sort'} ? " (even after sorting)" : "" ) . ":\n";
511 map { print "$_\n" } split( /\n/, $diff );
512 if ( $cmd{'sort'} ) {
513 print "WARNING: Both the observed output and expected output were sorted as requested.\n";
514 print "WARNING: Output were only sorted using the $sort_prefix first chars.\n"
515 if ( $sort_prefix > 0 );
516 print "WARNING: Use <! output sort 19> to sort by simulated date and process ID only.\n";
518 # print "----8<--------------- Begin of unprocessed observed output (as it should appear in file):\n";
519 # map {print "> $_\n"} @{$cmd{'unsorted got'}};
520 # print "--------------->8---- End of the unprocessed observed output.\n";
523 print "Test suite `$cmd{'file'}': NOK (<$cmd{'file'}:$cmd{'line'}> output mismatch)\n";
529 my $infh; # The file descriptor from which we should read the teshfile
530 if ( $tesh_file eq "(stdin)" ) {
533 open $infh, $tesh_file
534 or die "[Tesh/CRITICAL] Unable to open $tesh_file: $!\n";
537 my %cmd; # everything about the next command to run
539 LINE: while ( defined( my $line = <$infh> ) and not $error ) {
544 print "[TESH/debug] $line_num: $line\n" if $opts{'debug'};
546 # deal with line continuations
547 while ( $line =~ /^(.*?)\\$/ ) {
549 die "[TESH/CRITICAL] Continued line at end of file\n"
550 unless defined($next);
553 print "[TESH/debug] $line_num: $next\n" if $opts{'debug'};
557 # Push delayed commands on empty lines
558 unless ( $line =~ m/^(.)(.*)$/ ) {
559 if ( defined( $cmd{'cmd'} ) ) {
563 print $diff_tool_tmp_fh "$line\n" if ($diff_tool);
567 my ( $cmd, $arg ) = ( $1, $2 );
568 print $diff_tool_tmp_fh "$line\n" if ( $diff_tool and $cmd ne '>' );
573 # handle the commands
574 if ( $cmd =~ /^#/ ) { # comment
575 } elsif ( $cmd eq '>' ) { # expected result line
576 print "[TESH/debug] push expected result\n" if $opts{'debug'};
577 push @{ $cmd{'out'} }, $arg;
579 } elsif ( $cmd eq '<' ) { # provided input
580 print "[TESH/debug] push provided input\n" if $opts{'debug'};
581 push @{ $cmd{'in'} }, $arg;
583 } elsif ( $cmd eq 'p' ) { # comment
584 print "[$tesh_name:$line_num] $arg\n";
586 } elsif ( $cmd eq '$' ) { # Command
587 # if we have something buffered, run it now
588 if ( defined( $cmd{'cmd'} ) ) {
592 if ( $arg =~ /^\s*mkfile / ) { # "mkfile" command line
593 die "[TESH/CRITICAL] Output expected from mkfile command!\n"
594 if scalar @{ cmd { 'out' } };
597 $cmd{'arg'} =~ s/\s*mkfile //;
601 } elsif ( $arg =~ /^\s*cd / ) {
602 die "[TESH/CRITICAL] Input provided to cd command!\n"
603 if scalar @{ cmd { 'in' } };
604 die "[TESH/CRITICAL] Output expected from cd command!\n"
605 if scalar @{ cmd { 'out' } };
611 } else { # regular command
613 $cmd{'file'} = $tesh_file;
614 $cmd{'line'} = $line_num;
616 } elsif ( $cmd eq '&' ) { # background command line
618 if ( defined( $cmd{'cmd'} ) ) {
622 $cmd{'background'} = 1;
624 $cmd{'file'} = $tesh_file;
625 $cmd{'line'} = $line_num;
627 } elsif ( $line =~ /^!\s*output sort/ ) { #output sort
628 if ( defined( $cmd{'cmd'} ) ) {
633 if ( $line =~ /^!\s*output sort\s+(\d+)/ ) {
636 } elsif ( $line =~ /^!\s*output ignore/ ) { #output ignore
637 if ( defined( $cmd{'cmd'} ) ) {
641 $cmd{'output ignore'} = 1;
642 } elsif ( $line =~ /^!\s*output display/ ) { #output display
643 if ( defined( $cmd{'cmd'} ) ) {
647 $cmd{'output display'} = 1;
648 } elsif ( $line =~ /^!\s*expect signal (\w*)/ ) { #expect signal SIGABRT
649 if ( defined( $cmd{'cmd'} ) ) {
653 $cmd{'expect'} = "$1";
654 } elsif ( $line =~ /^!\s*expect return/ ) { #expect return
655 if ( defined( $cmd{'cmd'} ) ) {
659 $line =~ s/^! expect return //g;
661 $cmd{'return'} = $line;
662 } elsif ( $line =~ /^!\s*setenv/ ) { #setenv
663 if ( defined( $cmd{'cmd'} ) ) {
667 $line =~ s/^! setenv //g;
670 } elsif ( $line =~ /^!\s*timeout/ ) { #timeout
671 if ( defined( $cmd{'cmd'} ) ) {
675 $line =~ s/^! timeout //;
677 $cmd{'timeout'} = $line;
679 die "[TESH/CRITICAL] parse error: $line\n";
683 # We're done reading the input file
684 close $infh unless ( $tesh_file eq "(stdin)" );
686 # Deal with last command
687 if ( defined( $cmd{'cmd'} ) ) {
694 analyze_result( \%test );
698 close $diff_tool_tmp_fh;
699 system("$diff_tool $diff_tool_tmp_filename $tesh_file");
700 unlink $diff_tool_tmp_filename;
705 } elsif ( $tesh_file eq "(stdin)" ) {
706 print "Test suite from stdin OK\n";
708 print "Test suite `$tesh_name' OK\n";
714 #### Helper functions
719 my $diff = Diff->new(@_);
721 $diff->Base(1); # Return line numbers, not indices
722 my $chunk_count = $diff->Next(-1); # Compute the amount of chuncks
723 return "" if ( $chunk_count == 1 && $diff->Same() );
725 while ( $diff->Next() ) {
726 my @same = $diff->Same();
727 if ( $diff->Same() ) {
728 if ( $diff->Next(0) > 1 ) { # not first chunk: print 2 first lines
729 $res .= ' ' . $same[0] . "\n";
730 $res .= ' ' . $same[1] . "\n" if ( scalar @same > 1 );
732 $res .= "...\n" if ( scalar @same > 2 );
734 # $res .= $diff->Next(0)."/$chunk_count\n";
735 if ( $diff->Next(0) < $chunk_count ) { # not last chunk: print 2 last lines
736 $res .= ' ' . $same[ scalar @same - 2 ] . "\n"
737 if ( scalar @same > 1 );
738 $res .= ' ' . $same[ scalar @same - 1 ] . "\n";
741 next if $diff->Same();
742 map { $res .= "- $_\n" } $diff->Items(1);
743 map { $res .= "+ $_\n" } $diff->Items(2);
748 # Helper function replacing any occurence of variable '$name' by its '$value'
749 # As in Bash, ${$value:=BLABLA} is rewritten to $value if set or to BLABLA if $value is not set
751 my ( $text, $name, $value ) = @_;
753 $text =~ s/\${$name(?::[=-][^}]*)?}/$value/g;
754 $text =~ s/\$$name(\W|$)/$value$1/g;
756 $text =~ s/\${$name:=([^}]*)}/$1/g;
757 $text =~ s/\${$name}//g;
758 $text =~ s/\$$name(\W|$)/$1/g;
763 ################################ The possible commands ################################
766 my %cmd = %{ $_[0] };
767 my $file = $cmd{'arg'};
768 print STDERR "[Tesh/INFO] mkfile $file. Ctn: >>".join( '\n', @{ $cmd{'in'} })."<<\n"
772 open( FILE, ">$file" )
773 or die "[Tesh/CRITICAL] Unable to create file $file: $!\n";
774 print FILE join( "\n", @{ $cmd{'in'} } );
775 print FILE "\n" if ( scalar @{ $cmd{'in'} } > 0 );
779 # Command CD. Just change to the provided directory
781 my $directory = shift;
783 if ( -e $directory && -d $directory ) {
785 print "[Tesh/INFO] change directory to $directory\n";
787 } elsif ( -e $directory ) {
788 print "Cannot change directory to '$directory': it is not a directory\n";
790 print "Chdir to $directory failed: No such file or directory\n";
792 if ( $failure == 1 ) {
793 print "Test suite `$tesh_file': NOK (system error)\n";
798 # Command setenv. Gets "variable=content", and update the environment accordingly
801 if ( $arg =~ /^(.*)=(.*)$/ ) {
802 my ( $var, $ctn ) = ( $1, $2 );
803 print "[Tesh/INFO] setenv $var=$ctn\n";
804 $environ{$var} = $ctn;
807 die "[Tesh/CRITICAL] Malformed argument to setenv: expected 'name=value' but got '$arg'\n";