Test::Cmd - Perl module for portable testing of commands and scripts
use Test::Cmd;
$test = Test::Cmd->new(prog => 'program_or_script_to_test', interpreter => 'script_interpreter', string => 'identifier_string', workdir => '', subdir => 'dir', match_sub => $code_ref, verbose => 1);
$test->verbose(1);
$test->prog('program_or_script_to_test');
$test->basename(@suffixlist);
$test->interpreter('script_interpreter');
$test->string('identifier string');
$test->workdir('prefix');
$test->workpath('subdir', 'file');
$test->subdir('subdir', ...); $test->subdir(['sub', 'dir'], ...);
$test->write('file', <<'EOF'); contents of file EOF $test->write(['subdir', 'file'], <<'EOF'); contents of file EOF
$test->read(\$contents, 'file'); $test->read(\@lines, 'file'); $test->read(\$contents, ['subdir', 'file']); $test->read(\@lines, ['subdir', 'file']);
$test->writable('dir', rwflag);
$test->preserve(condition, ...);
$test->cleanup(condition);
$test->run(prog => 'program_or_script_to_test', interpreter => 'script_interpreter', chdir => 'dir', args => 'arguments', stdin => <<'EOF'); input to program EOF
$test->pass(condition); $test->pass(condition, funcref);
$test->fail(condition); $test->fail(condition, funcref); $test->fail(condition, funcref, caller);
$test->no_result(condition); $test->no_result(condition, funcref); $test->no_result(condition, funcref, caller);
$test->stdout; $test->stdout(run_number);
$test->stderr; $test->stderr(run_number);
$test->match(\@lines, \@matches); $test->match($lines, $matches);
$test->match_exact(\@lines, \@matches); $test->match_exact($lines, $matches);
$test->match_regex(\@lines, \@regexes); $test->match_regex($lines, $regexes);
$test->diff_exact(\@lines, \@matches, \@output); $test->diff_exact($lines, $matches, \@output);
$test->diff_regex(\@lines, \@regexes, \@output); $test->diff_regex($lines, $regexes, \@output);
sub func { my ($self, $lines, $matches) = @_; # code to match $lines and $matches } $test->match_sub(\&func); $test->match_sub(sub { code to match $_[1] and $_[2] });
$test->here;
The Test::Cmd
module provides a framework for portable automated
testing of executable commands and scripts (in any language, not
just Perl), especially commands and scripts that require file system
interaction.
In addition to running tests and evaluating conditions, the Test::Cmd
module manages and cleans up one or more temporary workspace directories,
and provides methods for creating files and directories in those workspace
directories from in-line data (that is, here-documents), allowing tests
to be completely self-contained.
The Test::Cmd
module inherits File::Spec methods
(file_name_is_absolute()
, catfile()
, etc.) to support writing
tests portably across a variety of operating and file systems.
A Test::Cmd
environment object is created via the usual invocation:
$test = Test::Cmd->new();
Arguments to the Test::Cmd::new
method are keyword-value pairs that
may be used to initialize the object, typically by invoking the same-named
method as the keyword.
No Test::Cmd
methods (including the new()
method) exit, die
or throw any other sorts of exceptions (but they all do return useful
error indications). Exceptions should be handled by the test itself or
a subclass specific to the program under test.
The Test::Cmd
module may be used in conjunction with the Test
module
to report test results in a format suitable for the Test::Harness
module. A typical use would be to call the Test::Cmd
methods to
prepare and execute the test, and call the ok()
method exported by the
Test
module to test the conditions:
use Test; use Test::Cmd; BEGIN { $| = 1; plan => 3 } $test = Test::Cmd->new(prog => 'test_program', workdir => ''); ok($test); $wrote_file = $test->write('input_file', <<'EOF'); This is input to test_program, which we expect to process this and exit successfully (status 0). EOF ok($wrote_file); $test->run(args => 'input_file'); ok($? == 0);
Alternatively, the Test::Cmd
module provides pass()
, fail()
,
and no_result()
methods that report test results for use with the Aegis
change management system. These methods terminate the test immediately,
reporting PASSED, FAILED, or NO RESULT respectively, and exiting with
status 0 (success), 1 or 2 respectively. This allows for a distinction
between an actual failed test and a test that could not be properly
evaluated because of an external condition (such as a full file system
or incorrect permissions):
use Test::Cmd; $test = Test::Cmd->new(prog => 'test_program', workdir => ''); Test::Cmd->no_result(! $test); $wrote_file = $test->write('input_file', <<'EOF'); This is input to test_program, which we expect to process this and exit successfully (status 0). EOF $test->no_result(! $wrote_file); $test->run(args => 'input_file'); $test->fail($? != 0); $test->pass;
It is not a good idea to intermix the two reporting models. If you use
the Test
module and its ok
method, do not use the Test::Cmd
pass
, fail
or no_result
methods, and vice versa.
Methods supported by the Test::Cmd
module include:
new
Test::Cmd
environment. Arguments with which to initialize
the environment are passed in as keyword-value pairs. Fails if a
specified temporary working directory or subdirectory cannot be created.
Does NOT die or exit on failure, but returns FALSE if the test environment
object cannot be created.
verbose
prog
basename
interpreter
prog
as a script.
Returns the current value of interpreter
.
string
workdir
testcmd
by default, followed by the
unique ID of the executing process.
Returns the absolute pathname to the temporary working directory, or FALSE if the directory could not be created.
workpath
subdir
File::Spec-&
catfile>
method. Subdirectories multiple levels deep must be created via a
separate argument for each level:
$test->subdir('sub', ['sub', 'dir'], [qw(sub dir ectory)]);
Returns the number of subdirectories actually created.
write
read
Returns TRUE on successfully opening and reading the file, FALSE otherwise.
writable
rwflag
== TRUE) or not
writable (rwflag
== FALSE).
preserve
Test::Cmd
environment to be preserved for one or more conditions.
If no conditions are specified, arranges for the temporary working
directories to be preserved for all conditions.
cleanup
Test::Cmd
environment. If the environment variable PRESERVE
was set when
the Test::Cmd
module was loaded, temporary working directories are
not removed. If any of the environment variables PRESERVE_PASS
,
PRESERVE_FAIL
, or PRESERVE_NO_RESULT
were set when the Test::Cmd
module was loaded, then temporary working directories are not removed
if the test passed, failed, or had no result, respectively. Temporary
working directories are also preserved for conditions specified via the
preserve
method.
Typically, this method is not called directly, but is used when the script exits to clean up temporary working directories as appropriate for the exit status.
run
stdout
and stderr
methods.
Arguments are supplied as keyword-value pairs:
args
$test->run(args => 'arg1 arg2');
chdir
$test->run(chdir => 'xyzzy');
If the specified path is not an absolute path name (begins with '/'
on Unix systems), then the subdirectory is relative to the temporary
working directory for the environment ($test-&
workdir>). Note that,
by default, the Test::Cmd
module does NOT chdir to the temporary
working directory, so to execute the test under the temporary working
directory, you must specify an explicit chdir
to the current directory:
$test->run(chdir => '.'); # Unix-specific
$test->run(chdir => $test->curdir); # portable
interpreter
prog
as a script,
for this run only. This does not change the $test-&
interpreter>
value of the test environment.
prog
$test-&
prog> value of the test environment.
stdin
$test->run(stdin => <<_EOF_); input to the program under test _EOF_
Returns the exit status of the program or script.
pass
fail
no_result
stdout
undef
if
there has been no test run.
stderr
undef
if there has
been no test run.
match
match_regex
method, which means that
the default is to match lines against regular expressions.
match_exact
Returns TRUE if each line matched its corresponding line in the other array, FALSE otherwise.
match_regex
Returns TRUE if each line matched each regular expression, FALSE otherwise.
diff_exact
diff(1)
utility.
If the Algorithm::DiffOld
package is installed on the local system,
output describing the differences between the input lines and the
matching lines, in diff(1)
format, is saved to the $output
array
reference. In the diff output, the expected output lines are considered
the ``old'' (left-hand) file, and the actual output is considered the
``new'' (right-hand) file.
If the Algorithm::DiffOld
package is not installed on the local
system, the Expected and Actual contents are saved as-is to the
$output
array reference.
The lines
and matches
arguments are passed in as either scalars,
in which case each is split on newline boundaries, or as array
references. Trailing newlines are stripped from each line and regular
expression.
Returns TRUE if each line matched its corresponding line in the expected
matches, FALSE otherwise, in order to conform to the conventions of the
match
method.
Typical invocation:
if (! $test->diff_exact($test->stdout, \@expected_lines, \@diff)) { print @diff; }
diff_regex
diff(1)
utility.
If the Algorithm::DiffOld
package is installed on the local system,
output describing the differences between the input lines and the
matching lines, in diff(1)
format, is saved to the $output
array
reference. In the diff output, the expected output lines are considered
the ``old'' (left-hand) file, and the actual output is considered the
``new'' (right-hand) file.
If the Algorithm::DiffOld
package is not installed on the local
system, the Expected and Actual contents are saved as-is to the
$output
array reference.
The lines
and regexes
arguments are passed in as either scalars,
in which case each is split on newline boundaries, or as array
references. Trailing newlines are stripped from each line and regular
expression. Comparison is performed for each entire line, that is, with
each regular expression anchored at both the start of line (^) and end
of line ($).
Returns TRUE if each line matched each regular expression, FALSE
otherwise, in order to conform to the conventions of the match
method.
Typical invocation:
if (! $test->diff_regex($test->stdout, \@expected_lines, \@diff)) { print @diff; }
match_sub
match
method. This can be a user-supplied
subroutine, or the match_exact
, match_regex
, diff_exact
, or
diff_regex
methods supplied by the Test::Cmd
module:
$test->match_sub(\&Test::Cmd::match_exact);
$test->match_sub(\&Test::Cmd::match_regex);
$test->match_sub(\&Test::Cmd::diff_exact);
$test->match_sub(\&Test::Cmd::diff_regex);
The match_exact
, match_regex
, diff_exact
and diff_regex
subroutine names are exportable from the Test::Cmd
module, and may be
specified at object initialization:
use Test::Cmd qw(match_exact match_regex diff_exact diff_regex); $test_exact = Test::Cmd->new(match_sub => \&match_exact); $test_regex = Test::Cmd->new(match_sub => \&match_regex); $test_exact = Test::Cmd->new(match_sub => \&diff_exact); $test_regex = Test::Cmd->new(match_sub => \&diff_regex);
here
Cwd::cwd
method, except that the
Test::Cmd::here
method preserves the directory separators exactly
as returned by the underlying operating-system-dependent method.
The Cwd::cwd
method canonicalizes all directory separators to '/',
which makes for consistent path name representations within Perl, but may
mess up another program or script to which you try to pass the path name.)
Several environment variables affect the default values in a newly created
Test::Cmd
environment object. These environment variables must be set
when the module is loaded, not when the object is created.
PRESERVE
PRESERVE_FAIL
PRESERVE_NO_RESULT
PRESERVE_PASS
VERBOSE
Although the Test::Cmd
module is intended to make it easier to write
portable tests for portable utilities that interact with file systems,
it is still very easy to write non-portable tests if you're not careful.
The best and most comprehensive set of portability guidelines is the standard ``Writing portable Perl'' document at:
http://www.perl.com/pub/doc/manual/html/pod/perlport.html
To reiterate one important point from the ``WpP'' document: Not all Perl
programs have to be portable. If the program or script you're testing
is UNIX-specific, you can (and should) use the Test::Cmd
module to
write UNIX-specific tests.
That having been said, here are some hints that may help keep your tests portable, if that's a requirement.
Test::Cmd-&
here> method for current directory path.Cwd::cwd
method. Unfortunately, the Cwd::cwd
method canonicalizes
the path name it returns, changing the native directory separators into
the forward slashes favored by Perl and UNIX. For most Perl scripts,
this makes a great deal of sense and keeps code uncluttered.
Passing in a file name that has had its directory separators altered,
however, may confuse the command or script under test, or make it
difficult to compare output from the command or script with an expected
result. The Test::Cmd::here
method returns the absolute path name of
the current working directory, like Cwd::cwd
, but does not manipulate
the returned path in any way.
File::Spec
methods for manipulating path names.File::Spec
module provides a system-independent interface for
manipulating path names. Because the Test::Cmd
class is a sub-class
of the File::Spec
class, you can use these methods directly as follows:
if (! Test::Cmd->file_name_is_absolute($prog)) { my $prog = Test::Cmd->catfile(Test::Cmd->here, $prog); }
For details about the available methods and their use, see the
documentation for the File::Spec
module and its sub-modules, especially
the File::Spec::Unix
modules.
Config
for file-name suffixes, where possible.Config
module provides values that reflect the file-name
suffixes on the system for which the Perl executable was built.
This provides convenient portability for situations where a file name
may have different extensions on different systems:
$foo_exe = "foo$Config{_exe}"; ok(-f $foo_exe);
(Unfortunately, there is no existing $Config
value that specifies
the suffix for a directly-executable Perl script.)
If your test somehow requires executing a script that you generate
from the test itself, the best way is to generate the script in Perl
and then explicitly feed it to the Perl executable on the local system.
To be maximally portable, use the $^X
variable instead of hard-coding
``perl'' into the string you execute:
$line = "This is output from the generated perl script."; $test->write('script', <<EOF); print STDOUT "$line\\n"; EOF $output = `$^X script`; ok($output eq "$line\n");
This completely avoids having to make the script
file itself
executable. (Since you're writing your test in Perl, it's safe to assume
that Perl itself is executable.)
If you must generate a directly-executable script, then use the
$Config{'startperl'}
variable at the start of the script to generate
the appropriate magic that will execute it as a Perl script:
use Config; $line = "This is output from the generated perl script."; $test->write('script', <<EOF); $Config{'startperl'}; print STDOUT "$line\\n"; EOF chdir($test->workdir); chmod(0755, 'script'); # POSIX-SPECIFIC $output = `script`; ok($output eq "$line\n");
Addtional hints on writing portable tests are welcome.
perl(1), Algorithm::DiffOld(3), File::Find(3), File::Spec(3), Test(3), Test::Harness(3).
A rudimentary page for the Test::Cmd module is available at:
http://www.baldmt.com/Test-Cmd/
The most involved example of using the Test::Cmd package to test a
real-world application is the cons-test
testing suite for the
Cons software construction utility. The suite sub-classes Test::Cmd
to provide common, application-specific infrastructure across a
large number of end-to-end application tests. The suite, and other
information about Cons, is available at:
http://www.dsmit.com/cons
Steven Knight, knight@baldmt.com
Copyright 1999-2001 Steven Knight. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
Thanks to Greg Spencer for the inspiration to create this package and the initial draft of its implementation as a specific testing package for the Cons software construction utility. Information about Cons is available at:
http://www.dsmit.com/cons/
The general idea of managing temporary working directories in this way,
as well as the test reporting of the pass
, fail
and no_result
methods, come from the testing framework invented by Peter Miller for
his Aegis project change supervisor. Aegis is an excellent bit of work
which integrates creation and execution of regression tests into the
software development process. Information about Aegis is available at:
http://www.tip.net.au/~millerp/aegis.html