@c DO NOT EDIT!  Generated automatically by munge-texi.

@c Copyright (C) 2005-2011 David Bateman
@c Copyright (C) 2002-2005 Paul Kienzle
@c
@c This file is part of Octave.
@c
@c Octave is free software; you can redistribute it and/or modify it
@c under the terms of the GNU General Public License as published by the
@c Free Software Foundation; either version 3 of the License, or (at
@c your option) any later version.
@c 
@c Octave is distributed in the hope that it will be useful, but WITHOUT
@c ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
@c FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
@c for more details.
@c 
@c You should have received a copy of the GNU General Public License
@c along with Octave; see the file COPYING.  If not, see
@c <http://www.gnu.org/licenses/>.

@node Test and Demo Functions
@appendix Test and Demo Functions
@cindex test functions

Octave includes a number of functions to allow the integration of testing
and demonstration code in the source code of the functions themselves.

@menu
* Test Functions::
* Demonstration Functions::
@end menu

@node Test Functions
@section Test Functions

@c test scripts/testfun/test.m
@anchor{doc-test}
@deftypefn  {Command} {} test @var{name}
@deftypefnx {Command} {} test @var{name} quiet|normal|verbose
@deftypefnx {Function File} {} test ('@var{name}', 'quiet|normal|verbose', @var{fid})
@deftypefnx {Function File} {} test ([], 'explain', @var{fid})
@deftypefnx {Function File} {@var{success} =} test (@dots{})
@deftypefnx {Function File} {[@var{n}, @var{max}] =} test (@dots{})
@deftypefnx {Function File} {[@var{code}, @var{idx}] =} test ('@var{name}', 'grabdemo')

Perform tests from the first file in the loadpath matching @var{name}.
@code{test} can be called as a command or as a function.  Called with
a single argument @var{name}, the tests are run interactively and stop
after the first error is encountered.

With a second argument the tests which are performed and the amount of
output is selected.

@table @asis
@item 'quiet'
 Don't report all the tests as they happen, just the errors.

@item 'normal'
Report all tests as they happen, but don't do tests which require
user interaction.

@item 'verbose'
Do tests which require user interaction.
@end table

The argument @var{fid} can be used to allow batch processing.  Errors
can be written to the already open file defined by @var{fid}, and
hopefully when Octave crashes this file will tell you what was happening
when it did.  You can use @code{stdout} if you want to see the results as
they happen.  You can also give a file name rather than an @var{fid}, in
which case the contents of the file will be replaced with the log from
the current test.

Called with a single output argument @var{success}, @code{test} returns
true if all of the tests were successful.  Called with two output arguments
@var{n} and @var{max}, the number of successful tests and the total number
of tests in the file @var{name} are returned.

If the second argument is the string 'grabdemo', the contents of the demo
blocks are extracted but not executed.  Code for all code blocks is
concatenated and returned as @var{code} with @var{idx} being a vector of
positions of the ends of the demo blocks.

If the second argument is 'explain', then @var{name} is ignored and an
explanation of the line markers used is written to the file @var{fid}.
@seealso{@ref{doc-assert,,assert}, @ref{doc-fail,,fail}, @ref{doc-error,,error}, @ref{doc-demo,,demo}, @ref{doc-example,,example}}
@end deftypefn


@code{test} scans the named script file looking for lines which
start with @code{%!}.  The prefix is stripped off and the rest of the
line is processed through the Octave interpreter.  If the code
generates an error, then the test is said to fail.

Since @code{eval()} will stop at the first error it encounters, you must
divide your tests up into blocks, with anything in a separate
block evaluated separately.  Blocks are introduced by the keyword
@code{test} immediately following the @code{%!}.  For example:

@example
@group
   %!test error ("this test fails!");
   %!test "test doesn't fail. it doesn't generate an error";
@end group
@end example

When a test fails, you will see something like:

@example
@group
     ***** test error ('this test fails!')
   !!!!! test failed
   this test fails!
@end group
@end example

Generally, to test if something works, you want to assert that it
produces a correct value.  A real test might look something like

@example
@group
   %!test
   %! @var{a} = [1, 2, 3; 4, 5, 6]; B = [1; 2];
   %! expect = [ @var{a} ; 2*@var{a} ];
   %! get = kron (@var{b}, @var{a});
   %! if (any(size(expect) != size(get)))
   %!    error ("wrong size: expected %d,%d but got %d,%d",
   %!           size(expect), size(get));
   %! elseif (any(any(expect!=get)))
   %!    error ("didn't get what was expected.");
   %! endif
@end group
@end example

To make the process easier, use the @code{assert} function.  For example,
with @code{assert} the previous test is reduced to:

@example
@group
   %!test
   %! @var{a} = [1, 2, 3; 4, 5, 6]; @var{b} = [1; 2];
   %! assert (kron (@var{b}, @var{a}), [ @var{a}; 2*@var{a} ]);
@end group
@end example

@code{assert} can accept a tolerance so that you can compare results
absolutely or relatively.  For example, the following all succeed:

@example
@group
   %!test assert (1+eps, 1, 2*eps)          # absolute error
   %!test assert (100+100*eps, 100, -2*eps) # relative error
@end group
@end example

You can also do the comparison yourself, but still have assert
generate the error:

@example
@group
   %!test assert (isempty([]))
   %!test assert ([ 1,2; 3,4 ] > 0)
@end group
@end example

Because @code{assert} is so frequently used alone in a test block, there
is a shorthand form:

@example
   %!assert (@dots{})
@end example

@noindent
which is equivalent to:

@example
   %!test assert (@dots{})
@end example

Sometimes during development there is a test that should work but is
known to fail.  You still want to leave the test in because when the
final code is ready the test should pass, but you may not be able to
fix it immediately.  To avoid unnecessary bug reports for these known
failures, mark the block with @code{xtest} rather than @code{test}:

@example
@group
   %!xtest assert (1==0)
   %!xtest fail ('success=1','error'))
@end group
@end example

Another use of @code{xtest} is for statistical tests which should
pass most of the time but are known to fail occasionally.

Each block is evaluated in its own function environment, which means
that variables defined in one block are not automatically shared
with other blocks.  If you do want to share variables, then you
must declare them as @code{shared} before you use them.  For example, the
following declares the variable @var{a}, gives it an initial value (default
is empty), then uses it in several subsequent tests.

@example
@group
   %!shared @var{a}
   %! @var{a} = [1, 2, 3; 4, 5, 6];
   %!assert (kron ([1; 2], @var{a}), [ @var{a}; 2*@var{a} ]);
   %!assert (kron ([1, 2], @var{a}), [ @var{a}, 2*@var{a} ]);
   %!assert (kron ([1,2; 3,4], @var{a}), [ @var{a},2*@var{a}; 3*@var{a},4*@var{a} ]);
@end group
@end example

You can share several variables at the same time:

@example
   %!shared @var{a}, @var{b}
@end example

You can also share test functions:

@example
@group
   %!function @var{a} = fn(@var{b})
   %!  @var{a} = 2*@var{b};
   %!assert (@var{a}(2),4);
@end group
@end example

Note that all previous variables and values are lost when a new 
shared block is declared.

Error and warning blocks are like test blocks, but they only succeed 
if the code generates an error.  You can check the text of the error
is correct using an optional regular expression @code{<pattern>}.  
For example:

@example
   %!error <passes!> error('this test passes!');
@end example

If the code doesn't generate an error, the test fails.  For example:

@example
   %!error "this is an error because it succeeds.";
@end example

@noindent
produces

@example
@group
   ***** error "this is an error because it succeeds.";
   !!!!! test failed: no error
@end group
@end example

It is important to automate the tests as much as possible, however
some tests require user interaction.  These can be isolated into
demo blocks, which if you are in batch mode, are only run when 
called with @code{demo} or @code{verbose}.  The code is displayed before
it is executed.  For example,

@example
@group
   %!demo
   %! @var{t}=[0:0.01:2*pi]; @var{x}=sin(@var{t});
   %! plot(@var{t},@var{x});
   %! you should now see a sine wave in your figure window
@end group
@end example

@noindent
produces

@example
@group
   > @var{t}=[0:0.01:2*pi]; @var{x}=sin(@var{t});
   > plot(@var{t},@var{x});
   > you should now see a sine wave in your figure window
   Press <enter> to continue: 
@end group
@end example

Note that demo blocks cannot use any shared variables.  This is so
that they can be executed by themselves, ignoring all other tests.

If you want to temporarily disable a test block, put @code{#} in place
of the block type.  This creates a comment block which is echoed
in the log file, but is not executed.  For example:

@example
@group
   %!#demo
   %! @var{t}=[0:0.01:2*pi]; @var{x}=sin(@var{t});
   %! plot(@var{t},@var{x});
   %! you should now see a sine wave in your figure window
@end group
@end example

Block type summary:

@table @code
@item %!test
check that entire block is correct

@item %!error
check for correct error message

@item %!warning
check for correct warning message

@item %!demo
demo only executes in interactive mode

@item %!#
comment: ignore everything within the block

@item %!shared x,y,z
declares variables for use in multiple tests

@item %!function
defines a function value for a shared variable

@item %!assert (x, y, tol)
shorthand for %!test assert (x, y, tol)
@end table

You can also create test scripts for builtins and your own C++
functions.  Just put a file of the function name on your path without
any extension and it will be picked up by the test procedure.  You
can even embed tests directly in your C++ code:

@example
@group
   #if 0
   %!test disp('this is a test')
   #endif
@end group
@end example

@noindent
or

@example
@group
   /*
   %!test disp('this is a test')
   */
@end group
@end example

@noindent
but then the code will have to be on the load path and the user 
will have to remember to type test('name.cc').  Conversely, you
can separate the tests from normal Octave script files by putting
them in plain files with no extension rather than in script files.
@c DO I WANT TO INCLUDE THE EDITOR SPECIFIC STATEMENT BELOW???
@c Don't forget to tell emacs that the plain text file you are using
@c is actually octave code, using something like:
@c   -*-octave-*-

@c assert scripts/testfun/assert.m
@anchor{doc-assert}
@deftypefn  {Function File} {} assert (@var{cond})
@deftypefnx {Function File} {} assert (@var{cond}, @var{errmsg}, @dots{})
@deftypefnx {Function File} {} assert (@var{cond}, @var{msg_id}, @var{errmsg}, @dots{})
@deftypefnx {Function File} {} assert (@var{observed}, @var{expected})
@deftypefnx {Function File} {} assert (@var{observed}, @var{expected}, @var{tol})

Produces an error if the condition is not met.  @code{assert} can be
called in three different ways.

@table @code
@item assert (@var{cond})
@itemx assert (@var{cond}, @var{errmsg}, @dots{})
@itemx assert (@var{cond}, @var{msg_id}, @var{errmsg}, @dots{})
Called with a single argument @var{cond}, @code{assert} produces an
error if @var{cond} is zero.  If called with a single argument a
generic error message.  With more than one argument, the additional
arguments are passed to the @code{error} function.

@item assert (@var{observed}, @var{expected})
Produce an error if observed is not the same as expected.  Note that
observed and expected can be strings, scalars, vectors, matrices,
lists or structures.

@item assert(@var{observed}, @var{expected}, @var{tol})
Accept a tolerance when comparing numbers.
If @var{tol} is positive use it as an absolute tolerance, will produce an
error if
@code{abs(@var{observed} - @var{expected}) > abs(@var{tol})}.
If @var{tol} is negative use it as a relative tolerance, will produce an
error if
@code{abs(@var{observed} - @var{expected}) > abs(@var{tol} *
@var{expected})}.  If @var{expected} is zero @var{tol} will always be used as
an absolute tolerance.
@end table
@seealso{@ref{doc-test,,test}}
@end deftypefn


@c fail scripts/testfun/fail.m
@anchor{doc-fail}
@deftypefn  {Function File} {} fail (@var{code}, @var{pattern})
@deftypefnx {Function File} {} fail (@var{code}, 'warning', @var{pattern})

Return true if @var{code} fails with an error message matching
@var{pattern}, otherwise produce an error.  Note that @var{code}
is a string and if @var{code} runs successfully, the error produced is:

@example
          expected error but got none
@end example

If the code fails with a different error, the message produced is:

@example
@group
          expected <pattern>
          but got <text of actual error>
@end group
@end example

The angle brackets are not part of the output.

Called with three arguments, the behavior is similar to
@code{fail(@var{code}, @var{pattern})}, but produces an error if no
warning is given during code execution or if the code fails.

@end deftypefn


@node Demonstration Functions
@section Demonstration Functions

@c demo scripts/testfun/demo.m
@anchor{doc-demo}
@deftypefn {Function File} {} demo ('@var{name}', @var{n})

Runs any examples associated with the function '@var{name}'.
Examples are stored in the script file, or in a file with the same
name but no extension somewhere on your path.  To keep them separate
from the usual script code, all lines are prefixed by @code{%!}.  Each
example is introduced by the keyword 'demo' flush left to the prefix,
with no intervening spaces.  The remainder of the example can contain
arbitrary Octave code.  For example:

@example
@group
   %!demo
   %! t=0:0.01:2*pi; x = sin(t);
   %! plot(t,x)
   %! %-------------------------------------------------
   %! % the figure window shows one cycle of a sine wave
@end group
@end example

Note that the code is displayed before it is executed, so a simple
comment at the end suffices.  It is generally not necessary to use
disp or printf within the demo.

Demos are run in a function environment with no access to external
variables.  This means that all demos in your function must use
separate initialization code.  Alternatively, you can combine your
demos into one huge demo, with the code:

@example
   %! input("Press <enter> to continue: ","s");
@end example

@noindent
between the sections, but this is discouraged.  Other techniques
include using multiple plots by saying figure between each, or
using subplot to put multiple plots in the same window.

Also, since demo evaluates inside a function context, you cannot
define new functions inside a demo.  Instead you will have to
use @code{eval(example('function',n))} to see them.  Because eval only
evaluates one line, or one statement if the statement crosses
multiple lines, you must wrap your demo in "if 1 <demo stuff> endif"
with the 'if' on the same line as 'demo'.  For example:

@example
@group
  %!demo if 1
  %!  function y=f(x)
  %!    y=x;
  %!  endfunction
  %!  f(3)
  %! endif
@end group
@end example
@seealso{@ref{doc-test,,test}, @ref{doc-example,,example}}
@end deftypefn


@c rundemos scripts/testfun/rundemos.m
@anchor{doc-rundemos}
@deftypefn {Function File} {} rundemos (@var{directory})
@end deftypefn


@c runtests scripts/testfun/runtests.m
@anchor{doc-runtests}
@deftypefn {Function File} {} runtests (@var{directory})
Execute built-in tests for all function files in the specified directory.
@end deftypefn


@c example scripts/testfun/example.m
@anchor{doc-example}
@deftypefn  {Function File} {} example ('@var{name}', @var{n})
@deftypefnx {Function File} {[@var{x}, @var{idx}] =} example ('@var{name}', @var{n})

 Display the code for example @var{n} associated with the function
'@var{name}', but do not run it.  If @var{n} is not given, all examples
are displayed.

Called with output arguments, the examples are returned in the form of
a string @var{x}, with @var{idx} indicating the ending position of the
various examples.

See @code{demo} for a complete explanation.
@seealso{@ref{doc-demo,,demo}, @ref{doc-test,,test}}
@end deftypefn


@c speed scripts/testfun/speed.m
@anchor{doc-speed}
@deftypefn  {Function File} {} speed (@var{f}, @var{init}, @var{max_n}, @var{f2}, @var{tol})
@deftypefnx {Function File} {[@var{order}, @var{n}, @var{T_f}, @var{T_f2}] =} speed (@dots{})

Determine the execution time of an expression (@var{f}) for various input
values (@var{n}).  The @var{n} are log-spaced from 1 to @var{max_n}.  For
each @var{n}, an initialization expression (@var{init}) is computed to
create any data needed for the test.  If a second expression (@var{f2}) is
given then the execution times of the two expressions are compared.  When
called without output arguments the results are displayed graphically.

@table @code
@item @var{f}
The expression to evaluate.

@item @var{max_n}
The maximum test length to run.  Default value is 100.  Alternatively,
use @code{[min_n, max_n]} or specify the @var{n} exactly with
@code{[n1, n2, @dots{}, nk]}.

@item @var{init}
Initialization expression for function argument values.  Use @var{k}
for the test number and @var{n} for the size of the test.  This should
compute values for all variables used by @var{f}.  Note that @var{init} will
be evaluated first for @math{k = 0}, so things which are constant throughout
the test series can be computed once.  The default value is
@code{@var{x} = randn (@var{n}, 1)}.

@item @var{f2}
An alternative expression to evaluate, so that the speed of two
expressions can be directly compared.  The default is @code{[]}.

@item @var{tol}
Tolerance used to compare the results of expression @var{f} and expression
@var{f2}.  If @var{tol} is positive, the tolerance is an absolute one.
If @var{tol} is negative, the tolerance is a relative one.  The default is
@code{eps}.  If @var{tol} is @code{Inf}, then no comparison will be made.

@item @var{order}
The time complexity of the expression @math{O(a*n^p)}.  This
is a structure with fields @code{a} and @code{p}.

@item @var{n}
The values @var{n} for which the expression was calculated AND
the execution time was greater than zero.

@item @var{T_f}
The nonzero execution times recorded for the expression @var{f} in seconds.

@item @var{T_f2}
The nonzero execution times recorded for the expression @var{f2} in seconds.
If required, the mean time ratio is simply @code{mean (T_f./T_f2)}.

@end table

The slope of the execution time graph shows the approximate
power of the asymptotic running time @math{O(n^p)}.  This
power is plotted for the region over which it is approximated
(the latter half of the graph).  The estimated power is not
very accurate, but should be sufficient to determine the
general order of an algorithm.  It should indicate if, for
example, the implementation is unexpectedly @math{O(n^2)}
rather than @math{O(n)} because it extends a vector each
time through the loop rather than pre-allocating storage.
In the current version of Octave, the following is not the
expected @math{O(n)}.

@example
speed ("for i = 1:n, y@{i@} = x(i); endfor", "", [1000, 10000])
@end example

@noindent
But it is if you preallocate the cell array @code{y}:

@example
@group
speed ("for i = 1:n, y@{i@} = x(i); endfor", ...
       "x = rand (n, 1); y = cell (size (x));", [1000, 10000])
@end group
@end example

An attempt is made to approximate the cost of individual
operations, but it is wildly inaccurate.  You can improve the
stability somewhat by doing more work for each @code{n}.  For
example:

@example
speed ("airy(x)", "x = rand (n, 10)", [10000, 100000])
@end example

When comparing two different expressions (@var{f}, @var{f2}), the slope
of the line on the speedup ratio graph should be larger than 1 if the new
expression is faster.  Better algorithms have a shallow slope.  Generally,
vectorizing an algorithm will not change the slope of the execution
time graph, but will shift it relative to the original.  For
example:

@example
@group
speed ("v = sum (x)", "", [10000, 100000], ...
       "v = 0; for i = 1:length (x), v += x(i); end")
@end group
@end example

The following is a more complex example.  If there was an original version
of @code{xcorr} using for loops and a second version using an FFT, then
one could compare the run speed for various lags as follows, or for a fixed
lag with varying vector lengths as follows:

@example
@group
speed ("v = xcorr (x, n)", "x = rand (128, 1);", 100,
       "v2 = xcorr_orig (x, n)", -100*eps)
speed ("v = xcorr (x, 15)", "x = rand (20+n, 1);", 100,
       "v2 = xcorr_orig (x, n)", -100*eps)
@end group
@end example

Assuming one of the two versions is in xcorr_orig, this
would compare their speed and their output values.  Note that the
FFT version is not exact, so we specify an acceptable tolerance on
the comparison @code{100*eps}, and that the errors should be computed
relatively, as @code{abs ((@var{x} - @var{y}) ./ @var{y})} rather than
absolutely as @code{abs (@var{x} - @var{y})}.

Type @code{example('speed')} to see some real examples.  Note that for
obscure reasons, examples 1 and 2 can not be run directly using
@code{demo('speed')}.  Instead use, @code{eval ( example('speed', 1) )}
or @code{eval ( example('speed', 2) )}.
@end deftypefn

