Just a Theory

By David E. Wheeler

Posts about TAP

Test Everything with TAP Source Handlers

I’ve just arrived in Japan with my family. We’re going to be spending several days in Tokyo, during which time I’ll be at the JPUG 10th Anniversary PostgreSQL Conference for a couple of days (giving the usual talk), but mainly I’ll be on vacation. We’ll be visiting Kyoto, too. We’re really excited about this trip; it’ll be a great experience for Anna. I’ll be back in the saddle in December, so for those of you anxiously awaiting the next installment of my Catalyst tutorial, I’m afraid you’ll have to wait a bit longer.

In the meantime, I wanted to write about a little something that’s been cooking for a while. Over the last several months, Steve Purkis has been working on a new feature for TAP::Parser: source handlers. The idea is to make it easier for developers to add support for TAP emitters other than Perl. The existing implementation did a decent job of handling Perl test scripts, of course, and executable files (useful for compiled tests in C using libtap, for example), but anything else was difficult.

As the author of pgTAP, I was of course greatly interested in this work, because I had to bend over backwards to get pg_prove to work nicely. It’s even uglier to get a Module::Build-based distribution to run pgTAP and Perl tests all at once in during ./Build test: You had to subclass Module::Build to do it.

Steve wanted to solve this problem, and he did. Then he was kind enough to listen to my bitching an moaning and rewrite his fix so that it was simpler for third parties (read: me) to add new source handlers. What’s a source handler, you ask? Check out the latest dev release of Test::Harness and you’ll see it: TAP::Parser::SourceHandler. As soon as Steve committed it, I jumped in and implemented a new handler for pgTAP. The cool thing is that it took me only three hours to do, including tests. And here’s how you use it in a Build.PL, so that you can have pgTAP tests named *.pg run at the same time as your *.t Perl tests:

Module::Build->new(
    module_name        => 'MyApp',
    test_file_exts     => [qw(.t .pg)],
    use_tap_harness    => 1,
    tap_harness_args   => {
        sources => {
            Perl  => undef,
            pgTAP => {
                dbname   => 'try',
                username => 'postgres',
                suffix   => '.pg',
            },
        }
    },
    build_requires     => {
        'Module::Build'                      => '0.30',
        'TAP::Parser::SourceHandler::pgTAP' => '3.19',
    },
)->create_build_script;

To summarize, you just have to:

  • Tell Module::Build the extensions of your test scripts (that’s qw(.t .pg) here)
  • Specify the Perl source with its defaults (that’s what the undef does)
  • Specify the pgTAP options (database name, username, suffix, and lots of other potential settings)

And that’s it. You’re done! Run your tests with the usual incantation:

perl Build.PL
./Build test

You can use pgTAP and its options with prove, too, via the --source and --pgtap-option options:

prove --source pgTAP --pgtap-option dbname=try \
                     --pgtap-option username=postgres \
                     --pgtap-option suffix=.pg \
                     t/sometest.pg

It’s great that it’s now so much easier to support pgTAP tests, but what if you want to have Ruby tests? Or PHP? Well, it’s a simple process to write your own source handler. Here’s how:

  • Subclass TAP::Parser::SourceHandler. The final part of the package name is the name of the source. Thus if you wrote TAP::Parser::SourceHandler::Ruby, the name of your source would be “Ruby”.

  • Load the necessary modules and register your source handler. For a Ruby source handler, it might look like this:

    package TAP::Parser::SourceHandler::Ruby;
    use strict;
    use warnings;
    
    use TAP::Parser::IteratorFactory   ();
    use TAP::Parser::Iterator::Process ();
    TAP::Parser::IteratorFactory->register_handler(__PACKAGE__);
  • Implement the can_handle() method. The task of this method is to return a score between 0 and 1 for how likely it is that your source handler can handle a given source. A bunch of information is passed in a hash to the method, so you can check it all out. For example, if you wanted to run Ruby tests ending in .rb, you might write something like this:

    sub can_handle {
        my ( $class, $source ) = @_;
        my $meta = $source->meta;
    
        # If it's not a file (test script), we're not interested.
        return 0 unless $meta->{is_file};
    
        # Get the file suffix, if any.
        my $suf = $meta->{file}{lc_ext};
    
        # If the config specifies a suffix, it's required.
        if ( my $config = $source->config_for('Ruby') ) {
            if ( defined $config->{suffix} ) {
                # Return 1 for a perfect score.
                return $suf eq $config->{suffix} ? 1 : 0;
            }
        }
    
        # Otherwise, return a score for our supported suffix.
        return $suf eq '.rb' ? 0.8 : 0;
    }

    The last line is the most important: it returns 0.8 if the suffix is .rb, saying that it’s likely that this handler can handle the test. But the middle bit is interesting, too. The $source->config_for('Ruby') call is seeing if the user specified a suffix, either via the command-line or in the options. So in a Build.PL, that might be:

        tap_harness_args => {
            sources => {
                Perl => undef,
                Ruby => { suffix => '.rub' },
            }
        },

    Meaning that the user wanted to run tests ending in .rub as Ruby tests. It can also be done on the command-line with prove:

    prove --source Ruby --ruby-option suffix=.rub
    

    Cool, eh? We have a reasonable default for Ruby tests, .rb, but the user can override however she likes.

  • And finally, implement the make_iterator() method. The job of this method is simply to create a TAP::Parser::Iterator object to actually run the test. It might look something like this:

    sub make_iterator {
        my ( $class, $source ) = @_;
        my $config = $source->config_for('Ruby');
    
        my $fn = ref $source->raw ? ${ $source->raw } : $source->raw;
        $class->_croak(
            'No such file or directory: ' . defined $fn ? $fn : ''
        ) unless $fn && -e $fn;
    
        return TAP::Parser::Iterator::Process->new({
            command => [$config->{ruby} || 'ruby', $fn ],
            merge   => $source->merge
        });
    }

    Simple, right? Just make sure we have a valid file to execute, then instantiate and return a TAP::Parser::Iterator::Process object to actually run the test.

That’s it. Just two methods and you’re ready to go. I’ve even added support for a suffix option and a ruby option (so that you can point to the ruby executable in case it’s not in your path). Using it is easy. I wrote a quick TAP-emitting Ruby script like so:

puts 'ok 1 - This is a test'
puts 'ok 2 - This is another test'
puts 'not ok 3 - This is a failed test'

And to run this test (assuming that TAP::Parser::SourceHandler::Ruby has been installed somewhere where Perl can find it), it’s just:

% prove --source Ruby ~/try.rb --verbose
/Users/david/try.rb .. 
ok 1 - This is a test
ok 2 - This is another test
not ok 3 - This is a failed test
Failed 1/3 subtests 

Test Summary Report
-------------------
/Users/david/try.rb (Wstat: 0 Tests: 3 Failed: 1)
  Failed test:  3
  Parse errors: No plan found in TAP output
Files=1, Tests=3,  0 wallclock secs ( 0.02 usr +  0.01 sys =  0.03 CPU)
Result: FAIL

It’s so easy to create new source handlers now, especially if all you have to do is support a new dynamic language. I’ve put the simple Ruby example over here; feel free to take it and run with it!

Looking for the comments? Try the old layout.

pgTAP Best Practices Slides Available

Last month I gave two presentations at the PostgreSQL Conference West. The first was a variation on Unit Test Your Database!, which I’ve now given about six times (and will at least two more times, including tomorrow night for Portland PLUG and in two weeks at the JPUG 10th Anniversary Conference). The second was a new talk, a 90-minute tutorial, called “pgTAP Best Practices.” And here it is, published on Slideshare. Enjoy.

You can also download a 76MB PDF if that’s more your style.

Enjoy!

Looking for the comments? Try the old layout.

pgTAP 0.21 Drops

Mmmmm…pgTAP

I just dropped a new version of pgTAP following a few marathon hack sessions since my talk at PGCon (movie here, BTW). Actually, the new performs_ok() function came about as I wrote the presentation, but all the rest came on the flight home and in the few days since. Not sure when I’ll hack on it this much again (it’s getting a bit big at 5,758 lines of PL/pgSQL and SQL).

Overall I’m very happy with this release, as it adds a lot of new assertion functions. In particular, I added a slew of functions that test that the objects in a given schema (or visible in the search path, if you prefer) are exactly the objects that should be there. This is useful for a couple of things. For one, Norman Yamada, in his PGCon talk, mentioned that his team was using pgTAP to compare database objects between replicated databases. I like this because it’s a great example of using pgTAP for system testing, rather than just unit testing as I’ve been advocating. See, pgTAP can be used for any kind of testing!

Another use for these functions is in a large organization where many different people might be making changes to a schema. In this scenario, you might have application developers adding new objects to the database (or dropping objects) without necessarily informing the DBAs. Using, for example, tables_are() and functions_are() and continuous testing, the DBAs can see when objects have been modified by the developers. Even better, if the developers are running the pgTAP tests themselves (as they should be!), they will be reminded to add new tests for their changes when the existing tests notice that things have been added or dropped and thus fail.

Beyond that, I added a bunch of new functions for testing functions and a number of other goodies. Check out the release notes for all the details.

With these changes, I’ve finished nearly everything I’ve thought of for pgTAP. There are only a few sequence-testing functions left on the To Do list, as well as a call to add a throws_like() function, which I’ll throw in soon if no one else steps up. Beyond these changes, I have a few ideas of where to take it next, but so far I’m kind of stumped. Mainly what I think should be done is to add an interface that makes it easier to compare relations (or result sets, if you prefer). Epic does this by allowing query strings to be passed to a function, but I’d really like to keep queries in SQL rather than in SQL strings. I’ll be giving it some more thought and will post about it soon.

Looking for the comments? Try the old layout.

pgTAP 0.20 Infiltrates Community

I did all I could to stop it, but it just wasn’t possible. pgTAP 0.20 has somehow made its way from my Subversion server and infiltrated the PostgreSQL community. Can nothing be done to stop this menace? Its use leads to cleaner, more stable, and more-safely refactored code. This insanity must be stopped! Please review the following list of its added vileness since 0.19 to determine how you can stop the terrible, terrible influence on your PostgreSQL unit-testing practices that is pgTAP:

  • Changed the names of the functions tested in sql/do_tap.sql and sql/runtests.sql so that they are less likely to be ordered differently given varying collation orders provided in different locales and by different vendors. Reported by Ingmar Brouns.
  • Added the --formatter and --archive options to pg_prove.
  • Fixed the typos in pg_prove where the output of --help listed --test-match and --test-schema instead of --match and --schema.
  • Added has_cast(), hasnt_cast(), and cast_context_is().
  • Fixed a borked function signature in has_trigger().
  • Added has_operator(), has_leftop(), and has_rightop().
  • Fixed a bug where the order of columns found for multicolum indexes by has_index() could be wrong. Reported by Jeff Wartes. Thanks to Andrew Gierth for help fixing the query.

Don’t make the same mistake I did, where I wrote a lot of pgTAP tests for a client, and now testing database upgrades from 8.2 to 8.3 is just too reliable! YOU HAVE BEEN WARNED.

Good luck with your mission.

Looking for the comments? Try the old layout.

pgTAP 0.16 in the Wild

I’ve been writing a lot tests for a client in pgTAP lately. It’s given me a lot to think about in terms of features I need and best practices in writing tests. I’m pleased to say that, overall, it has been absolutely invaluable. I’m doing a lot of database refactoring, and having the safety of solid test coverage has been an absolute godsend. pgTAP has done a lot to free me from worry about the effects of my changes, as it ensures that everything about the databases continue to just work.

Of course, that’s not to say that I don’t scew up. There are times when my refactorings have introduced new bugs or incompatibilities; after all, the tests I write of existing functionality extend only so far as I can understand that functionality. But as such issues come up, I just add regression tests, fix the issues, and move on, confident in the knowledge that, as long as the tests continue to be run regularly, those bugs will never come up again. Ever.

As a result, I’ll likely be posting a bit on best practices I’ve found while writing pgTAP tests. As I’ve been writing them, I’ve started to find the cow paths that help me to keep things sane. Most helpful is the large number of assertion functions that pgTAP offers, of course, but there are a number of techniques I’ve been developing as I’ve worked. Some are better than others, and still others suggest that I need to find other ways to do things (you know, when I’m cut-and-pasting a lot, there must be another way, though I’ve always done a lot of cut-and-pasting in tests).

In the meantime, I’m happy to announce the release of pgTAP 0.16. This version includes a number of improvements to the installer (including detection of Perl and TAP::Harness, which are required to use the included pg_prove test harness app. The installer also has an important bug fix that greatly increases the chances that the os_name() function will actually know the name of your operating system. And of course, there are new test functions:

  • has_schema() and hasnt_schema(), which test for the presence of absence of a schema
  • has_type() and hasnt_type(), which test for the presence and absence of a data type, domain, or enum
  • has_domain() and hasnt_domain(), which test for the presence and absence of a data domain
  • has_enum() and hasnt_enum(), which test for the presence and absence of an enum
  • enum_has_lables() which tests that an enum has an expected list of labels

As usual, you can download the latest release from pgFoundry. Visit the pgTAP site for more information and for documentation.

Looking for the comments? Try the old layout.

pgTAP 0.14 Released

I’ve just released pgTAP 0.14. This release focuses on getting more schema functions into your hands, as well as fixing a few issues. Changes:

  • Added SET search_path statements to uninstall_pgtap.sql.in so that it will work properly when TAP is installed in its own schema. Thanks to Ben for the catch!
  • Added commands to drop pg_version() and pg_version_num() touninstall_pgtap.sql.in.
  • Added has_index(), index_is_unique(), index_is_primary(), is_clustered(), and index_is_type().
  • Added os_name(). This is somewhat experimental. If you have uname, it’s probably correct, but assistance in improving OS detection in the Makefile would be greatly appreciated. Notably, it does not detect Windows.
  • Made ok() smarter when the test result is passed as NULL. It was dying, but now it simply fails and attaches a diagnostic message reporting that the test result was NULL. Reported by Jason Gordon.
  • Fixed an issue in check_test() where an extra character was removed from the beginning of the diagnostic output before testing it.
  • Fixed a bug comparing name[]s on PostgreSQL 8.2, previously hacked around.
  • Added has_trigger() and trigger_is().
  • Switched to pure SQL implementations of the pg_version() and pg_version_num() functions, to simplify including pgTAP in module distributions.
  • Added a note to README.pgtap about the need to avoid pg_typeof() and cmp_ok() in tests run as part of a distribution.

Enjoy!

Looking for the comments? Try the old layout.

pgTAP 0.12 Released

In anticipation of my PostgreSQL Conference West 2008 talk on Sunday, I’ve just released pgTAP 0.12. This is a minor release with just a few tweaks:

  • Updated plan() to disable warnings while it creates its tables. This means that plan() no longer send NOTICE messages when they run, although tests still might, depending on the setting of client_min_messages.
  • Added hasnt_table(), hasnt_view(), and hasnt_column().
  • Added hasnt_pk(), hasnt_fk(), col_isnt_pk(), and col_isnt_fk().
  • Added missing DROP statements to uninstall_pgtap.sql.in.

I also have an idea to add functions that return the server version number (and each of the version number parts) and an OS string, to make testing things on various versions of PostgreSQL and on various operating systems a lot simpler.

I think I’ll also spend some time in the next few weeks on an article explaining exactly what pgTAP is and why you’d want to use it. Provided, of course, I can find the tuits for that.

Looking for the comments? Try the old layout.

pgTAP 0.11 Released

So I’ve just released pgTAP 0.11. I know I said I wasn’t going to work on it for a while, but I changed my mind. Here’s what’s changed:

  • Simplified the tests so that they now load test_setup.sql instead of setting a bunch of stuff themselves. Now only test_setup.sql needs to be created from test_setup.sql.in, and the other .sql files depend on it, meaning that one no longer has to specify TAPSCHEMA for any make target other than the default.
  • Eliminated all uses of E'' in the tests, so that we don’t have to process them for testing on 8.0.
  • Fixed the spelling of ON_ROLLBACK in the test setup. Can’t believe I had it with one L in all of the test files before! Thanks to Curtis “Ovid” Poe for the spot.
  • Added a couple of variants of todo() and skip(), since I can never remember whether the numeric argument comes first or second. Thanks to PostgreSQL’s functional polymorphism, I don’t have to. Also, there are variants where the numeric value, if not passed, defaults to 1.
  • Updated the link to the pgTAP home page in pgtap.sql.in.
  • TODO tests can now nest.
  • Added todo_start(), todo_end(), and in_todo().
  • Added variants of throws_ok() that test error messages as well as error codes.
  • Converted some more tests to use check_test().
  • Added can() and can_ok().
  • Fixed a bug in check_test() where the leading whitespace for diagnostic messages could be off by 1 or more characters.
  • Fixed the installcheck target so that it properly installs PL/pgSQL into the target database before the tests run.

Now I really am going to do some other stuff for a bit, although I do want to see what I can poach from Epic Test. And I do have that talk on pgTAP next month. So I’ll be back with more soon enough.

Looking for the comments? Try the old layout.

Introducing pgTAP

So I started working on a new PostgreSQL data type this week. More on that soon; in the meantime, I wanted to create a test suite for it, and wasn’t sure where to go. The only PostgreSQL tests I’ve seen are those distributed with Elein Mustain’s tests for the email data type she created in a PostgreSQL General Bits posting from a couple of years ago. I used the same approach myself for my GTIN data type, but it was rather hard to use: I had to pay very close attention to what was output in order to tell the description output from the test output. It was quite a PITA, actually.

This time, I started down the same path, then then started thinking about Perl testing, where each unit test, or “assertion,” in the xUnit parlance, triggers output of a single line of information indicating whether or not a test succeeded. It occurred to me that I could just run a bunch of queries that returned booleans to do my testing. So my first stab looked something like this:

\set ON_ERROR_STOP 1
\set AUTOCOMMIT off
\pset format unaligned
\pset tuples_only
\pset pager
\pset null '[NULL]'

SELECT foo() = 'bar';
SELECT foo(1) = 'baz';
SELECT foo(2) = 'foo';

The output looked like this:

% psql try -f ~/Desktop/try.sql
t
t
t

Once I started down that path, and had written ten or so tests, It suddenly dawned on me that the Perl Test::More module and its core ok() subroutine worked just like that. It essentially just depends on a boolean value and outputs text based on that value. A couple minutes of hacking and I had this:

CREATE TEMP SEQUENCE __tc__;
CREATE OR REPLACE FUNCTION ok ( boolean, text ) RETURNS TEXT AS $$
    SELECT (CASE $1 WHEN TRUE THEN '' ELSE 'not ' END) || 'ok'
        || ' ' || NEXTVAL('__tc__')
        || CASE $2 WHEN '' THEN '' ELSE COALESCE( ' - ' || $2, '' ) END;
$$ LANGUAGE SQL;

I then rewrote my test queries like so:

\echo 1..3
SELECT ok( foo() = 'bar'   'foo() should return "bar"' );
SELECT ok( foo(1) = 'baz', 'foo(1) should return "baz"' );
SELECT ok( foo(2) = 'foo', 'foo(2) should return "foo"' );

Running these tests, I now got:

% psql try -f ~/Desktop/try.sql
1..3
ok 1 - foo() should return "bar"
ok 2 - foo(1) should return "baz"
ok 3 - foo(2) should return "foo"

And, BAM! I had the beginning of a test framework that emits pure TAP output.

Well, I was so excited about this that I put aside my data type for a few hours and banged out the rest of the framework. Why was this exciting to me? Because now I can use a standard test harness to run the tests, even mix them in with other TAP tests on any project I might work on. Just now, I quickly hacked together a quick script to run the tests:

use TAP::Harness;

my $harness = TAP::Harness->new({
    timer   => $opts->{timer},
    exec    => [qw( psql try -f )],
});

$harness->runtests( @ARGV );

Now I’m able to run the tests like so:

% try ~/Desktop/try.sql        
/Users/david/Desktop/try........ok   
All tests successful.
Files=1, Tests=3,  0 wallclock secs ( 0.00 usr  0.00 sys +  0.01 cusr  0.00 csys =  0.01 CPU)
Result: PASS

Pretty damn cool! And lest you wonder whether such a suite of TAP-emitting test functions is suitable for testing SQL, here are a few examples of tests I’ve written:

-- Plan the tests.
SELECT plan(4);

-- Emit a diagnostic message for users of different locales.
SELECT diag(
    E'These tests expect LC_COLLATE to be en_US.UTF-8,\n'
    || 'but yours is set to ' || setting || E'.\n'
    || 'As a result, some tests may fail. YMMV.'
)
    FROM pg_settings
    WHERE name = 'lc_collate'
    AND setting <> 'en_US.UTF-8';

SELECT is( 'a', 'a', '"a" should = "a"' );
SELECT is( 'B', 'B', '"B" should = "B"' );

CREATE TEMP TABLE try (
    name lctext PRIMARY KEY
);

INSERT INTO try (name)
VALUES ('a'), ('ab'), ('â'), ('aba'), ('b'), ('ba'), ('bab'), ('AZ');

SELECT ok( 'a' = name, 'We should be able to select the value' )
    FROM try
    WHERE name = 'a';

SELECT throws_ok(
    'INSERT INTO try (name) VALUES (''a'')',
    '23505',
    'We should get an error inserting a lowercase letter'
);

-- Finish the tests and clean up.
SELECT * FROM finish();

As you can see, it’s just SQL. And yes, I have ported most of the test functions from Test::More, as well as a couple from Test::Exception.

So, without further ado, I’d like to introduce pgTAP, a lightweight test framework for PostgreSQL implemented in PL/pgSQL and PL/SQL. I’ll be hacking on it more in the coming days, mostly to get a proper client for running tests hacked together. Then I think I’ll see if pgFoundry is interested in it.

Whaddya think? Is this something you could use? I can see many uses, myself, not only for testing a custom data type as I develop it, but also custom functions in PL/pgSQL or PL/Perl, and, heck, just regular schema stuff. I’ve had to write a lot of Perl tests to test my database schema (triggers, rules, functions, etc.), all using the DBI and being very verbose. Being able to do it all in a single psql script seems so much cleaner. And if I can end up mixing the output of those scripts in with the rest of my unit tests, so much the better!

Anyway, feedback welcome. Leave your comments, suggestions, complaints, patches, etc., below. Thanks!

Looking for the comments? Try the old layout.

Test.Simple 0.20 Released

It gives me great pleasure, not to mention a significant amount of pride, to announce the release of Test.Simple 0.20. There are quite a few changes in this release, including a few that break backwards compatibility—but only you’re writing your own Test.Builder-based test libraries (and I don’t think anyone has done so yet) or if you’re subclassing Test.Harness (and there’s only one of those, that I know of).

The biggest change is that Test.Harness.Browser now supports pure .js script files in addition to the original .html files. This works best in Firefox, of course, but with a lot of help from Pawel Chmielowski (“prefiks” on #jsan), it also works in Safari, Opera, and IE 6 (though not in XP Service Pack 2; I’ll work on that after I get my new PC in the next few days). The trick with Firefox (and hopefully other browsers in the future, since it feels lest hackish to me), is that it uses the DOM to create a new HTML document in a hidden iframe, and that document loads the .js. Essentially, it just uses the DOM to mimic the structure of a typical .html test file. For the other browsers, the hidden iframe uses XMLHttpRequest to load and eval the .js test file. Check it out (verbosely)!

I think that this will greatly enhance the benefits of Test.Simple, as it makes writing tests really simple. All you have to do is create a single .html file that looks something like this:

<html>
<head>
    <script type="text/javascript" src="./lib/JSAN.js"></script>
</head>
<body>
<script type="text/javascript">
    new JSAN("../lib").use("Test.Harness.Browser");
    new Test.Harness.Browser('./lib/JSAN.js').encoding('utf-8').runTests(
        'foo.js',
        'bar.js'
    );
</script>
</body>
</html>

In fact, that’s pretty much exactly what Test.Simple’s new harness looks like, now that I’ve moved all of the old tests into .js files (although there is still a simpl.html test file to ensure that .html test files still work!). Here I’m using JSAN to dynamically load the libraries I need. I use it to load Test.Harness.Browser (which then uses it to load Test.Harness), and then I tell the Test.Harness.Browser object where it is so that it can load it for each .js script. The test script itself can then look something like this:

new JSAN('../lib').use('Test.Simple');
plan({tests: 3});
ok(1, 'compile');
ok(1);
ok(1, 'foo');

And that’s it! Just use JSAN to load the appropriate test library or libraries and go! I know that JSAN is already loaded because Test.Harness.Browser loads it for me before it loads and runs my .js test script. Nice, eh?

Of course, you don’t have to use JSAN to run pure .js tests, although it can be convenient. Instead, you can just pass a list of files to the harness to have it load them for each test:

<html>
<head>
    <script type="text/javascript" src="./lib/Test/Harness.js"></script>
    <script type="text/javascript" src="./lib/Test/Harness/Browser.js"></script>
</head>
<body>
<script type="text/javascript">
    new Test.Harness.Browser(
        'lib/Test/Builder.js',
        'lib/Test/More.js',
        '../lib/MY/Library.js'
    ).runTests(
        'foo.js',
        'bar.js'
    );
</script>
</body>
</html>

This example tells Test.Harness.Browser to load Test.Builder and Test.More, and then to run the tests in foo.js and bar.js. No need for JSAN if you don’t want it. The test script is exactly the same as the above, only without the line with JSAN loading your test library.

Now, as I’ve said, this is imperfect. It’s surprisingly difficult to get browsers to do this properly, and it’s likely that it won’t work at all in many browsers. I’m sure that I broke the Directory harness, too. Nevertheless, I’m pleased that I got as many to work as I did (again, with great thanks to Pawel Chmielowski for all the great hacks), but at this point, I’ll probably only focus on adding support for Windows XP Service Pack 2. But as you might imagine, I’d welcome patches from anyone who wants to add support for other browsers.

There are a lot of other changes in this release. Here’s the complete list:

  • Fixed verbose test output to be complete in the harness in Safari and IE.
  • Fixed plan() so that it doesn’t die if the object is passed with an unknown attribute. This can happen when JS code has altered Object.prototype (shame on it!). Reported by Rob Kinyon.
  • Fixed some errors in the POD documentation.
  • Updated JSAN to 0.10.
  • Added documentation for Test.Harness.Director, complements of Gordon McCreight.
  • Fixed line endings in Konqueror and Opera and any other browser other than MSIE that supports document.all. Reported by Rob Kinyon.
  • Added support to Test.Harness.Browser for .js test files in addition to .html test files. Thanks to Pawel Chmielowski for helping me to overcome the final obstacles to actually getting this feature to work.
  • Added missing variable declarations. Patch from Pawel Chmielowski.
  • More portable fetching of the body element in Test.Builder. Based on patch from Pawel Chmielowski.
  • Added an encoding attribute to Test.Harness. This is largely to support pure JS tests, so that the browser harness can set up the proper encoding for the script elements it creates.
  • Added support for Opera, with thanks to Pawel Chmielowski.
  • Fixed the output from skipAll in the test harness.
  • Fixed display of summary of failed tests after all tests have been run by the browser harness. They are now displayed in a nicely formatted table without a NaN stuck where it doesn’t belong.
  • COMPATIBILITY CHANGE: The browser harness now outputs failure information bold-faced and red. This required changing the output argument to the outputResults() method to an object with two methods, pass() and fail(). Anyone using Test.Harness.outputResults() will want to make any changes accordingly.
  • COMPATIBILITY CHANGE: new Test.Builder() now always returns a new Test.Builder object instead of a singleton. If you want the singleton, call Test.Builder.instance(). Test.Builder.create() has been deprecated and will be removed in a future release. This is different from how Perl’s Test::Builder works, but is more JavaScript-like and sensible, so we felt it was best to break things early on rather than later. Suggested by Bob Ippolito.
  • Added beginAsync() and endAsync() functions to Test.More. Suggested by Bob Ippolito.

As always, feedback/comments/suggestions/winges welcome. Enjoy!

Looking for the comments? Try the old layout.

Test.Simple 0.11 Released

I’m pleased to announce the release of Test.Simple 0.11. This release fixes a number of bugs in the framework’s IE and Safari support, adds JSAN support, and includes an experimental harness for Macromedia\^Adobe Director. You can download it from JSAN, and all future releases will be available on JSAN. See the harness in action here (or verbosely!). This release has the following changes:

  • The browser harness now works more reliably in IE and Safari.
  • Fixed syntax errors in tests/harness.html that IE and Safari care about.
  • Various tweaks for Director compatibility from Gordon McCreight.
  • Removed debugging output from Test.More.canOK().
  • Fixed default output so that it doesn’t re-open a closed browser document when there is a “test” element.
  • Added experimental Test.Harness.Director, complements of Gordon McCreight. This harness is subject to change.
  • Added Test.PLATFORM, containing a string defining the platform. At the moment, the only platforms listed are “browser” or “director”.
  • Added support for Casey West’s JSAN. All releases of Test.Simple will be available on JSAN from now on.
  • The iframe in the browser harness is no longer visible in IE. Thanks to Marshall Roch for the patch.
  • Noted addition of Test.Harness and Test.Harness.Browser in the README.

I’ve been getting more and more excited about Casey West’s work on JSAN. It gets better every day, and I hope that it attracts a lot of hackers who want to distribute open source JavaScript modules. You should check it out! I’ve been working on a Perl module to simplify the creation of JSAN distributions. Look for it on CPAN soonish.

Looking for the comments? Try the old layout.

More about…

Test.Simple 0.10 Released

I’m pleased to announce the first beta release of Test.Simple, the port of Test::Builder, Test::Simple, Test::More, and Test::Harness to JavaScript. You can download it here. See the harness in action here (or verbosely!). This release has the following changes:

  • Changed the signature of functions passed to output() and friends to accept a single argument rather than a list of arguments. This allows custom functions to be much simpler.
  • Added support for Macromedia Director. Patch from Gordon McCreight.
  • Backwards Incompatibility change: moved all “modules” into Test “namespace” by using an object for the Test namespace and assigning the Build() constructor to it. See http://xrl.us/fy4h for a description of this approach.
  • Fixed the typeOf() class method in Test.Builder to just return the value returned by the typeof operator if the class constructor is an anonymous function.
  • Changed for (var in in someArray) to for (var i = 0; i < someArray.length; i++) for iterating through arrays, since the former method will break if someone has changed the prototype for arrays. Thanks to Bob Ippolito for the spot!
  • The default output in browsers is now to append to an element with the ID “test” or, failing that, to use document.write. The use of the “test” element allows output to continue to be written to the browser window even after the document has been closed. Reported by Adam Kennedy.
  • Changed the default endOutput() method to be the same as the other outputs.
  • Backwards incompatibility change: Changed semantics of plan() so that it takes an object for an argument. This allows multiple commands to be passed, where the object attribute keys are the command and their values are the arguments.
  • Backwards incompatibility change: Changed the “no_plan”, “skip_all”, and “no_diag” (in Test.More only) options to plan() to their studlyCap alternatives, “noPlan”, “skipAll”, and “noDiag”. This makes them consistent with JavaScript attribute naming convention.
  • Added beginAsync() and endAsync() methods to Test.Builder to allow users to put off the ending of a script until after asynchronous tests have been run. Suggested by Adam Kennedy.
  • Backwards incompatibility change: Changed the signature for the output() method and friends to take only a single anonymous function as its argument. If you still need to call a method, pass an anonymous function that calls it appropriately.
  • Changed handling of line-endings to be browser-specific. That is, if the current environment is Internet Explorer, we use “\r” for line endings. Otherwise we use “\n”. Although IE properly interprets \n as a line ending when it’s passed to document.write(), it doesn’t when passed to a DOM text node. No idea why not.
  • Added a browser harness. Now you can run all of your tests in a single browser window and get a summary at the end, including a list of failed tests and the time spent running the tests.
  • Fixed calls to warn() in Test.More.
  • Output to the browser now causes the window to scroll when the length of the output is greater than the height of the window.
  • Backwards incompatibility change: Changed all instances of “Ok” to “OK”. So this means that the new Test.More function names are canOK(), isaOK(), and cmpOK(). Sorry ‘bout that, won’t happen again.
  • Ported to Safari (though there are issues–see the “Bugs” section of the Test.Harness.Browser docs for details).

Obviously this is a big release. I bumped up the version number because there are a fair number of backwards incompatibilities. But I’m reasonably confident that they wont’ change so much in the future. And with the addition of the harness, it’s getting ready for prime time!

Next up, I’ll finish porting the test from Test::Harness (really!) and add support for JSAN (look for a JSAN announcement soon). But in the meantime, feedback, bug reports, kudos, complaints, etc.warmly welcomed!

Looking for the comments? Try the old layout.

TestSimple 0.02 Released

I’m pleased to announce the second alpha release of TestSimple, the port of Test::Builder, Test::Simple, and Test::More to JavaScript. You can download it here. This release has the following changes:

  • Removed eqArray() and eqAssoc() functions from TestMore per suggestion from Michael Schwern. The problem is that these are not test functions, and so are inconsistent with the way the rest of the functions work. isDeeply() is the function that users really want.
  • Changed eqSet() to isSet() and made it into a real test function.
  • Implemented skip(), todoSkip(), and todo(). These are a bit different than the Perl originals originals so read the docs!
  • Implemented skipAll() and BAILOUT() using exceptions and an exception handler installed in window.onerror.
  • The final message of a test file now properly outputs in the proper place. Tests must be run inside an element its “id” attribute set to “test”, such as <pre id="test">. The window.onload handler will find it and append the final test information.
  • Implemented skipRest() in TestBuilder and TestMore. This method is stubbed out the Perl original, but not yet implemented there!

The only truly outstanding issues I see before I would consider these “modules” ready for production use are:

  • Figure out how to get at file names and line numbers for better diagnostic messages. Is this even possible in JavaScript?
  • Decide where to send test output, and where to allow other output to be sent. Test::Builder clones STDERR and STDOUT for this purpose. We’ll probably have to do it by overriding document.write()>, but it’d be good to allow users to define alternate outputs (tests may not always run in a browser, eh?). Maybe we can use an output object? Currently, a browser and its DOM are expected to be present. I could really use some advice from real JavaScript gurus on this one.
  • Write tests!

Feedback/advice/insults welcome!

Looking for the comments? Try the old layout.