Just a Theory

By David E. Wheeler

Posts about mod_perl

Moving Towards Bricolage 2.0

Today I’ve finished just about over two and a half weeks of hacking on Bricolage. It has been a couple of years since I gave it much attention, but there was so much good stuff that other people have contributed that, since I had a little time, it seemed worth it to give it some love. So here’s a quick list of all that I’ve done in the last two weeks:

  • Fixed all reported issues with Bricolage 1.10. Scott Lanning kindly released 1.10.5 yesterday with all of those fixes.

  • I integrated the element occurrence branch that Christian Muise had worked on as his 2006 Google Summer of Code project. Christian’s project added support for maximum and minimum specifications for subelements in Bricolage, which allows administrators to define how many fields and elements can occur in a story or media document. All I had to do was add a few UI tweaks to support the new fields and their specification in the story profile, and all was ready to go. Oh, and I did have to go back and make the SOAP interface work with the feature, but the only reason it never did was lazy hacking of the SOAP interface (way before Christian’s time). Nice work, Christian, and thank you for your contribution!

  • I fixed a few bugs with Arsu Andrei’s port of Bricolage to MySQL, which was his 2006 Google Summer of Code project. Arsu did a terrific job with the port, with only a few minor things missed that he likely could not have caught anyway. This work had already been merged into the trunk. Thanks Arsu!

  • I fixed a bunch of bugs from Marshall Roch’s AJAXification of Bricolage, carried out during his 2006 Google Summer of Code project. Marshall actually did a lot more stuff than he’d planned, as it all went quite smoothly. I found only a few minor oversights that I was able to easily address. This work represents the single most visible change to how users user Bricolage since we launched the project back in 2001. Editing stories, in particular, is now a lot cleaner, with far fewer page loads. Thanks a million, Marshall!

  • I completed the work started by Chris Heiland of the University of Washington, Bothell, and Scott Lanning of the World Health Organization to port Bricolage to Apache 2. They really did most of the hard work, and I just spent several days integrating everything, making sure all the features work, and updating the installer to handle differences in configuration. I thought this would take me a day or two, but it actually took the better part of a week! So much has changed, but in truth Bricolage is now better for running on mod_perl 2. Expect to see Apache 2 bet the recommended platform for Bricolage in a release in the near future.

  • I integrated a number of patches from Brian Smith of Gossamer Threads to allow the installer to be run as a non-root user. The key here is if the installer has to become the database super user, which is required for ident authentication, and of course whether files are to be installed somewhere on the system requiring super user access. This work is not done, yet, as make upgrade and make uninstall are not quite there yet. But we’re getting there, and it should be all done in time for 2.0, thanks to Brian.

  • I added support for a whole slew of environment variables to the installer. Now you can set environment variables to override default settings for installation parameters, such as choice of RDBMS, Apache, location of an SSL cert and key, whether to support SLL, and lots of other stuff, besides. This is all documented in the “Quick Installation Instructions” section of Bric::Admin/INSTALL.

  • I fully tested and fixed a lot of bugs leftover from making the installer database- and Apache-neutral. Now all of these commands should work perfectly:

    • make
    • make cpan
    • make test
    • make install
    • make devtest
    • make clone
    • make uninstall
  • I improved the DHTML functionality of the “Add More” widget, which is used to add contact information to users and contributors, rules to alert types, and extensions to media types. I think it’s pretty slick, now! This was built on Marshall’s AJAX work.

All of these changes have been integrated into the Bricolage trunk and I’ve pushed out a developer release today. Please do check out all the goodness on a test box and send feedback or file bug reports! There are only a couple of other features waiting to go into Bricolage before we start the release candidate process. And, oh yeah, tht title of this blog post? It’s not a lie. The next production release of Bricolage, based on all this work, will be Bricolage 2.0. Enough of the features we’d planned for Bricolage lo these many years ago are in the trunk that the new version number is warranted. I for one will be thrilled to see 2.0 ship in time for OSCON.

And in case it isn’t already clear, many thanks to the Google Summer of Code and participating students for the great contributions! This release would not have been possible without them.

Also in the news today, the Bricolage server has been replaced! The new server, which hosts the Web site, the wiki and the instance of Bricolage used to manage the site itself, is graciously provided by the kind folks at Gossamer Threads. The server is Gossamer Threads’s way of giving back to the Bricolage community as they prepare to launch a hosted Bricolage solution. Thaks GT!

The old Bricolage server was provided by pair Networds for the last five years. I’d just like to thank pair for the generous five-year loan of that box, which helped provided infrastructure for both Bricolage and Kineticode. Thank you, pair!

And with that, I’m going heads-down on some other projects. I’ll pop back up to make sure that Bricolage 2.0 is released in a few months, but otherwise, I’m on to other things again for a while. Watch this space for details!

Looking for the comments? Try the old layout.

What's With These CPAN-Testers Failures?

So I just learned about and subscribed to the CPAN-Testers feed for my modules. There appear to be a number of odd failures. Take this one. It says,“Can’t locate Algorithm/Diff.pm,” despite the fact that I have properly specified the requirement for Text::Diff, which itself properly requires Algorithm::Diff.. Is this an instance of CPAN.pm or CPANPLUS not following all prerequisites, or what?

Or take this failure. It says, “[CP_ERROR] [Mon Sep 5 09:32:08 2005] No such module ‘mod_perl’ found on CPAN”. Yet here it is. Maybe the CPANPLUS indexer has a bug? Or are people’s configurations just horked? Or am I just doing something braindead?

Opinions welcomed.

Looking for the comments? Try the old layout.

More about…

Bricolage Now has PHP 5 Templating

Well, I’ve finally gone and done it. I’ve released Bricolage 1.9.0, the first development release towards 1.10.0, which I plan to get out sometime next month. Among the new features in this release that I’m most excited about are a revamped UI, LDAP authentication, and PHP 5 templating support.

The new UI is really nice. Marshall Roch ported it all to XHTML 1.0 strict plus CSS. All the layout is done with CSS now, instead of the old 1999-era table layouts we had before. The result is much smaller page loads, sometimes up to 70% smaller, Marshall tells me, and therefore much faster loads. Playing around with the new version (now powering the Kineticode and Bricolage Web sites, I do indeed find it to be a lot zippier. Couple the new UI with the new mod_gzip support and static Mason components, and things just perk right along! Oh, and the expandable interfaces for both lists of objects and for editing interfaces is very nice, indeed. Thank you Marshall for the great work!

Kineticode added the LDAP authentication support. We’ve been using a patch for it against 1.8.x for our internal version of Bricolage and it works great. Now we authenticate off of an OpenLDAP server, using the same usernames and passwords we use for our email, Subversion, and RT servers. The configuration is simple, via bricolage.conf directives, and you can even limit users who authenticate to members of a particular LDAP group. Users must exist in Bricolage, and if LDAP authentication fails, you can fall back on Bricolage’s internal authentication. I’m hoping that the LDAP support goes a long way towards attracting more enterprise customers with single sign-on requirements.

And finally—yes, you heard right—Bricolage now supports PHP 5 templating in addition to the existing Perl-based templating architectures (Mason, Template Toolkit, and HTML::Template). So how did we add PHP 5 templating to a mod_perl application? Easy: we hired George Schlossnagle of Omni TI to write PHP::Interpreter, an embedded PHP 5 interpreter. Now anyone can natively execute PHP 5 code from a Perl application. Not only that, but the PHP 5 code can reach back into the Perl interpreter to use Perl modules and objects! Here’s an example that I like to show off to the PHP crowd:

<?php
    $perl = Perl::getInstance();
    $perl->eval("use DBI");
    $perl->eval("use DateTime");
    $dbh = $perl->call("DBI::connect", "DBI", "dbi:SQLite:dbname=dbfile");
    $dbh->do("CREATE TABLE foo (bar TEXT, time DATETIME)");
    $now = $perl->call("DateTime::now", "DateTime");
    $ins = $dbh->prepare("INSERT INTO foo VALUES (?, ?)");
    $ins->execute("This is a test", $now);
    $sel = $dbh->prepare("SELECT bar, time FROM foo");
    $sel->execute();
    $a = array("foo", "bar");
    foreach ($sel->fetch() as $val) {
        echo "$val\n";
    }
    $sel->finish();
    $dbh->do("DROP TABLE foo");
    $dbh->disconnect();
?>

Note that George plans to add convenience methods to load Perl modules and call Perl class methods. Now, to execute this code from Perl, all you have to do is write a little script, call it pphp, like so:

use strict;
use PHP::Interpreter;
my $php = PHP::Interpreter->new;
$php->include(shift);

Then just execute your PHP code with the script: pphp try.php. Yes, this does work! For years, when I’ve run across a PHP coder who wanted to try to tell me that PHP was better than Perl, I always had a one-word reply that left him cursing and conceding defeat: “CPAN.” Well no more. Now PHP hackers can use any module on CPAN, too!

And as for Bricolage, the integration of PHP 5 templating is completely transparent. Users just write PHP 5 templates instead of Mason templates and that’s it! For example, this is a fairly common style Bricolage Mason template:

<%perl>;
for my $e ($element->get_elements(qw(header para _pull_quote_))) {
    my $kn = $e->get_key_name;
    if ($kn eq "para") {
        $m->print("<p>", $e->get_data, "</p>\n");
    } elsif ($kn eq "header") {
        # Test sdisplay_element() on a field.
        $m->print("<h3>", $burner->sdisplay_element($e), "</h3>\n");
    } elsif ($kn eq "_pull_quote_" && $e->get_object_order > 1) {
        # Test sdisplay_element() on a container.
        $m->print($burner->sdisplay_element($e));
    } else {
        # Test display_element().
        $burner->display_element($e);
    }
}
$burner->display_pages("_page_");
</%perl>

The same template in PHP 5 looks like this:

<?php
    # Convenience variables.
    $story   = $BRIC["story"];
    $element = $BRIC["element"];
    $burner  = $BRIC["burner"];
    foreach ($element->get_elements("header", "para", "_pull_quote_") as $e) {
        $kn = $e->get_key_name();
        if ($kn == "para") {
            echo "<p>", $e->get_data(), "</p>\n";
        } else if ($kn == "header") {
            # Test sdisplay_element() on a field.
            echo "<h3>", $burner->sdisplay_element($e), "</h3>\n";
        } else if ($kn == "_pull_quote_" && $e->get_object_order() > 1) {
            # Test sdisplay_element() on a container.
            echo $burner->sdisplay_element($e);
        } else {
            # Test display_element().
            $burner->display_element($e);
        }
    }
    $burner->display_pages("_page_");
?>

Yes, you are seeing virtually the same thing. But this is just a simple template from Bricolage’s test suite. The advantage is that PHP 5 coders who are familiar with all the ins and outs of PHP 5 can just jump in an get started writing Bricolage templates without having to learn any Perl! The Bricolage objects have exactly the same API as they do in Perl, because they are exactly the same objects! So everyone uses the same API documentation for the same tasks. The only issue I’ve noticed so far is that PHP 5 does not yet have proper Unicode support. Since all content in Bricolage is stored as UTF-8, this means that the PHP 5 templates must treat it as binary data. But this is okay as long as templaters use the mb_* PHP 5 functions to parse text.

Overall I’m very excited about this, and hope that it helps Bricolage to reach a whole new community of users. I’d like to thank Portugal Telecom—SAPO.pt for sponsoring the development of PHP::Interpreter and its integration into Bricolage. I believe that they’ve really done the Bricolage community a great service, and I hope that the Perl and PHP communities likewise benefit from the integration possible with PHP::Interpreter.

And just so that the other templating architectures don’t feel left out, here is how the above template looks in Template Toolkit:

[% FOREACH e = element.get_elements("header", "para", "_pull_quote_") %]
    [% kn = e.get_key_name %]
    [% IF kn == "para" %]
<p>[% e.get_data %]</p>
    [% ELSIF kn == "header" %]
        [% # display_element() should just return a value. %]
<h3>[% burner.display_element(e) %]</h3>
    [% ELSIF kn == "_pull_quote_" && e.get_object_order > 1 %]
        [% PERL %]
          # There is no sdisplay_element() in the TT burner, but we"ll just
          # Play with it, anyway.
          print $stash->get("burner")->display_element($stash->get("e"));
        [% END %]
    [% ELSE %]
        [% # Test display_element(). %]
        [% burner.display_element(e) %]
    [% END %]
[% END %]
[% burner.display_pages("_page_") %]

And here it is in HTML::Template:

<tmpl_if _page__loop>
<tmpl_loop _page__loop>
<tmpl_include name="testing/sub/util.tmpl">
<tmpl_var _page_>
<tmpl_var page_break>
</tmpl_loop>
<tmpl_else>
<tmpl_include name="testing/sub/util.tmpl">
</tmpl_if>

I had to use the utility template with HTML::Template to get it to work right, don’t ask why. It looks like this:

<tmpl_loop element_loop>
<tmpl_if is_para>
<p><tmpl_var para></p>
</tmpl_if>
<tmpl_if is_header>
<h3><tmpl_var header></h3>
</tmpl_if>
<tmpl_if is__pull_quote_>
<tmpl_var _pull_quote_>
</tmpl_if>
</tmpl_loop>

These templates come from the Bricolage test suite, where until this release, there were never any template tests before. So you can see that the PHP 5 templating initiative has had major benefits for the stability of Bricolage, too. Now that I’ve really worked with all four templating architectures in Bricolage, I can now say that my preference for which to do goes in this order:

  1. Mason, because of its killer autohandler and inheritance architecture
  2. PHP 5 or Template Toolkit are tied for second place
  3. HTML::Template

In truth, all four are capable and have access to the entire Bricolage API so that they can output anything. So what are you waiting for? Download Bricolage and give it a try!

Looking for the comments? Try the old layout.

Bricolage 1.8.5 Released

The Bricolage development team is pleased to announce the release of Bricolage 1.8.5. This maintenance release addresses a number of issues in Bricolage 1.8.3 and adds a number of improvements (there was no announcement for the short-lived 1.8.4 release). The SOAP server in particular sees improvements in this release, with improved character set support; better support for related stories and media using URIs in addition to IDs; and as support for top-level element relations. Issues with the ordering of story elements have also been corrected, as well as errors when attempting to revert a story or media document or template. Here are the other highlights of this release:

Improvements

  • Added Linux startup script contrib/start_scripts/linux. [David]
  • Related story and media elements managed through the SOAP server can now use a combination of URI and site ID to identify related assets in addition to the existing approach of using story and media IDs. [David]
  • A list of subelements is now less likely to mysteriously become out of order and thus lead to strange action-at-a-distance errors. And even if they do become out of order, the error message will be more appropriate (“Warning! State inconsistent” instead of “Can’t call method “get_name” on an undefined value”). Reported by Curtis Poe. [David]
  • The SOAP media interface now supports creating relationships between the media documents elements and other story and media documents, just like the SOAP story interface does. [David]
  • The SOAP interface now supports Related stories and media on story type and media type elements just as in the UI. This involved the somewhat hackish necessity for including the “related_story_id” and “related_media_id” (or “related_story_uri” and “related_media_uri”) attributes in the “elements” XML element, but it does the trick. [David]

Bug Fixes

  • Calls to publish documents via SOAP will no longer fail if the published_version attribute is not specified and the document to be published has never been published before. [David]
  • The Bricolage virtual FTP server will no longer fail to start if Template Toolkit is installed but its version number is less than 2.14. Reported by Adam Rinehart. [David]
  • Stories and Media created or updated via the SOAP interface will now associate contributors of the appropriate type, instead of “All Contributors”. [Scott & David]
  • Deleting an element that has a template no longer causes an error. Thanks to Susan for the spot! [David]
  • Eliminated encoding errors when using the SOAP interface to output stories, media, or templates with wide characters. Reported by Scott Lanning. [David]
  • Reverting (stories, media, templates) no longer gives an error. Reported by Simon Wilcox, Rachel Murray, and others. [David]
  • Publishing a published version of a document that has a later version in workflow will no longer cause that later version to be mysteriously removed from workflow. This could be caused by passing a document looked up using the published_version to list() to $burner->publish_another in a template. [David]
  • The SOAP server story and media interfaces now support elements that contain both related stories and media, rather than one or the other. [David]
  • Attempting to preview a story or media document currently checked out to another user no longer causes an error. Reported by Paul Orrock. [David]
  • Custom fields with default values now have their values included when they are added to stories and media. Thanks to Clare Parkinson for the spot! [David]
  • The bric_queued script now requires a username and password and will authenticate the user. This user will then be used for logging events. All events logged when a job is run via the UI are now also logged by bric_queued. [Mark and David]
  • Preview redirections now use the protocol setting of the preview output channel if it’s available, and falls back on using “http://” when it’s not, instead of using the hard-coded “http://”. Thanks to Martin Bacovsky for the spot! [David]
  • The has_keyword() method in the Business class (from which the story and media classes inherit) now works. Thanks to Clare Parkinson for the spot! [David]
  • Clicking a link in the left-side navigation after the session has expired now causes the whole window to show the login form, rather than it showing inside the nav frame, which was useless. [Marshall]
  • The JavaScript that validates form contents once again works with htmlArea, provided htmlArea itself is patched. See the relevant htmlArea bug report for the patch. As of this writing, you must run the version of htmlArea in CVS. [David & Marshall]
  • The JavaScript that handles the double list manager has been vastly optimized. It should now be able to better handle large lists, such as a list of thousands of categories. Reported by Scott. [Marshall]
  • Uploading a new image to a media document with a different media type than the previous image no longer causes an Imager error. [David]

For a complete list of the changes, see the changes. For the complete history of ongoing changes in Bricolage, see Bric::Changes.

Download Bricolage 1.8.5 now from the Bricolage Web site Downloads page, from the SourceForge download page, and from the Kineticode download page.

About Bricolage

Bricolage is a full-featured, enterprise-class content management and publishing system. It offers a browser-based interface for ease-of use, a full-fledged templating system with complete HTML::Mason, HTML::Template, and Template Toolkit support for flexibility, and many other features. It operates in an Apache/mod_perl environment and uses the PostgreSQL RDBMS for its repository. A comprehensive, actively-developed open source CMS, Bricolage was hailed as “quite possibly the most capable enterprise-class open-source application available” by eWEEK.

Looking for the comments? Try the old layout.

Bricolage 1.8.3 Released

The Bricolage development team is pleased to announce the release of Bricolage 1.8.3. This maintenance release addresses a number of issues in Bricolage 1.8.2. The most important changes eliminate or greatly reduce the number of deadlocks caused during bulk publishes of many documents. Other changes include new contributed scripts for importing contributors and for generating thumbnail images, Russian localization, and various fixes for database transaction, template formatting, and various user interface fixes. Here are the other highlights of this release:

Improvements

  • Added contrib/thumbnails/precreate-thumbs.pl script to pre-create thumbnails from images. Useful for upgraders. [Scott]
  • Added contrib/bric_import_contribs to import contributors from a tab-delimited file. Development by Kineticode, sponsored by the RAND Corporation. [David]
  • Added the published_version parameter to the list() methods of the story, media, and template classes. This parameter forces the search to return the versions of the assets as they were last published, rather than the most recent version. This will be most useful to those looking up other documents in templates and publishing them, as a way of avoiding pulling documents out from other anyone who might have them checked out! [David]
  • All publishing and distribution jobs are now executed in their own transactions when they are triggered by the user interface. This is to reduce the chances of a deadlock between long-running publishing transactions. [David]
  • Optimized SQL queries for key names or that order by string values to use indexes in the list() and list_ids() methods of the story, media, and template classes. [David]
  • Added Russian localization. [Sergey Samoilenko].
  • Changed the foreign keys in the story, media, and formatting (template) tables so that DELETEs do not cascade, but are restricted. This means that before deleting any source, element, site, workflow, or other related object that has a foreign key reference in an asset table, those rows must be deleted. Otherwise, PostgreSQL will throw an exception. Hopefully, this will put a stop to the mysterious but very rare disappearance of stories from Bricolage. [David]
  • A call to $burner->burn_another in a template that passes in a date/time string in the future now causes a publish job to be scheduled for that time, rather than immediate burning the document and then scheduling the distribution to take place in the future. Reported by Ashlee Caul. [David]
  • Changing the sort order of a list of items in a search interface now properly reverses the entire collection of object over the pages, rather than just the objects for the current page. Thanks to Marshall for the spot! [David]

Bug Fixes

  • Publishing stories not in workflow via the SOAP server works again. [David]
  • The Burner object’s encoding attribute is now setable as well as readable. [David]
  • The category browser works again. [David]
  • Fixed Media Upload bug where the full local path was being used, by adding a “winxp” key to Bric::Util::Trans::FS to account for an update to HTTP::BrowserDetect. [Mark Kennedy]
  • Instances of a required custom field in story elements is no longer required once it has been deleted from the element definition in the element manager. Reported by Rod Taylor. [David]
  • A false value passed to the checked_out parameter of the list() and list_ids() methods of the story, media, and template (formatting) classes now properly returns only objects or IDs for assets that are not checked out. [David]
  • The cover date select widget now works properly in the clone interface when a non-ISO style date preference is selected. Thanks to Susan G. for the spot! [David]
  • Sorting templates based on Asset Type (Element) no longer causes an error. [David]
  • Fixed a number of the callbacks in the story, media, and template profiles so that they didn’t clear out the session before other callbacks were done with it. Most often seen as the error “Can’t call method ‘get_tiles’ on an undefined value” in the media profile, especially with IE/Windows (for some unknown reason). Reported by Ed Stevenson. [David]
  • Fixed typo in clone page that caused all output channels to be listed rather than only those associated with the element itself. [Scott]
  • Fixed double listing of the “All” group in the group membership double list manager. [Christian Hauser]
  • Image buttons now correctly execute the onsubmit() method for forms that define an onsubmit attribute. This means that, among other things, changes to a group profile will persist when you click the “Permissions” button. [David]
  • Simple search now works when it is selected when the “Default Search” preference is set to “Advanced”. Reported by Marshall Roch. [David]
  • Multiple alert types set up to trigger alerts for the same event will now all properly execute. Thanks to Christian Hauser for the spot! [David]
  • Publishing stories or media via SOAP with the published_only parameter (--published-only for bric_republish) now correctly republishes the published versions of documents even if the current version is in workflow. Reported by Adam Rinehart. [David]
  • Users granted a permission greater than READ to the members of the “All Users” group no longer get such permission to any members of the “Global Admins” group unless they have specifically been granted such permission to the members of the “Global Admins” group. Thanks to Marshall Roch for the spot! [David]

For a complete list of the changes, see the changes. For the complete history of ongoing changes in Bricolage, see Bric::Changes.

Download Bricolage 1.8.3 now from the Bricolage Website Downloads page, from the SourceForge download page, and from the Kineticode download page.

About Bricolage

Bricolage is a full-featured, enterprise-class content management and publishing system. It offers a browser-based interface for ease-of use, a full-fledged templating system with complete HTML::Mason, HTML::Template, and Template Toolkit support for flexibility, and many other features. It operates in an Apache/mod_perl environment and uses the PostgreSQL RDBMS for its repository. A comprehensive, actively-developed open source CMS, Bricolage was hailed as “quite possibly the most capable enterprise-class open-source application available” by eWEEK.

Enjoy!

–The Bricolage Team

Looking for the comments? Try the old layout.

Apache::Util::escape_html() Doesn't Like Perl UTF-8 Strings

I got bit by a bug with Apache::Util’s escape_html() function in mod_perl 1. It seems that it doesn’t like Perl’s Unicode encoded strings! This patch demonstrates the issue (be sure that your editor understands utf-8):

--- modperl/t/net/perl/util.pl.~1.18.~  Sun May 25 03:54:08 2003+++ modperl/t/net/perl/util.pl  Thu Sep  9 19:38:40 2004@@ -74,6 +74,25 @@  #print $esc_2; test ++$i, $esc eq $esc_2;++# Make sure that escape_html() understands multibyte characters.+my $utf8 = '<專輯>';+my $esc_utf8 = '<專輯>';+my $test_esc_utf8 = Apache::Util::escape_html($utf8);+test ++$i, $test_esc_utf8 eq $esc_utf8;+#print STDERR "Compare '$test_esc_utf8'\n     to '$esc_utf8'\n";++eval { require Encode };+unless ($@) {+    # Make sure escape_html() properly handles strings with Perl's+    # Unicode encoding.+    $utf8 = Encode::decode_utf8($utf8);+    $esc_utf8 = Encode::decode_utf8($esc_utf8);+    $test_esc_utf8 = Apache::Util::escape_html($utf8);+    test ++$i, $test_esc_utf8 eq $esc_utf8;+    #print STDERR "Compare '$test_esc_utf8'\n     to '$esc_utf8'\n";+}+ use Benchmark;  =pod

If I enable the print statements and look at the log, I see this:

Compare '<專輯>'
     to '<專輯>'
Compare '<å°è¼¯>'
     to '<專輯>'

The first escape appears to work correctly, but when I decode the string to Perl’s Unicode representation, you can see how badly escape_html() munges the text!

Curiously, both tests fail, although the first conversion appears to be correct. This could be due to the behavior of eq, though I’m not sure why. But it’s the second test that’s the more interesting, since it really screws things up.

Looking for the comments? Try the old layout.

More about…

Apache::TestMB Released!

As I mentioned last week, I’ve been working on a subclass of Module::Build that supports testing with Apache::Test. Today, Geoff announced the release of Apache::Test 1.12. This release includes the new Module::Build subclass, Apache::TestMB. Now anyone using Apache::Test to test their module can convert the build system to Module::Build.

To set an example, I’ve just released MasonX::Interp::WithCallbacks using the new build module. The conversion was simple; in fact, I think that Apache::TestMBis easier to use than Apache::TestMM (which integrates Apache::Test with ExtUtils::MakeMaker). My Makefile.PL had looked like this:

#!perl -w

use strict;
use ExtUtils::MakeMaker;
use File::Spec::Functions qw(catfile catdir);
use constant HAS_APACHE_TEST => eval {require Apache::Test};

# Set up the test suite.
if (HAS_APACHE_TEST) {
    require Apache::TestMM;
    require Apache::TestRunPerl;
    Apache::TestMM->import(qw(test clean));
    Apache::TestMM::filter_args();
    Apache::TestRunPerl->generate_script();
} else {
    print "Skipping Apache test setup.\n";
}

my $clean = join ' ', map { catfile('t', $_) }
  qw(mason TEST logs);

WriteMakefile(
    NAME        => 'MasonX::Interp::WithCallbacks',
    VERSION_FROM    => 'lib/MasonX/Interp/WithCallbacks.pm',
    PREREQ_PM       => { 'HTML::Mason'             => '1.23',
                                'Test::Simple'            => '0.17',
                                'Class::Container'        => '0.09',
                                'Params::CallbackRequest' => '1.11',
                              },
    clean               => { FILES => $clean },
    ($] >= 5.005 ?    ## Add these new keywords supported since 5.005
      (ABSTRACT_FROM    => 'lib/MasonX/Interp/WithCallbacks.pm',
        AUTHOR           => 'David Wheeler <david@kineticode.com>') : ()),
);

The new Build.PL simplifies things quite a bit. It looks like this:

use Module::Build;

my $build_pkg = eval { require Apache::TestMB }
  ? 'Apache::TestMB' : 'Module::Build';

$build_pkg->new(
    module_name        => 'MasonX::Interp::WithCallbacks',
    license            => 'perl',
    requires           => { 'HTML::Mason'             => '1.23',
                                'Test::Simple'            => '0.17',
                                'Class::Container'        => '0.09',
                                'Params::CallbackRequest' => '1.11'
                              },
    build_requires     => { Test::Simple => '0.17' },
    create_makefile_pl => 'passthrough',
    add_to_cleanup     => ['t/mason'],
)->create_build_script;

Much nicer, eh?

Looking for the comments? Try the old layout.

Module::Build + Apache::Test is Nearly Here

Over the last couple of days, I whipped up a new class to be added to the Apache HTTP Test Project. The new class, Apache::TestMB, is actually a subclass of Module::Build, and finally provides support for using Apache::Test with Module::Build. You use it just like Module::Build; however, since a lot of modules choose to install themselves even if Apache isn’t installed (because they can be used both inside and outside of mod_perl, e.g., HTML::Mason), I’m suggesting that Build.PL files look like this:

use Module::Build;

my $build_pkg = eval { require Apache::TestMB }
  ? "Apache::TestMB" : "Module::Build";

my $build = $build_pkg->new(
  module_name => "My::Module",
)->create_build_script;

Pretty simple, huh? To judge by the discussion, it will soon be committed to the Apache::Test repository and released to CPAN. My MasonX::Interp::WithCallbacks module will debut with a new Apache::TestMB-powered Build.PL soon afterward.

Looking for the comments? Try the old layout.

My Adventures with Mac OS X

I recently decided to make the leap from Yellow Dog Linux to Mac OS X on my Titanium PowerBook. Getting everything to work the way I wanted proved to be a challenge, but well worth it. This document outlines all that I learned, so that neither you nor I will have to experience such pain again. The overall goal was to get Bricolage up and running, figuring that if it worked, then just about any mod_perl based solution would run. I’m happy to say that I was ultimately successful. You can be, too.

In the descriptions below, I provide links to download the software you’ll need, as well as the shell commands I used to compile and install each package. In all cases (except for the installation of the Developer Tools), I saved each package’s sources to /usr/local/src and gunzipped and untarred them there. I also carried out each step as root, by running sudo -s. If you’re not comfortable using a Unix shell, you might want to read up on it, first. All of my examples also assume a sh-compatible shell, such as bash or zsh. Fortunately, zsh comes with OS X, so you can just enable it for yourself in NetInfo Manager by setting users -> <username> -> shell to “/bin/zsh”, where <username> is your user name.

Developer Tools

All of the software that I describe installing below must be compiled. To compile software on Mac OS X, you need to install the Mac OS X Developer Tools. These provide the cc compiler and many required libraries. Conveniently, these come on a CD-ROM with the Mac OS X Version 10.1 upgrade kit. I just popped in the CD and installed them like you’d install any other OS X software. I needed administrative access to OS X to install the Developer Tools (or, indeed, to install any of the other software I describe below), but otherwise it posed no problems.

The best time to install the Developer Tools is immediately after upgrading to OS X version 10.1. Then run the Software Update applet in the System preferences to get your system completely up-to-date. By the time I was done, I had the system updated to version 10.1.3.

Emacs

The first step I took in the process of moving to OS X was to get working the tools I needed most. Essentially, what this meant was GNU Emacs. Now I happen to be a fan of the X version of Emacs – not XEmacs, but GNU Emacs with X support built in. I wasn’t relishing the idea of having to install X on OS X (although there are XFree86 ports that do this), so I was really pleased to discover the Mac-Emacs project. All I had to do was patch the GNU Emacs 21.1 sources and compile them, and I was ready to go! GNU Emacs works beautifully with the OS X Aqua interface.

There were a few configuration issues for me to work out, however. I have become addicted to the green background that an old RedHat .XConfig file had set, and I wanted this feature in OS X, too. Plus, the default font was really ugly (well, too big, really – anyone know how to make it smaller in Emacs?) and the Mac command key was working as the Emacs META key, rather than the option key. So I poked around the net until I found the settings I needed and put them into my .emacs file:

(custom-set-faces
'(default ((t (:stipple nil
  :background "DarkSlateGrey"
  :foreground "Wheat"
  :inverse-video nil
  :box nil
  :strike-through nil
  :overline nil
  :underline nil
  :slant normal
  :weight normal
  :height 116
  :width normal
  :family "apple-andale mono"))))
'(cursor ((t (:background "Wheat"))))
; Use option for the meta key.
(setq mac-command-key-is-meta nil)

Installing Emacs is not required for installing any of the other packages described below – it just happens to be my favorite text editor and IDE. So I don’t provide the instructions here; the Mac-Emacs project does a plenty good job. If you’re not comfortable with Unix editors, you can use whatever editor you like. BBEdit is a good choice.

GDBM

Mac OS X doesn’t come with a DBM! But since mod_ssl needs it, we have to install it. Fortunately, I found this PDF detailing someone else’s adventures with mod_ssl on OS X, and it provided decent instructions for installing GDBM. First, I created a new user for GDBM. In NetInfoManager, I created a duplicate of the “unknown” user and named it “bin”. Then, I downloaded GDBM from the FSF, and installed it like this:

cd /usr/local/src/gdbm-1.8.0
cp /usr/libexec/config* .
./configure
make
make install
ln -s /usr/local/lib/libgdbm.a /usr/local/lib/libdbm.a

That did the trick. Nothing else was involved, fortunately.

Expat

Who doesn’t do something with XML these days? If your answer is, “not me!”, then you’ll need to install the Expat library in order to work with XML::Parser in Perl. Fortunately it’s relatively easy to install, although support for the -static flag appears to be broken in cc on OS X, so it needs to be stripped out. I downloaded it from its project bpage, and then did this:

cd /usr/local/src/expat-1.95.2
./configure
perl -i.bak -p -e \
  's/LDFLAGS\s*=\s*-static/LDFLAGS=/' \
  examples/Makefile
perl -i.bak -p -e \
  's/LDFLAGS\s*=\s*-static/LDFLAGS=/' \
  xmlwf/Makefile
make
make install

Perl

Although Mac OS X ships with Perl (Yay!), it’s the older 5.6.0 version. There have been many bug fixes included in 5.6.1, so I wanted to make sure I got the latest stable version before I built anything else around it (mod_perl, modules, etc.).

Being a Unix program, Perl doesn’t expect to run into the problems associated with a case-insensitive file system like that Mac OS X’s HFS Plus. So there are a couple of tweaks to the install process that make it slightly more complicated than you would typically expect. Fortunately, many have preceded us in doing this, and the work-arounds are well-known. Basically, it comes down to this:

cd /usr/local/src/perl-5.6.1/
export LC_ALL=C
export LANG=en_US
perl -i.bak -p -e 's|Local/Library|Library|g' hints/darwin.sh
sh Configure -des -Dfirstmakefile=GNUmakefile -Dldflags="-flat_namespace"
make
make test
make install

There were a few errors during make test, but none of them seems to be significant. Hopefully, in the next version of Perl, the build will work just as it does on other platforms.

Downloads

Before installing Open SSL, mod_ssl, mod_perl, and Apache, I needed to get all the right pieces in place. The mod_ssl and mod_perl configure processes patch the Apache sources, so the Apache sources have to be downloaded and gunzipped and untarred into an adjacent directory. Furthermore, the mod_ssl version number corresponds to the Apache version number, so you have to be sure that they match up. Normally, I would just download the latest versions of all of these pieces and run with it.

However, Bricolage requires the libapreq library and its supporting Perl modules to run, and these libraries have not yet been successfully ported to Mac OS X. But worry not; fearless mod_perl hackers are working on the problem even as we speak, and there is an interim solution to get everything working.

As of this writing, the latest version of Apache is 1.3.24. But because I needed libapreq, I had to use an experimental version of Apache modified to statically compile in libapreq. Currently, only version 1.3.23 has been patched for libapreq, so that’s what I had to use. I discovered this experimental path thanks to a discussion on the Mac OS X Perl mail list.

So essentially what I did was download the experimental apache.tar.gz and the experimental lightweight apreq.tar.gz packages and gunzip and untar them into /usr/local/src. Then I was ready to move on to Open SSL, mod_ssl, and mod_perl.

Open SSL

Compiling Open SSL was pretty painless. One of the tests fails, but it all seems to work out, anyway. I download the sources from the Open SSL site, and did this:

cd /usr/local/src/openssl-0.9.6c
./config
make
make test

mod_ssl

The mod_ssl Apache module poses no problems whatsoever. I simply downloaded mod_ssl-2.8.7-1.3.23 from the mod_ssl site (note that the “1.3.23” at the end matches the version of Apache I downloaded) and gunzipped and untarred it into /usr/local/src/. Then I simply excuted:

./configure --with-apache=/usr/local/src/apache_1.3.23

mod_perl

Configuring and installing mod_ssl was, fortunately, a relatively straight-forward process. Getting Apache compiled with mod_perl and mod_ssl, however, was quite tricky, as you’ll see below. A number of braver folks than I have preceded me in installing mod_perl, so I was able to rely on their hard-earned knowledge to get the job done. For example, Randal Schwartz posted instructions to the mod_perl mail list, and his instructions worked well for me. So I downloaded the sources from the mod_perl site, and did this:

cd /usr/local/src/mod_perl-1.26
perl Makefile.PL \
  APACHE_SRC=/usr/local/src/apache_1.3.23/src \
  NO_HTTPD=1 \
  USE_APACI=1 \
  PREP_HTTPD=1 \
  EVERYTHING=1
make
make install

Apache

Getting Apache compiled just right was the most time-consuming part of this process for me. Although many had gone before me in this task, everybody seems to do it differently. I had become accustomed to just allowing Apache to use most of its defaults when I compiled under Linux, but now I was getting all kinds of errors while following different instructions from different authorities from around the web. Sometimes Apache wouldn’t compile at all, and I’d get strange errors. Other times it would compile, pass all of its tests, and install, only to offer up errors such as

dyld: /usr/local/apache/bin/httpd Undefined symbols: _log_config_module

when I tried to start it. It turns out that the problem there was that I had a number of modules compiled as DSOs – that is, libraries that can be loaded into Apache dynamically – but wasn’t loading them properly in my httpd.conf. This was mainly because I’ve grown accustomed to Apache having all the libraries I needed compiled in statically, so I simply didn’t have to worry about them.

But I finally hit on the right incantation to get Apache to compile with everything I need added statically, but still with support for DSOs by compiling in mod_so. I present it here for your viewing pleasure:

SSL_BASE=/usr/local/src/openssl-0.9.6c/ \
    ./configure \
    --with-layout=Apache \
    --enable-module=ssl \
    --enable-module=rewrite \
    --enable-module=so \
    --activate-module=src/modules/perl/libperl.a \
    --disable-shared=perl \
    --without-execstrip
  make
  make certificate TYPE=custom 
  make install

This series of commands successfully compiled Apache with mod_perl and mod_ssl support statically compiled in, along with most of the other default modules that come with Apache. In short, everything is there that you need to run a major application with security such as Bricolage.

Note that make certificate will lead you through the process of creating an SSL certificate. I like to use the “custom” type so that it reflects the name of my organization. But you can use whatever approach you’re most comfortable with. Consult the mod_ssl INSTALL file for more information.

libapreq

Once Apache is installed with mod_perl and mod_ssl, the rest is gravy! The experimental libapreq library I downloaded installed without a hitch:

cd /usr/local/src/httpd-apreq
perl Makefile.PL
make
make install

PostgreSQL

PostgreSQL is a sophisticated open-source Object-Relational DBMS. I use it a lot in my application development, and it, too, is required by Bricolage. I was a bit concerned about how well it would compile and work on Mac OS X, but I needn’t have worried. First of all, Apple has provided some pretty decent instructions. Although they mainly document how to install MySQL, a competing open-source RDBMS, many of the same concepts apply to PostgreSQL.

The first thing I had to do was to create the “postgres” user. This is the system user that PostgreSQL typically runs as. I followed Apple’s instructions, using NetInfo Manager to duplicate the default “www” group and “www” user and give the copies the name “postgres” and a new gid and uid, respectively.

Next I downloaded the PostgreSQL version 7.2.1 sources. Version 7.2 is the first to specifically support Mac OS X, so going about the install was as simple as it is on any Unix system:

./configure --enable-multibyte=UNICODE
make
make install

That was it! PostgreSQL was now installed. Next I had to initialize the PostgreSQL database directory. Again, this works much the same as it does on any Unix system:

sudo -u postgres /usr/local/pgsql/bin/initdb \
  -D /usr/local/pgsql/data

The final step was to start PostgreSQL and try to connect to it:

sudo -u postgres /usr/local/pgsql/bin/pg_ctl start \
  -D /usr/local/pgsql/data /usr/local/pgsql/bin/psql -U postgres template1

If you follow the above steps and find yourself at a psql prompt, you’re in business! Because I tend to use PostgreSQL over TCP, I also enabled TCP connectivity by enabling the “tcpip_socket” option in the postgresql.conf file in the data directory created by initdb:

tcpip_socket = true

If you’re like me, you like to have servers such as PostgreSQL start when your computer starts. I enabled this by creating a Mac OS X PostgreSQL startup bundle. It may or may not be included in a future version of PostgreSQL, but in the meantime, you can download it from here. Simply download it, gunzip and untar it into /Library/StartupItems, restart OS X, and you’ll see it start up during the normal Mac OS X startup sequence. I built this startup bundle by borrowing from the existing FreeBSD PostgreSQL startup script, the Apache startup script that ships with OS X, and by reading the Creating SystemStarter Startup Item Bundles HOWTO.

XML::Parser

At this point, I had most

of the major pieces in place, and it was time for me to install the Perl modules I needed. First up was XML::Parser. For some reason, XML::Parser can’t find the expat libraries, even though the location in which I installed them is pretty common. I got around this by installing XML::Parser like this:

perl Makefile.PL EXPATLIBPATH=/usr/local/lib \
  EXPATINCPATH=/usr/local/include
make
make test
make install

Text::Iconv

In Bricolage, Text::Iconv does all the work of converting text between character sets. This is because all of the data is stored in the database in Unicode, but we wanted to allow users to use the character set with which they’re accustomed in the UI. So I needed to install Text::Iconv. Naturally, Mac OS X doesn’t come with libiconv – a library on which Text::Iconv depends – so I had to install it. Fortunately, it was a simple process to download it and do a normal build:

cd /usr/local/src/libiconv-1.7
./configure
make
make install

Now, Text::Iconv itself was a little more problematic. You have to tell it to look for libiconv by adding the -liconv option to the LIBS key in Makefile.PL. I’ve simplified doing this with the usual Perl magic:

perl -i.bak -p -e \
  "s/'LIBS'\s*=>\s*\[''\]/'LIBS' => \['-liconv'\]/" \
  Makefile.PL
perl Makefile.PL
make
make test
make install

DBD::Pg

Although the DBI installed via the CPAN module without problem, DBD::Pg wanted to play a little less nice. Of course I specified the proper environment variables to install it (anyone know why DBD::Pg’s Makefile.PL script can’t try to figure those out on its own?), but still I got this error during make:

/usr/bin/ld: table of contents for archive:
/usr/local/pgsql/lib/libpq.a is out of date;
rerun  ranlib(1) (can't load from it)

But this was one of those unusual situations in which the error message was helpful. So I took the error message’s advice, and successfully compiled and installed DBD::Pg like this:

ranlib /usr/local/pgsql/lib/libpq.a
export POSTGRES_INCLUDE=/usr/local/pgsql/include
export POSTGRES_LIB=/usr/local/pgsql/lib
perl Makefile.PL
make
make test
make install

LWP

The last piece I needed to worry about customizing when I installed it was LWP. Before installing, back up /usr/bin/head. The reason for this is that LWP will install /usr/bin/HEAD, and because HFS Plus is a case-insensitive file system, it’ll overwrite /usr/bin/head! This is a pretty significant issue, since many configure scripts use /usr/bin/head. So after installing LWP, move /usr/bin/HEAD, GET, & POST to /usr/local/bin. Also move /usr/bin/lwp* to /usr/local/bin. Then move your backed-up copy of head back to /usr/bin.

Naturally, I didn’t realize that this was necessary until it was too late. I installed LWP with the CPAN module, and it wiped out /usr/bin/head. Fortunately, all was not lost (though it took me a while to figure out why my Apache compiles were failing!): I was able to restore head by copying it from the Mac OS X installer CD. I Just popped it in an executed the command:

cp "/Volumes/Mac OS X Install CD/usr/bin/head" /usr/bin

And then everything was happy again.

Bricolage

And finally, the pièce de résistance: Bricolage! All of the other required Perl modules installed fine from Bundle::Bricolage:

perl -MCPAN -e 'install Bundle::Bricolage'

Then I simply followed the directions in Bricolage’s INSTALL file, and started ‘er up! I would document those steps here, but the install process is currently in flux and likely to change soon. The INSTALL file should always be current, however – check it out!

To Be Continued

No doubt my adventures with Unix tools on Mac OS X are far from over. I’ve reported to various authors on the issues I’ve described above, and most will soon be releasing new versions to address those issues. As they do, I’ll endeavor to keep this page up-to-date. In the meantime, I am thoroughly enjoying working with the first really solid OS that Apple has released in years, and thrilled that I can finally have the best of both worlds: a good, reliable, and elegant UI, and all the Unix power tools I can stand! I hope you do, too.

Looking for the comments? Try the old layout.