Just a Theory

By David E. Wheeler

Posts about Perl

SVN::Notify 2.20 Adds Colorized Diffs

After getting prodded by Erik Hatcher, I went ahead and added another subclass to SVN::Notify. This one adds a pretty colorized diff to the message, instead of just the plain text one. See an example here. I’ve also added links from the lists of affected files into the diffs in the HTML and new HTML::ColorDiff layouts.

Enjoy!

Update: And now I’ve released SVN::Notify 2.21 with a few minor fixes, including XHTML 1.1 compliance.

Looking for the comments? Try the old layout.

SVN::Notify 2.10 Generalizes Behavior

It’s all Autrijus’ fault.

As I mentioned last week when I released SVN::Notify 2.0, Autrijus has suggested using SVN::Notify as the base class for modules that do other things, such as send instant messages or update a checkout for backup purposes. Instantly seeing the value in this, I further realized that I could greatly simplify the support for HTML notification emails by moving the HTML-specific code to a subclass and then just let polymorphism do the work.

The result SVN::Notify 2.10. To simplify the move to a subclass for the HTML notifications, I broke up the old send() method into a large number of other methods that affect various parts of the composition of the email, such as headers, starting the message, outputting the log message, the file list, and outputting or attaching the diff. Then I just overrode the few methods that need different behavior in the subclass, and it all worked!

I realized, as I worked on it, I also realized that I was following the same principals that Ovid has written about with regard to the use of if. I was able to remove quite a few of them by moving HTML to a subclass. Of course, there are still some to enable diffs to be either included in an email or attached, but I didn’t want to split things up too much, or I’d have a geometric explosion of subclasses!

The svnnotify script, in the meantime, remains largely unmodified. The only change is the deprecation of the --format option in favor of a new option, --handler. Use this option to specify what subclass of SVN::Notify should handle the notification. So far, there’s just one, --format HTML, but I’m sure that Autrijus will soon add --format Jabber, and I’d like to add --format HTML::ColorDiff, myself. I might have to move the processing of command-line arguments out of svnnotify and into SVN::Notify, instead, so that subclasses can add new options. We’ll see what comes up.

Other changes to SVN::Notify include:

  • Added code to Build.PL to set the shebang line in the test scripts. Reported by Robert Spier.
  • Changed name of attached diff file to be named for the revision and the committer, rather than the committer and the date. Suggested by Robert Spier.
  • Added Author, Date, and Revision information to the top of each message.
  • The ViewCVS URL is no longer output for each file. A single link for the entire revision number is put at the top of the email, instead. ViewCVS Revision URL syntax pointed out by Peter Valdemar Morch.
  • Changed the send() method to execute() to better reflect its generalized use as the method that executes actions in response to Subversion activity.
  • The tests no longer require HTML::Entities to run. The HTML email tests will be skipped if it is not installed.
  • Added accessor methods for the attributes of SVN::Notify.

Enjoy!

Looking for the comments? Try the old layout.

SVN::Notify 2.0 Hitting CPAN

My latest Perl module, SVN::Notify 2.00, has hit CPAN. This is a port of my widely-used activitymail CVS notification script to Subversion. But it underwent quite a few changes over the port, including:

Modularization
The old monolithic activitymail script is gone. It has been replaced with a Perl class, SVN::Notify, that does most of the work. The new script, svnnotify, is essentially just a wrapper around the class; all it does is process command-line arguments and then pass the results to SVN::Notify.
Simplification
Subversion’s system for hooking in to commit transactions is far better thought-out than that of CVS. It’s now easy to capture the results of an entire commit in a single transaction, without having to write out temp files to keep track of where we are and to concatenate diffs. As a result, SVN::Notify has a much simpler architecture and implementation that requires fewer third-party modules to do its work. In addition, the move to a class should make it much easier to build on SVN::Notify in the future than it was with activitymail. Autrijus Tang already suggested a number of ideas on IRC, including SVN::Notify::Jabber or SVN::Notify::Export. Have at it, everyone!
Reduced Resource Usage
I had heard some complaints that, on very large commits, activitymail could end up taking up a huge amount of memory. As best I could figure, this was because it was loading everything into memory, including the diff for the commit! SVN::Notify avoids this problem by using a file handle to read in a diff an print it to sendmail one line at a time. This should keep resource usage by SVN::Notify way below what activitymail used.
Context-Specific Notifications
SVN::Notify has added support for mapping email addresses to regular expressions. Whenever a regular expression matches the name of one or more of the directories affected in a single commit, the corresponding email address will be added to the list of recipients of the notification. This is a great way to get notification messages sent to particular email addressed based on what part of the Subversion tree was affected by a commit. I intend to use this to set it up so that a list of translators only get notification about a commit when it changes a directory related to localization in my projects, so that they can ignore commits to other parts of the application.

These are the major changes, but SVN::Notify also features a number of smaller improvements over its activitymail ancestor, including character set support, user domain support for the “From” header, explicit specification of a “From” header, properly escaped content when sending HTML-formatted notifications, and a maximum subject length configuration.

So what did it lose? Just a few things:

  • syncmail-like behavior. Did anyone ever use this? If so, feel free to implement SVN::Notify::Syncmail.
  • Arguments to diff. SVN::Notify just uses svnlook diff to generate a diff. Support for other diffs could be added in a future version, if people really need it.
  • New directories and imports can no longer be ingored, because in Subversion they’re really no different from any other commit.
  • Limit on the maximum size of the email. This is because SVN::Notify no longer loads the entire email into memory to measure it.
  • Excluding certain files from the diff. Subversion handles this itself by paying attention to the media type of each file.
  • Windows support. Actually, I’m not sure if activitymail was ever used on Windows, but the new method of using pipes to communicate with other processes isn’t supported by Windows, as near as I can tell. There are comments in the code for those who wish to do the port; it would probably be easy using Win32::Process.

Not too much, eh? Let me know what you think, and send feedback!

Looking for the comments? Try the old layout.

Lessons Learned with Perl and UTF-8

I learned quite a lot last week as I was making Bricolage much more Unicode-aware. Bricolage has always managed Unicode content and stored it in a PostgreSQL Unicode-encoded database. And by “Unicode” I of course mean “UTF-8”. By far the biggest nightmare was figuring out the bug with Apache::Util::escape_html(), but ultimately it came down to an interesting lesson.

Why was I making Bricolage Unicode-aware? Well, it all started with a bug report from Kang-min Liu (a.k.a. “Gugod”). I had naïvely thought that if strings were Unicode that Perl would know it and do the right thing. It turns out I was wrong. Perl assumes that everything is binary unless you tell it otherwise. This means that Perl operators such as length and substr will count bytes instead of characters. And in the case of Unicode, where characters can be multiple bytes, this can cause serious problems. Not only were strings improperly concatenated mid-character for Gugod, but PostgreSQL could refuse to accept such strings, since a chopped-up multibyte character isn’t valid Unicode!

So I had to make some decisions: Either stop using Perl operators that count bytes, or let Perl know that all the strings that Bricolage deals with are Unicode strings. The former wasn’t really an option, of course, since users can specify that certain content fields be a certain length of characters. So with a lot of testing help from Gugod and his Bricolage install full of multibyte characters, I set about doing so. The result is in the recently released Bricolage 1.8.2 and I’m blogging what I learned for both your reference and mine.

Perl considers its internal representation of strings to be UTF-8 strings, and it knows what variables contain valid UTF-8 strings because they have a special flag set on them, called, strangely enough, utf8. This flag isn’t set by default, but can be set in a number of ways. The ways I’ve found so far are:

  • Using Encode::decode() to decode a string from binary to Perl’s internal representation. The use of the word “decode” here had confused me for a while, because I thought it was a special encoding. But the truth is that it’s not. Strings can have any number of encodings, such as “ISO-8859-1”, “GB3212”, “EUC-KR”, “UTF-8”, and the like. But when you “decode” a string, you’re telling Perl that it’s not any of those encodings, but Perl’s own representation. I was confused because Perl’s internal representation is UTF-8, which is an encoding. But really it’s not UTF-8, It’s “utf8”, which isn’t an encoding, but Perl’s own thing.

  • Cheat: Use Encode::_set_utf8_on(). This private function is nevertheless documented by the Encode module, and therefore usable. What it does is simply turn on the utf8 flag on a variable. You need be confident that the variable contains only valid UTF-8 characters, but if it does, then you should be pretty safe.

  • Using the three-argument version of open, such as

    open my $fh, "<utf8", "/foo/bar" or die "Cannot open file: $!\n"

    Now when you read lines from this file, they will automatically be decoded to utf8.

  • Using binmode to set the mode on a file handle:

    binmode $fh, ":utf8";

    As with the three-argument version of open this forces Perl to decode the strings read from the file handle.

  • use utf8;. This Perl pragma indicates that everything within its scope is UTF-8, and therefore should be decoded to utf8.

So I started applying these approaches in various places. The first thing I did was to set the utf8 flag on data coming from the browser with Encode::_set_utf8_on(). Shitty browsers can of course send shitty data, but I’m deciding, for the moment at least, to trust browser to send only UTF-8 when I tell them that’s what I want. This solved Gugod’s immediate problem, and I happily closed the bug. But then he started to run into places where strings appeared properly in some places but not in others. We spent an entire day (night for Gugod–I really appreciated the help!) tracking down the problem, and there turned out to be two of them. One was the the bug with Apache::Util::escape_html() that I’ve [described elsewhere]the bug with Apache::Util::escape_html(), but the other proved more interesting.

It seems that if you concatenate a UTF-8 string with the utf8 flagged turned on with a UTF-8 string without utf8 turned on, the text in the unflagged variable turns to crap! I have no idea why this is, but Gugod noticed that strings pulled into the UI from the Bricolage zh_tw localization library simply didn’t display properly. I had him add use utf8; to the zh_tw module, and the problem went away!

So the lesson learned here is: If you’re going to make Perl strings Unicode-aware, then all of your Perl strings need to be Unicode-aware. It’s an all or nothing kind of thing.

So while setting the utf8 flag on browser submits and adding use utf8; to the localization modules got us part of the way toward a solution, it turned out to be trickier than I expected to get the utf8 flag set on everything. The places I needed to get it working were in the UI Mason components, in templates, and in strings pulled from the database.

It took a bit of research, but I think I successfully figured out how to make the UI Mason components UTF-8 aware. I just added preamble => "use utf8\n;" to the creation of the Mason interpretor. This gets passed on to is compiler, and now that string is added to the beginning of every template. This made things behave better in the UI. I applied the same approach to the interpreter created for Mason templates with equal success.

I’m less confident that I pulled it off for the HTML::Template and Template Toolkit templating architectures. In a discussion on the templates mailing list, Andy Wardley suggested that it wasn’t currently possible. But I wasn’t so sure. It seemed to me that, since Bricolage reads in the templates and asks TT to execute them within a certain scope, that I could just set the mode to utf8 on the file handle and then execute the template within the scope of a use utf8; statement. So that’s what I did. Feedback on whether it works or not would be warmly welcomed.

I tried a similar approach with the HTML::Template burner. Again, the burner reads the templates from files and passes them to HTML::Template for execution (as near as I could tell, anyway; I’m not an HTML::Template template user). Hopefully it’ll just work.

So that just left the database. Since the database is Unicode-only, all I needed to do was to turn on the utf8 flag for all content pulled from the database. Amazingly, this hasn’t come up as an issue for people very much, because DBI doesn’t do anything about Unicode. I picked up an older discussion started by Matt Sergeant on the dbi-dev mail list, but it looks like it might be a while before DBI has fast, integrated support for turning utf8 on and off for various database handles and columns. I look forward to it, though, because it’s likely to be very efficient. I greatly look forward to seeing the results of Tim’s work in the next release of DBI. I opened another bug report to remind myself to take advantage of the new feature when it’s ready.

So in the meantime, I needed to find another solution. Fortunately, my fellow PostgreSQL users had run into it before, and added what I needed to DBD::Pg back in version 1.22. The pg_enable_utf8 database handle parameter forces the utf8 flag to be turned on for all string data returned from the database. I added this parameter to Bricolage, and now all data pulled from the database is utf8. And so are the UI components, templates, localization libraries, and data submitted from browsers. I think that nailed everything, but I know that Unicode issues are a slippery slope. I can’t wait until I have to deal with them again!

Not.

Looking for the comments? Try the old layout.

More about…

Apache::Util::escape_html() Doesn't Like Perl UTF-8 Strings

I got bit by a bug with Apache::Util’s escape_html() function in mod_perl 1. It seems that it doesn’t like Perl’s Unicode encoded strings! This patch demonstrates the issue (be sure that your editor understands utf-8):

--- modperl/t/net/perl/util.pl.~1.18.~  Sun May 25 03:54:08 2003+++ modperl/t/net/perl/util.pl  Thu Sep  9 19:38:40 2004@@ -74,6 +74,25 @@  #print $esc_2; test ++$i, $esc eq $esc_2;++# Make sure that escape_html() understands multibyte characters.+my $utf8 = '<專輯>';+my $esc_utf8 = '<專輯>';+my $test_esc_utf8 = Apache::Util::escape_html($utf8);+test ++$i, $test_esc_utf8 eq $esc_utf8;+#print STDERR "Compare '$test_esc_utf8'\n     to '$esc_utf8'\n";++eval { require Encode };+unless ($@) {+    # Make sure escape_html() properly handles strings with Perl's+    # Unicode encoding.+    $utf8 = Encode::decode_utf8($utf8);+    $esc_utf8 = Encode::decode_utf8($esc_utf8);+    $test_esc_utf8 = Apache::Util::escape_html($utf8);+    test ++$i, $test_esc_utf8 eq $esc_utf8;+    #print STDERR "Compare '$test_esc_utf8'\n     to '$esc_utf8'\n";+}+ use Benchmark;  =pod

If I enable the print statements and look at the log, I see this:

Compare '<專輯>'
     to '<專輯>'
Compare '<å°è¼¯>'
     to '<專輯>'

The first escape appears to work correctly, but when I decode the string to Perl’s Unicode representation, you can see how badly escape_html() munges the text!

Curiously, both tests fail, although the first conversion appears to be correct. This could be due to the behavior of eq, though I’m not sure why. But it’s the second test that’s the more interesting, since it really screws things up.

Looking for the comments? Try the old layout.

More about…

Introduction to Bricolage Published by Perl.com

Perl.com has published the first in a series of articles I’ve promised to write, “Content Management with Bricolage.” The article is targeted at organizational decision makers who need to evaluate Bricolage as part of their selection of a content management solution. So I’ve spent some time discussing what content management is and how Bricolage fits in. I mention a lot of other CMSes as for comparative purposes, to try to really give people a feel for how Bricolage can meet their needs.

I also go over some of Bricolage’s most important features, such as multisite management, document modeling, Perl templating, meaningful URLs, workflow, and any number of outputs via “output channels.” The article concludes with a list of sites known to be managed by Bricolage, as well as the promise for more.

The next article in the series will cover installation and configuration. Subsequent articles will cover document analysis and modeling, Mason templating, and fun with the SOAP server. Enjoy!

Looking for the comments? Try the old layout.

Nicholas Clark

Nick Clark goes Wild!

I just had to share this lovely picture of Nick Clark, taken on the Friday night of OSCon 2004 at Matt Sergeant’s party. I honestly have no idea what Nick was doing, but it was worth it for the photo, don’t you think?

Looking for the comments? Try the old layout.

Apache::TestMB Released!

As I mentioned last week, I’ve been working on a subclass of Module::Build that supports testing with Apache::Test. Today, Geoff announced the release of Apache::Test 1.12. This release includes the new Module::Build subclass, Apache::TestMB. Now anyone using Apache::Test to test their module can convert the build system to Module::Build.

To set an example, I’ve just released MasonX::Interp::WithCallbacks using the new build module. The conversion was simple; in fact, I think that Apache::TestMBis easier to use than Apache::TestMM (which integrates Apache::Test with ExtUtils::MakeMaker). My Makefile.PL had looked like this:

#!perl -w

use strict;
use ExtUtils::MakeMaker;
use File::Spec::Functions qw(catfile catdir);
use constant HAS_APACHE_TEST => eval {require Apache::Test};

# Set up the test suite.
if (HAS_APACHE_TEST) {
    require Apache::TestMM;
    require Apache::TestRunPerl;
    Apache::TestMM->import(qw(test clean));
    Apache::TestMM::filter_args();
    Apache::TestRunPerl->generate_script();
} else {
    print "Skipping Apache test setup.\n";
}

my $clean = join ' ', map { catfile('t', $_) }
  qw(mason TEST logs);

WriteMakefile(
    NAME        => 'MasonX::Interp::WithCallbacks',
    VERSION_FROM    => 'lib/MasonX/Interp/WithCallbacks.pm',
    PREREQ_PM       => { 'HTML::Mason'             => '1.23',
                                'Test::Simple'            => '0.17',
                                'Class::Container'        => '0.09',
                                'Params::CallbackRequest' => '1.11',
                              },
    clean               => { FILES => $clean },
    ($] >= 5.005 ?    ## Add these new keywords supported since 5.005
      (ABSTRACT_FROM    => 'lib/MasonX/Interp/WithCallbacks.pm',
        AUTHOR           => 'David Wheeler <david@kineticode.com>') : ()),
);

The new Build.PL simplifies things quite a bit. It looks like this:

use Module::Build;

my $build_pkg = eval { require Apache::TestMB }
  ? 'Apache::TestMB' : 'Module::Build';

$build_pkg->new(
    module_name        => 'MasonX::Interp::WithCallbacks',
    license            => 'perl',
    requires           => { 'HTML::Mason'             => '1.23',
                                'Test::Simple'            => '0.17',
                                'Class::Container'        => '0.09',
                                'Params::CallbackRequest' => '1.11'
                              },
    build_requires     => { Test::Simple => '0.17' },
    create_makefile_pl => 'passthrough',
    add_to_cleanup     => ['t/mason'],
)->create_build_script;

Much nicer, eh?

Looking for the comments? Try the old layout.

Module::Build + Apache::Test is Nearly Here

Over the last couple of days, I whipped up a new class to be added to the Apache HTTP Test Project. The new class, Apache::TestMB, is actually a subclass of Module::Build, and finally provides support for using Apache::Test with Module::Build. You use it just like Module::Build; however, since a lot of modules choose to install themselves even if Apache isn’t installed (because they can be used both inside and outside of mod_perl, e.g., HTML::Mason), I’m suggesting that Build.PL files look like this:

use Module::Build;

my $build_pkg = eval { require Apache::TestMB }
  ? "Apache::TestMB" : "Module::Build";

my $build = $build_pkg->new(
  module_name => "My::Module",
)->create_build_script;

Pretty simple, huh? To judge by the discussion, it will soon be committed to the Apache::Test repository and released to CPAN. My MasonX::Interp::WithCallbacks module will debut with a new Apache::TestMB-powered Build.PL soon afterward.

Looking for the comments? Try the old layout.

My Adventures with Mac OS X

I recently decided to make the leap from Yellow Dog Linux to Mac OS X on my Titanium PowerBook. Getting everything to work the way I wanted proved to be a challenge, but well worth it. This document outlines all that I learned, so that neither you nor I will have to experience such pain again. The overall goal was to get Bricolage up and running, figuring that if it worked, then just about any mod_perl based solution would run. I’m happy to say that I was ultimately successful. You can be, too.

In the descriptions below, I provide links to download the software you’ll need, as well as the shell commands I used to compile and install each package. In all cases (except for the installation of the Developer Tools), I saved each package’s sources to /usr/local/src and gunzipped and untarred them there. I also carried out each step as root, by running sudo -s. If you’re not comfortable using a Unix shell, you might want to read up on it, first. All of my examples also assume a sh-compatible shell, such as bash or zsh. Fortunately, zsh comes with OS X, so you can just enable it for yourself in NetInfo Manager by setting users -> <username> -> shell to “/bin/zsh”, where <username> is your user name.

Developer Tools

All of the software that I describe installing below must be compiled. To compile software on Mac OS X, you need to install the Mac OS X Developer Tools. These provide the cc compiler and many required libraries. Conveniently, these come on a CD-ROM with the Mac OS X Version 10.1 upgrade kit. I just popped in the CD and installed them like you’d install any other OS X software. I needed administrative access to OS X to install the Developer Tools (or, indeed, to install any of the other software I describe below), but otherwise it posed no problems.

The best time to install the Developer Tools is immediately after upgrading to OS X version 10.1. Then run the Software Update applet in the System preferences to get your system completely up-to-date. By the time I was done, I had the system updated to version 10.1.3.

Emacs

The first step I took in the process of moving to OS X was to get working the tools I needed most. Essentially, what this meant was GNU Emacs. Now I happen to be a fan of the X version of Emacs – not XEmacs, but GNU Emacs with X support built in. I wasn’t relishing the idea of having to install X on OS X (although there are XFree86 ports that do this), so I was really pleased to discover the Mac-Emacs project. All I had to do was patch the GNU Emacs 21.1 sources and compile them, and I was ready to go! GNU Emacs works beautifully with the OS X Aqua interface.

There were a few configuration issues for me to work out, however. I have become addicted to the green background that an old RedHat .XConfig file had set, and I wanted this feature in OS X, too. Plus, the default font was really ugly (well, too big, really – anyone know how to make it smaller in Emacs?) and the Mac command key was working as the Emacs META key, rather than the option key. So I poked around the net until I found the settings I needed and put them into my .emacs file:

(custom-set-faces
'(default ((t (:stipple nil
  :background "DarkSlateGrey"
  :foreground "Wheat"
  :inverse-video nil
  :box nil
  :strike-through nil
  :overline nil
  :underline nil
  :slant normal
  :weight normal
  :height 116
  :width normal
  :family "apple-andale mono"))))
'(cursor ((t (:background "Wheat"))))
; Use option for the meta key.
(setq mac-command-key-is-meta nil)

Installing Emacs is not required for installing any of the other packages described below – it just happens to be my favorite text editor and IDE. So I don’t provide the instructions here; the Mac-Emacs project does a plenty good job. If you’re not comfortable with Unix editors, you can use whatever editor you like. BBEdit is a good choice.

GDBM

Mac OS X doesn’t come with a DBM! But since mod_ssl needs it, we have to install it. Fortunately, I found this PDF detailing someone else’s adventures with mod_ssl on OS X, and it provided decent instructions for installing GDBM. First, I created a new user for GDBM. In NetInfoManager, I created a duplicate of the “unknown” user and named it “bin”. Then, I downloaded GDBM from the FSF, and installed it like this:

cd /usr/local/src/gdbm-1.8.0
cp /usr/libexec/config* .
./configure
make
make install
ln -s /usr/local/lib/libgdbm.a /usr/local/lib/libdbm.a

That did the trick. Nothing else was involved, fortunately.

Expat

Who doesn’t do something with XML these days? If your answer is, “not me!”, then you’ll need to install the Expat library in order to work with XML::Parser in Perl. Fortunately it’s relatively easy to install, although support for the -static flag appears to be broken in cc on OS X, so it needs to be stripped out. I downloaded it from its project bpage, and then did this:

cd /usr/local/src/expat-1.95.2
./configure
perl -i.bak -p -e \
  's/LDFLAGS\s*=\s*-static/LDFLAGS=/' \
  examples/Makefile
perl -i.bak -p -e \
  's/LDFLAGS\s*=\s*-static/LDFLAGS=/' \
  xmlwf/Makefile
make
make install

Perl

Although Mac OS X ships with Perl (Yay!), it’s the older 5.6.0 version. There have been many bug fixes included in 5.6.1, so I wanted to make sure I got the latest stable version before I built anything else around it (mod_perl, modules, etc.).

Being a Unix program, Perl doesn’t expect to run into the problems associated with a case-insensitive file system like that Mac OS X’s HFS Plus. So there are a couple of tweaks to the install process that make it slightly more complicated than you would typically expect. Fortunately, many have preceded us in doing this, and the work-arounds are well-known. Basically, it comes down to this:

cd /usr/local/src/perl-5.6.1/
export LC_ALL=C
export LANG=en_US
perl -i.bak -p -e 's|Local/Library|Library|g' hints/darwin.sh
sh Configure -des -Dfirstmakefile=GNUmakefile -Dldflags="-flat_namespace"
make
make test
make install

There were a few errors during make test, but none of them seems to be significant. Hopefully, in the next version of Perl, the build will work just as it does on other platforms.

Downloads

Before installing Open SSL, mod_ssl, mod_perl, and Apache, I needed to get all the right pieces in place. The mod_ssl and mod_perl configure processes patch the Apache sources, so the Apache sources have to be downloaded and gunzipped and untarred into an adjacent directory. Furthermore, the mod_ssl version number corresponds to the Apache version number, so you have to be sure that they match up. Normally, I would just download the latest versions of all of these pieces and run with it.

However, Bricolage requires the libapreq library and its supporting Perl modules to run, and these libraries have not yet been successfully ported to Mac OS X. But worry not; fearless mod_perl hackers are working on the problem even as we speak, and there is an interim solution to get everything working.

As of this writing, the latest version of Apache is 1.3.24. But because I needed libapreq, I had to use an experimental version of Apache modified to statically compile in libapreq. Currently, only version 1.3.23 has been patched for libapreq, so that’s what I had to use. I discovered this experimental path thanks to a discussion on the Mac OS X Perl mail list.

So essentially what I did was download the experimental apache.tar.gz and the experimental lightweight apreq.tar.gz packages and gunzip and untar them into /usr/local/src. Then I was ready to move on to Open SSL, mod_ssl, and mod_perl.

Open SSL

Compiling Open SSL was pretty painless. One of the tests fails, but it all seems to work out, anyway. I download the sources from the Open SSL site, and did this:

cd /usr/local/src/openssl-0.9.6c
./config
make
make test

mod_ssl

The mod_ssl Apache module poses no problems whatsoever. I simply downloaded mod_ssl-2.8.7-1.3.23 from the mod_ssl site (note that the “1.3.23” at the end matches the version of Apache I downloaded) and gunzipped and untarred it into /usr/local/src/. Then I simply excuted:

./configure --with-apache=/usr/local/src/apache_1.3.23

mod_perl

Configuring and installing mod_ssl was, fortunately, a relatively straight-forward process. Getting Apache compiled with mod_perl and mod_ssl, however, was quite tricky, as you’ll see below. A number of braver folks than I have preceded me in installing mod_perl, so I was able to rely on their hard-earned knowledge to get the job done. For example, Randal Schwartz posted instructions to the mod_perl mail list, and his instructions worked well for me. So I downloaded the sources from the mod_perl site, and did this:

cd /usr/local/src/mod_perl-1.26
perl Makefile.PL \
  APACHE_SRC=/usr/local/src/apache_1.3.23/src \
  NO_HTTPD=1 \
  USE_APACI=1 \
  PREP_HTTPD=1 \
  EVERYTHING=1
make
make install

Apache

Getting Apache compiled just right was the most time-consuming part of this process for me. Although many had gone before me in this task, everybody seems to do it differently. I had become accustomed to just allowing Apache to use most of its defaults when I compiled under Linux, but now I was getting all kinds of errors while following different instructions from different authorities from around the web. Sometimes Apache wouldn’t compile at all, and I’d get strange errors. Other times it would compile, pass all of its tests, and install, only to offer up errors such as

dyld: /usr/local/apache/bin/httpd Undefined symbols: _log_config_module

when I tried to start it. It turns out that the problem there was that I had a number of modules compiled as DSOs – that is, libraries that can be loaded into Apache dynamically – but wasn’t loading them properly in my httpd.conf. This was mainly because I’ve grown accustomed to Apache having all the libraries I needed compiled in statically, so I simply didn’t have to worry about them.

But I finally hit on the right incantation to get Apache to compile with everything I need added statically, but still with support for DSOs by compiling in mod_so. I present it here for your viewing pleasure:

SSL_BASE=/usr/local/src/openssl-0.9.6c/ \
    ./configure \
    --with-layout=Apache \
    --enable-module=ssl \
    --enable-module=rewrite \
    --enable-module=so \
    --activate-module=src/modules/perl/libperl.a \
    --disable-shared=perl \
    --without-execstrip
  make
  make certificate TYPE=custom 
  make install

This series of commands successfully compiled Apache with mod_perl and mod_ssl support statically compiled in, along with most of the other default modules that come with Apache. In short, everything is there that you need to run a major application with security such as Bricolage.

Note that make certificate will lead you through the process of creating an SSL certificate. I like to use the “custom” type so that it reflects the name of my organization. But you can use whatever approach you’re most comfortable with. Consult the mod_ssl INSTALL file for more information.

libapreq

Once Apache is installed with mod_perl and mod_ssl, the rest is gravy! The experimental libapreq library I downloaded installed without a hitch:

cd /usr/local/src/httpd-apreq
perl Makefile.PL
make
make install

PostgreSQL

PostgreSQL is a sophisticated open-source Object-Relational DBMS. I use it a lot in my application development, and it, too, is required by Bricolage. I was a bit concerned about how well it would compile and work on Mac OS X, but I needn’t have worried. First of all, Apple has provided some pretty decent instructions. Although they mainly document how to install MySQL, a competing open-source RDBMS, many of the same concepts apply to PostgreSQL.

The first thing I had to do was to create the “postgres” user. This is the system user that PostgreSQL typically runs as. I followed Apple’s instructions, using NetInfo Manager to duplicate the default “www” group and “www” user and give the copies the name “postgres” and a new gid and uid, respectively.

Next I downloaded the PostgreSQL version 7.2.1 sources. Version 7.2 is the first to specifically support Mac OS X, so going about the install was as simple as it is on any Unix system:

./configure --enable-multibyte=UNICODE
make
make install

That was it! PostgreSQL was now installed. Next I had to initialize the PostgreSQL database directory. Again, this works much the same as it does on any Unix system:

sudo -u postgres /usr/local/pgsql/bin/initdb \
  -D /usr/local/pgsql/data

The final step was to start PostgreSQL and try to connect to it:

sudo -u postgres /usr/local/pgsql/bin/pg_ctl start \
  -D /usr/local/pgsql/data /usr/local/pgsql/bin/psql -U postgres template1

If you follow the above steps and find yourself at a psql prompt, you’re in business! Because I tend to use PostgreSQL over TCP, I also enabled TCP connectivity by enabling the “tcpip_socket” option in the postgresql.conf file in the data directory created by initdb:

tcpip_socket = true

If you’re like me, you like to have servers such as PostgreSQL start when your computer starts. I enabled this by creating a Mac OS X PostgreSQL startup bundle. It may or may not be included in a future version of PostgreSQL, but in the meantime, you can download it from here. Simply download it, gunzip and untar it into /Library/StartupItems, restart OS X, and you’ll see it start up during the normal Mac OS X startup sequence. I built this startup bundle by borrowing from the existing FreeBSD PostgreSQL startup script, the Apache startup script that ships with OS X, and by reading the Creating SystemStarter Startup Item Bundles HOWTO.

XML::Parser

At this point, I had most

of the major pieces in place, and it was time for me to install the Perl modules I needed. First up was XML::Parser. For some reason, XML::Parser can’t find the expat libraries, even though the location in which I installed them is pretty common. I got around this by installing XML::Parser like this:

perl Makefile.PL EXPATLIBPATH=/usr/local/lib \
  EXPATINCPATH=/usr/local/include
make
make test
make install

Text::Iconv

In Bricolage, Text::Iconv does all the work of converting text between character sets. This is because all of the data is stored in the database in Unicode, but we wanted to allow users to use the character set with which they’re accustomed in the UI. So I needed to install Text::Iconv. Naturally, Mac OS X doesn’t come with libiconv – a library on which Text::Iconv depends – so I had to install it. Fortunately, it was a simple process to download it and do a normal build:

cd /usr/local/src/libiconv-1.7
./configure
make
make install

Now, Text::Iconv itself was a little more problematic. You have to tell it to look for libiconv by adding the -liconv option to the LIBS key in Makefile.PL. I’ve simplified doing this with the usual Perl magic:

perl -i.bak -p -e \
  "s/'LIBS'\s*=>\s*\[''\]/'LIBS' => \['-liconv'\]/" \
  Makefile.PL
perl Makefile.PL
make
make test
make install

DBD::Pg

Although the DBI installed via the CPAN module without problem, DBD::Pg wanted to play a little less nice. Of course I specified the proper environment variables to install it (anyone know why DBD::Pg’s Makefile.PL script can’t try to figure those out on its own?), but still I got this error during make:

/usr/bin/ld: table of contents for archive:
/usr/local/pgsql/lib/libpq.a is out of date;
rerun  ranlib(1) (can't load from it)

But this was one of those unusual situations in which the error message was helpful. So I took the error message’s advice, and successfully compiled and installed DBD::Pg like this:

ranlib /usr/local/pgsql/lib/libpq.a
export POSTGRES_INCLUDE=/usr/local/pgsql/include
export POSTGRES_LIB=/usr/local/pgsql/lib
perl Makefile.PL
make
make test
make install

LWP

The last piece I needed to worry about customizing when I installed it was LWP. Before installing, back up /usr/bin/head. The reason for this is that LWP will install /usr/bin/HEAD, and because HFS Plus is a case-insensitive file system, it’ll overwrite /usr/bin/head! This is a pretty significant issue, since many configure scripts use /usr/bin/head. So after installing LWP, move /usr/bin/HEAD, GET, & POST to /usr/local/bin. Also move /usr/bin/lwp* to /usr/local/bin. Then move your backed-up copy of head back to /usr/bin.

Naturally, I didn’t realize that this was necessary until it was too late. I installed LWP with the CPAN module, and it wiped out /usr/bin/head. Fortunately, all was not lost (though it took me a while to figure out why my Apache compiles were failing!): I was able to restore head by copying it from the Mac OS X installer CD. I Just popped it in an executed the command:

cp "/Volumes/Mac OS X Install CD/usr/bin/head" /usr/bin

And then everything was happy again.

Bricolage

And finally, the pièce de résistance: Bricolage! All of the other required Perl modules installed fine from Bundle::Bricolage:

perl -MCPAN -e 'install Bundle::Bricolage'

Then I simply followed the directions in Bricolage’s INSTALL file, and started ‘er up! I would document those steps here, but the install process is currently in flux and likely to change soon. The INSTALL file should always be current, however – check it out!

To Be Continued

No doubt my adventures with Unix tools on Mac OS X are far from over. I’ve reported to various authors on the issues I’ve described above, and most will soon be releasing new versions to address those issues. As they do, I’ll endeavor to keep this page up-to-date. In the meantime, I am thoroughly enjoying working with the first really solid OS that Apple has released in years, and thrilled that I can finally have the best of both worlds: a good, reliable, and elegant UI, and all the Unix power tools I can stand! I hope you do, too.

Looking for the comments? Try the old layout.