Just a Theory

Black lives matter

Posts about postgres

No UTF-8 Support on Windows?

This just blows. It will be a while before Bricolage runs on Windows, then. The PostgreSQL team is understandably reluctant to simply include the whole ICU library in PostgreSQL. Maybe it could be compiled into the binaries, though?

Looking for the comments? Try the old layout.

Bricolage Now has PHP 5 Templating

Well, I’ve finally gone and done it. I’ve released Bricolage 1.9.0, the first development release towards 1.10.0, which I plan to get out sometime next month. Among the new features in this release that I’m most excited about are a revamped UI, LDAP authentication, and PHP 5 templating support.

The new UI is really nice. Marshall Roch ported it all to XHTML 1.0 strict plus CSS. All the layout is done with CSS now, instead of the old 1999-era table layouts we had before. The result is much smaller page loads, sometimes up to 70% smaller, Marshall tells me, and therefore much faster loads. Playing around with the new version (now powering the Kineticode and Bricolage Web sites, I do indeed find it to be a lot zippier. Couple the new UI with the new mod_gzip support and static Mason components, and things just perk right along! Oh, and the expandable interfaces for both lists of objects and for editing interfaces is very nice, indeed. Thank you Marshall for the great work!

Kineticode added the LDAP authentication support. We’ve been using a patch for it against 1.8.x for our internal version of Bricolage and it works great. Now we authenticate off of an OpenLDAP server, using the same usernames and passwords we use for our email, Subversion, and RT servers. The configuration is simple, via bricolage.conf directives, and you can even limit users who authenticate to members of a particular LDAP group. Users must exist in Bricolage, and if LDAP authentication fails, you can fall back on Bricolage’s internal authentication. I’m hoping that the LDAP support goes a long way towards attracting more enterprise customers with single sign-on requirements.

And finally—yes, you heard right—Bricolage now supports PHP 5 templating in addition to the existing Perl-based templating architectures (Mason, Template Toolkit, and HTML::Template). So how did we add PHP 5 templating to a mod_perl application? Easy: we hired George Schlossnagle of Omni TI to write PHP::Interpreter, an embedded PHP 5 interpreter. Now anyone can natively execute PHP 5 code from a Perl application. Not only that, but the PHP 5 code can reach back into the Perl interpreter to use Perl modules and objects! Here’s an example that I like to show off to the PHP crowd:

    $perl = Perl::getInstance();
    $perl->eval("use DBI");
    $perl->eval("use DateTime");
    $dbh = $perl->call("DBI::connect", "DBI", "dbi:SQLite:dbname=dbfile");
    $dbh->do("CREATE TABLE foo (bar TEXT, time DATETIME)");
    $now = $perl->call("DateTime::now", "DateTime");
    $ins = $dbh->prepare("INSERT INTO foo VALUES (?, ?)");
    $ins->execute("This is a test", $now);
    $sel = $dbh->prepare("SELECT bar, time FROM foo");
    $a = array("foo", "bar");
    foreach ($sel->fetch() as $val) {
        echo "$val\n";
    $dbh->do("DROP TABLE foo");

Note that George plans to add convenience methods to load Perl modules and call Perl class methods. Now, to execute this code from Perl, all you have to do is write a little script, call it pphp, like so:

use strict;
use PHP::Interpreter;
my $php = PHP::Interpreter->new;

Then just execute your PHP code with the script: pphp try.php. Yes, this does work! For years, when I’ve run across a PHP coder who wanted to try to tell me that PHP was better than Perl, I always had a one-word reply that left him cursing and conceding defeat: “CPAN.” Well no more. Now PHP hackers can use any module on CPAN, too!

And as for Bricolage, the integration of PHP 5 templating is completely transparent. Users just write PHP 5 templates instead of Mason templates and that’s it! For example, this is a fairly common style Bricolage Mason template:

for my $e ($element->get_elements(qw(header para _pull_quote_))) {
    my $kn = $e->get_key_name;
    if ($kn eq "para") {
        $m->print("<p>", $e->get_data, "</p>\n");
    } elsif ($kn eq "header") {
        # Test sdisplay_element() on a field.
        $m->print("<h3>", $burner->sdisplay_element($e), "</h3>\n");
    } elsif ($kn eq "_pull_quote_" && $e->get_object_order > 1) {
        # Test sdisplay_element() on a container.
    } else {
        # Test display_element().

The same template in PHP 5 looks like this:

    # Convenience variables.
    $story   = $BRIC["story"];
    $element = $BRIC["element"];
    $burner  = $BRIC["burner"];
    foreach ($element->get_elements("header", "para", "_pull_quote_") as $e) {
        $kn = $e->get_key_name();
        if ($kn == "para") {
            echo "<p>", $e->get_data(), "</p>\n";
        } else if ($kn == "header") {
            # Test sdisplay_element() on a field.
            echo "<h3>", $burner->sdisplay_element($e), "</h3>\n";
        } else if ($kn == "_pull_quote_" && $e->get_object_order() > 1) {
            # Test sdisplay_element() on a container.
            echo $burner->sdisplay_element($e);
        } else {
            # Test display_element().

Yes, you are seeing virtually the same thing. But this is just a simple template from Bricolage’s test suite. The advantage is that PHP 5 coders who are familiar with all the ins and outs of PHP 5 can just jump in an get started writing Bricolage templates without having to learn any Perl! The Bricolage objects have exactly the same API as they do in Perl, because they are exactly the same objects! So everyone uses the same API documentation for the same tasks. The only issue I’ve noticed so far is that PHP 5 does not yet have proper Unicode support. Since all content in Bricolage is stored as UTF-8, this means that the PHP 5 templates must treat it as binary data. But this is okay as long as templaters use the mb_* PHP 5 functions to parse text.

Overall I’m very excited about this, and hope that it helps Bricolage to reach a whole new community of users. I’d like to thank Portugal Telecom—SAPO.pt for sponsoring the development of PHP::Interpreter and its integration into Bricolage. I believe that they’ve really done the Bricolage community a great service, and I hope that the Perl and PHP communities likewise benefit from the integration possible with PHP::Interpreter.

And just so that the other templating architectures don’t feel left out, here is how the above template looks in Template Toolkit:

[% FOREACH e = element.get_elements("header", "para", "_pull_quote_") %]
    [% kn = e.get_key_name %]
    [% IF kn == "para" %]
<p>[% e.get_data %]</p>
    [% ELSIF kn == "header" %]
        [% # display_element() should just return a value. %]
<h3>[% burner.display_element(e) %]</h3>
    [% ELSIF kn == "_pull_quote_" && e.get_object_order > 1 %]
        [% PERL %]
          # There is no sdisplay_element() in the TT burner, but we"ll just
          # Play with it, anyway.
          print $stash->get("burner")->display_element($stash->get("e"));
        [% END %]
    [% ELSE %]
        [% # Test display_element(). %]
        [% burner.display_element(e) %]
    [% END %]
[% END %]
[% burner.display_pages("_page_") %]

And here it is in HTML::Template:

<tmpl_if _page__loop>
<tmpl_loop _page__loop>
<tmpl_include name="testing/sub/util.tmpl">
<tmpl_var _page_>
<tmpl_var page_break>
<tmpl_include name="testing/sub/util.tmpl">

I had to use the utility template with HTML::Template to get it to work right, don’t ask why. It looks like this:

<tmpl_loop element_loop>
<tmpl_if is_para>
<p><tmpl_var para></p>
<tmpl_if is_header>
<h3><tmpl_var header></h3>
<tmpl_if is__pull_quote_>
<tmpl_var _pull_quote_>

These templates come from the Bricolage test suite, where until this release, there were never any template tests before. So you can see that the PHP 5 templating initiative has had major benefits for the stability of Bricolage, too. Now that I’ve really worked with all four templating architectures in Bricolage, I can now say that my preference for which to do goes in this order:

  1. Mason, because of its killer autohandler and inheritance architecture
  2. PHP 5 or Template Toolkit are tied for second place
  3. HTML::Template

In truth, all four are capable and have access to the entire Bricolage API so that they can output anything. So what are you waiting for? Download Bricolage and give it a try!

Looking for the comments? Try the old layout.

Bricolage 1.8.5 Released

The Bricolage development team is pleased to announce the release of Bricolage 1.8.5. This maintenance release addresses a number of issues in Bricolage 1.8.3 and adds a number of improvements (there was no announcement for the short-lived 1.8.4 release). The SOAP server in particular sees improvements in this release, with improved character set support; better support for related stories and media using URIs in addition to IDs; and as support for top-level element relations. Issues with the ordering of story elements have also been corrected, as well as errors when attempting to revert a story or media document or template. Here are the other highlights of this release:


  • Added Linux startup script contrib/start_scripts/linux. [David]
  • Related story and media elements managed through the SOAP server can now use a combination of URI and site ID to identify related assets in addition to the existing approach of using story and media IDs. [David]
  • A list of subelements is now less likely to mysteriously become out of order and thus lead to strange action-at-a-distance errors. And even if they do become out of order, the error message will be more appropriate (“Warning! State inconsistent” instead of “Can’t call method “get_name” on an undefined value”). Reported by Curtis Poe. [David]
  • The SOAP media interface now supports creating relationships between the media documents elements and other story and media documents, just like the SOAP story interface does. [David]
  • The SOAP interface now supports Related stories and media on story type and media type elements just as in the UI. This involved the somewhat hackish necessity for including the “related_story_id” and “related_media_id” (or “related_story_uri” and “related_media_uri”) attributes in the “elements” XML element, but it does the trick. [David]

Bug Fixes

  • Calls to publish documents via SOAP will no longer fail if the published_version attribute is not specified and the document to be published has never been published before. [David]
  • The Bricolage virtual FTP server will no longer fail to start if Template Toolkit is installed but its version number is less than 2.14. Reported by Adam Rinehart. [David]
  • Stories and Media created or updated via the SOAP interface will now associate contributors of the appropriate type, instead of “All Contributors”. [Scott & David]
  • Deleting an element that has a template no longer causes an error. Thanks to Susan for the spot! [David]
  • Eliminated encoding errors when using the SOAP interface to output stories, media, or templates with wide characters. Reported by Scott Lanning. [David]
  • Reverting (stories, media, templates) no longer gives an error. Reported by Simon Wilcox, Rachel Murray, and others. [David]
  • Publishing a published version of a document that has a later version in workflow will no longer cause that later version to be mysteriously removed from workflow. This could be caused by passing a document looked up using the published_version to list() to $burner->publish_another in a template. [David]
  • The SOAP server story and media interfaces now support elements that contain both related stories and media, rather than one or the other. [David]
  • Attempting to preview a story or media document currently checked out to another user no longer causes an error. Reported by Paul Orrock. [David]
  • Custom fields with default values now have their values included when they are added to stories and media. Thanks to Clare Parkinson for the spot! [David]
  • The bric_queued script now requires a username and password and will authenticate the user. This user will then be used for logging events. All events logged when a job is run via the UI are now also logged by bric_queued. [Mark and David]
  • Preview redirections now use the protocol setting of the preview output channel if it’s available, and falls back on using “http://” when it’s not, instead of using the hard-coded “http://”. Thanks to Martin Bacovsky for the spot! [David]
  • The has_keyword() method in the Business class (from which the story and media classes inherit) now works. Thanks to Clare Parkinson for the spot! [David]
  • Clicking a link in the left-side navigation after the session has expired now causes the whole window to show the login form, rather than it showing inside the nav frame, which was useless. [Marshall]
  • The JavaScript that validates form contents once again works with htmlArea, provided htmlArea itself is patched. See the relevant htmlArea bug report for the patch. As of this writing, you must run the version of htmlArea in CVS. [David & Marshall]
  • The JavaScript that handles the double list manager has been vastly optimized. It should now be able to better handle large lists, such as a list of thousands of categories. Reported by Scott. [Marshall]
  • Uploading a new image to a media document with a different media type than the previous image no longer causes an Imager error. [David]

For a complete list of the changes, see the changes. For the complete history of ongoing changes in Bricolage, see Bric::Changes.

Download Bricolage 1.8.5 now from the Bricolage Web site Downloads page, from the SourceForge download page, and from the Kineticode download page.

About Bricolage

Bricolage is a full-featured, enterprise-class content management and publishing system. It offers a browser-based interface for ease-of use, a full-fledged templating system with complete HTML::Mason, HTML::Template, and Template Toolkit support for flexibility, and many other features. It operates in an Apache/mod_perl environment and uses the PostgreSQL RDBMS for its repository. A comprehensive, actively-developed open source CMS, Bricolage was hailed as “quite possibly the most capable enterprise-class open-source application available” by eWEEK.

Looking for the comments? Try the old layout.

MySQL’s REPLACE Considered Harmful

So we’ve set up a client with an online poll application using MySQL. Polls are created in Bricolage, and when they’re published, rather than writing data to files, the template writes data to the MySQL database. PHP code on the front-end server then uses the database records to manage the polls.

On the recommendation of one of my colleagues, I was using the MySQL REPLACE statement to insert and update poll answers in the database. At first, this seemed like a cool idea. All I had to do was create a unique index on the story_id and ord (for answer order) columns and I was set. Any time someone reordered the answers or changed their wording in Bricolage, the REPLACE statement would change the appropriate records and just do the right thing.

Or so I thought.

Come the day after the launch of the new site, I get a complaint from the customer that the percentage spread between the answers doesn’t add up to 100%. After some investigation, I realized that the poll_results table is using the ID of each question to identify the votes submitted by readers. This makes sense, of course, and is excellent relational practice, but I have overlooked the fact that REPLACE essentially replaces rows every time it is used. This means that even when a poll answer hasn’t changed, it gets a new ID. Yes, that’s right, its primary key value was changing. Yow!

Now we might have caught this earlier, but the database was developed on MySQL 3.23.58 and, as is conventional among MySQL developers, there were no foreign key constraints. So the poll results were still happily pointing to non-existent records. So a poll might appear to have 800 votes, but the percentages might be counted for only 50 votes. Hence the problem with the percentages not adding up to 100% (nowhere near it, in fact).

Fortunately, the production application is on a MySQL 4.1 server, so I made a number of changes to correct this issue:

  • Added foreign key constraints
  • Exploited a little-known (mis)feature of Bricolage to store primary keys for all poll answers (and questions, for that matter)
  • Switched from REPLACE to INSERT, UPDATE, and DELETE statements using the primary keys

I also started using transactions when making all these updates when a poll is published so that changes are always atomic. Now it works beautifully.

But the lesson learned is that REPLACE is a harmful construct. Yes, it was my responsibility to recognize that it would create new rows and therefore new primary keys. But any construct that changes primary keys should be stricken from any database developer’s toolbox. The fact that MySQL convention omits the use of foreign key constraints makes this a particularly serious issue that can appear to have mysterious consequences.

So my advice to you, gentle reader, is don’t use it.

Looking for the comments? Try the old layout.

Bricolage 1.8.3 Released

The Bricolage development team is pleased to announce the release of Bricolage 1.8.3. This maintenance release addresses a number of issues in Bricolage 1.8.2. The most important changes eliminate or greatly reduce the number of deadlocks caused during bulk publishes of many documents. Other changes include new contributed scripts for importing contributors and for generating thumbnail images, Russian localization, and various fixes for database transaction, template formatting, and various user interface fixes. Here are the other highlights of this release:


  • Added contrib/thumbnails/precreate-thumbs.pl script to pre-create thumbnails from images. Useful for upgraders. [Scott]
  • Added contrib/bric_import_contribs to import contributors from a tab-delimited file. Development by Kineticode, sponsored by the RAND Corporation. [David]
  • Added the published_version parameter to the list() methods of the story, media, and template classes. This parameter forces the search to return the versions of the assets as they were last published, rather than the most recent version. This will be most useful to those looking up other documents in templates and publishing them, as a way of avoiding pulling documents out from other anyone who might have them checked out! [David]
  • All publishing and distribution jobs are now executed in their own transactions when they are triggered by the user interface. This is to reduce the chances of a deadlock between long-running publishing transactions. [David]
  • Optimized SQL queries for key names or that order by string values to use indexes in the list() and list_ids() methods of the story, media, and template classes. [David]
  • Added Russian localization. [Sergey Samoilenko].
  • Changed the foreign keys in the story, media, and formatting (template) tables so that DELETEs do not cascade, but are restricted. This means that before deleting any source, element, site, workflow, or other related object that has a foreign key reference in an asset table, those rows must be deleted. Otherwise, PostgreSQL will throw an exception. Hopefully, this will put a stop to the mysterious but very rare disappearance of stories from Bricolage. [David]
  • A call to $burner->burn_another in a template that passes in a date/time string in the future now causes a publish job to be scheduled for that time, rather than immediate burning the document and then scheduling the distribution to take place in the future. Reported by Ashlee Caul. [David]
  • Changing the sort order of a list of items in a search interface now properly reverses the entire collection of object over the pages, rather than just the objects for the current page. Thanks to Marshall for the spot! [David]

Bug Fixes

  • Publishing stories not in workflow via the SOAP server works again. [David]
  • The Burner object’s encoding attribute is now setable as well as readable. [David]
  • The category browser works again. [David]
  • Fixed Media Upload bug where the full local path was being used, by adding a “winxp” key to Bric::Util::Trans::FS to account for an update to HTTP::BrowserDetect. [Mark Kennedy]
  • Instances of a required custom field in story elements is no longer required once it has been deleted from the element definition in the element manager. Reported by Rod Taylor. [David]
  • A false value passed to the checked_out parameter of the list() and list_ids() methods of the story, media, and template (formatting) classes now properly returns only objects or IDs for assets that are not checked out. [David]
  • The cover date select widget now works properly in the clone interface when a non-ISO style date preference is selected. Thanks to Susan G. for the spot! [David]
  • Sorting templates based on Asset Type (Element) no longer causes an error. [David]
  • Fixed a number of the callbacks in the story, media, and template profiles so that they didn’t clear out the session before other callbacks were done with it. Most often seen as the error “Can’t call method ‘get_tiles’ on an undefined value” in the media profile, especially with IE/Windows (for some unknown reason). Reported by Ed Stevenson. [David]
  • Fixed typo in clone page that caused all output channels to be listed rather than only those associated with the element itself. [Scott]
  • Fixed double listing of the “All” group in the group membership double list manager. [Christian Hauser]
  • Image buttons now correctly execute the onsubmit() method for forms that define an onsubmit attribute. This means that, among other things, changes to a group profile will persist when you click the “Permissions” button. [David]
  • Simple search now works when it is selected when the “Default Search” preference is set to “Advanced”. Reported by Marshall Roch. [David]
  • Multiple alert types set up to trigger alerts for the same event will now all properly execute. Thanks to Christian Hauser for the spot! [David]
  • Publishing stories or media via SOAP with the published_only parameter (--published-only for bric_republish) now correctly republishes the published versions of documents even if the current version is in workflow. Reported by Adam Rinehart. [David]
  • Users granted a permission greater than READ to the members of the “All Users” group no longer get such permission to any members of the “Global Admins” group unless they have specifically been granted such permission to the members of the “Global Admins” group. Thanks to Marshall Roch for the spot! [David]

For a complete list of the changes, see the changes. For the complete history of ongoing changes in Bricolage, see Bric::Changes.

Download Bricolage 1.8.3 now from the Bricolage Website Downloads page, from the SourceForge download page, and from the Kineticode download page.

About Bricolage

Bricolage is a full-featured, enterprise-class content management and publishing system. It offers a browser-based interface for ease-of use, a full-fledged templating system with complete HTML::Mason, HTML::Template, and Template Toolkit support for flexibility, and many other features. It operates in an Apache/mod_perl environment and uses the PostgreSQL RDBMS for its repository. A comprehensive, actively-developed open source CMS, Bricolage was hailed as “quite possibly the most capable enterprise-class open-source application available” by eWEEK.


–The Bricolage Team

Looking for the comments? Try the old layout.

How to Determine Your Transaction ID

Today I had reason to find out what PostgreSQL transaction I was in the middle of at any given moment in Bricolage. Why? I wanted to make sure that a single request was generating multiple transactions, instead of the normal one. It’s a long story, but suffice it to say that lengthy transactions were causing deadlocks. Read this if you’re really interested.

Anyway, here’s how to determine your current transaction using DBI. The query will be the same for any client, of course.

my $sth = $dbh->prepare(qq{
    SELECT transaction
    FROM   pg_locks
    WHERE  pid = pg_backend_pid()
            AND transaction IS NOT NULL
    LIMIT  1

$sth->bind_columns(\my $txid);
print "Transaction: $txid\n";

Looking for the comments? Try the old layout.

Compiling libreadline on Mac OS X

I just realized that I never posted my recipe for configuring and installing libreadline on Mac OS X. I need it for use with PostgreSQL, and don’t fully understand why Apple has not yet included it with Mac OS X. Maybe it’ll be in Tiger?

In the meantime, it turns out to be pretty easy to configure it and build it yourself, assuming you have the developer tools (“Xcode”) installed. The only thing that’s different from any other Unix is that the support/shobj-conf must be modified to be able to find other libraries installed on Mac OS X. Here’s a shell script I whipped up that can do the whole thing for you, soup-to-nuts.

export VERSION=4.3
curl -O ftp://ftp.gnu.org/pub/gnu/readline/readline-$VERSION.tar.gz
tar zxvf readline-$VERSION.tar.gz
cd readline-$VERSION
perl -i.bak -p -e \
  "s/SHLIB_LIBS=.*/SHLIB_LIBS='-lSystem -lncurses -lcc_dynamic'/g" \
sudo make install

Hope that this helps others!

Looking for the comments? Try the old layout.

Always use the C Locale with PostgreSQL

I ran into the weirdest bug with Bricolage today. We use the LIKE operator to do string comparisons throughout Bricolage. In one usage, the code checks to see if there’s a record in the “keyword” table before creating it. This is because keyword names are unique. So it looks for a keyword record like this:

SELECT name, screen_name, sort_name, active
  FROM   keyword

If it finds a keyword, it creates a relationship between it and a story document. If it doesn’t find it, it creates a new keyword record and then associates the new keyword with a story document.

However, one of our customers was getting SQL errors when attempting to add keywords to a story, and it took me a while to figure out what the problem was. This is because I couldn’t replicate the problem until I started trying to create multibyte keywords. Now, Bricolage uses a UTF-8 PostgreSQL database, but something very odd was going on. When I attempted to add the keyword “북한의”, it didn’t find an existing keyword, but then threw an error when the unique index thought it existed already! Running tests in psql, I found that = would find the existing record, but LIKE wouldn’t!

Once I posted a query on the pgsql-general list, someone noticed that the record returned when using = actually had a different value than was actually queried for. I had searched for “북한의”, but the database found “국방비”. It seems that = compares bytes, while LIKE compares characters. The error I was getting meant that the unique index was also using bytes. And because of the locale used when initdb was run, PostgreSQL thought that they actually were the same!

The solution to this problem, it turns out, was to dump the database, shut down PostgreSQL, move the old data directory, and create a new one with initdb -locale=C. I then restored the database, and suddenly = and LIKE (and the unique index) were doing the same thing. Hallelujah!

Naturally, I’m not the first to notice this issue. It’s particularly an issue with RedHat Linux installations, since RedHat has lately decided to set a system-wide locale. In my case, it was “en_US.UTF-8.” This apparently can break collations in other languages, and this affects indices, of course. So I was led to wonder if initdb shouldn’t default to a locale of C instead of the system default. What do you think?

You can read the whole thread here.

Looking for the comments? Try the old layout.

My Adventures with Mac OS X

I recently decided to make the leap from Yellow Dog Linux to Mac OS X on my Titanium PowerBook. Getting everything to work the way I wanted proved to be a challenge, but well worth it. This document outlines all that I learned, so that neither you nor I will have to experience such pain again. The overall goal was to get Bricolage up and running, figuring that if it worked, then just about any mod_perl based solution would run. I’m happy to say that I was ultimately successful. You can be, too.

In the descriptions below, I provide links to download the software you’ll need, as well as the shell commands I used to compile and install each package. In all cases (except for the installation of the Developer Tools), I saved each package’s sources to /usr/local/src and gunzipped and untarred them there. I also carried out each step as root, by running sudo -s. If you’re not comfortable using a Unix shell, you might want to read up on it, first. All of my examples also assume a sh-compatible shell, such as bash or zsh. Fortunately, zsh comes with OS X, so you can just enable it for yourself in NetInfo Manager by setting users -> <username> -> shell to “/bin/zsh”, where <username> is your user name.

Developer Tools

All of the software that I describe installing below must be compiled. To compile software on Mac OS X, you need to install the Mac OS X Developer Tools. These provide the cc compiler and many required libraries. Conveniently, these come on a CD-ROM with the Mac OS X Version 10.1 upgrade kit. I just popped in the CD and installed them like you’d install any other OS X software. I needed administrative access to OS X to install the Developer Tools (or, indeed, to install any of the other software I describe below), but otherwise it posed no problems.

The best time to install the Developer Tools is immediately after upgrading to OS X version 10.1. Then run the Software Update applet in the System preferences to get your system completely up-to-date. By the time I was done, I had the system updated to version 10.1.3.


The first step I took in the process of moving to OS X was to get working the tools I needed most. Essentially, what this meant was GNU Emacs. Now I happen to be a fan of the X version of Emacs – not XEmacs, but GNU Emacs with X support built in. I wasn’t relishing the idea of having to install X on OS X (although there are XFree86 ports that do this), so I was really pleased to discover the Mac-Emacs project. All I had to do was patch the GNU Emacs 21.1 sources and compile them, and I was ready to go! GNU Emacs works beautifully with the OS X Aqua interface.

There were a few configuration issues for me to work out, however. I have become addicted to the green background that an old RedHat .XConfig file had set, and I wanted this feature in OS X, too. Plus, the default font was really ugly (well, too big, really – anyone know how to make it smaller in Emacs?) and the Mac command key was working as the Emacs META key, rather than the option key. So I poked around the net until I found the settings I needed and put them into my .emacs file:

'(default ((t (:stipple nil
  :background "DarkSlateGrey"
  :foreground "Wheat"
  :inverse-video nil
  :box nil
  :strike-through nil
  :overline nil
  :underline nil
  :slant normal
  :weight normal
  :height 116
  :width normal
  :family "apple-andale mono"))))
'(cursor ((t (:background "Wheat"))))
; Use option for the meta key.
(setq mac-command-key-is-meta nil)

Installing Emacs is not required for installing any of the other packages described below – it just happens to be my favorite text editor and IDE. So I don’t provide the instructions here; the Mac-Emacs project does a plenty good job. If you’re not comfortable with Unix editors, you can use whatever editor you like. BBEdit is a good choice.


Mac OS X doesn’t come with a DBM! But since mod_ssl needs it, we have to install it. Fortunately, I found this PDF detailing someone else’s adventures with mod_ssl on OS X, and it provided decent instructions for installing GDBM. First, I created a new user for GDBM. In NetInfoManager, I created a duplicate of the “unknown” user and named it “bin”. Then, I downloaded GDBM from the FSF, and installed it like this:

cd /usr/local/src/gdbm-1.8.0
cp /usr/libexec/config* .
make install
ln -s /usr/local/lib/libgdbm.a /usr/local/lib/libdbm.a

That did the trick. Nothing else was involved, fortunately.


Who doesn’t do something with XML these days? If your answer is, “not me!”, then you’ll need to install the Expat library in order to work with XML::Parser in Perl. Fortunately it’s relatively easy to install, although support for the -static flag appears to be broken in cc on OS X, so it needs to be stripped out. I downloaded it from its project bpage, and then did this:

cd /usr/local/src/expat-1.95.2
perl -i.bak -p -e \
  's/LDFLAGS\s*=\s*-static/LDFLAGS=/' \
perl -i.bak -p -e \
    's/LDFLAGS\s*=\s*-static/LDFLAGS=/' \
make install


Although Mac OS X ships with Perl (Yay!), it’s the older 5.6.0 version. There have been many bug fixes included in 5.6.1, so I wanted to make sure I got the latest stable version before I built anything else around it (mod_perl, modules, etc.).

Being a Unix program, Perl doesn’t expect to run into the problems associated with a case-insensitive file system like that Mac OS X’s HFS Plus. So there are a couple of tweaks to the install process that make it slightly more complicated than you would typically expect. Fortunately, many have preceded us in doing this, and the work-arounds are well-known. Basically, it comes down to this:

cd /usr/local/src/perl-5.6.1/
export LC_ALL=C
export LANG=en_US
perl -i.bak -p -e 's|Local/Library|Library|g' hints/darwin.sh
sh Configure -des -Dfirstmakefile=GNUmakefile -Dldflags="-flat_namespace"
make test
make install

There were a few errors during make test, but none of them seems to be significant. Hopefully, in the next version of Perl, the build will work just as it does on other platforms.


Before installing Open SSL, mod_ssl, mod_perl, and Apache, I needed to get all the right pieces in place. The mod_ssl and mod_perl configure processes patch the Apache sources, so the Apache sources have to be downloaded and gunzipped and untarred into an adjacent directory. Furthermore, the mod_ssl version number corresponds to the Apache version number, so you have to be sure that they match up. Normally, I would just download the latest versions of all of these pieces and run with it.

However, Bricolage requires the libapreq library and its supporting Perl modules to run, and these libraries have not yet been successfully ported to Mac OS X. But worry not; fearless mod_perl hackers are working on the problem even as we speak, and there is an interim solution to get everything working.

As of this writing, the latest version of Apache is 1.3.24. But because I needed libapreq, I had to use an experimental version of Apache modified to statically compile in libapreq. Currently, only version 1.3.23 has been patched for libapreq, so that’s what I had to use. I discovered this experimental path thanks to a discussion on the Mac OS X Perl mail list.

So essentially what I did was download the experimental apache.tar.gz and the experimental lightweight apreq.tar.gz packages and gunzip and untar them into /usr/local/src. Then I was ready to move on to Open SSL, mod_ssl, and mod_perl.

Open SSL

Compiling Open SSL was pretty painless. One of the tests fails, but it all seems to work out, anyway. I download the sources from the Open SSL site, and did this:

cd /usr/local/src/openssl-0.9.6c
make test


The mod_ssl Apache module poses no problems whatsoever. I simply downloaded mod_ssl-2.8.7-1.3.23 from the mod_ssl site (note that the “1.3.23” at the end matches the version of Apache I downloaded) and gunzipped and untarred it into /usr/local/src/. Then I simply excuted:

./configure --with-apache=/usr/local/src/apache_1.3.23


Configuring and installing mod_ssl was, fortunately, a relatively straight-forward process. Getting Apache compiled with mod_perl and mod_ssl, however, was quite tricky, as you’ll see below. A number of braver folks than I have preceded me in installing mod_perl, so I was able to rely on their hard-earned knowledge to get the job done. For example, Randal Schwartz posted instructions to the mod_perl mail list, and his instructions worked well for me. So I downloaded the sources from the mod_perl site, and did this:

cd /usr/local/src/mod_perl-1.26
perl Makefile.PL \
  APACHE_SRC=/usr/local/src/apache_1.3.23/src \
  NO_HTTPD=1 \
make install


Getting Apache compiled just right was the most time-consuming part of this process for me. Although many had gone before me in this task, everybody seems to do it differently. I had become accustomed to just allowing Apache to use most of its defaults when I compiled under Linux, but now I was getting all kinds of errors while following different instructions from different authorities from around the web. Sometimes Apache wouldn’t compile at all, and I’d get strange errors. Other times it would compile, pass all of its tests, and install, only to offer up errors such as

dyld: /usr/local/apache/bin/httpd Undefined symbols: _log_config_module

when I tried to start it. It turns out that the problem there was that I had a number of modules compiled as DSOs – that is, libraries that can be loaded into Apache dynamically – but wasn’t loading them properly in my httpd.conf. This was mainly because I’ve grown accustomed to Apache having all the libraries I needed compiled in statically, so I simply didn’t have to worry about them.

But I finally hit on the right incantation to get Apache to compile with everything I need added statically, but still with support for DSOs by compiling in mod_so. I present it here for your viewing pleasure:

SSL_BASE=/usr/local/src/openssl-0.9.6c/ \
  ./configure \
  --with-layout=Apache \
  --enable-module=ssl \
  --enable-module=rewrite \
  --enable-module=so \
  --activate-module=src/modules/perl/libperl.a \
  --disable-shared=perl \
make certificate TYPE=custom 
make install

This series of commands successfully compiled Apache with mod_perl and mod_ssl support statically compiled in, along with most of the other default modules that come with Apache. In short, everything is there that you need to run a major application with security such as Bricolage.

Note that make certificate will lead you through the process of creating an SSL certificate. I like to use the “custom” type so that it reflects the name of my organization. But you can use whatever approach you’re most comfortable with. Consult the mod_ssl INSTALL file for more information.


Once Apache is installed with mod_perl and mod_ssl, the rest is gravy! The experimental libapreq library I downloaded installed without a hitch:

cd /usr/local/src/httpd-apreq
perl Makefile.PL
make install


PostgreSQL is a sophisticated open-source Object-Relational DBMS. I use it a lot in my application development, and it, too, is required by Bricolage. I was a bit concerned about how well it would compile and work on Mac OS X, but I needn’t have worried. First of all, Apple has provided some pretty decent instructions. Although they mainly document how to install MySQL, a competing open-source RDBMS, many of the same concepts apply to PostgreSQL.

The first thing I had to do was to create the “postgres” user. This is the system user that PostgreSQL typically runs as. I followed Apple’s instructions, using NetInfo Manager to duplicate the default “www” group and “www” user and give the copies the name “postgres” and a new gid and uid, respectively.

Next I downloaded the PostgreSQL version 7.2.1 sources. Version 7.2 is the first to specifically support Mac OS X, so going about the install was as simple as it is on any Unix system:

./configure --enable-multibyte=UNICODE
make install

That was it! PostgreSQL was now installed. Next I had to initialize the PostgreSQL database directory. Again, this works much the same as it does on any Unix system:

sudo -u postgres /usr/local/pgsql/bin/initdb \
  -D /usr/local/pgsql/data

The final step was to start PostgreSQL and try to connect to it:

sudo -u postgres /usr/local/pgsql/bin/pg_ctl start \
  -D /usr/local/pgsql/data /usr/local/pgsql/bin/psql -U postgres template1

If you follow the above steps and find yourself at a psql prompt, you’re in business! Because I tend to use PostgreSQL over TCP, I also enabled TCP connectivity by enabling the “tcpip_socket” option in the postgresql.conf file in the data directory created by initdb:

tcpip_socket = true

If you’re like me, you like to have servers such as PostgreSQL start when your computer starts. I enabled this by creating a Mac OS X PostgreSQL startup bundle. It may or may not be included in a future version of PostgreSQL, but in the meantime, you can download it from here. Simply download it, gunzip and untar it into /Library/StartupItems, restart OS X, and you’ll see it start up during the normal Mac OS X startup sequence. I built this startup bundle by borrowing from the existing FreeBSD PostgreSQL startup script, the Apache startup script that ships with OS X, and by reading the Creating SystemStarter Startup Item Bundles HOWTO.


At this point, I had most

of the major pieces in place, and it was time for me to install the Perl modules I needed. First up was XML::Parser. For some reason, XML::Parser can’t find the expat libraries, even though the location in which I installed them is pretty common. I got around this by installing XML::Parser like this:

perl Makefile.PL EXPATLIBPATH=/usr/local/lib \
make test
make install


In Bricolage, Text::Iconv does all the work of converting text between character sets. This is because all of the data is stored in the database in Unicode, but we wanted to allow users to use the character set with which they’re accustomed in the UI. So I needed to install Text::Iconv. Naturally, Mac OS X doesn’t come with libiconv – a library on which Text::Iconv depends – so I had to install it. Fortunately, it was a simple process to download it and do a normal build:

cd /usr/local/src/libiconv-1.7
make install

Now, Text::Iconv itself was a little more problematic. You have to tell it to look for libiconv by adding the -liconv option to the LIBS key in Makefile.PL. I’ve simplified doing this with the usual Perl magic:

perl -i.bak -p -e \
  "s/'LIBS'\s*=>\s*\[''\]/'LIBS' => \['-liconv'\]/" \
perl Makefile.PL
make test
make install


Although the DBI installed via the CPAN module without problem, DBD::Pg wanted to play a little less nice. Of course I specified the proper environment variables to install it (anyone know why DBD::Pg’s Makefile.PL script can’t try to figure those out on its own?), but still I got this error during make:

/usr/bin/ld: table of contents for archive:
/usr/local/pgsql/lib/libpq.a is out of date;
rerun  ranlib(1) (can't load from it)

But this was one of those unusual situations in which the error message was helpful. So I took the error message’s advice, and successfully compiled and installed DBD::Pg like this:

ranlib /usr/local/pgsql/lib/libpq.a
export POSTGRES_INCLUDE=/usr/local/pgsql/include
export POSTGRES_LIB=/usr/local/pgsql/lib
perl Makefile.PL
make test
make install


The last piece I needed to worry about customizing when I installed it was LWP. Before installing, back up /usr/bin/head. The reason for this is that LWP will install /usr/bin/HEAD, and because HFS Plus is a case-insensitive file system, it’ll overwrite /usr/bin/head! This is a pretty significant issue, since many configure scripts use /usr/bin/head. So after installing LWP, move /usr/bin/HEAD, GET, & POST to /usr/local/bin. Also move /usr/bin/lwp* to /usr/local/bin. Then move your backed-up copy of head back to /usr/bin.

Naturally, I didn’t realize that this was necessary until it was too late. I installed LWP with the CPAN module, and it wiped out /usr/bin/head. Fortunately, all was not lost (though it took me a while to figure out why my Apache compiles were failing!): I was able to restore head by copying it from the Mac OS X installer CD. I Just popped it in an executed the command:

cp "/Volumes/Mac OS X Install CD/usr/bin/head" /usr/bin

And then everything was happy again.


And finally, the pièce de résistance: Bricolage! All of the other required Perl modules installed fine from Bundle::Bricolage:

perl -MCPAN -e 'install Bundle::Bricolage'

Then I simply followed the directions in Bricolage’s INSTALL file, and started ’er up! I would document those steps here, but the install process is currently in flux and likely to change soon. The INSTALL file should always be current, however – check it out!

To Be Continued

No doubt my adventures with Unix tools on Mac OS X are far from over. I’ve reported to various authors on the issues I’ve described above, and most will soon be releasing new versions to address those issues. As they do, I’ll endeavor to keep this page up-to-date. In the meantime, I am thoroughly enjoying working with the first really solid OS that Apple has released in years, and thrilled that I can finally have the best of both worlds: a good, reliable, and elegant UI, and all the Unix power tools I can stand! I hope you do, too.

Looking for the comments? Try the old layout.