Just a Theory

By David E. Wheeler

Posts about Bricolage

iovationeering

Since June, as part of my work for PGX, I’ve been doing on-site full-time consulting for iovation here in Portland. iovation is in the business of deterring online fraud via device identification and reputation. Given the nature of that business, a whole lot of data arrives every day, and I’ve been developing PostgreSQL-based solutions to help get a handle on it. The work has been truly engaging, and a whole hell of a lot of fun. And there are some really great, very smart people at iovation, whom I very much like and respect.

iovation

So much so, in fact, that I decided to accept their offer of a full time position as “Senior Data Architect.” I started on Monday.

I know, crazy, right? They’ve actually been talking me up about it for a long time. In our initial contact close to two years ago, as I sought to land them as a PGX client, they told me they wanted to hire someone, and was I interested. I said “no.” I said “no” through four months of contracting this summer and fall, until one day last month I said to myself, “wait, why don’t I want this job?” I had been on automatic, habitually insisting I wasn’t interested in a W2 position. And with good reason. Aside from 15 months as CTO at values of n, I’ve been an independent consultant since I founded Kineticode in November of 2001. Yeah. Ten Years.

Don’t get me wrong, those ten years have been great! Not only have I been able to support myself doing the things I love—and learned a ton in the process—but I’ve managed to write a lot of great code. Hell, I will be continuing as an associate with PGX, though with greatly reduced responsibilities. And someday I may go indy again. But in the meantime, the challenges, opportunities, and culture at iovation are just too good to pass up. I’m loving the work I’m doing there, and expect to learn a lot over the next few years.

Kineticode

So what, you might ask, does this mean for Kineticode, the company I founded to offer support, consulting, and training services for Bricolage CMS? The truth is that Kineticode has only a few technical support customers left; virtually all of my work for the last two years has been through PGX. So I’ve decided to shut Kineticode down. I’m shifting the Bricolage tech support offerings over to PGX and having Kineticode’s customers move there as their contacts come up for renewal. They can expect the same great service as always. Better even, as there are 10 associates in PGX, and, lately, only me at Kineticode. Since Kineticode itself is losing its Raison d’être, it’s going away.

PGX

I intend to remain involved in the various open-source projects I work on. I still function as the benevolent dictator of Bricolage CMS, though other folks have stepped up their involvement quite a lot in the last few years. And I expect to keep improving [PGXN] and DesignScene as time allows (I’ve actually been putting some effort into both in the last few weeks; watch for PGXN and Lunar/Theory announcements in the coming weeks and months!). And, in fact, I expect that a fair amount of the work I do at iovation will lead to blog posts, conference presentations, and more open-source code.

This is going to be a blast. Interested in a front-row seat? Follow me on Twitter.

Looking for the comments? Try the old layout.

Bricolage 2.0 Drops

Bricolage 2.0 was released today. This is a pretty big deal, and a long time coming. The most important changes, from my point of view, are:

  • Revamped UI. As a 2006 Google Summer of Code project, Marshall Roch added a slew of Ajaxy goodness to the Bricolage UI. It used to be that, to dig down into a document, you needed to click through reloads for every level. Now the entire structure of a document is available on a single screen, and digging down takes place in situ. This means faster, easier document editing.

    There are other niceties too, thanks to Marshall, like as-you-type autocompletion of category URIs and keywords, popups for associating related documents, dynamic field generation for document keywords and user contacts, and animated workflow actions for moving, deleting, and publishing documents.

    These changes mark a truly impressive improvement in usability for the people who use Bricolage every day, and will by far be the most welcome change for our users.

  • Finer content control. Thanks to another 2006 Google Summer of Code project, Christian Muise implemented what we call “element occurrence specification.” Bricolage document structure is controlled by administrators creating document types with hierarchies of elements. Elements may contain fields—the types and value of which may also be specified (text, textarea, select list, etc.)—and other elements.

    In previous versions of Bricolage, if an element was a subelement of a document, one could add any number of that element to a document. Fields were a bit more controlled: you could only say whether one or many instances of a field were allowed in a given element.

    Element occurrence specification allows administrators to have much finer control over document elements by specifying the minimum and maximum number of instances of an element or field may occur. For example, one can say that a document may have only one instance of a field, or must have three, or may have between 3 and 5, or may have at least 3, or may have any number, including none.

    Bret Dawson put it really well in the Bricolage 2.0 Changes:

    Want every book review you publish to contain at least three but no more than 10 ISBN numbers? Want exactly four pull-quotes in every article? You can do that in Bricolage 2.

  • MySQL support. This, too, was a 2006 Google Summer of Code project, by Andrei Arsu. Yes, you can run Bricolage 2.0 on MySQL 5.0 if you want. This was a pretty big project, and I’m surprisingly pleased at how well it works now that all the kinks have been worked out (special thanks to Waldo Jaquith for being brave (foolish?) enough to start a Bricolage project on MySQL and thus to shake out some bugs).

  • Apache 2 support. This was started quite some time ago by Chris Heiland, hacked on later by Scott Lanning, and finally finished by yours truly. I look forward to dumping Apache 1 in the future.

There’s other stuff, too, lots of little things and not-so-little things. Altogether they go a long way toward making Bricolage better.

It’s been quite a day, and I’m glad to have it out the door. Four years is a long time to wait for a major release, and it happened not because of me, but thanks to the work of others who have picked up the gauntlet. Huge thanks especially to:

Many others provided feedback, patches, and bug reports, and I appreciate all the help. I hope to see you all for Bricolage 2.2!

Looking for the comments? Try the old layout.

Git-R-Done

Just a quick followup on the completion of the Bricolage Git migration last week, today I completed writing up a set of GitHub wiki documents explaining to my fellow Bricoleurs how to start hacking. The most important bits are:

  • Working with Git, explaining how to get set up with a forked Bricolage repository
  • Contributing a Bug Fix, an intro to the Git way of doing things (as far as I understand it)
  • Working with Branches, describing how to track a maintenance branch in your fork
  • Merging with Git, to cover the frequent merging from Bricolage maintenance branches into master, and how to get said merges pushed upstream
  • Starting a Project Branch, which you’d need to read if you were taking on a major development task, such as a Summer of Code project
  • Contributing via Email, for those who don’t want a GitHub account (needs fleshing out)
  • Creating a Release, in which the fine art of branching, tagging, and releasing is covered

If you’re familiar with the “Git way,” I would greatly appreciate your feedback on these documents. Corrections and comments would be greatly appreciated.

I also just wanted to say that the process of reconstructing the merge history from CVS and Subversion was quite an eye-opener for me. Not because it was difficult (it was) and required a number of hacks (it did), but because it highlighted just how much better a fit Git is for the way in which we do Open Source software development. Hell, probably closed-source, too, for that matter. I no longer will have to think about what revisions to include in a merge, or create a branch just to “tag” a merge. Hell, I’ll probably be doing merges a hell of a lot more often, just because it’s so easy, the history remains intact, and everything just stays more up-to-date and closely integrated.

But I also really appreciate the project-based emphasis of Git. A Subversion repository, I now realize, is really very much like a versioned file system. That means where things go is completely ad-hoc, or convention-driven at best. And god forbid if you decide to change the convention and move stuff around! It’s just so much more sane to get a project repository, with all of the history, branches, tags, merges, and everything else, all in one package. It’s more portable, it’s a hell of a lot faster (ever tried to check out a Subversion repository with 80 tags?), and just tighter. it encourages modularization, which can only be good. I’ll tell you, I expect to have some frustrations and challenges as I learn more about using Git, but I’m already very much happier with the overall philosophy.

Enough evangelizing. As a last statement on this, I’ve uploaded the Perl scripts I wrote to do this migration, just in case someone else finds them useful:

  • bric_cvs_to_git migrated a CVS backup to Git.
  • bric_to_git migrated Subversion from r5517 to Git.
  • stitch stitched the CVS-migrated Git repository into the Subversion-migrated Git repository for a final product.

It turned out that there were a few files lost in the conversion, which I didn’t notice until after all was said and done, but overall I’m very happy. My thanks again to Ask and the denizens of #git for all the help.

Looking for the comments? Try the old layout.

Migrating Bricolage CVS and SVN to Git

Now that I’ve successfully migrated the old Bricolage SourceForge CVS repository to Git, and also migrated Subversion to Git, it’s time to stitch the two repositories together into one with all history intact. I’m glad to say that figuring out how to do so took substantially less time than the first two steps, thanks in large part to the great help from “doener,” “Ilari,” and “Fissure” on the Freenode #git channel.

Actually, they helped me with a bit more tweaking of my CVS and Subversion conversions. One thing I realized after writing [yesterday’s post]migrated Subversion was that, after running git filter-branch, I had twice as many commits as I should have had. It turns out that git filter-branch rewrites all commits, but keeps the old ones around in case you mess something up. doener also pointed out that I wasn’t having all grafts properly applied, because git filter-branch only applies to the currently checked-out branch. To get all of the branches, he suggested that I read the git-filter-branch documentation, where I’ll find that git filter-branch --tag-name-filter cat -- --all would hit all branches. Actually, such was not clear to me from the documentation, but I took his word for it. Once I did that, to get rid of the dupes, all I had to do was git clone the repository to a new repository. And that was that.

This worked great for my CVS migration, but I realized that I also wanted to clean out metadata from the Subversion migration. Of course, git clone throws out most of the metadata, but git svn also stores some metadata at the end of every commit log message, like this:

git-svn-id: file:///Users/david/svn/bricolage/trunk@8581 e630fb3e-2914-11de-beb6-f300316f8eb1

This had been very handy as I looked through commits in GitX to find parents to set up for grafts, but with that done and everything grafted, I no longer needed it. Ilari helped me to figure out how to properly use git filter-branch to get rid of those. To do it, all I had to do was add a filter for commit messages, like so:

git filter-branch --msg-filter \
'perl -0777 -pe "s/\r?\ngit-svn-id:.+\n//ms"' \
--tag-name-filter cat -- --all

This properly strips out that ugly bit of metadata and finalizes the grafts all at the same time. Very nice.

Now it was time to combine these two repositories for a single unified history. I wasn’t able to find a good tutorial for this on the web, other than one that used a third-party Debian utility and only hooked up the master branch, using a bogus intermediary commit to do it. On the other hand, simply copying the pack files, as mentioned in the Git Wiki–and demonstrated by the scripts linked from there–also appeared to be suboptimal: The new commits were not showing up in GitX! And besides, Ilari said, “just copying packs might not suffice. There can also be loose objects.” Well, we can’t have that, can we?

Ilari suggested git-fetch, the documentation for which says that it will “download objects and refs from another repository.” Perfect! I wanted to copy the objects from my CVS migration to the Subversion migration.

My first attempt failed: some commits showed up, but not others. Ilari pointed out that it wouldn’t copy remote branches unless you asked it to do so, via “refspecs.” Since I’d cloned the repositories to get rid of the duplicate commits created by git filter-branch, all of my lovingly recreated local branches were now remote branches. Actually, this is what I want for the final repository, so I just had to figure out how to copy them. What I came up with was this:

chdir $cvs;
my @branches = map { s/^\s+//; s/\s+$//; $_ } `git branch -r`;

chdir $svn;
system qw(git fetch --tags), $cvs;

for my $branch (@branches) {
    next if $branch eq 'origin/HEAD';
    my $target = $branch =~ m{/master|rev_1_[68]$} ? "$branch-cvs" : $branch;
    system qw(git fetch --tags), $cvs,
        "refs/remotes/$branch:refs/remotes/$target";
}

It took me a while to figure out the proper incantation for referencing and creating remote branches. Once I got the refs/remotes part figured out, I found that the master, rev_1_6, and rev_1_8 branches from CVS were overwriting the Subversion branches with the same names. What I really needed was to have the CVS branches grafted as parents to the Subversion branches. The #git channel again came to my rescue, where Fissure suggested that I rename those branches when importing them, do the grafts, and then drop the renamed branches. Hence the line above that adds “-cvs” to the names of those branches.

Once the branches were imported, I simply looked for the earliest commits to those branches in Subversion and mapped it to the latest commits to the same branches in CVS, then wrote their SHA1 IDs to .git/info/grafts, like so:

open my $fh, '>', ".git/info/grafts" or die "Cannot open grafts: $!\n";
print $fh '77a35487f18d68b96d294facc1f1a41745ad914c '
        => "835ff47ee1e3d1bf228b8d0976fbebe3c7f02ae6\n", # rev_1_6
            '97ef646f5c2a7c6f47c2046c8d289c1dfc30a73d '
        => "2b9f3c5979d062614ef54afd0a01631f746fa3cb\n", # rev_1_8
            'b3b2e7f53d789bea962fe8047e119148e28865c0 '
        => "8414b64a6a434b2117294c0568c1012a17bc863b\n", # master
    ;
close $fh;

With the branches all imported and the grafts created, I simply had to run git filter-branch to make them permanent and drop the temporary CVS branches:

system qw(git filter-branch --tag-name-filter cat -- --all);
unlink '.git/info/grafts';
system qw(git branch -r -D), "origin/$_-cvs" for qw(rev_1_6 rev_1_8 master);

Now I had a complete repository, but with duplicate commits left over by git-filter-branch. To get rid of those, I need to clone the repository. But before I clone, I need the remote branches to be local branches, so that the clone will see them as remotes. For this, I wrote the following function:

sub fix_branches {
    for my $remote (map { s/^\s+//; s/\s+$//; $_ } `git branch -r`) {
        (my $local = $remote) =~ s{origin/}{};
        next if $local eq 'master';
        next if $local eq 'HEAD';
        system qw(git checkout), $remote;
        system qw(git checkout -b), $local;
    }
    system qw(git checkout master);
}

It’s important to skip the master and HEAD branches, as they’ll automatically be created by git clone. So then I call the function and and run git gc to take out trash, and then clone:

fix_branches();

run qw(git gc);
chdir '..';
run qw(git clone), "file://$svn", 'git_final';

It’s important to use the file:/// URL to clone so as to get a real clone; just pointing to the directory instead makes hard links.

Now I that I had the final repository with all history intact, I was ready to push it to GitHub! Well, almost ready. First I needed to make the branches local again, and then see if I could get the repository size down a bit:

chdir 'git_final';
fix_branches();
system qw(git remote rm origin);
system qw(git remote add origin git@github.com:bricoleurs/bricolage.git);
system qw(git gc);
system qw(git repack -a -d -f --depth 50 --window 50);

And that’s it! My new Bricolage Git repository is complete, and I’ve now pushed it up to its new home on GitHub. I pushed it like this:

git push origin --all
git push origin --tags

Damn I’m glad that’s done! I’ll be getting the Subversion repository set to read-only next, and then writing some documentation for my fellow Bricoleurs on how to work with Git. For those of you who already know, fork and enjoy!

Looking for the comments? Try the old layout.

Migrating Bricolage Subversion to Git

Following up on last week’s post on migrating the old Bricolage SourceForge CVS repository to Git, here are my notes on migrating the current Bricolage Subversion repository to Git.

It turns out that migrating from Subversion is much more of a pain than migrating from CVS. Why? Because CVS has real tags, while Subversion does not. So while git-svn tries to identify all of your tags and branches, it’s really relying on your Subversion repository using standard directories for all of your branches and tags. And while we’ve used a standard for branches directory, our tags setup is a bit more complicated.

The problem was that we used tags every time we merged between branches. This meant that we ended up with a lot of tags with names like “merge_rev_1_10_5665” to indicate a merge from the “rev_1_10” branch into trunk at r5665. Plus we had tags for releases. So Marshall took it upon himself to reorganize the tags in the Subversion tree so that all release tags went into the “releases” subdirectory, and merges went into subdirectories named for the branch from which the merge derived. Those subdirectories went into the “merges” subdirectory. We ended up with a directory structure organized like this:

/tags/
  /releases/
    /1.10.1/
    /1.10.2/
    /1.10.3/
  /merges/
    /dev_ajax/
      /trunk-7890
    /rev_1_10/
      /trunk-7043/
      /trunk-7194/
      /trunk-7300/

This was useful for keeping things organized in Subversion, so that we could easily find a tag for a previous merge in order to determine the revisions to specify for a new merge. But because older tags were moved from previous locations, and because newer tags were in subdirectories of the “tags” directory, git-svn did not identify them as tags. Well, that’s not really fair. It did identify earlier tags, before they were moved, but all the other tags were not found. Instead I ended up with tags in Git named tags/releases and tags/merges, which was useless. But even if all of our tags had been identified as tags, none had parent commit IDs, so there was no place to see where they actually came from.

So to rebuild the commit, release, and merge history from Subversion, I first created a local copy of the subversion repository using svnsync. Then I cloned it to Git like so:

SVNREPO=file:///Users/david/svn_bricolage_cc
git svn init $SVNREPO --stdlayout
git config svn.authorsfile /Users/david/bric_authors.txt
git svn fetch --no-follow-parent --revision 5517:HEAD

By starting with r5517, which was the first real commit to Subversion, I avoided the git-svn error I reported last week. In truth, though, I ended up running this clone many, many times. The first few times, I ran it with --no-metadata, as recommended in various HOWTOs. But then I kept getting errors such as:

git svn log
fatal: bad default revision 'refs/remotes/git-svn'
----------------------------------------------------

This was more than a little annoying, and it took me a day or so to realize that this was because I had been using --no-metadata. Once I killed off that option, things worked much better

Furthermore, by starting at r5517 and passing the --no-follow-parent option, git-svn ran much more quickly. Rather than taking 30 hours to get all revisions including stuff that had been moved around (and then failing), it now took around 90 minutes to do the export. Much more manageable, although I also started making backup copies and restoring from them as I experimented with fixing branches and tags. Ultimately, I ended up also passing the --ignore-paths option, to exclude various branches that were never really used or that I had already fetched in their entirety from CVS:

git svn fetch --no-follow-parent --revision 5517:HEAD \
--ignore-paths '(David|Kineticode|Release_|dev_(callback|(media_)?templates)|rev_1_([024]|[68]_temp)|tags/(Dev-|Release_|Start|help|mark|rel_1_([24567]|8_0)|rev_1_([26]|8_merge-2004-05-04)))|tmp'
svn2git --no-clone

The call to svn2git converts remote branches to local tags and branches. Now I had a reasonably clean copy of the repository (aside from the 120 or so commits from when Marshall did the tags reorganization) for me to work with. I opened it up with GitX and started scripting out merges.

To assist in this, I took a hint from Ask Bjørn Hansen, sent in email in response to a Tweet, and tagged every single commit with its corresponding Subversion revision number, like so (in Perl):

for my $c (`git rev-list --all --date-order --timestamp | sort -n | awk '{print \$2}'`) {
    chomp $c;
    my ($svnid) = `git show -s $c | tail -1` =~ /[@](\d+)\s+/;
    system qw(git tag -f), $svnid, $c;
}

The nice thing about this is that it made it easy for me to scan through the commits in GitX and see where things were. It also meant that I could reference these tags when I wrote the code to manage the merges. So what I did was sort the commits in reverse chronological order, and then search for those with the word “merge” in their subjects. When one was clearly for a merge (as opposed to simply using the word “merge”), I would disable the search, scroll through the commits until I found the selected commit, and then look for a likely prior commit that it merged from.

This was a bit of pain in the ass, because, unfortunately, GitX doesn’t keep the selected commit record in the middle of the screen when you cancel the search. Mail.app does this right: If I do a search, select a message, then cancel the search, the selected message is still in the middle of the screen. But with GitX, as I said, I have to scroll to find it. This wasn’t going to scale very well. So what I did instead was search for “merge”, then I took a screen shot of the results and cancelled the merge. Then I just opened the screenshot in Preview, looked at the records there, then found them in GitX. This made things go quite a bit faster.

Commits that mention merging in GitX

As a result, I added a migration function to properly tag merges. It looked like this:

sub graft_merges {
    print "Grafting merges\n";
    # Handle the merges.
    for my $graft (
        [qw( trunk@5524   rev_1_8@5523 )],
        [qw( trunk@5614   rev_1_8@5613 )],
        [qw( rev_1_8@5591 trunk@5590   )],
    ) {
        my ($commit, $parent) = map { s/.+[@]//; $_ } @$graft;
        my $cmd = "\$(git rev-parse $commit) "
                . "\$(git rev-parse $commit^) "
                . "\$(git rev-parse $parent)";
        `echo "$cmd" >> .git/info/grafts`;
    }
}

By referencing revision tags explicitly, I was able to just use git rev-parse to look up SHA1 hash IDs to put into .git/info/grafts. This saved me the headache of dealing with very long IDs, but also allowed me to easily keep track of revision numbers and branches (the branch information is actually superfluous here, but I kept it for my sanity). So, basically, for [qw( trunk@5524 rev_1_8@5523 )], it ends up writing the SHA1 hashes for r5524, the existing parent commit for r5524 (that’s the $commit^ bit), and for the new parent, r5523. I ended up with 73 merges that needed to be properly recorded.

With the merges done, I next dove into branches. For some reason, git-svn failed to identify a parent commit for any branch. Maybe because I started with r5517? I have no idea. So I had to search through the commits to see when branches were started. I mainly did this by looking at the branches in ViewVC. By clicking each one, I was able to see the earliest commit, which usually had a name like “Created a branch for my SoC project.” I would then look up that commit in ViewVC, such as r7423, which started the “dev_ajax” branch, just to make sure that it was copied from trunk. Then I simply went into GitX, found r7423, then looked back to the last commit to trunk before r7423. That was the parent of the branch. With such data, I was able to write a function like this:

sub graft_branches {
    print "Grafting branches\n";
    for my $graft (
        [qw( dev_ajax@7423            trunk@7301 )],
        [qw( dev_mysql@7424           trunk@7301 )],
        [qw( dev_elem_occurrence@7427 trunk@7301 )],
    ) {
        my ($commit, $parent) = map { s/.+[@]//; $_ } @$graft;
        my $cmd = "\$(git rev-parse $commit) "
                . "\$(git rev-parse $parent)";
        `echo "$cmd" >> .git/info/grafts`;
    }
}

Here I only needed to look up the revision and its parent and write it to .git/info/grafts. Then all of my branches had parents. Or nearly all of them; those that were also in the old CVS repository will have to wait until the two are stitched together to find their parents.

Next I needed to get releases properly tagged. This was not unlike the merge tag work: I just had to find the proper revision and tag it. This time, I looked through the commits in GitX for those with “tag for” in their subjects because, conveniently, I nearly always used this phrase in a release tag, as in “Tag for the 1.8.11 release of Bricolage.” Then I just looked back from the tag commit to find the commit copied to the tag, and that commit would be tagged with the release tag. The function to create the tags looked like this:

sub tag_releases {
    print "Tagging releases\n";
    for my $spec (
        [ 'rev_1_8@5726' => 'v1.8.1'  ],
        [ 'rev_1_8@5922' => 'v1.8.2'  ],
        [ 'rev_1_8@6073' => 'v1.8.3'  ],
    ) {
        my ($where, $tag) = @{$spec};
        my ($branch, $rev) = split /[@]/, $where;
        my $tag_date = `git show --pretty=format:%cd -s $rev`;
        chomp $tag_date;
        local $ENV{GIT_COMMITTER_DATE} = $tag_date;
        system qw(git tag -fa), $tag, '-m', "Tag for $tag release of Bricolage.", $rev;
    }
}

I am again indebted to Ask for the code here, especially to set the date for the tag.

Since I had created new release tags and recreated the merge history in Git, I no longer needed the old tags from Subversion, so next I rewrote the --ignore-paths option to exclude all of the tags directories, as well as some branches that were never used:

SVNREPO=file:///Users/david/svn_bricolage_cc
git svn init $SVNREPO --stdlayout
git config svn.authorsfile /Users/david/bric_authors.txt
git svn fetch --no-follow-parent --revision 5517:HEAD
git svn fetch --no-follow-parent --revision 5517:HEAD \
--ignore-paths '(David|Kineticode|Release_|dev_(callback|(media_)?templates)|rev_1_([024]|[68]_temp)|tags/)|tmp';

With this in hand, I killed off the call to svn2git, opting to convert trunk and the remote branches myself (easily done by copying-and-pasting the relevant Perl code). Then all I needed to do was clean up the extant tags and run git-filter-branch to make the grafts permanent:

sub finish {
    print "Deleting old tags\n";
    my @tags = grep m{^tags/}, map { s/^\s+//; s/\s+$//; $_ } `git branch -a`;
    system qw(git branch -r -D), $_ for @tags;

    print "Deleting revision tags\n";
    @tags_to_delete = grep { /^\d+$/ } map { s/^\s+//; s/\s+$//; $_ } `git tag`;
    system qw(git tag -d), $_ for @tags_to_delete;

    print "Grafting...\n";
    system qw(git filter-branch);
    system qw(git gc);
}

And now I have a nicely organized Git repository based on the Bricolage Subversion repository, with all (or most) merges in their proper places, release tags, and branch tracking. Now all I have to do is stitch it together with the repository based on CVS and I’ll be ready to put this sucker on GitHub! More on that in my next post.

Looking for the comments? Try the old layout.

More about…

Migrating Bricolage CVS to Git

Following a discussion on the Bricolage developers mail list, I started down the path last week of migrating the Bricolage Subversion repository to Git. This turned out to be much more work than I expected, but to the benefit of the project, I think. Since I had a lot of questions about how to do certain things and how Git thinks about certain things, I wanted to record what I worked out here over the course of a few entries. Maybe it will help you manage your migration to Git.

The first thing I tried to do was use git-svn to migrate Bricolage to Git. I pointed it to the root directory and let it rip. I immediately saw that it noticed that the root was originally at the root of the repository, rather than the “bricolage” subdirectory, and so followed that path and started pulling stuff down. In a separate terminal window, I was watching the branches build up, and there were a lot of them, many named like:

David
David@5248
David@584
tags/Release_1_2_1
tags/Release_1_2_1@5249
tags/Release_1_2_1@577

Although many of those branches and tags hadn’t been used since the beginning of time, and certainly not since Bricolage was moved to Subversion from its original home in SourceForge CVS, because Subversion has no real concept of branches or tags, git-svn was duly copying them all, including the separate histories for each. Yow.

I could have dealt with that, renaming things, deleting others, and grafting where appropriate (more on grafting in a minute), but then I got this error from git-svn:

bricolage/branches/rev_1_8/lib/Bric/App/ApacheConfig.pm was not
found in commit e5145931069a511e98a087d4cb1a8bb75f43f899 (r5256)

This was annoying, especially since the file clearly does exist in that commit:

svn list -r5256 http://svn.bricolage.cc/bricolage/branches/rev_1_8/lib/Bric/App/ApacheConfig.pm
ApacheConfig.pm

I posted to the Git mail list about this issue, but unfortunately got no reply. Given that it was taking around 30 hours(!) to get to that point (and about 18 hours once I started using a local copy of the Subversion repository, thank to a suggestion from Ask Bjørn Hansen), I started thinking about how to simplify things a bit.

Since most of the moving stuff around happened immediately after the move to Subversion, and before we started committing working code to the repository, it occurred to me that I could probably go back to the original Bricolage CVS Repository on SourceForge, migrate that to Git, and then just migrate from Subversion starting from the first real commit there. Then I could just stitch the two repositories together.

From CVS to Git

Thanks to advice from IRC, I used cvs2git to build a repository from a dump from CVS. Apparently, git cvsimport makes a lot of mistakes, while cvs2git does a decent job keeping branches and tags where they should be. It’s also pretty fast; once I set up its configuration and ran it, it took only around 5 minutes for it to build import files for git fast-import. It also has some nice features to rename symbols (tags), ignore tags, assign authors, etc. I’m aware of not tool to migrate Subversion to Git that does the same thing.

Once I had my dump, I started writing a script to import it into Git. The basic import looks like this:

GITREPO=/Users/david/Desktop/git_from_cvs
rm -rf $GITREPO
mkdir $GITREPO
chdir $GITREPO
git init
cat ../cvs2svn-tmp/git-blob.dat ../cvs2svn-tmp/git-dump.dat | git fast-import
svn2git --no-clone
git gc
git reset --hard

I used svn2git to convert remote branches to local tags and branches The --no-clone option is what keeps it from doing the Subversion stuff; everything else is the same for a new conversion from CVS. I also had to run git reset --hard to throw out uncommitted local changes. What changes? I’m not sure where they came from, but after the last commit is imported from CVS, all of the local files in the master branch are deleted, but that change is not committed. Strange, but by doing a hard reset, I reverted that change with no harm done.

Next, I started looking at the repository in GitX, which provides a decent graphical interface for browsing around a Git repository on Mac OS X. There I discovered that a major benefit to importing from CVS rather than Subversion is that, because CVS has real tags, those tags are properly migrated to Git. What this means is that, because the Bricolage project (nearly) always tagged merges between branches and included the name of the appropriate tag name in a merge commit message, I was able to reconstruct the merge history in Git.

For example, there were a lot of tags named like so:

% git tag
rev_1_8_merge-2004-05-04
rev_1_6_merge-2004-05-02
rev_1_6_merge-2004-04-10
rev_1_6_merge-2004-04-09
rev_1_6_merge-2004-03-16

So if I wanted to find the merge commit that corresponded to that first tag, all I had to do was sort the commits in GitX by date and look near 2004-05-04 for a commit message that said something like:

Merge from rev_1_8. Will tag that branch "rev_1_8_merge-2004-05-04".

That commit’s SHA key is “b786ad1c0eeb9df827d658a81dc2d32ec6108e92”. Its parent’s SHA key is “11dbbd49644aaa607bd83f8d542d37fcfbd5e63b”. So then all I had to do was to tell git that there is a second parent for that commit. Looking in GitX for the commit tagged “rev_1_8_merge-2004-05-04”, I found that its SHA key is “4fadb117a71a49add69950eccc14b77a04c8ec68”. So to assign that as a second parent, I write a line to the file .git/info/grafts that describes its parentage:

b786ad1c0eeb9df827d658a81dc2d32ec6108e92 11dbbd49644aaa607bd83f8d542d37fcfbd5e63b 4fadb117a71a49add69950eccc14b77a04c8ec68

Once I had all the grafts written, I just ran git filter-branch and they were permanently rewritten to the new hierarchy.

And that’s it! The parentage is now correct. It was a lot of busy work to create the mapping between tags and merges, but it’s nice to have it all done and properly mapped out historically in Git. I even found a bunch merges with no corresponding tags and figured out the proper commit to link them up to (though I stopped when I got back to 2002 and things get really confusing). And now, because the merge relationships are now properly recorded in Git, I can drop those old merge tags: as workarounds for a lack of merge tracking in CVS, they are no longer necessary in Git.

Next up, how I completed the merge from Subversion. I’ll write that once I’ve finally got it nailed down. Unfortunately, it takes an hour or two to export from Subversion to Git, and I’m having to do it over and over again as I figure stuff out. But it will be done, and you’ll hear more about it here.

Looking for the comments? Try the old layout.

More about…

Moving Towards Bricolage 2.0

Today I’ve finished just about over two and a half weeks of hacking on Bricolage. It has been a couple of years since I gave it much attention, but there was so much good stuff that other people have contributed that, since I had a little time, it seemed worth it to give it some love. So here’s a quick list of all that I’ve done in the last two weeks:

  • Fixed all reported issues with Bricolage 1.10. Scott Lanning kindly released 1.10.5 yesterday with all of those fixes.

  • I integrated the element occurrence branch that Christian Muise had worked on as his 2006 Google Summer of Code project. Christian’s project added support for maximum and minimum specifications for subelements in Bricolage, which allows administrators to define how many fields and elements can occur in a story or media document. All I had to do was add a few UI tweaks to support the new fields and their specification in the story profile, and all was ready to go. Oh, and I did have to go back and make the SOAP interface work with the feature, but the only reason it never did was lazy hacking of the SOAP interface (way before Christian’s time). Nice work, Christian, and thank you for your contribution!

  • I fixed a few bugs with Arsu Andrei’s port of Bricolage to MySQL, which was his 2006 Google Summer of Code project. Arsu did a terrific job with the port, with only a few minor things missed that he likely could not have caught anyway. This work had already been merged into the trunk. Thanks Arsu!

  • I fixed a bunch of bugs from Marshall Roch’s AJAXification of Bricolage, carried out during his 2006 Google Summer of Code project. Marshall actually did a lot more stuff than he’d planned, as it all went quite smoothly. I found only a few minor oversights that I was able to easily address. This work represents the single most visible change to how users user Bricolage since we launched the project back in 2001. Editing stories, in particular, is now a lot cleaner, with far fewer page loads. Thanks a million, Marshall!

  • I completed the work started by Chris Heiland of the University of Washington, Bothell, and Scott Lanning of the World Health Organization to port Bricolage to Apache 2. They really did most of the hard work, and I just spent several days integrating everything, making sure all the features work, and updating the installer to handle differences in configuration. I thought this would take me a day or two, but it actually took the better part of a week! So much has changed, but in truth Bricolage is now better for running on mod_perl 2. Expect to see Apache 2 bet the recommended platform for Bricolage in a release in the near future.

  • I integrated a number of patches from Brian Smith of Gossamer Threads to allow the installer to be run as a non-root user. The key here is if the installer has to become the database super user, which is required for ident authentication, and of course whether files are to be installed somewhere on the system requiring super user access. This work is not done, yet, as make upgrade and make uninstall are not quite there yet. But we’re getting there, and it should be all done in time for 2.0, thanks to Brian.

  • I added support for a whole slew of environment variables to the installer. Now you can set environment variables to override default settings for installation parameters, such as choice of RDBMS, Apache, location of an SSL cert and key, whether to support SLL, and lots of other stuff, besides. This is all documented in the “Quick Installation Instructions” section of Bric::Admin/INSTALL.

  • I fully tested and fixed a lot of bugs leftover from making the installer database- and Apache-neutral. Now all of these commands should work perfectly:

    • make
    • make cpan
    • make test
    • make install
    • make devtest
    • make clone
    • make uninstall
  • I improved the DHTML functionality of the “Add More” widget, which is used to add contact information to users and contributors, rules to alert types, and extensions to media types. I think it’s pretty slick, now! This was built on Marshall’s AJAX work.

All of these changes have been integrated into the Bricolage trunk and I’ve pushed out a developer release today. Please do check out all the goodness on a test box and send feedback or file bug reports! There are only a couple of other features waiting to go into Bricolage before we start the release candidate process. And, oh yeah, tht title of this blog post? It’s not a lie. The next production release of Bricolage, based on all this work, will be Bricolage 2.0. Enough of the features we’d planned for Bricolage lo these many years ago are in the trunk that the new version number is warranted. I for one will be thrilled to see 2.0 ship in time for OSCON.

And in case it isn’t already clear, many thanks to the Google Summer of Code and participating students for the great contributions! This release would not have been possible without them.

Also in the news today, the Bricolage server has been replaced! The new server, which hosts the Web site, the wiki and the instance of Bricolage used to manage the site itself, is graciously provided by the kind folks at Gossamer Threads. The server is Gossamer Threads’s way of giving back to the Bricolage community as they prepare to launch a hosted Bricolage solution. Thaks GT!

The old Bricolage server was provided by pair Networds for the last five years. I’d just like to thank pair for the generous five-year loan of that box, which helped provided infrastructure for both Bricolage and Kineticode. Thank you, pair!

And with that, I’m going heads-down on some other projects. I’ll pop back up to make sure that Bricolage 2.0 is released in a few months, but otherwise, I’m on to other things again for a while. Watch this space for details!

Looking for the comments? Try the old layout.

I'm Ba-aack!

Sandy - your free personal email assistant

Yes indeed, I am back. Was I ever gone? Well, yes, I’ve been rather busy for the last 15 months.

But December 31 was my last day at Values of n. I’m really pleased with the work I did there. Sandy in particular, was a pleasure to work with. I really think that the work that Rael and I did with Sandy has been important work. Dare I say potentially paradigm-shifting? At any rate, I’m convinced that Sandy is really going to go places. If you haven’t signed on to become her client, do try. Though I will no longer be as intimate with her as I have in the past, we’re still going to keep in touch—I’m still her client. And Rael will do a great job pushing forward with her.

So why did I leave? Well, the truth is that, after Julie’s dad died in July, I found that I no longer had the resources to commit to the 80-100 hours/week required to work in a startup. It was just no longer that important to me. Don’t get me wrong, it was rewarding work, but my priorities completely realigned. It was vital that I continue helping Rael to get Sandy’s career launched, but once that was done, it was time for me to move on.

And what am I doing now? Well, first and foremost, I’m taking a few months off. I’m going to spend a lot more time re-acquainting myself with the two terrific women with whom I share a house, and just generally reset myself. Take a few deep breaths. Relax and enjoy life a bit. Sleep in now and then. That sort of thing.

That’s not to say that I’ll be sitting on my ass all the time. I have a very long list of things I want to do during this time, including catch up on my blogging (hence the title of this post), fix some bugs in Bricolage and help get 2.0 out the door, update my Perl libraries (I’ve got some great ideas for improving SVN::Notify), finally get all my digital photos organized, etc. I already spent much of last week revamping our mail system (I outsourced it to FuseMail). And all that’s leaving aside all the things Julie and I want to get done around the house. That’s the really important stuff.

But do watch for more blog posts in the coming months, too. There are a few interesting things I want to write about, and I’ve got some serious catching-up to do. Interested in following along on my adventures? Follow me via Twitter.

Looking for the comments? Try the old layout.

Bricolage GSoC Projects Completed

I’m very pleased to report that the Google Summer of Code Bricolage projects have all been successfully completed. The contributions of the three Summer of Coders, Marshall Roch, Christian Muise, and Andrei Arsu, will be included in the next major release of Bricolage. On behalf of the Bricolage community, like to extend my gratitude to Google for sponsoring these three excellent students to dramatically improve the interface, capabilities, and compatibility of Bricolage.

So what got done? Here’s a rundown:

  • Marshall Roch added many slick Ajax features to Bricolage. The story profile now manages the editing of all elements and subelements in a single screen, with no loading of a separate screen for subelements. You can navigate to subelements by clicking on a tree structure right in the story profile. Subelements more than three levels down will be loaded dynamically when you get to them. You can also drag and drop fields and elements to reorder them.

    Other stuff that Marshall Ajaxified:

    • Document and category keyword editing

    • Document category association

    • Document output channel associations

    • Organizations in the source profile

    • The “Add More” sections of the user, contributor, media type, and alert type profiles

    • Roles in the contributor profile

    • Assets on desks and My Workspace

    Marshall worked hard to integrate more interactive features into this 2000-era application, and I, for one, appreciate his hard work. Great job, Marshall!

  • Christian Muise added support for an occurrence specification to element types and field types. That means that when you make an element type a subelement of another element type, you can specify the minimum and/or maximum number of times that it can be a subelement. So when an element of the parent type is created, it will automatically add the minimum number of instances of a subelement specified for that parent type. This will allow an entire element tree to be pre-populated as soon as you create a new story or media document. Leaving the min and max occurrence set to 0 (zero) maintains the old behavior (no required subelements and an unlimited number can be added).

    Christian did the same for field types, too. The old “Required” and “Repeatable” attributes are gone; now you just specify a minimum number to require that number of instances of a field, and a maximum number to limit the number of instances. Together with the element type occurrence specification, this functionality allows Bricolage administrators to have a lot more control over the structure of the documents created by editors.

    Christian worked hard to complete this project, despite other huge demands on his time this summer (including a full-time job!). But thanks to his active participation on the developer mail list and his willingness to ask questions of his mentor, Scott Lanning, and myself, he overcame all obstacles to implement these features. He even wrote a number of new tests to ensure that it works properly and will continue to do so for the foreseeable future.

    Excellent work, Christian, and thank you so much for your contribution!

  • Andrei Arsu ported Bricolage to MySQL 5. Bricolage has always run on PostgreSQL and used a number of PostgreSQL-specific features to ensure that it ran properly and well. Andrei took these on, converting the existing PostgreSQL DDL files to work on MySQL, figuring out how to convince MySQL to work with some of their idiosyncrasies, and writing compatibility functions in the MySQL driver and upgrade module so that things should largely “just work.” As a result, for the first time ever, you can now build and run Bricolage on MySQL. Can compatibility with other databases be far behind?

    Andrei picked up Perl very quickly during this project, and was able to understand how such horrible code as the Bricolage installer worked without running screaming from the project. His code was well-written and his approaches to compatibility flexible and scalable. Well done, Andrei!

Future Plans

The next tasks toward getting this code integrated and released are as follows:

  • Andrei will merge his MySQL port into subversion trunk. This should actually be fairly straight-forward.

  • Marshall will merge his Ajaxification work into trunk. I don’t expect that there will be any conflicts with Andrei’s work, as the two projects were orthogonal.

  • Christian will merge his occurrence specification work into trunk. This will require that he work some with Andrei to ensure that his changes to the PostgreSQL DDLs are propagated to the new MySQL DDLS. He will also then need to work with Marshall to make sure that the occurrence specification works properly with the Ajaxified UI.

Once these tasks have been completed, we’ll be ready to release a development version of Bricolage with all three of these major improvements. The development release will allow members of the Bricolage community to start to play with the new features, report bugs, and make further suggestions for improvement. Expect the release sometime in the next six weeks or so.

Again, my thanks to Marshall, Christian, and Andrei for their hard work this summer, and for all that they have contributed to the Bricolage community and project. I hope that each will remain involved in the community, not only to support the features they’ve added, but to work with other members of the community to add new features, help newbies, and generally to spread the word.

Looking for the comments? Try the old layout.

Why Bricolage documents have UUIDs

Some time ago, I decided that all objects in Bricolage 2 would have Universally Unique Identifiers, also known as “UUIDs.” A UUID is guaranteed to be universally unique, never to be generated again by the same or any other system now or in the future. As anyone using Bricolage knows, all stories and media already have IDs, so why have UUIDs, as well? How does their purpose differ?

Well, first of all, the existing IDs are not really identifiers. What they are, instead, are primary keys. However, a primary key should ideally be a surrogate key, meaning that it has no other meaning outside of identifying a single database row. Sometimes you can use an “intelligent key,” meaning an attribute of the object being stored (such as a user login), for the primary key. But the problem with intelligent keys is that, should their values ever be changed (say a user’s name changes and company login name conventions dictate that the login must be changed to represent the new name), all foreign key references will be broken. It is therefore easier, and more agile, to use a surrogate key with no inherent meaning to the object with which it is associated.

Now, once you start using an object ID that is actually a surrogate key for something other than identifying a row in a database, you add new meaning to it. At that point, it is no longer a surrogate. In Bricolage, this comes up when users want to use IDs for story URIs. At that point, the ID is no longer just a primary key identifying a database row, but it is also an object identifier. What happens if that identifier changes? Well, in general, it won’t, so you’d be safe to use it for both purposes. But sometimes it does.

When? Some Bricolage users have decided to upgrade to a newer version of Bricolage by setting up the new version on a different server, exporting their data from the old server, and then importing it into the new. This can work reasonably well, but it has what may be an unintended side-effect for those who use the ID in the URI: all objects will get new primary keys when they’re inserted into the new system.

What? you cry! Yes, that’s right. Because the ID is used solely to identify a row in a database, when you insert an existing object into a new database, it gets stored in a new row. It therefore gets a new ID, and your URIs suddenly start to 404. Ouch.

The solution to this problem is to give Bricolage objects a universally unique identifier that can work anywhere, that means nothing other than “this is a unique identifier for this object,” and which are guaranteed not to change when you move an object from one system to another. Happily, the UUID standard exists for just this sort of thing. You are free to use a story’s UUID in its URI without having to worry about it ever changing. IDs may change, but you don’t have to worry about those.

For these reason, the forthcoming Bricolage 1.10.0 has added UUIDs to story and media objects, these being the objects most in need of UUIDs, and they are available for use in URIs. Looking to the future, the Kinetic Platform, currently under development and the platform to which Bricolage 2.0 will be ported, never exposes the primary key IDs at all. There is only the UUID for referencing objects externally. I judge this a very good thing.

Looking for the comments? Try the old layout.

More about…

TKP Permissions Design

So, I’m thinking of implementing permissions in the Kinetic Platform differently than they’re implemented in Bricolage. Bricolage has a number of fixed permissions: READ, EDIT, RECALL, CREATE, PUBLISH, and DENY. These permissions are cumulative, so that if you have EDIT permission, it implies READ, and if you have CREATE, it implies RECALL, EDIT, and READ.

This design was based on Windows NT permissions (roughly), and has worked reasonably, well, but is annoying for various reasons. The most serious drawback is that it’s difficult to understand. I always tell people who need to manage Bricolage permissions to read Bric::Security, and then read six more times. But aside from the impenetrability of the current permissions design, it’s also difficult to add new permissions: where should they fit into the hierarchy? This is what happened with RECALL and PUBLISH, which were added in a later version of Bricolage. To this day, it’s a bit confusing to some that, with RECALL permission, you can RECALL a story but not CREATE one.

So I’m looking around for other permissions patterns. Unix is nice, in that READ, WRITE, and EXECUTE permissions are all entirely independent, and apply to three classes of objects (file owner, file group, everyone). But Unix only needs to worry about files; Kinetic applications will have many more objects for which permissions will need to be specified. RT uses discreet permissions with names like “AdminQueue”, “CommentOnTicket”, “CreateTicket”, and “StealTicket” to be applied to every user or group. This strikes me as somewhat more useful, since the permissions are much more descriptive and can be targeted to particular objects. In fact, the permission names even indicate to what types of objects permissions apply!

So I’m thinking of leaving the cumulative permissions model behind and switching to more descriptive, discreet, and potentially numerous permissions. I’m not, however particularly fond of RT’s approach of storing the permissions as strings in the database. Now, I could keep them as numbers, where each permission has its own unique number. This is similar to how Bricolage permissions work. Only I’d have to always use List::MoreUtil’s any() function to see if a permission was in a list.

For example, say that an object had permissions with the numbers 1, 2, 5, 8, 12, 49, and 50. If these were stored in an array, then every time I had to check permissions, the can_do() function would have to search through those numbers:

sub can_do {
    my ($self, $to_check) = @_;
    my $perms = $self->perms;
    return any { $_ == $to_check } @$perms;
}

This isn’t the most efficient approach, unfortunately. If there are a lot of numbers, and you were always checking one that was towards the end of the list of permissions, it’d always take a long time. So, an alternate approach–one that conveniently works well with discreet permissions–is to use powers of two for the permissions numbers: 1, 2, 4, 8, 16, 32, 64, 128, 256, and so on. Then, for a given object, these can just be stored in a single number that’s created by bit-wise ORing them together: 1 | 2 | 8 | 16. Fortunately, PostgreSQL has a nice aggregate function for this, bit_or(). (I’ll have to create my own for SQLite.) Anyway, then the can_do() function becomes much simpler and more efficient:

sub can_do {
    my ($self, $to_check) = @_;
    return $self->perms & $to_check;
}

If the value of $to_check was in the list bit-ORed into their permissions number, it will be returned; otherwise, 0 will be returned. Not bad, eh?

So anyway, I’m soliciting feedback. Are discreet permissions better than cumulative permissions? And if so, are bit-wise ORed numbers the best way to represent an object ACLs?

Looking for the comments? Try the old layout.

More about…

10001

% make devtest
PERL_DL_NONLAZY=1 /usr/bin/perl inst/runtests.pl -d
t/Bric/Test/Runner....ok                                                     
        1801/10001 skipped: various reasons
All tests successful, 1801 subtests skipped.
Files=1, Tests=10001, 247 wallclock secs (110.30 cusr +  8.90 csys = 119.20 CPU)

How ‘bout them apples?

Looking for the comments? Try the old layout.

More about…

My OSCON 2005 Presentation Slides Posted

Kineticode has posted PDF versions of my my OSCON 2005 presentation slides. Direct links:

Enjoy!

Looking for the comments? Try the old layout.

Bricolage Now has PHP 5 Templating

Well, I’ve finally gone and done it. I’ve released Bricolage 1.9.0, the first development release towards 1.10.0, which I plan to get out sometime next month. Among the new features in this release that I’m most excited about are a revamped UI, LDAP authentication, and PHP 5 templating support.

The new UI is really nice. Marshall Roch ported it all to XHTML 1.0 strict plus CSS. All the layout is done with CSS now, instead of the old 1999-era table layouts we had before. The result is much smaller page loads, sometimes up to 70% smaller, Marshall tells me, and therefore much faster loads. Playing around with the new version (now powering the Kineticode and Bricolage Web sites, I do indeed find it to be a lot zippier. Couple the new UI with the new mod_gzip support and static Mason components, and things just perk right along! Oh, and the expandable interfaces for both lists of objects and for editing interfaces is very nice, indeed. Thank you Marshall for the great work!

Kineticode added the LDAP authentication support. We’ve been using a patch for it against 1.8.x for our internal version of Bricolage and it works great. Now we authenticate off of an OpenLDAP server, using the same usernames and passwords we use for our email, Subversion, and RT servers. The configuration is simple, via bricolage.conf directives, and you can even limit users who authenticate to members of a particular LDAP group. Users must exist in Bricolage, and if LDAP authentication fails, you can fall back on Bricolage’s internal authentication. I’m hoping that the LDAP support goes a long way towards attracting more enterprise customers with single sign-on requirements.

And finally—yes, you heard right—Bricolage now supports PHP 5 templating in addition to the existing Perl-based templating architectures (Mason, Template Toolkit, and HTML::Template). So how did we add PHP 5 templating to a mod_perl application? Easy: we hired George Schlossnagle of Omni TI to write PHP::Interpreter, an embedded PHP 5 interpreter. Now anyone can natively execute PHP 5 code from a Perl application. Not only that, but the PHP 5 code can reach back into the Perl interpreter to use Perl modules and objects! Here’s an example that I like to show off to the PHP crowd:

<?php
    $perl = Perl::getInstance();
    $perl->eval("use DBI");
    $perl->eval("use DateTime");
    $dbh = $perl->call("DBI::connect", "DBI", "dbi:SQLite:dbname=dbfile");
    $dbh->do("CREATE TABLE foo (bar TEXT, time DATETIME)");
    $now = $perl->call("DateTime::now", "DateTime");
    $ins = $dbh->prepare("INSERT INTO foo VALUES (?, ?)");
    $ins->execute("This is a test", $now);
    $sel = $dbh->prepare("SELECT bar, time FROM foo");
    $sel->execute();
    $a = array("foo", "bar");
    foreach ($sel->fetch() as $val) {
        echo "$val\n";
    }
    $sel->finish();
    $dbh->do("DROP TABLE foo");
    $dbh->disconnect();
?>

Note that George plans to add convenience methods to load Perl modules and call Perl class methods. Now, to execute this code from Perl, all you have to do is write a little script, call it pphp, like so:

use strict;
use PHP::Interpreter;
my $php = PHP::Interpreter->new;
$php->include(shift);

Then just execute your PHP code with the script: pphp try.php. Yes, this does work! For years, when I’ve run across a PHP coder who wanted to try to tell me that PHP was better than Perl, I always had a one-word reply that left him cursing and conceding defeat: “CPAN.” Well no more. Now PHP hackers can use any module on CPAN, too!

And as for Bricolage, the integration of PHP 5 templating is completely transparent. Users just write PHP 5 templates instead of Mason templates and that’s it! For example, this is a fairly common style Bricolage Mason template:

<%perl>;
for my $e ($element->get_elements(qw(header para _pull_quote_))) {
    my $kn = $e->get_key_name;
    if ($kn eq "para") {
        $m->print("<p>", $e->get_data, "</p>\n");
    } elsif ($kn eq "header") {
        # Test sdisplay_element() on a field.
        $m->print("<h3>", $burner->sdisplay_element($e), "</h3>\n");
    } elsif ($kn eq "_pull_quote_" && $e->get_object_order > 1) {
        # Test sdisplay_element() on a container.
        $m->print($burner->sdisplay_element($e));
    } else {
        # Test display_element().
        $burner->display_element($e);
    }
}
$burner->display_pages("_page_");
</%perl>

The same template in PHP 5 looks like this:

<?php
    # Convenience variables.
    $story   = $BRIC["story"];
    $element = $BRIC["element"];
    $burner  = $BRIC["burner"];
    foreach ($element->get_elements("header", "para", "_pull_quote_") as $e) {
        $kn = $e->get_key_name();
        if ($kn == "para") {
            echo "<p>", $e->get_data(), "</p>\n";
        } else if ($kn == "header") {
            # Test sdisplay_element() on a field.
            echo "<h3>", $burner->sdisplay_element($e), "</h3>\n";
        } else if ($kn == "_pull_quote_" && $e->get_object_order() > 1) {
            # Test sdisplay_element() on a container.
            echo $burner->sdisplay_element($e);
        } else {
            # Test display_element().
            $burner->display_element($e);
        }
    }
    $burner->display_pages("_page_");
?>

Yes, you are seeing virtually the same thing. But this is just a simple template from Bricolage’s test suite. The advantage is that PHP 5 coders who are familiar with all the ins and outs of PHP 5 can just jump in an get started writing Bricolage templates without having to learn any Perl! The Bricolage objects have exactly the same API as they do in Perl, because they are exactly the same objects! So everyone uses the same API documentation for the same tasks. The only issue I’ve noticed so far is that PHP 5 does not yet have proper Unicode support. Since all content in Bricolage is stored as UTF-8, this means that the PHP 5 templates must treat it as binary data. But this is okay as long as templaters use the mb_* PHP 5 functions to parse text.

Overall I’m very excited about this, and hope that it helps Bricolage to reach a whole new community of users. I’d like to thank Portugal Telecom—SAPO.pt for sponsoring the development of PHP::Interpreter and its integration into Bricolage. I believe that they’ve really done the Bricolage community a great service, and I hope that the Perl and PHP communities likewise benefit from the integration possible with PHP::Interpreter.

And just so that the other templating architectures don’t feel left out, here is how the above template looks in Template Toolkit:

[% FOREACH e = element.get_elements("header", "para", "_pull_quote_") %]
    [% kn = e.get_key_name %]
    [% IF kn == "para" %]
<p>[% e.get_data %]</p>
    [% ELSIF kn == "header" %]
        [% # display_element() should just return a value. %]
<h3>[% burner.display_element(e) %]</h3>
    [% ELSIF kn == "_pull_quote_" && e.get_object_order > 1 %]
        [% PERL %]
          # There is no sdisplay_element() in the TT burner, but we"ll just
          # Play with it, anyway.
          print $stash->get("burner")->display_element($stash->get("e"));
        [% END %]
    [% ELSE %]
        [% # Test display_element(). %]
        [% burner.display_element(e) %]
    [% END %]
[% END %]
[% burner.display_pages("_page_") %]

And here it is in HTML::Template:

<tmpl_if _page__loop>
<tmpl_loop _page__loop>
<tmpl_include name="testing/sub/util.tmpl">
<tmpl_var _page_>
<tmpl_var page_break>
</tmpl_loop>
<tmpl_else>
<tmpl_include name="testing/sub/util.tmpl">
</tmpl_if>

I had to use the utility template with HTML::Template to get it to work right, don’t ask why. It looks like this:

<tmpl_loop element_loop>
<tmpl_if is_para>
<p><tmpl_var para></p>
</tmpl_if>
<tmpl_if is_header>
<h3><tmpl_var header></h3>
</tmpl_if>
<tmpl_if is__pull_quote_>
<tmpl_var _pull_quote_>
</tmpl_if>
</tmpl_loop>

These templates come from the Bricolage test suite, where until this release, there were never any template tests before. So you can see that the PHP 5 templating initiative has had major benefits for the stability of Bricolage, too. Now that I’ve really worked with all four templating architectures in Bricolage, I can now say that my preference for which to do goes in this order:

  1. Mason, because of its killer autohandler and inheritance architecture
  2. PHP 5 or Template Toolkit are tied for second place
  3. HTML::Template

In truth, all four are capable and have access to the entire Bricolage API so that they can output anything. So what are you waiting for? Download Bricolage and give it a try!

Looking for the comments? Try the old layout.

Bricolage 1.8.6 Released

The Bricolage development team is pleased to announce the release of Bricolage 1.8.6. This maintenance release addresses numerous minor issues in Bricolage 1.8.5 and adds a number of improvements, including SOAP, document expiration, and bric_queued fixes. The most important changes include:

Improvements

  • Added JavaScript code to validate that the username in the user profile does not have leading or trailing spaces. [David]
  • Events in the event log are now returned (and displayed) in reverse chronological order. [David]
  • The SOAP server now uses a user’s template sandbox when executing previews (such as with bric_soap --to-preview workflow publish). Reported by Marshall. [David]
  • Bric::Biz::Workflow now caches calls to allowed_desks(). This will allow desks to render much Faster, since most assets on a desk will list the same desks in the “Move to” select lists. [David]
  • When the PUBLISH_RELATED_ASSETS bricolage.conf directive is enabled, aliases are now also republished. Only aliases that have previously been published will be republished, and only the last published version will be republished, rather than any versions created since the last publish. Suggested by Serge Sozonoff. [David]
  • A story or media document published with an expire date earlier than the scheduled publish time no longer bothers with the publish but just expires the story or media document. [David]
  • Media documents without an associated media file will no longer be displayed in the search results when attempting to relate a media document to an element. Reported by Adam Rinehart. [David]

Bug Fixes

  • Form validation and group management now properly work in the user profile. [David]
  • The SFTP mover now works with bric_queued. [David]
  • Cloned stories now properly set the published_version attribute to undef rather than the value of the original story, thus preventing the clone from having a published version number greater than its current version number. Reported by Nate Perry-Thistle and Joshua Edelstein. [David and Nate Perry-Thistle]
  • When a category is added to a story that creates a URI conflict, the new category does not remain associated with the story in the story profile after the conflict error has been thrown. Reported by Paul Orrock. [David]
  • Contributor groups created in the contributor profile are no longer missing from the contributor manager search interface. Reported by Rachel Murray and Scott. [David]
  • The favicon.ico works again. [David]
  • Stories are now properly expired when the BRIC_QUEUED bricolage.conf directive is enabled. Reported by Scott. [David]
  • When a template is checked out of the library and then the checkout is canceled, it is no longer left on the desk it was moved into upon the checkout, but properly reshelved. Reported by Marshall. [David]
  • Super Bulk Edit now works for media as well as stories. Reported by Scott. [David]
  • When a template is moved to a new category, the old version of the template is undeployed when the new version is deployed to the new category. The versions in the sandbox are properly synced, as well.

For a complete list of the changes, see the changes. For the complete history of ongoing changes in Bricolage, see Bric::Changes.

Download Bricolage 1.8.6 now from the Bricolage Web site Downloads page, from the SourceForge download page, and from the Kineticode download page.

About Bricolage

Bricolage is a full-featured, enterprise-class content management and publishing system. It offers a browser-based interface for ease-of use, a full-fledged templating system with complete HTML::Mason, HTML::Template, and Template Toolkit support for flexibility, and many other features. It operates in an Apache/mod_perl environment and uses the PostgreSQL RDBMS for its repository. A comprehensive, actively-developed open source CMS, Bricolage has been hailed as “quite possibly the most capable enterprise-class open-source application available” by eWEEK.

Looking for the comments? Try the old layout.

How the Bricolage Summer of Code Projects were Selected

As you may have read, we got quite a number of applications from students wishing to contribute to Bricolage as part of Google’s Summer of Code initiative. Quite a few of them were very good. There were eight projects I wanted to accept, but, Bricolage was allocated only four projects. Of course, this is four more than we would have had otherwise, and I’m really excited to be working with them this summer.

The four winning projects are:

  • Add Input Channels, by Marshall Roch
  • New Sample Document types and templates, by Scott Loyd
  • Port Bricolage to Apache 2/mod_perl 2 and Windows, by Sam Strasser
  • Port Bricolage to MySQL, by Tamas Mezei

The other projects I wanted to get but could not were:

  • Add Bulk Edit, Bulk Media Upload, and Site Tags, by Andreas Hofmeister
  • Element Occurrence Specification, by Christian James Muise
  • Add JSP Templating, by Adrian Fernandez
  • Update and Modernize the Installer, by Yiannis Valassakis

I am hoping that some of these students might want to work on their projects, anyway. I’ve even found other developers to help with the mentoring of JSP templating (Patrick LeBoutillier with Perl/Java voodo) and the installer modernization (Sam Tregar of Matchstick fame). Unfortunately, I’ve not heard back from any of them after sending them an invite to participate in the project. C’est la vie, I guess

The hardest part of the proposal evaluation process was selecting from the 20 proposals to port Bricolage to MySQL. Ultimate, there were four excellent proposals for this project. Reading the proposals over and over, I couldn’t decide between them. Ultimately, I sent an email to the four top contenders with the following items for them to reply to:

  1. Please describe in a line or two your Perl knowledge or experience (if any).
  2. Please describe in a line or two your MySQL and PostgreSQL knowledge or experience (if any).
  3. Please describe any previous Bricolage usage experience.
  4. Please describe any previous Bricolage development experience.
  5. Please describe any previous Open Source development experience.
  6. What school do you attend?
  7. What is your specialty at the school?
  8. How many years have you attended there?
  9. How much time do you expect to have for this project?
  10. Have you applied for any other Summer of Code projects? If so, which ones?
  11. Your personal or professional web page URL (if any).
  12. Would you be willing to collaborate with another developer who might be working on a SQLite port to ensure that your changes can fully inter-operate?
  13. Please outline your project plan for porting Bricolage to MySQL, including a description of what parts of the Bricolage API, DDLs, installer, and upgrader would need to be modified to complete the project.

For better or for worse, all four applicants responded with detailed answers to my questions. They were all great, and that made it even harder to select just one of them. At this point, there were only a few hours left to rank applicants in Google’s SoC Admin Web app, so I figured I had to get more objective—or at least fool myself into thinking I was.

So I decided to rank each applicant from one to five for each question, and then add up all of the results and see who came out on top. So now I was comparing answers to a single question between applicants, and filling in scores for them in a spreadsheet. As it was, things were still really close; yes, all of the students where that good! Tamas came out on top with a score of 30, two others were tied at 28, and the fourth applicant scored 26. They were close enough that I wanted to review them all one more time, this time paying specific attention to the last item in my questionnaire, the project plan.

Each of the four applicants had taken the time to read the mail lists and had looked at the existing Bricolage code. But there were varying levels of detail and demonstration of knowledge for how to implement the MySQL port, but Tamas did come out slightly ahead on this item, so I gave his proposal the green light.

But in truth, I would have been happy with any one of those four applicants. I was only sorry I had to choose only one! If Google does this again, I think I’ll list many more project ideas on the Bricolage Web site, and try to steer people to the mail lists to discuss their ideas before sending them in. Then I might end up with 11 great applications!

Looking for the comments? Try the old layout.