I’ve been following the development of a few things in the Perl community lately, and it’s leaving me very frustrated. For years now, I’ve written modules that start with the same incantation:

package My::Module;

use strict;
our $VERSION = '0.01';

Pretty simple: declare the module name and version, and turn on strictures to make sure I’m not doing anything stupid. More recently I’ve added use warnings; as a best practice. And even more recently, I’ve started adding use utf8;, too, because I like to write my code in UTF-8. And I like to turn on all of the Perl 5.10 features. It’s mildly annoying to have the same incantation at the start of every module, but I could deal with it:

package My::Module;

use strict;
use warnings;
use feature ':5.10';
use utf8;

our $VERSION = '0.01';

Until now that is. Last year, chromatic started something with his Modern::Perl module. It was a decent idea for newbies to help them get started with Perl by having to have only one declaration at the tops of their modules:

package My::Module;

use Modern::Perl;
our $VERSION = '0.01';

Alas, it wasn’t really designed for me, but for more casual users of Perl, so that they don’t have to think about the pragmas they need to use. The fact that it doesn’t include the utf8 pragma also made it a non-starter for me. Or did it? Someone recently suggested that the utf8 pragma has problems (I can’t find the Perl Monks thread at the moment). Others report that the encoding pragma has issues, too. So what’s the right thing to do with regard to assuming everything is UTF8 in my program and its inputs (unless I say otherwise)? I’m not at all sure.

Not only that, but Modern::Perl has lead to an explosion of other pragma-like modules on CPAN that promise best pragma practices. There’s common::sense, which loads utf8 but only some of of the features of strict, warnings, and feature. uni::perl looks almost exactly the same. There’s also Damian Conway’s Toolkit, which allows you to write your own pragma-like loader module. There’s even Acme::Very::Modern::Perl, which is meant to be a joke, but is it really?

If I want to simplify the incantation at the top of every file, what do I use?

And now it’s getting worse. In addition to feature, Perl 5.11 introduces the legacy pragma, which allows one to get back behaviors from older Perls. For example, to get back the old Unicode semantics, you’d use legacy 'unicode8bit';. I mean, WTF?

I’ve had it. Please make the pragma explosion stop! Make it so that the best practices known at the time of the release of any given version of Perl can automatically imported if I just write:

package My::Module '0.01';
use 5.12;

That’s it. Nothing more. Whatever has been deemed the best practice at the time 5.12 is released will simply be used. If the best practices change in 5.14, I can switch to use 5.14; and get them, or just leave it at use 5.12 and keep what was the best practices in 5.12 (yay future-proofing!).

What should the best practices be? My list would include:

• strict
• warnings
• features — all of them
• UTF-8 — all input and output to the scope, as well as the source code

Maybe you disagree with that list. Maybe I’d disagree with what Perl 5 Porters settles on. But then you can I can read what’s included and just add or removed pragmas as necessary. But surely there’s a core list of likely candidates that should be included the vast majority of the time, including for all novices.

In personal communication, chromatic tells me, with regard to Modern::Perl, “Experienced Perl programmers know the right incantations to get the behavior they want. Novices don’t, and I think we can provide them much better defaults without several lines of incantations.â€? I’m fine with the second assertion, but disagree with the first. I’ve been hacking Perl for almost 15 years, and I no longer have any fucking idea what incantation is best to use in my modules. Do help the novices, and make the power tools available to experienced hackers, but please make life easier for the experienced hackers, too.

I think that declaring the semantics of a particular version of Perl is where the Perl 5 Porters are headed. I just hope that includes handling all of the likely pragmas too, so that I don’t have to.

Looking for the comments? Try the old layout.

Pondering Conservatism

I’ve been following chromatic’s new blog since it launched, and have read with particular interest his posts on the Perl 5 development and release process. The very long time between releases of stable versions of Perl has concerned me for a while, though I hadn’t paid much attention until recently. There has been a fair amount of discussion about what it means for a release to be “stable,” from, among others, now-resigned Perl Pumpking Rafael Garcia-Suarez and Perl 5 contributor chromatic. Reading this commentary, I started to ponder what other major open-source projects might consider “stable,” and how they manage stability in their development and release processes. And it occurred to me that the Perl 5 code base is simultaneously treated too conservatively and – more importantly – not conservatively enough. What open-source projects treat their code highly conservatively?

If you think about most software needs, you will seldom find more conservatism than in the use of relational databases. It’s not just that the code needs to continue to work version after version, but that the data needs to remain intact. If your application doesn’t work quite right on a new version of Perl, you might lose some time reverting to an older version of Perl. Hell, for a massive deployment, you might lose a lot of time. But if something happens to the data in your RDBMS, you could lose your whole business! So database developers need to be extremely careful about their releases – even more careful than Perl developers – so as not to kill off entire businesses with regressions.

The PostgreSQL RDBMS is especially noted for its stability over time. So I thought it might be worthwhile to look at how the PostgreSQL project works.

PostgreSQL Development Organization

First, there is no one person in charge of PostgreSQL development. There is no benevolent dictator, no Pumpking, just a 7-member core team that collectively makes decisions about release schedules, security issues, and, whatnot. Several members of the team are of course core hackers, but some handle PR or release management and are less familiar with internals. The core team has a moderated mail list on which they can discuss issues amongst themselves, such as when to put out new releases. But most development and contribution issues are discussed on the pgsql-hackers mail list, which corresponds to the perl5-porters list. The vast majority of the decisions about what does and does not get into the core takes place on this list, leaving the core team to take on only those issues which are irresolvable by the community at large.

After each major release of PostgreSQL (most recently, 8.4 two weeks ago), the pgsql-hackers list discusses and agrees to a commit fest and release schedule for the next major version of PostgreSQL. The schedule is typically for about a year, and despite the occasional worry about the increasing time between major releases (up to 16 months between 8.3 and 8.4), there is in fact a major new release of PostgreSQL – with significant new features – every 9-18 months. That’s an incredibly aggressive release cycle; I’d love to see Perl 5.10 followed by 5.12 just a year later.

This is the liberal part of the PostgreSQL development process: freely accepting patches and working them in to the core via commit fests over the course of 6-8 months and relying on the build farm to quickly address regressions. The commit fests were introduced for the 8.4 schedule to make it easier for the core hackers to track, review, and commit contributed patches. They last for a month and take place every other month, and while there were some hiccups with them the first time around, they were enough of a success that a dedicated Webapp has been built to manage them for 8.5 and beyond. Community members are encouraged to independently test patches, just to confirm that things work as expected before one of the committers dedicates the time. This allows new development to progress quickly over the course of 6-8 months before a feature freeze is declared and the project enters a beta- and release-candidate release cycle. Once a release candidate goes for two weeks or so without a major regression, it’s declared done, the x.x.0 release drops, and CVS HEAD is opened for development of the next major version.

PostgreSQL’s Code Conservatism

With all the activity around adding new features and the occasional backward incompatibility to PostgreSQL, you might wonder wherein lies the conservatism I mentioned. Well, it’s this: every time a major new version of PostgreSQL ships, a maintenance branch is forked for it; and thereafter only bug fixes and security issues are committed to it. Nothing else. PostgreSQL’s maintenance branches are treated very conservatively; even documentation patches are accepted only for CVS HEAD.

How do things get applied to maintenance branches? When a committer applies a patch to CVS HEAD, he or she also evaluates whether the patch is relevant to actively maintained earlier versions of PostgreSQL, and applies the patch (with appropriate modifications) to those versions. All such changes are committed all at once, or in a series of commits with exactly the same commit message. For example, here’s a commit by Tom Lane fixing a bug in PL/pgSQL last October and back-patching it through all supported versions except 7.4, which did not have the problem. As you can see, there is no cherry-picking of commits from HEAD here. It is the responsibility of the committer to ensure that bug fixes are applied to all supported branches of PostgreSQL – at the same time.

The upshot of this approach is that the PostgreSQL project can be explicit about what versions of PostgreSQL it maintains (in terms of regular releases with bug fixes and security patches) and can quickly deliver new releases of those versions. Because so little changes in maintenance branches other than demonstrable bug fixes, there is little concern over breaking people’s installations. For example, on March 2 of this year, Tom Lane fixed a bug in all supported versions of PostgreSQL that addressed a security vulnerability. The core hackers decided that this issue was important enough that they delivered new releases of all of those versions of PostgreSQL (8.3.7, 8.2.13, 8.1.17, 8.0.21 and 7.4.25 – yes, there have been 26 releases of 7.4!) on March 17, just two weeks later. More serious security issues have been addressed by new releases within a day or two of being fixed.

In short, thanks to its formal support policy and its insistence on applying only bug fixes to supported versions and applying them to all maintenance branches at once, the PostgreSQL project can regularly and quickly deliver stable new releases of PostgreSQL.

An Insight

Now let’s contrast PostgreSQL development practice with the Perl approach. I had assumed that major versions of Perl (5.8.x, 5.10.x) were maintained as stable releases, with just bug fixes going into releases after the .0 versions. After all, the PostgreSQL practice isn’t so uncommon; we do much the same thing for Bricolage. So I was stunned last weekend to see this post, by Perl 5.10 Pumpking Dave Mitchell, suggesting that inclusion of autodie in the core be pushed back from 5.10.1 to 5.10.2. The fact that a major new module/pragma is being added to a minor release (and it looks like things were worked out so that autodie can stay in 5.10.1) highlights the fact that minor releases of Perl are not, in fact, maintenance releases. They are, rather, major new versions of Perl.

This probably should have been obvious to me already because, notwithstanding Nicholas Clark’s heroic delivery of new versions of Perl 5.8 every three months for close to two years, minor releases of Perl tend to come out infrequently. Perl 5.10.0 was released over a year and a half ago, and it looks like 5.10.1 will be out in August. That’s a standard timeline for major releases. No wonder it’s so bloody much work to put together a new release of Perl! This insight also helps to explain David Golden’s suggestion to change Perl version number scheme to support, for example, 5.10.0.1. I couldn’t see the point at first, but now I understand the motivation. I’m not sure it’s the best idea, but the ability to have proper bug-fix-only maintenance releases of officially supported versions of Perl would be a definite benefit.

Last week came a new surprise: Rafael Garcia-Suarez has resigned as Perl 5.12 Pumpking. This is a sad event for Perl 5, as Rafael has done a ton of great work over the last five years – most recently with the smart-match operator borrowed from Perl 6. But I think that it’s also an opportunity, a time to re-evaluate how Perl 5 development operates, and to consider organizational and structural changes to development and release management. (And, yes, I also want to see 5.10.1 out the door before just about anything else.)

Modest Proposals

I’m a newcomer to the Perl 5 Porters list, but not to Perl (I started hacking Perl in 1995, my first programming language). So I hope that it’s not too impertinent of me to draw on the example of PostgreSQL to make some suggestions as to how things might be reorganized to the benefit of the project and the community.

Create a cabal.

It seems to me that the pressure of managing the direction of Perl development is too great for one person. The history of Perl is littered with the remains of Perl Pumpkings. I can think of only two former Pumpkings who are still actively involved in the project: Nicholas Clark and Chip Salzenberg. Most (all?) of the others are gone, and even Chip took a break for a few years. Tim Bunce is still active in the project, but not in core development. I’m loathe to recommend design-by-committee, but the nature of the perl5-porters list reveals that such is already the case, and the committee is too big. Why should one person alone take on the stress of dealing with it all, of defending executive decisions?

I think that PostgreSQL gets this one right (or at least more right) with its core team. It’s intentionally limited to a very small group, and each of the members has equal say. The group sets parameters for things like release scheduling and makes decisions that the community at large can’t agree to, but otherwise is fairly hands-off. Responsibility is shared by all members, and they help each other or refer to decisions made between them in the context of heated discussions on the pgsql-hackers list. It’s more of a guiding structure than a leadership role, and it works well for an unstructured project like open-source development.

Rather than make just one person responsible for each major version of Perl, handling all executive decisions, managing commits and back-patches and defending decisions, wouldn’t it work better to have a small group of people doing it? Couldn’t you see RGS, Dave Mitchell (who, it seems, has also suggested breaking up the Pumpking role), Chip, Nicholas, and a few other parties with a significant investment in the development and maintenance of the Perl core (mst? Jesse Vincent? Larry???) gently guiding development and community participation, not to mention maintenance and release management? Perl is a big project: the huge responsibility for maintaining it should be distributed among a number of people, rather than be a heavy burden for one person who then burns out.

Establish a policy for supported versions.

What is the oldest major version of Perl that’s officially supported by the project? I don’t know, either. I guess it’s 5.8, but only because Nicholas picked up the gauntlet and got 5.8.9 out last year. 5.6? Not so much (we got 5.6.2 a couple years back, but will there be a 5.6.3?). 5.4? Forget about it. I can guess what’s supported because of my familiarity with the project, but who knows for sure? What does the community (read: perl5-porters) commit to continuing to fix and release? There is no official statement.

It would be really beneficial to know – that is, for an explicit maintenance policy to be articulated and maintained. Such a policy would allow third parties to know exactly what versions of Perl will continue to work and what versions will be deprecated and dropped. Of course, to do this realistically, it will have to get easier to deliver maintenance releases, and that means the project will have to…

Use minor versions for bug fixes only.

The fact that there are effectively no bug-fix-only releases of Perl is, in my opinion, a huge problem. Regressions can sit for months or even years with fixes without seeing a release. You can’t just tell people to apply a patch or rely on distribution packagers to fix up the patches (hell, certain packagers tend to break Perl by leaving such patches in place for years!).

So the Perl project needs maintenance branches that are actively maintained by back-patching all bug fixes as appropriate as they are committed to blead. The maintenance branches always ought to be in a state such that they’re identical to their initial releases plus bug fixes. This also goes for any dual-life modules: no new features, just bug fixes. By adhering to a strict regimen for maintaining such branches, the core team can tag a release at any time with just a few steps. Such will be important to fix serious security issues, bugs, or performance regressions in a timely manner, and will likely help prevent package maintainers from wandering too far from core releases.

Ideally, such branches would be for a major version number. For example, there would be a branch for 5.10 and one for 5.8. For the 5.10 branch, maintenance releases would be 5.10.2, 5.10.3, etc. If that’s not do-able because of the current practice of the minor release numbers actually being major releases, perhaps the branch would be 5.10.1 and maintenance releases would be 5.10.1.1, 5.10.1.2, etc. Such is the path the Git project follows, for example. Or perhaps we could change the numbers altogether: make the next major release “Perl 5 v10.1.0,” the maintenance branch v10.1, and the next maintenance release 10.1.2. The next major release would be 10.2.0 or, eventually, 12.0.0.

That last suggestion probably won’t fly, and the first option would, frankly, be more to my liking, but the point is to have some logical versioning system to make it easy to identify major releases and maintenance releases. Ultimately it doesn’t really matter what version numbers are used, as long as their use is consistent.

Update smoke testers to simplify regression tracking.

Like the PostgreSQL build farm, we need a way to quickly see what works and what doesn’t for all maintained versions of Perl. I’m not familiar with the smoke testing configuration, so maybe it does this already. But ideally, the system would be easy to set up, would check out and build every officially supported version of Perl, run the test suite, and send the results back to a central database. Via the interface for that database, you could see what versions and revisions of Perl are passing or failing tests on every reporting platform at any moment in time. And finally, you’d be able to see the full TAP output of all tests (or maybe just particular test scripts?) so that it’s easy to jump down into the test results and see failure diagnostics, to allow a developer go get an early start on fixing failures without having to ask the server owner to run the tests again.

Bonus points for plugging in results from cpan-testers for each version, too.

Fix and record as you go.

I alluded to this already, but it deserves its own section: Back-patch bug fixes to all appropriate maintenance branches as you go. And as you make those fixes, record them in a changes file, so that the release manager doesn’t have to dig through the commit logs to figure out what’s changed from version to version. The existing practice – where the Pumpking decides it’s time for a release and spends weeks or months cherry-picking fixes from blead and trolling through the logs for changes – just doesn’t scale: it puts all the work onto one person, leading directly to the very real possibility for burnout. Getting a release ready is hard enough without all the extra busy work. The only effective way to keep things up-to-date and well recorded at all times is to, well, keep things up-to-date and well recorded at all times.

If the project committers adhere to this practice, it will always be easy to get a maintenance release out with just a day’s worth of work – and perhaps less. If the code is always ready for release, it can always be released. Perhaps the smoke farm is given a day or two to show that there are no regressions, but otherwise, release early, release often.

The Goal

These are some of the lessons I take away from observing the differences between PostgreSQL development and Perl development. There are other changes that might be worthwhile, such as eliminating the overhead created by dual-life modules and articulating an explicit deprecation policy. Such issues have been covered elsewhere, however, and not practiced by or relevant to the PostgreSQL example.

As for the comparison, I recognize that there are no exact parallels (one hacker I know who has worked on both projects says that the PostgreSQL source code is a lot cleaner and easier to work with than the Perl soure, and therefore it’s easier to maintain and prep for release), but surely ideas can be borrowed and put to good use. Ultimately, I’d really like to see changes to the Perl development and release process to enable:

• More frequent stable releases of Perl
• More rapid development and delivery of major releases of Perl
• Less work and stress for core maintainers
• Greater predictability and accessibility for users

There’s a lot here, but if you take only two things away from this essay, let them be these suggestions:

1. establish a cabal to spread the burden of responsibility and decision making, and
2. maint should be much more conservative about changes

Both are very simple and very effective. What do you think?

My thanks to Bruce Momjian, Tim Bunce, chromatic, and Nicholas Clark for reviewing earlier drafts of this essay and providing invaluable feedback and suggestions – many of which I accepted. Any errors of course remain completely my own.

Looking for the comments? Try the old layout.

Tonight we had a Portland Perl Mongers Perl 5.10 code sprint. This was the first time I’ve ever worked with the Perl 5 core code, and while things weren’t exactly intuitive, it wasn’t bad to get started, either. I already had a clone of the Perl Git repository, so here’s what I did to get started.

Here’s how to clone the Git repository, and then check out the maint-5.10 branch and build and test Perl:

git clone git://perl5.git.perl.org/perl.git
cd perl
git checkout -b maint-5.10 origin/maint-5.10
sh Configure -Ode \
 -DDEBUGGING \
 -Dprefix=/usr/local/perl/blead \
 -Dusedevel \
 -Duseithreads \
 -Dccflags='-I/usr/local/include' -Dldflags='-L/usr/local/lib' \
 -Dlibpth='/usr/local/lib /usr/lib' \
 -Uversiononly \
 -Uinstallusrbinperl $@
make
make test

I had two test failures:

#   at ../lib/ExtUtils/CBuilder/t/00-have-compiler.t line 39.
#          got: '0'
#     expected: '1'
# Looks like you failed 1 test of 4.
FAILED at test 4

And:

#   at pod/pod2usage2.t line 235.
# Got:
# #Usage:
# #    This is a test for CPAN#33020
# #
# #Usage:
# #    And this will be also printed.
# #
# 

So the first thing I want to do is fix those failures. It took a bit of fiddling to figure out how to get it to run a single test. On the advice of Paul Fenwick and Schwern, I tried:

./perl -Ilib lib/ExtUtils/CBuilder/t/00-have-compiler.t

But then the test passed! So something has to be different when running under make test. With help from Duke Leto, I finally figured out the proper incantation:

make test TEST_FILES=../lib/ExtUtils/CBuilder/t/00-have-compiler.t

So the TEST_FILES option tells make test what tests to run, relative to the t/ directory. Shortly thereafter, Schwern told me about:

cd t
./perl TEST ../lib/ExtUtils/CBuilder/t/00-have-compiler.t

Which is nice, because it does a lot less work.

Now I was ready to figure out the ExtUtils::CBuilder bug and fix it. This is good, I thought, because ExtUtils::CBuilder is a Perl module, and while my C is teh suck, my Perl skillz are mad. So I do some poking around, and finally pinned down the failure to the have_compiler() method in ExtUtils::CBuilder::Base. A bit more poking and I had the error printed out:

# error building /tmp/compilet-1529033816.o from '/tmp/compilet-1529033816.c'
at ../../ExtUtils/CBuilder/Base.pm line 110.

Well fuck, that sure looks like a C problem. But then I added some more debugging code to see what command it’s actually running, and from where. So I added this code:

use Cwd;
print STDERR "# CWD: ", getcwd, $/;
print STDERR "# ", join( ' ', @cc, @flags), $/;

And then I got this output:

# CWD: /Users/david/dev/perl/lib/ExtUtils/CBuilder
# ./perl -e1 -- -I/Users/david/dev/perl/perl -c -fno-common -DPERL_DARWIN -no-cpp-precomp -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -O3 -o ./compilet-892348855.o

And of course, there’s no perl in lib/ExtUtils/CBuilder. Why does it think there is? The test file has this code:

$b->{config}{cc} = "$^X -e1 --";
$b->{config}{ld} = "$^X -e1 --";

Well, that cinches it. $^X is the path to the currently-executing Perl, but in core–for me at least–that’s a relative path. The test changes directories away from that path, and so it can’t find it. So I just needed to get it to use the proper Perl. Frankly, this test should fail on every platform that runs it, so I’m not sure how it got committed to the maint-5.10 branch, but here we are.

So I moved up use File::Spec; and assigned File::Spec->rel2abs($^X) to a variable before the chdir, and used that variable for the cc mock. And that was it, the test passed. Yay!

Now it was time to submit a patch to p5p. I committed the fix to my local clone, and then, with help from Jacob, generated a patch with git format-patch -1. The -1 means just a single commit, which is all I’d committed. This generated a file, named 0001-Fixed-failing-test-for-ExtUtils-CBuilder.patch. I edited it to add a comment to the effect that the patch is targeted at the maint-5.10 branch.

Next Jacob helped me get git configured to send mail through my gmail account. I just added these lines to ~/.gitconfig:

[sendemail]
        smtpserver = smtp.gmail.com
        smtpserverport = 587
        smtpuser = justatheory
        smtppass = idontthinkso!
        smtpencryption = tls

I then sent the patch with:

git send-email --to perl5-porters@perl.org 0001-Fixed-failing-test-for-ExtUtils-CBuilder.patch

And that was it! You can see the result here. I started to fiddle with the pod2usage failure, but it was a bit obscure for me. Perhaps next time.

And speaking of next time, I’m forking the Perl repository on GitHub, and will probably start to use my own repository for stuff (if it can be kept up to date; RJBS points out that it’s currently way behind the core repo). I’ll likely also create a topic branch before fixing a bug, and then send the patch from there. That way, when it’s committed to the remote branch in core, I won’t have to rebase my local copy: I can just delete the topic branch. This is a nice way to track individual bug fixes that I’ve worked on, while letting the maint-5.10 branch just track the core repository, with no local changes.

Looking for the comments? Try the old layout.