Testing

When backward compatibility became an advantage

Mon, 12 Jan 2026 12:07:49 -0500

Twenty years ago, I argued passionately that breaking backward compatibility was one of Drupal's core values:

The only viable long-term strategy is to focus exclusively on getting the technology right. The only way to stay competitive is to have the best product. [...] If you start dragging baggage along, your product will, eventually, be replaced by something that offers the same functionality but without the baggage.

I warned that preserving backward compatibility would be the beginning of the end:

I fear that this will be the end of Drupal as we have come to know it. Probably not immediately, maybe not even for several years, but eventually Drupal will be surpassed by technology that can respond more quickly to change.

Twenty years later, I have to admit I was wrong.

So what changed?

In 2006, Drupal had almost no automated tests. We couldn't commit to backward compatibility because we had no way to know when we broke it. Two years later in 2008, we embraced test-driven development.

Drupal's test code now exceeds production code by more than two to one. Source: Drupal Core Metrics.

By 2013, we had built up some test coverage, and with that foundation we adopted semantic versioning and committed to backward compatibility. It transformed how we innovate in Drupal. We can mark old code for removal and clear it out every two years with each major release. The baggage I feared never really accumulated.

Today, according to the Drupal Core Metrics dashboard, Drupal Core has more than twice as much test code as production code. I didn't fully appreciate how much that would change things. You can't promise backward compatibility at Drupal's scale without extensive automated testing.

Our upgrades are now the smoothest in the project's history. And best of all, Drupal didn't end. It's still a top choice for organizations that need flexibility, security, and scale.

I recently came across an interview with Richard Hipp, SQLite's creator. SQLite has 90 million lines of tests for 150,000 lines of production code. That is a whopping 600-to-1 ratio. Hipp calls it "aviation-grade testing" and says it's what lets a team of three maintain billions of installations.

I suspect our test coverage will continue to grow over time. But Drupal can't match SQLite's ratio, and it doesn't need to. What matters is that we built the habits and discipline that work for us.

In 2006, I thought backward compatibility would be the end of Drupal. In 2026, I think it might be what keeps us here for another twenty years.

Thank you to everyone who wrote those tests.

It does make me wonder: what are we wrong about now? What should we be investing in today that will slowly reshape how we work and become an obvious advantage twenty years from now? And who is already saying it while the rest of us aren't listening?

PHPUnit tests for Drupal

Mon, 31 Oct 2022 09:15:23 -0400

Writing tests can be hard, but the key is to get started and take that first step. When I start a new Drupal module, I begin with a simple test and make sure it passes. Once I have one simple test passing, it becomes easy to add more.

This page documents how I use PHPUnit with Drupal. Since I sometimes take breaks from working on my Drupal site, I've documented the process of installing, configuring, running and debugging tests as a quick reference. I often return to this guide to refresh my memory.

The most basic PHPUnit test for Drupal

The most basic PHPUnit test for Drupal looks something like this:

<?php
namespace Drupal\Tests\my_module\Functional;

use Drupal\Tests\BrowserTestBase;

class MyModuleTest extends BrowserTestBase {

  /**
  * Modules to enable.
  *
  * @var array<string>
  */
  protected static $modules = ['my_module'];

  /**
  * Theme to enable. This field is mandatory.
  *
  * @var string
  */
  protected $defaultTheme = 'stark';

  /**
  * The simplest assert possible.
  */
  public function testOne() {
   $this->assertTrue(TRUE);
  }
}

The test lives in docroot/modules/custom/my_module/tests/src/Functional/MyModuleTest.php.

Installing PHPUnit for Drupal

Drupal does not ship with PHPUnit out-of-the-box, so it needs to be installed.

The best way to install PHPUnit on a Drupal site is by installing the 1 package. It can be installed using Composer:

$ composer require drupal/core-dev --dev --update-with-all-dependencies

The command above will download and install PHPUnit, along with other development dependencies that are considered a best-practice for Drupal development (e.g. PHPStan).

Configuring PHPUnit for Drupal

Once installed, you still have to configure PHPUnit. All you need to do is set two variables in your shell:

SIMPLETEST_BASE_URL should be the URL of your site (e.g. http://localhost/).
SIMPLETEST_DB should be your database settings (e.g. mysql://username:password@localhost/database).

I use DDEV for my local development environment and follow these steps to log into DDEV's web server and configure the PHPUnit testing variables:

$ ddev ssh
$ export SIMPLETEST_BASE_URL="https://dri.es.ddev.site/"
$ export SIMPLETEST_DB="mysql://db:db@db/db"

Running PHPUnit test for Drupal

To run all PHPUnit tests in the ./docroot/modules/custom directory, run the following command from your web server's container:

$ ./vendor/bin/phpunit docroot/modules/custom

To run all tests from a specific class, use the following command:

$ ./vendor/bin/phpunit --filter MyModuleTest

To run a single test method from a specific class, use this command:

$ ./vendor/bin/phpunit --filter MyModuleTest::testOne

Debugging PHPUnit tests in Drupal

When tests fail, the default PHPUnit output can make it hard to debug the problem. A good first step is to enable the --debug flag:

$ ./vendor/bin/phpunit --filter MyModuleTest::testOne --debug

This shows all the steps your test goes through, including setup, data preparation, and each assertion. Often, this extra detail is enough to identify the problem.

If you still encounter unclear errors, like a "500 Internal Server Error", inspecting the response from your application can help. You can temporarily modify your test to print the response body or headers by adding this code:

$content = $this->getSession()->getPage()->getContent(); 
var_dump($content);

If PHP is configured to display a stack trace (by setting display_errors = On and error_reporting = E_ALL in php.ini), it will also appear in your terminal.

Drupal 7 testing: status update and next steps

Tue, 16 Jun 2009 15:07:57 -0400

The first version of Drupal's automated test framework was developed in 2004 by Moshe Weitzman. It started as a simple wrapper around the SimpleTest unit testing framework. Later, Thomas Ilsche and Rok Zlender extended it as part of the Google Summer of Code projects of 2005 and 2006. NowPublic and others continued to sponsor Rok's work into 2008. Today, Jimmy Berry is the principal contributor of the Drupal test framework, as well as the main developer and maintainer of Drupal.org's automated test infrastructure. Behind the scenes, Kieran Lal was instrumental in helping to ensure our test framework received financial support, project management, hardware resources, and server administrators.

In the February 2008, I declared that Drupal should embrace test-driven development, and shortly thereafter, we added the test framework to Drupal 7 core. In addition to a test framework for core, Jimmy Berry and Chad Phillips developed a second generation automated testing framework for drupal.org – so far, it has tested over 8500 unique patches on Drupal.org.

Yesterday, Jimmy announced that he will be working part-time as an intern for Acquia this summer. At Acquia, we like what Jimmy is doing, and because he has been looking for a financial sponsor for a while now, I decided Acquia should step up and help Jimmy for part of the summer. Frankly put, Jimmy's work has the potential of doubling the velocity of our developer community, and that is well worth sponsoring. Jimmy will spend his time with Acquia to continue his work on Drupal 7's test framework and the automated test infrastructure for drupal.org.

For well over a year now, core development has used a test-driven development strategy combined with automated testing. I think all core developers working on Drupal 7 will unanimously agree when I say that our test infrastructure has drastically improved our velocity and effectiveness. Overall, patches get accepted faster. The automated tests allow us to focus on the architectural and the algorithmic changes introduced by a patch, rather than having to worry about unexpected side-effects. This helps both patch reviewers and core developers contributing patches. Furthermore, thanks to the tests, we have a much better feel about the stability and the health of Drupal 7. I am optimistic that our code freeze period for Drupal 7 could be shorter than prior releases. As the project lead, our test framework helps me sleep better at night. The stability and health of our code base is important to me, but frankly, it is at least as important for the many Drupal users, or for those people and organizations looking to adopt Drupal.

By adopting a test-driven development strategy we raised the bar for core development, and I strongly believe we have to do the same for contributed modules. While the community cannot force module maintainers to write tests for their modules, I do want to support those that do write tests. The best way to support them, is to provide them the same tools and infrastructure that we use for core development. And as the maintainer of a contributed module myself (i.e. the Mollom module), I can't wait to have the same tools available that I have available when working on Drupal core.

Jimmy is working on making drupal.org's automated test infrastructure available to contributed modules, integrating the test results in the project pages as an incentive tool, improving the documentation to help people get started, and closing some functional gaps such as the ability to test Drupal's installer and upgrade system. We're also working on deploying the third generation of the automated testing framework. This new version will make it easier to add testing servers so we can triple the number of servers needed to run tests, and start testing the non-MySQL backends that are supported by Drupal 7.

All in all, I'm very optimistic that we'll succeed in our goal to adopt a test-driven development strategy.

Usability work at DrupalCon DC

Tue, 03 Mar 2009 23:12:54 -0500

I recently announced that one of the ways in which Acquia is contributing to Drupal is by providing funds for Mark Boulton and Leisa Reichelt to help the Drupal community improve usability in Drupal 7. Mark and Leisa are beginning their work at DCDC this week!

One of the things they're hosting, with the help of Jeff Noyes, are "Blue Sky Design Workshops". The workshops are Wednesday 3-4pm and Thursday 9-10am and 11:30-12:30am. The "Blue Sky" part of the title notes that these sessions are designed to push the boundaries of usability in Drupal, to get the community together, to brainstorm about new possibilities and to dream up solutions.

Furthermore, the Drupal usability team has just completed a second round of formal usability testing at the University of Baltimore and I'm really looking forward to learn more about the results of their work.

To keep up the involvement after DCDC, or if you're following along from home, consider joining this Mark Boulton D7 UX group as well as the regular usability group on groups.drupal.org for updates. Make sure to tune in!

Usability is one of those areas where we can really break new ground with our next release. We've already done much work, but have more to do. Through activities like this, and with the feedback from some "fresh eyes" through research and user testing, I'm very hopeful and excited about where we're headed.

We have a test framework in Drupal 7

Mon, 21 Apr 2008 11:10:13 -0400

Every major Drupal release we should ask ourselves what steps we can take to double the capacity of our community.

I spent the weekend in Paris where we had a two day code sprint. Our main accomplishment was getting Drupal's test framework into Drupal core – the culmination of three years of hard work carried out by many people and companies in the Drupal community.

Here is why a strong investment in testing will help double the capacity of our community:

For developers, upgrading, maintaining and releasing your modules becomes easier. The combination of test results and code coverage reports makes it easier to determine the release readiness of your code. This translates to fewer betas, shorter code freeze periods and more frequent releases. Furthermore, design for testability leads to easier to use and more complete APIs. It is guaranteed to make Drupal a better development platform.
For end users, it is important that we provide quantifiable reporting on the health of Drupal core and the many contributed modules.
For patch reviewers, tests are great because it allows them to focus on the architectural and the algorithmic changes that the patch introduces. With good test coverage, we can rely on the tests to discover any unwanted side-effects. Patches can be committed faster.
For people new to Drupal, tests lower the barrier to entry and encourage collaboration and innovation, two of Drupal's core values. With good test coverage, you don't necessarily have to understand the entire code base before you can comfortably propose changes or help maintain a module.
For me, it means I'll sleep better at night. ;-) With hundreds of thousands of people using Drupal, the availability of a test framework takes some pressure of my shoulders.

As of today, it is expected that you submit test cases with your patches for Drupal 7. Writing good tests takes time: it is not unlikely that you'll spent twice as long working on a patch. This might take some time getting used to but you'll find that it pays off and that it is actually very rewarding.

Thanks to Ori Pekelman of AF83 for being such a great host when in Paris. And a special thanks to everyone who helped get the test framework in Drupal 7.

First results from usability testing

Mon, 10 Mar 2008 05:48:25 -0400

Two weeks ago, we conducted formal usability testing for Drupal at the University of Minnesota Office for Information Technology's usability lab. As I wrote earlier, the university has a professional usability lab that allowed us to record eye-tracking data and video.

The heat map shows where the users look the first 5 seconds after landing on Drupal's main administration page. The red X's show where the users clicked.

The blue dot in the video shows what the user is looking at on Drupal's user account creation page. Several users backed out in a panic because they thought they caused an error condition due to the red text.

Seeing users consistently fail at what we consider to be basic tasks (e.g. creating a new content type or adding a new user account) is a true eye opener. Let's be clear about this: this is Drupal's fault, not the users' fault. The good news is that we came out of this with a long list of usability problems that we can fix.

In fact, the Drupal team that participated in Minnesota tried to process much of the data prior to our DrupalCon Boston presentation on usability testing: check out our preliminary report from the usability testing (PDF, 6MB) or participate in the Drupal usability group where more information and results will be made available.

Massive thanks to Chad and Cody of the University of Minnesota for making this possible, and a big thumbs up for all the Drupal participants that are committed to making Drupal easier to use. Let's start addressing those problems!

Embracing test-driven development

Fri, 15 Feb 2008 02:37:31 -0500

At DrupalCon Barcelona about 20 people gathered together for the Quality Assurance Birds of a Feather session. This led to the development of a continuous testing infrastructure for Drupal. People that want to write tests for Drupal can, but don't have to.

Second, in my State of Drupal presentation at DrupalCon Barcelona, I shared a list of the top-10 most requested Drupal improvements. On spot 9 we had "better internal APIs" and on spot 10 we had "better external APIs (import/export, web services)". APIs that are designed for testability, are often better APIs. This comes from the fact that testability leads to more cohesive code with loose coupling. Important for readable, agile code.

Third, Drupal 6 has been in a code freeze for 6 month. Often times, the behavior introduced by one patch inadvertently breaks the behavior of existing functionality. In absence of tests, it takes a while to shake out all the bugs and before you're actually confident to roll a new major Drupal release.

As a result, people in the community have suggested to put more emphasis on testing in Drupal. Writing tests is expensive so it is hard to model the cost and benefits. However, the only way to past the costs and to experience the true benefits, is to take testing to the extreme. There is no middle ground. If we want to be serious about testing, we need to enforce, not just encourage, testing in the Drupal 7 development cycle and beyond. (And maybe we should consider introducing tests for Drupal 5 and Drupal 6 maintenance releases.)

In such a world, patches that fix bugs should come with a test case that demonstrates the behavior being fixed. That patch would then be added to the regression test suite and run continuously to avoid that the bug gets re-introduced. In addition, patches that add new features or enhancements should come with test cases for this new functionality. Everything should be tested.

Of course, this would require a significant commitment from the whole Drupal developer community. There is bound to be some controversy about the need to learn how to use Drupal's test framework and the overhead and complications of having to write and submit a test. But over time, fewer people will be frustrated by bugs, the burden on patch reviewers will go down, code freezes will become shorter, APIs will continuously improve, and we would have more confidence in the stability of our releases. Last but not least, it actually improves overal productivity and encourages change and experimentation – something the Drupal community really takes pride in.

As we learned before: sweeping changes are required to make major advances in technology, and often times there is a lot of pain before the pay-off. Are we willing to take some pain? Are we willing to enforce tests to be written as part of the Drupal 7 development process? I'm willing to fully embrace testing, but only if you are willing to write tests ...

Drupal usability testing

Sun, 03 Feb 2008 07:58:09 -0500

When I predicted that there would be a big and concentrated effort to further improve Drupal's ease of use in 2008, I was cheating a bit ... The past months we've been preparing some formal usability testing for Drupal that will be conducted at the University of Minnesota Office for Information Technology's usability lab. The university has a professional usability lab that will allow us to record eye-tracking data and video which will be provided to the Drupal community. Très cool!

Chad Fennel and Cody Hanson of the University of Minnesota Libraries have taken an incredible lead in this; they secured the lab, they prepared test scenarios, they recruited test users, they are coaching us how to do usability research, and last but not least, they are footing part of the bills. Needless to say, this is a significant contribution to the Drupal project – and hopefully one that will have big impact on Drupal 7.

Needless to say, formal usability testing is only one way to make Drupal easier of use. There are a lot of usability findings out there already (example), and many of us know first-hand which changes need to occur to improve Drupal's usability. All it takes is a knack for usability, some development skills and a lot of time and effort. Ultimately, everyone can help – with or without a professional usability lab.

The past years, I've pushed hard to make usability reviews part of the development and patch review process. I strongly believe that usability reviews should be tightly integrated with the software development process; rapid usability feedback and incremental usability improvements allows us to build better software faster. And it is working – every major Drupal release has become significantly easier to use.

Nonetheless, there is a lot of work left to be done. I hope that the formal usability testing at the University of Minnesota Libraries will provide us with a fresh perspective and that it helps train the Drupal community's eye for usability. Because more than anything else, I want to us to flatten the Drupal learning curve.

More information about the usability testing will be made available shortly. The goal is to share some first results at DrupalCon Boston next month.