Mnmlst Dvlpr

Stop rerunning your tests

2018-09-08T00:00:00+00:00

Tests are usually the longest running operation in your development process. Running them unnecessarily is the ultimate time waster. Gradle helps you avoid this cost with its build cache and incremental build features. It knows when any of your test inputs, like your code, your dependencies or system properties, have changed. If everything stays the same, Gradle will skip the test run, saving you a lot of time.

So you can imagine my desparation when I see snippets like this on StackOverflow:

tasks.withType(Test) {
    outputs.upToDateWhen { false }
}

Let’s talk about what this means and why it is a bad idea.

Communicating intent

The above snippet just says “Never reuse this test’s output”. But why? Is it because there is some hidden input that Gradle doesn’t know about? Or is it because the test produces random outputs? The reader can’t tell.

Deterministic tests don’t need reruns

“Insanity is doing the same thing over and over and expecting different results”

- Not Albert Einstein

The vast majority of your tests should be deterministic, i.e. given the same inputs they should produce the same result. If this is not the case, your project is in serious trouble. Stop reading this post and start fixing your code!

Rerunning deterministic tests is a waste of your team’s time.

Non-deterministic tests

There are a few reasons why you might want to rerun some tests in some cases even though none of the code changed. In these cases, you should model the additional input properly. Tell Gradle what exactly makes your tests non-deterministic.

Randomized tests

Some tests use randomization to improve the quality of your software.

Random testing can be used to ensure that the production code can handle all kinds of inputs, not just the ones that the developer came up with.
Mutation testing changes the production code in subtle way (e.g. introducing off-by-one errors) and checks whether your test suite catches these mistakes.

Make this randomization explicit by making the random seed an input to your task:

task randomizedTest(type: Test) {
    systemProperty "random.testing.seed", new Random().nextInt()
}

This will force Gradle to always rerun that test, because it will always have a different seed. Even better, you could make the seed user-configurable to locally reproduce bugs that were found on your build server.

System integration tests

System integration tests verify your application against realistic versions of other applications that are controlled by other teams. They may fail even if you didn’t change anything, e.g. because another team broke an API. They also tend to be among the slowest tests in your code base, so you don’t want to rerun them just because you changed some documentation. A good compromise may be to check integration at least once a day, even if nothing on your side has changed.

Make this interval part of your test inputs:

task systemIntegrationTest(type: Test) {
    inputs.property "integration.date", LocalDate.now()
}

You can then set up an automated build that runs this test in the morning before everyone starts working and let it push the result to a shared build cache. When your team comes to work, the test result will be downloaded from the cache and the tests won’t have to rerun for that day. They can then use the saved time on more productive tasks like fixing bugs or developing new features.

Flaky tests

Sometimes you encounter a bug that only makes a test fail 1 out of 10 times. In order to analyze the situation, you want Gradle to rerun the test even if it was successful before. In this case it is reasonable to use the random input approach I showed above. However, I find it more productive to wrap the flaky test in an endless loop. This way I can keep the debugger in the IDE running and even make small changes on the fly without having to restart.

Model your tests properly

Properly modeling your requirements is just as important for your build logic as it is for your production code. Doing it right will make your builds faster and your team more productive.

The java plugin’s built-in test task is meant for deterministic and fast unit tests. It may seem convenient to put all your tests into this one task and rerun them on every build, because a few of them are non-deterministic. This lazy approach will save you a few minutes of thinking and coding, but the long term costs are huge, slowing everyone on your team down every day.

Instead, create additional Test tasks for different types of tests like functional, performance, or randomized tests. For each one of them, consider when it needs to rerun and model that as an input to the task.

Don’t waste your time.

Performance testing at Gradle

2018-05-10T00:00:00+00:00

Performance is one of Gradle’s most important features. This is why we go to great lengths to ensure that Gradle stays fast and improves with every release. I’m proud of the performance testing infrastructure that we’ve built over the last two years and would like to give you a tour of it.

Overview

We have about 150 performance tests in two categories: Regression tests and experiments.

Regression tests compare the current code against a previous version of Gradle. They fail as soon as a statistically relevant difference is detected. Most regression tests are run on every commit and need to pass before a branch can be merged.

Experiments compare different ways of using Gradle, e.g. comparing parallel vs. non-parallel builds or measuring the overhead of the build cache. They can either have hand-picked thresholds like “the cache overhead should be no more than X percent” or no threshold at all, serving only an informational purpose. Most experiments are run once a day, as they usually excercise more complex scenarios that take longer to run.

The tests are distributed over a farm of build agents to achieve an acceptable execution time.

The test results are stored in a database so we can render historical reports and see both positive and negative trends. The following graph shows the recent history of how long it takes to run gradle idea on a 500-subproject Java build.

Once a week, we run a historical build that compares the current version of Gradle against a wide selection of older versions. This shows us the bigger picture and allows us to measure how much we’ve improved, not just in the last release, but over the last year or more.

Scenarios

Every scenario consists of a test project and some tasks to run. Some scenarios also do intermediate steps like changing a source file or cleaning the cache. A typical performance test definition might look like this:

There are two kinds of test projects: Fully synthetic and representative.

Fully synthetic projects are generated to show how fast an ideal usage of Gradle can be. They are usually very big and used to explore scaling bottlenecks in Gradle.

Representative projects try to mimic some of the real-world builds we’ve seen at our customers, without exposing any of their proprietary details. These builds give us a reality check on which non-ideal usage patterns we should speed up or prevent.

Distributed execution

Running all our performance scenarios on one machine would take more than half a day. We want to run most of them on every commit, so we need to go much faster. We do this using a “coordinator” build, which discovers all the scenarios and schedules individual builds for each of them. The coordinator uses TeamCity’s REST API to schedule and wait for the scenarios. Once they are finished, the coordinator gets the scenario results and synthesizes a test report. From the TeamCity UI it looks just as if all the tests had run on the same machine. You can browse executed tests, see their execution time and how often they’ve failed in the past.

All the build agents are automatically managed, dedicated servers with exactly the same hardware and software configuration. This ensures that the results are reproducible. The full regression test suite takes about an hour to run, though the target is to get it down to 30 minutes.

Regression detection

For regression tests, we run the current code against a baseline, usually some recent nightly build that is known to be well-performing. Each version gets its own user home, temporary directory etc. to ensure isolation. We even go so far as to pad folder names so that each version under test needs to use the same amount of memory for its file system cache keys.

Each version is given ample warmup time and is then usually measured for 40 iterations. Afterwards, we use a Mann-Whitney u test to determine whether the current version is significantly slower than the baseline. If so, the test fails and someone has to investigate.

We require a confidence of 99%, so there is still a 1% chance that a test may fail due to pure bad luck. This is the price we’re ready to pay in order to detect even the smallest regressions. The infrastructure reliably detects performance regressions of as little as 1%. That means if a commit makes a 5s build just 50ms slower, the performance test will already sound the alarm.

Profiling

It is not always obvious what caused a performance regression. A commit may look unsuspicious at first, but have some unintended side effect. A regression may have gone undetected because it was very small and now another, equally small regression pushed the test over the edge. In both cases, the committer needs more data to find out what the problem is.

For this purpose, we can run any scenario with the Java Flight Recorder attached. This gives us highly accurate data on CPU usage, memory allocation, file and network IO, and blocked threads. The data is visualized using the jfr-flamegraph converter and the FlameGraph tool. You can zoom and search the stacks and quickly explore both the top-down (flame graph) and bottom-up (icicle graph) resource usage of the version under test. For instance, here is an icicle graph showing the allocation hot spots when calling gradle help on our own build. It is a simplified version since the fully detailed one is over 2MB large.

We had recently migrated most of our scripts from Groovy to Kotlin and didn’t see the full performance benefit we expected. A look at the graph quickly shows that inefficient classloader caching was to blame. This analysis took no more than a few minutes and uncovered an issue that may otherwise have led to disappointed users.

For each tested version, we also create a diff against every other tested version. This usually makes it pretty easy to spot where things got slower or faster. For instance, this flame graph diff shows what happened when we removed the rule-based model bridging from projects that aren’t using the rule-based model.

You can clearly see task creation in dark blue, meaning it is much faster now. Red, on the other hand, would mean that something got significantly slower. You can also look at the fully detailed version (1 MB).

Coming to you soon

The flame graphs you saw above will soon be added to the Gradle Profiler, a tool that deserves its own blog post in the future. With it, you’ll be able to easily create the same visualizations for your own performance scenarios. This will allow both plugin authors and build engineers optimize their to algorithms for the fastest possible builds.

The Gradle team will, of course, keep pushing for ever-faster builds. Gradle 4.8 will contain some nice configuration time improvements for large Android builds. These improvements were only possible thanks to JFR snapshots generously provided by our customers and the great work on flame graphs done by Brendan Gregg and Chrishantha Perera. Thank you so much!

The build-changing magic of tidying up

2018-03-21T00:00:00+00:00

Organizing your build logic properly can make the difference between a tool that your developers will love or a mess of spaghetti code that no one dares to touch.

Gradle offers different ways to organize build logic:

Scripting in a project’s build script
Writing scripts that are applied to multiple projects
Writing binary plugins in buildSrc
Writing binary plugins shared through a repository with other teams

Unfortunately, I often see users not making the leap from step 2 to step 3 as early as they should. Scripts should only contain declarative configuration. As soon as you need control flow, complex computations or external dependencies, you should move that logic to buildSrc

buildSrc is a special project which is built before your settings and build scripts are executed. This means it can contribute additional plugins and tasks that your build scripts can use. You can write these types in any JVM language you like. The dependencies of the buildSrc project are visible to all build scripts in your project.

Let’s see how these properties make buildSrc such a great tool.

Plugin classpath management

Without buildSrc

If you have ever tried splitting a complex build script into multiple smaller ones you may have run into classloading problems. Project build scripts cannot see classes loaded by scripts that they apply.

For instance, you may have a spring-boot.gradle script that loads the Spring Boot plugin and configures some defaults. Already there are some oddities - you have to apply the plugin by its class name instead of id.

spring-boot.gradle

buildscript {
    repositories {
        gradlePluginPortal()
    }
    dependencies {
        classpath 'org.springframework.boot:spring-boot-gradle-plugin:2.0.1.RELEASE'
    }
}

apply plugin: 'org.springframework.boot.gradle.plugin.SpringBootPlugin'

springBoot {
  //set some defaults
}

But even worse - when you try referencing any of the Spring Boot plugin’s types in a project script, you’ll be greeted with script compilation errors.

some-project.gradle

apply from: 'spring-boot.gradle'

tasks.withType(BootRun) { //won't compile :(
  //some project-specific config
}

With buildSrc

buildSrc on the other hand exposes all its types and dependencies to all scripts in your project. This means that buildSrc is the perfect place to centrally manage all your plugin dependencies.

buildSrc/build.gradle

apply plugin: 'java-library'
apply plugin: 'java-gradle-plugin'

repositories {
    gradlePluginPortal()
}
dependencies {
    api 'org.springframework.boot:spring-boot-gradle-plugin:2.0.1.RELEASE'
}

The two scripts from earlier can now be simplified and will work just as you’d expect.

spring-boot.gradle

apply plugin: 'org.springframework.boot'

springBoot {
  //set some defaults
}

some-project.gradle

apply from: 'spring-boot.gradle'

tasks.withType(BootRun) {
  //some project-specific config
}

IDE support

The example above still uses a script for configuring the Spring Boot defaults. I highly recommend moving this logic into buildSrc as soon as it gets more complex. Writing plugins in buildSrc gives you much better IDE support than writing scripts. Your IDE will offer you syntax highlighting, auto completion, source navigation and refactoring.

Don’t be afraid to make the move. Both scripts and plugins use the same Gradle API. A script is just a plugin without the class/method boilerplate. Here is how that plugin could look like when copy-pasted to buildSrc.

buildSrc/src/main/groovy/MySpringBootPlugin.groovy

class MySpringBootPlugin implements Plugin<Project> {
  void apply(Project project) {
    project.with {
      apply plugin: 'org.springframework.boot'
      springBoot {
        //some complex team/company-specific defaults
      }
    }
  }
}

You can then give this plugin an id and use the much nicer plugin DSL to apply it.

buildSrc/build.gradle

gradlePlugin {
    plugins {
        springBoot {
            id = 'spring-boot'
            implementationClass = 'MySpringBootPlugin'
        }
    }
}

some-project.gradle

plugins {
  id 'spring-boot'
}

tasks.withType(BootRun) {
  //some project-specific config
}

Testing

Another reason to keep complex logic out of scripts is that scripts cannot be tested. buildSrc allows you to write unit tests for low-level implementation details and functional tests for the user-facing behavior of your plugins and tasks. This is essential as your build grows more complex, as a good test suite allows you to refactor without fear of breaking your developers’ workflow.

Performance

While dynamic Groovy allows you to build very succinct and readable DSLs, it also has a rather high execution overhead. If you have a large project and scripts that you apply to all subprojects, consider using statically compiled buildSrc plugins instead. Of course, you could just switch the script over to the statically compiled Kotlin DSL. But moving it to buildSrc and turning it into a proper plugin gives you all the other benefits mentioned above. And if neither Kotlin nor Groovy are your cup of tea when it comes to writing complex logic, you can write it in plain old Java too. Any statically compiled language is great for performance.

For instance, here is the plugin from earlier, translated to Java.

buildSrc/src/main/groovy/MySpringBootPlugin.java

public class MySpringBootPlugin implements Plugin<Project> {
  public void apply(Project project) {
    project.plugins.apply("org.springframework.boot");
    project.extensions.configure(SpringBootExtension, () -> {
      //some complex team/company-specific defaults
    });
  }
}

Once you have extracted plugins to buildSrc, the next step towards making them a standalone project shared with other teams is much simpler. You’ll already have the project structure in place and a good suite of tests to show others what to expect of your plugin.

Readability

Extracting complex logic into buildSrc and only leaving declarative logic in your build scripts makes it much easier for others to customize the few details they care about. Don’t make your team members read a bunch of unrelated code just to change a dependency or add another subproject.

Conclusion

Properly structuring your build takes some discipline. You need to treat your build logic the same way you would write your production code. Complex control flow should be well-tested and remain separate from declarative configuration. In return you don’t just get a more manageable build, but better IDE support, performance and convenience. I hope I could inspire you to give your build a fresh look and ask yourself: “What will I extract today?”

Mnmlst Dvlpr

Stop rerunning your tests

Communicating intent

Deterministic tests don’t need reruns

Non-deterministic tests

Randomized tests

System integration tests

Flaky tests

Model your tests properly

Performance testing at Gradle

Overview

Scenarios

Distributed execution

Regression detection

Profiling

Coming to you soon

The build-changing magic of tidying up

Plugin classpath management

Without buildSrc

With buildSrc

IDE support

Testing

Performance

Sharing

Readability

Conclusion