Distributing Test Execution with SBT

The Problem: Long Time Test Execution in CI/CD

Test suites have grown in size and complexity. What used to be a quick validation step has become a bottleneck that can significantly impact development velocity. CI/CD pipelines that take a long time to complete are delaying feedback and deployments, creating frustration for developers and slowing down the entire development process. Here is the reality many teams face:

Large test suites with hundreds or thousands of tests
Long-running integration tests that interact with databases, APIs, or external services
CI/CD pipelines that take 20+ minutes to complete, delaying feedback and deployments

The Solution: Distributed Test Execution

The solution is to distribute test execution across multiple machines, running different test groups in parallel. This approach can reduce test execution time by 70-90%.

Results may vary

Key Benefits

Significantly faster feedback loops and shorter deployment cycles - From 20 minutes to 4 minutes in our example
Scalable - Easy to adjust number of runners based on test suite size and available resources

Implementation: `The parTestGroup` SBT Task

The core of this solution is a custom SBT task that divides tests into groups and allows running specific groups independently.

The SBT Task Implementation

1// Define a custom task
2lazy val parTestGroup = inputKey[Unit]("Runs a single test group")
3parTestGroup := (Def.inputTaskDyn {
4  // Takes two parameters: groupId (which group to run) and numberOfGroups (total groups)
5  val List(groupId, numberOfGroups) = complete.DefaultParsers
6	.spaceDelimited("<arg>")
7	.parsed
8	.map(_.toInt)
9
10  // Retrieves all available tests
11  val allTests = (Test / definedTests).value
12
13  // Calculates how many tests should be in each group
14  val numberOfTests = allTests.size
15  val numberOfTestsPerGroup =
16	if (numberOfTests % numberOfGroups == 0) {
17  	numberOfTests / numberOfGroups
18	} else { (numberOfTests / numberOfGroups) + 1 }
19
20  // Divides tests into groups
21  val groups = allTests.grouped(numberOfTestsPerGroup).toArray
22
23  val groupToRun 	= groups(groupId - 1)
24  val argForTestOnly = " " + groupToRun.map(_.name).mkString(" ")
25
26  streams.value.log.info(s"Running testOnly:$argForTestOnly")
27
28  // Runs only the specified group using SBT's testOnly task
29  Def.taskDyn {
30	(Test / testOnly).toTask(argForTestOnly)
31  }
32}).evaluated

Usage Examples

Example 1: Split tests into 3 groups and run the first group

1sbt "parTestGroup 1 3"

1sbt:root> parTestGroup 1 3
2[info] Running testOnly: io.github.stivens.example.suites.TestSuite5 io.github.stivens.example.suites.TestSuite7 io.github.stivens.example.suites.TestSuite2 io.github.stivens.example.suites.TestSuite6

Example 2: Split tests into 3 groups and run the third group

1sbt "parTestGroup 3 3"

1sbt:root> parTestGroup 3 3
2[info] Running testOnly: io.github.stivens.example.suites.TestSuite8 io.github.stivens.example.suites.TestSuite3

Complete CI/CD Implementation

Multi-Machine Workflow Structure

1jobs:
2  # Phase 1: Compile once
3  compilation:
4    name: Compile the project
5    runs-on: ubuntu-latest
6    timeout-minutes: 15
7    steps:
8      - name: Checkout
9        uses: actions/checkout@v4
10      - name: Setup JDK 21, Scala, SBT
11        uses: ./.github/actions/setup-scala
12      - name: Compile
13        uses: ./.github/actions/compile
14
15  # Phase 2: Run test groups in parallel
16  tests-group-1:
17	name: Run tests (1 of 10)
18	needs: [compilation]
19	uses: ./.github/workflows/run-tests-group.yml
20	with:
21  	group_id: 1
22  	num_groups: 10
23
24  tests-group-2:
25	name: Run tests (2 of 10)
26	needs: [compilation]
27	uses: ./.github/workflows/run-tests-group.yml
28	with:
29  	group_id: 2
30  	num_groups: 10
31
32  # ... additional groups ...
33
34  # Phase 3: Aggregate results
35aggregate-all:
36    name: compile and test the project
37    runs-on: ubuntu-latest
38    needs: [compilation, tests-group-1, tests-group-2, tests-group-3, tests-group-4, tests-group-5, tests-group-6, tests-group-7, tests-group-8, tests-group-9, tests-group-10]
39    if: ${{ always() }} # do not skip if one of the actions above has failed!
40    env:
41      everything_ok: ${{ !contains(needs.*.result, 'failure') && (!contains(needs.*.result, 'skipped') || needs.compilation.result == 'skipped') }}
42    steps:
43      - name: The workflow has succeeded
44        if: ${{ env.everything_ok == 'true' }}
45        run: exit 0
46      - name: The workflow has failed
47        if: ${{ env.everything_ok == 'false' }}
48        run: exit 1

1# .github/workflows/run-tests-group.yml
2on:
3  workflow_call:
4    inputs:
5      group_id:
6        required: true
7        type: string
8      num_groups:
9        required: true
10        type: string
11  
12jobs:
13  run_tests_group:
14    name: tests
15    runs-on: ubuntu-latest
16    timeout-minutes: 10
17    steps:
18    - uses: actions/checkout@v4
19    - uses: ./.github/actions/setup-scala
20    - uses: ./.github/actions/restore-compilation-cache
21    - name: Run tests
22      run: sbt 'parTestGroup ${{ inputs.group_id }} ${{ inputs.num_groups }}'

Performance Results

In our demonstration project with 70 test suites - each containing a simulated long-running test - the results are as follows:

Single-machine execution takes ~20 minutes

Parallel execution completes in ~4 minutes

Screenshot: Single-machine execution

Screenshot: Parallel execution

The Problem with Naive Distribution

When each machine independently compiles the entire codebase, you're essentially paying for the same compilation work multiple times across different runners. This not only increases your total billing time but also wastes valuable computing resources. Furthermore, each machine must download and resolve all dependencies from scratch, creating redundant network traffic and extending setup times. The result is a distributed system that performs more work than necessary, negating the potential benefits of parallelization.

Compile Once, Distribute Everywhere

Our implementation uses a two-phase approach:

The compilation job:

Restores cache
Runs incremental (re)compilation
Uploads build artifacts using GitHub Actions cache

Each test group job:

Ensures compilation completes first
Downloads cached compilation results
Runs only tests - no compilation overhead

Understanding Amdahl's Law and Test Optimization Limits

What is Amdahl's Law?

Amdahl's Law is a fundamental principle in parallel computing that describes the theoretical speedup achievable when parallelizing a computation. The law states that the overall speedup is limited by the portion of the computation that cannot be parallelized.

Key Formula:

1Speedup = 1 / ((1 - P) + P/N)

Where:

P is the proportion of the computation that can be parallelized
N is the number of processors/machines
(1 - P) is the sequential portion that cannot be parallelized

Implications for Test Distribution

When distributing test execution, Amdahl's Law reveals some limitations:

Example Scenario:

You have a test suite with 100 tests
99 tests take 1 minute each (parallelizable)
1 test takes 15 minutes (sequential bottleneck)

Sequential Execution: 99 + 15 = 114 minutes

Parallel Execution (10 machines):

99 tests distributed across 10 machines: ~10 minutes
1 long test still takes 15 minutes (cannot be parallelized)
Total time: 15 minutes (limited by the 15-minute test)

The Reality Check: Even with infinite parallelization, you cannot achieve execution time below 15 minutes because that's the duration of your longest sequential test.

If you have a problem with a long-running individual test, you should look for optimization opportunities in:

The source code of the test itself - Are there inefficient test setup/teardown operations?
The code being tested - Is the application code performing unnecessary operations, making slow API calls, or using inefficient algorithms?
Consider test isolation - can long tests be broken into smaller units?

Beyond SBT: Considering Bazel for Advanced Optimization

This article focuses on SBT and provides a simple, practical solution that should be applicable to every project using SBT. While our approach delivers significant performance improvements through distributed test execution, some projects may require further optimizations — particularly for build parallelization and advanced incremental compilation. In such cases, tools like Bazel can provide additional benefits.

If you're interested in exploring how Bazel can help overcome monorepo challenges and achieve even more efficient builds, we recommend reading our article: Overcoming monorepo challenges with Bazel for your projects The article covers real-world examples of how Bazel transforms large codebases into manageable, efficient ecosystems.

Conclusion

By implementing the parTestGroup task and leveraging test distribution, teams can achieve significant performance improvements — reducing test execution time from 20 minutes to just 4 minutes in our example, representing a 80% reduction in feedback time.

Whether you're dealing with hundreds of unit tests or complex integration test suites, this pattern can help you move faster and more efficiently, ultimately improving your team's development velocity and deployment confidence.

Links and Resources

Full Working Example: You can find a complete, working implementation of this approach on GitHub

Distributing Test Execution with SBT: A Complete Guide to Parallel CI/CD