Forcefully deterministic unit testing

Programming language design

I’m working a general-purpose programming language on my free time. I’m writing about the design of unit tests. Any mentions of “test” in the article refer to unit tests.

We’ll consider a unit test any test which does not interact with anything outside of the program. Reading a file? Forbidden. Writing to the console? Not allowed. No changes can be observed by running it.

It’d be unproductive to talk about testing without providing examples. Since the ideas I want to talk about are not specific to my language, we can imagine I’m building a new Typescript runtime, for the sake of the examples.

Hello world!

import { console } from "imaginary-standard-library";

function main() {
  console.log("Hello world!")
}

The example above assumes a couple of rules are in place:

• There’s no implicitly available global variables. We need to import `console`.

• When running a program, the runtime looks for the function named “main” and starts there.

Because we love tidy code, we’re going to move the business logic of our application into a separate function.

import { console } from "imaginary-standard-library";

function main() {
  greet()
}

function greet() {
  console.log("Hello world!")
}

Now we want to create a test for our business logic. Let’s introduce more rules to our imaginary runtime:

• Tests are all functions whose name starts with “test_”.

• Tests can be in any file. They can be in the same file as the functions being tested.

That should be enough to add our test:

import { console } from "imaginary-standard-library";

function main() {
  greet()
}

function greet() {
  console.log("Hello world!")
}

function test_greet() {
  greet()
}

Our test doesn’t test anything, besides the fact that function doesn’t crash! 😱

Even if it was testable, we haven’t introduced assertions yet. Because we don’t want `assert` being used outside of test code, it’ll be introduced within a function parameter, which we’ll call testkit.

import { console } from "imaginary-standard-library";
import { TestKit } from "imaginary-standard-library/test";

function main() {
  greet()
}

function greet() {
  console.log("Hello world!")
}

function test_greet(testkit: TestKit) {
  greet()
  testkit.assert.equals(true, true)
}

Now let’s get back to making our code testable. In order to do so, we need to introduce some interface to abstract away the console.

interface Console {
  log(input: string): void;
}

Now we can make our function testable:

import { console } from "imaginary-standard-library";
import { TestKit } from "imaginary-standard-library/test";

interface Console {
  log(input: string): void;
}

function main() {
  greet({ log: console.log })
}

function greet(console: Console) {
  console.log("Hello world!")
}

function test_greet(testkit: TestKit) {
  let fakeStdOut: String[] = []
  let fakeConsole = {
    log(input: String) {
      fakeStdOut.push(input)
    }
  }
  greet(fakeConsole)
  testkit.assert.equals(["Hello world!"], fakeStdOut)
}

We’ve achieved our goal of having our function testable. All it took was having interactions with the outside world abstracted in an interface. This is a pattern that can be used in nearly any programming language.

The success of this is still hinging on your discipline, of not having any outside world interactions being directly used. But we could enforce that responsibility, if it’s up to the runtime to pass it as a function parameter to main. So we move all of the functionality to interact with the outside world into an OutsideWorld instance and remove the ability to import things like console directly.

import { OutsideWorld, Console } from "imaginary-standard-library";
import { TestKit } from "imaginary-standard-library/test";

function main(outsideWorld: OutsideWorld) {
  greet(outsideWorld.console)
}

function greet(console: Console) {
  console.log("Hello world!")
}

function test_greet(testkit: TestKit) {
  let fakeStdOut: String[] = []
  let fakeConsole = {
    log(input: String) {
      fakeStdOut.push(input)
    }
  }
  greet(fakeConsole)
  testkit.assert.equals(["Hello world!"], fakeStdOut)
}

Now that the only place where the OutsideWorld instance comes from is the main function argument, we can no longer pass it to tests. This means all tests no longer have the ability to do anything related to the outside world.

As long as the programming language author keeps all of the functions that can interact with the outside world in the OutsideWorld instance, no discipline is required from the programmer anymore.

Until concurrency primitives are introduced, the tests are deterministic. Forcefully so. By design. That means I can reliably cache the test results and users of my language only run the relevant tests when code changes.

As a side-note, it would be annoying to write the fake instances all the time. I provide a fake OutsideWorld instance in the tests alongside other functions to make most things easy to test.