Sometimes you may need to test that an asynchronous event has been triggered in
your code but you can end up compromising on the quality of your tests when
This post will show one technique to test this behaviour whilst keeping your tests clean and not complicating your production code
It's assumed you're familiar with writing tests, interfaces, go routines, channels and
select from Go.
The code to test
We are writing a service which has a dependency of a repo passed to it. Plain and simple DI for separation of concerns.
The service has a method
FaveAndSave which takes a
string and it will do some amazing business logic and then save to the repo.
The code calls the
repo in a new go routine which makes testing a little tricky.
Let's test it
The benefit of using DI here is that we can now stub our
Repo, make it behave how we like and see how it's called.
This test fails wrongly because the save event is fired off in a separate go routine
which ends up being executed after our assertion.
This means the
saveCalled value is still set to
false when we make our assertions.
Solution 1 - Sleepy time
We can add a sleep into our test to let the go routine finish
before making assertions. But there are a few problems:
- The sleep length I put in is fairly arbitary and it's hard to know precisely what value it should be. You dont want to set it to be too long because then your test will be slow, but too short will make the test flaky.
- It's noise in our test. In this simple example it doesn't seem so bad but it is ultimately more cruft which we need to try and avoid.
- Sleeps always feel yucky!
Solution 2 - Use a channel in our stubs
We can embrace the asynchronousness of our production code so we dont have to rely on guessing how long the stub takes to get called.
Rather than setting a normal value we have a channel which gets written to when
the stub is called. This means our test can wait on a value to appear on the
channel, rather than using sleeps and hoping the value gets set.
This too has problems:
- If the stub isn't called like we expect then the test will fail, but because of
deadlock/test time out.
This means our test output is less helpful then it should be.
- The test now has channels and general asynchrous behaviour which is
implementation detail that we dont really need to care about, like the sleeps it
Solution 3 - Make the stub's value "blocking" with a timeout
It seems these days many developers automatically recoil in disgust when you say "blocking" but it can often be simpler than the alternative.
return and generally a lot of ansynchonous cruft because everything is asynchronous.
That can make writing tests quite taxing, especially if you are new to it. Common gotchas include tests not actually running assertions or tests running forever.
Let's keep our tests clean by putting just a little bit of code in to our stubs to make them block for the event we care about
We have moved the responsibility of "Has the stub been called in a timely manner?" to the stub rather than the test by simply adding the method
The code takes advantage of Go's
select syntax to block until the value is written or until the timeout occurs.
This feels like a good separation of concerns. Now if we look at the actual test we can see it reads very clearly and has no cruft around go routines or channels.
The code in the stub maybe looks a little involved but you will definitely move these into separate files, perhaps even
go generate can be used to auto generate them like goautomock.
This post was motivated by some work I am doing where we are writing
a "listener" to a RabbitMQ channel. The listener takes a
chan Message, gets
fired off in a go routine and processes messages as they come in to the system.
We went through similar iterations described above until we arrived at solution
Is this idiomatic Go? I'm not sure I know what idiomatic Go is yet but this
solution to me ticks a number of boxes I care about in writing software irrespective of the programming language.
- Easy to read and write test code
- Separation of concerns in regards to "has this function been called?"
- Good test output on failure
Hopefully this has been interesting. Know a better way? Flame me on twitter @quii
For the sake of terseness this example just sees if
Repo is called, but you can of course see what it was called with. In the real world
Repo would probably have return values that we'd stub for tests too.
I like to pass
t.Log to my stubs so they can log diagnostic info (such as "I timed out waiting for
x"), it has the added bonus they only appear if the test fails
Most property-based test examples you see on the web are contrived scenarios and people can be dismissive of them, claiming that they aren't actually practical. This post will have a simple example applied to one of my projects which will hopefully illustrate their value.
This is a follow up to Property-based testing in Go and if you're not familiar with these kind of tests I suggest you read that first, it's a quick read.
Although the examples are in Go there are QuickCheck frameworks for most mainstream programming languages and the concepts described here will still apply.
This post will:
- Briefly explain the domain
- Describe writing a real world property-based test
- Summarise my conclusions
It's important to have a good knowledge of the domain when writing property-based-tests, the following is an overview of the domain we will be testing.
mockingjay-server (MJ) is an application which is a lightweight HTTP server driven from configuration.
MJ has two "modes":
Serves responses matched to requests that you define.
Compatibility or CDC mode
Take that same config, executing each request against a given URL and checking the response is compatible with the response in configuration. This is known as a consumer-driven contract (CDC).
When writing integration tests against a HTTP service you will usually make a fake server (or some kind of stub around HTTP) to write tests against. You will then make your tests pass and be happy right?
- Fakes dont always correspond with the real behaviour
- Fakes might behave correctly at first, but if the downstream service changes the fake could be wrong
In both of these situations your build would be green but your software is broken.
mockingjay allows you to make a server and easily verify that it is equivalent to what you're testing against. You can then distribute this configuration as a CDC for the maintainer of the downstream service, so they dont accidentally break the service for your use case. The wiki goes into this more.
There is an inherent coupling between HTTP integration tests and consumer driven contracts. MJ leverages this in a single config.
A property of mockingjay
To write a property-based test we need to identify a property to throw lots of auto-generated data at, to make sure the property holds true.
A property of MJ is:
MJ should always be compatible with itself
Here's my thinking:
- Given Config A
- When You start an MJ server with config A
- And you run a CDC check using config A with the URL of the running MJ server
- Then the CDC check should pass
If it doesn't it either means there is a flaw in the CDC algorithm or in the way the server is behaving.
I have lots of example based tests for this but if I invest time writing a property-based test I can be really confident MJ is working.
Create a generator
For all but the basic types you will need to create a
Generate method for the input type in your test. This will allow the
quickcheck package to create thousands of different data points to check the property against.
I made a simple one to start with and other HTTP things like headers, forms, etc can be added later.
FakeEndpoint is a representation of the config from earlier.
Test the property
- Write a function which takes the randomly generated endpoint
- Start an MJ server using that endpoint
- Take the same config and run the CDC against the server
- If there are any errors then the check fails
When I ran the test I was pleasantly surprised in that the CDC check failed.
Couldn't reach real server: Post http://127.0.0.1:41006/snipped-really-long-random-url: 303 response missing Location header
This means the CDC tried to POST to the configured URL and Go's HTTP client returned an error.
This is quite exciting, I haven't seen MJ configured with any 3xx response codes so this points to some naivety in my code.
HTTP 303 is "See other" which relies on a location header to get redirected to another resource.
Investigating Go's HTTP client
Before writing any code for this I wanted to have a look at the Go source of
http.Client to see exactly how it works.
By following a few of the function calls we can see if it's a
POST or a
PUT and the status is a HTTP
Found 302 or
See Other 303 then it will expect a location header that it can parse with
I will have to add some additional validation to the configuration so that these rules can be respected or make Go's HTTP client not follow redirects.
The costs of writing property-based tests are low (this took me about 10 minutes) and can help give you a lot of confidence in your code (or not!).
What's great is that even on a very simple generator implementation the tests uncovered some bugs.
This style of testing can:
- Improve knowledge of the problem domain
- Find bugs you wouldn't have thought of otherwise
These things prove that property-based tests can improve the quality of your software.
Other examples of this style of testing in the real world:
In this talk John Hughes shows us how QuickCheck helped us to model Riak’s behaviour, improving understanding and revealing the occasional bug.
John Hughes - Testing the hard stuff and staying sane
Taking 3k lines of specification to create 20 lines of QuickCheck to test 1 million lines of code from 6 different vendors. Some real war stories in this video.