Rendered at 16:47:34 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
pfdietz 2 hours ago [-]
I had a test suite with thousands of tests. One way of running it was to take all the passing tests, and then run them repeatedly in random order.
This found new bugs involving unintended persistent state.
natbennett 1 hours ago [-]
After a bunch of experiences like this I will now basically only use test runners that automatically run the tests in randomized order.
Gotta catch that stuff early.
sholladay 42 minutes ago [-]
Shout out to AVA for running all tests async by default, which not only finds order-dependent issues, but also race condition and other timing related issues.
Ah Kent Beck, I see your style of writing trivial examples hasn't changed
mrkeen 2 hours ago [-]
> If a test runs by first setting up its own test fixture, creating from scratch all the data it will be using as input, then that test is guaranteed to be isolated. It doesn’t matter what order you run the tests, the results will be exactly the same.
Completely backward. This definition requires isolation, rather than granting it.
In reality, you (or your test framework) provides the isolation by hobbling along, executing only one test at a time.
aleksiy123 2 hours ago [-]
I don’t think that’s right?
The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation.
If I create a new in mem dB in the test there’s nothing stopping me from running it in parallel?
Nothing about that “requires” isolation. It is isolation.
mrkeen 1 hours ago [-]
Maybe I misread. When I see a setUp() in a test suite, 9 times out of 10 it's to handle something shared and bulky, like a database. Otherwise you'd just do the thing - assert(expected, myService.run(input)); If I saw an extra myService.reset() or myService.setUp() I would suspect it's either in anticipation of a bad state (from a previous test) or to be a good neighbour for the next test which will run.
In the cases where each unit test does have it's own individual Postgres or whatever, sure.
aleksiy123 54 minutes ago [-]
I guess it all depends, isolation isn’t binary.
you can share some things and be isolated across other dimensions.
You could share a Postgres dB connection but just isolate the data logically.
You can isolate or share across individual tests, across suites, across envs, across runs, across time.
More isolation is generally better unless the complexity or cost is too high.
But I guess I was just confused as to how a test requires isolation rather than have it.
locknitpicker 36 minutes ago [-]
> The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation.
I'm not sure what you mean by that. All the frameworks I ever used were designed with test isolation as the primary design goal. Even when you set shared test fixtures and setup/teardown code, all they provide is a way to share code across tests, which are by themselves independent and isolated. BDD-style frameworks are the notable exception in breaking away from the pattern of having isolation as a fundamental design trait. In fact, the whole reason why BDD-style tools require special support from testing frameworks is that they need to work around test isolation in order to share context across steps.
Groxx 26 minutes ago [-]
Ehhh... when it truly is two tests bodged into one, then sure.
But sometimes you do this kind of thing to avoid useless test brittleness: does your test check that `doSomething()` does what you expect, or do you have another test for that and this test only checks that `nowSomethingElse()` changes the object in a predictable way, e.g. updates a calculated field?
If it's the former, then it might be two tests masquerading as one, and this might make sense.
If it's the latter, you've changed a test that only checks what it cares about, and now you have a test that depends on unrelated implementation detail and will probably break unnecessarily in the future. Plus you've removed the assert that was documenting what it expects, so it's harder to tell if fixing it should mean updating both checks, or only the second one.
grim_io 2 hours ago [-]
I'm done listening to smart people proposing fixes for my dumb code.
Either I'm not smart or disciplined enough to make it work, or my colleagues are not. Mostly both.
OakNinja 2 hours ago [-]
Kent Beck is smart because he proposes fixes that work in real life, as in environments with dumb code and undisciplined people.
I can very much recommend his latest book ”Tidy first?”. It’s extremely short and concise, and is perfect for a very light book club within any tech team.
grim_io 1 hours ago [-]
I assure you, there is no such thing as a book club at my job, especially not one focused on technical writing :)
RHSeeger 2 hours ago [-]
I agree, both with your analysis of him and your review of "Tidy first?". We read and discussed it at work, and I enjoyed it a lot.
skydhash 2 hours ago [-]
I enjoyed the last (and theoretical) part of "Tidy first?" as I was already doing the recommended practices in the first two. But yes, very pragmatic advice and easy to apply to get your own corner of peace in a somewhat chaotic codebase.
One term I can use is "defensive programming" (not sure if the term has been used before). People are going to do awful stuff with their code, it's up to you to draw boundary of correctness to stop it from spilling on the part that you're responsible for. It should be automated (with tests and static analysis) as well as documented. I think of those boundary as hazmat suits and I take extra care of maintaining their integrity.
troupo 2 hours ago [-]
What he wrote is basically "don't repeat in test X what you already tested in test X-1".
It's not as much composition as compounding, and can work quite well.
Let's say something takes 20 steps, and you want to test all 20.
Instead of this:
test 1:
do step 1, assert step 1
test 2:
do step 1, assert step 1
do step 2, assert step 2
test 3:
do step 1, assert step 1
do step 2, assert step 2
do step 3, assert step 3
...
you do this:
test 1:
do step 1, assert step 1
test 2:
do step 1
do step 2, assert step 2
test 3:
do step 1
do step 2
do step 3, assert step 3
...
This works well in certain situations, as it skips diplicated redundant testing. Requires some discipline so that tests don't drift away from each other.
As most things, it depends on what those steps are. Perhaps you only need that one final (integration) test instead of 20 intermediate unit ones.
RHSeeger 2 hours ago [-]
I wonder if it would work do design something that was able to say
test 1:
do step 1, assert step 1
test 2:
requires: test 1
do step 2, assert step 2
test 3:
requires: test 2
do step 3, assert step 3
test 4:
requires: test 2
do step 3, assert step 3
If the assertion of each test doesn't change any state, that might make things easier to read. Though, given that I haven't spent much time pondering it, I expect it could have it's own problems.
But it could also do things like skip test 3 if tests 2 or 1 failed - because it knows about the relationship.
brabel 2 hours ago [-]
Yes that is the way to do it, in Spock that’s what @Stepwise does.
mamcx 53 minutes ago [-]
This make too much sense!
In special when testing against a DB.
troupo 2 hours ago [-]
Yup. I'd love to have dependency declarations for tests like this.
jaggederest 19 minutes ago [-]
That's... A thing in many frameworks. If it's not in yours, you could add it.
Forgive an old man some ruby:
it 'relies on mobile setup defined elsewhere', :mobile => true do
# test that relies on mobile setup here
end
latencyharbor 6 minutes ago [-]
[flagged]
jayd16 1 hours ago [-]
IMO it seems kind of terrible.
You're not going to remember what tests "test 3" relies on. They aren't actually linear progressions 123. They will be "test this", "test that". If 3 fails you're going to want to immediately go and add all those asserts back to help you debug your assumptions.
The tests _will_ drift and that should be fine. Implicitly depending on other tests doesn't really get me anything.
seanwilson 2 hours ago [-]
What's the problem of only keeping test 3 if it depends on test 2 and 1 passing anyway?
The article mentions not to do this because "Deleting test1 loses us another property from the Test Desiderata—tests should be specific. That’s the property of tests where, when one fails, you know exactly where the problem is." but you'll know what line it failed on. And some test runners let you break a test into steps, where groups of lines are given a description.
Or put each step + assert in a helper function (e.g. `doStep1AndAssert()`), and each test only calls these helper functions?
Nothing is perfect, but copy/pasting chunks between tests like this isn't great when you want to refactor it's repetitive to read.
troupo 2 hours ago [-]
> What's the problem of only keeping test 3 if it depends on test 2 and 1 passing anyway?
That depends.
Sometimes test 1 tests a combination of ways (e.g., property testing, or just going through a bunch of various inputs), and only a few of those are needed for test 2.
Sometimes you don't want your test 2 to be more complicated than it already is. Or the same things are needed checked in other tests. So you extract them into test 1.
And sometimes (and in some of code bases most of the time) test 1 is redundant and unnecessary. That's why I always advocate investing in integration tests (test 20) and skip all the intermediate tests.
skydhash 1 hours ago [-]
> That's why I always advocate investing in integration tests (test 20) and skip all the intermediate tests.
My approach is to have acceptance tests documented for any feature. Like how it would be from the user point of view to actually use the software. Then do Integration tests for each part of that workflows. That's usually the most ROI you will get for testing. Then I invest into unit tests for particular elements that are very important. I start from the middle of the pyramid because an actual e2e is expensive to setup (easy to maintain afterwards) and having lots of unit tests (easy to setup) is expensive to maintain.
mrkeen 2 hours ago [-]
I think I disagree with Kent, but your explanation is clearer, so I'll object here.
There's nothing wrong with hitting the same assertion multiple times, even if it doesn't sit nicely in your gut.
From a purely philosophical point of view: If I have testFoo(), testBar(), and testFooAndBar(), and my Foo is plain wrong, then both testFoo() and testFooAndBar() must fail. Anything less is misleading/dishonest.
From a practical side: Changes happen. Someone will remove testBar(), and then you're down to 0 assertions on Bar, even though you have a test claiming to testFooAndBar(). It's not even a crazy hypothetical. Someone with a different test philosophy will think (to quote TFA) "They are redundant! Something must be wrong." and delete testBar() because obviously testFooAndBar() already covers it.
Anyway, we all know how to deal with repetition. That's what programming is!
> Deleting test1 loses us another property from the Test Desiderata—tests should be specific. That’s the property of tests where, when one fails, you know exactly where the problem is.
Contra Kent and, it seems, prevailing wisdom, I think simply deleting test1 is by far the simplest, clearest and best way. Provided that your testing framework tells you which specific assertion failed (e.g., by telling you the line number in a stack trace), you do know exactly where the problem is. The only thing you lose is that a test function or method may now cover several related checks (they are related by "setup dependence"), meaning their names may need to be somewhat broader. But you can still describe the specific semantics of each assert() check in a one-line comment beforehand if you want. There's no need to cram it into a legal method name.
ETA: Prefer to write tests whose "arrange" steps are as simple as possible, to minimise unnecessary overlaps. But if the simplest possible "arrange" step for a test is something that itself needs to be checked for correctness, just do that check right there, and nowhere else. Anything beyond that is ceremony that adds nothing useful.
dack 35 minutes ago [-]
I think this could lead to the tests getting harder to reason about over time because individual test size will just grow (many iterations of deleting the simple test in favor of a more complicated test that also asserts the simple things)
and if you start simplifying the more complicated test later, you may not realize that you're accidentally deleting checks that don't exist anywhere else other than incidentally here. So it's less clear what's important without thinking it through each time.
now all that to say I don't think it's strictly that big of a deal either way - there's always tons of room for taste that can make one or the other way better in practice. But that was my gut-reaction when reading your comment.
skydhash 1 hours ago [-]
> Contra Kent and, it seems, prevailing wisdom, I think simply deleting test1 is by far the simplest, clearest and best way. Provided that your testing framework tells you which specific assertion failed (e.g., by telling you the line number in a stack trace), you do know exactly where the problem is.
The issue with your approach is that codepath execution is more of a graph than a linear timeline. With one test, you artificially constrains it to a linear timeline even if the assertions are correct. With multiples and independent you only assert one specific node. That lets you switch up how you do the preliminary steps. I much prefer an exhaustive test unit for step 1, and just a regular call in step 2 and step 3.
akoboldfrying 1 hours ago [-]
If you need test1's logic as setup for test2, then you already have that "linear timeline" -- for test2 all by itself. Additionally (that is, redundantly) running the same setup code "prefix" by itself in test1 doesn't remove test2's linear timeline.
ETA: I'm assuming your objection to a "linear timeline" is that it reduces the potential for running tests in parallel -- have I got that right? If not, what do you see as being the problem with it?
skydhash 40 minutes ago [-]
If I know that step 1 is correct for all the combinations (N) of its input (in test 1), in test 2, I only need to test the specific combinations (M) of step 2, treating step 1 as an axiom. So no need to have NxM in one test, or NxM tests.
Like if you were testing a drone stabilization software, testing the flying state can always assume that it has indeed taken off. No need to validate that it has done so for each scenarios, I can directly put it the correct value. It’s a contrived example, but that axiomatic aspect helps greatly when designing interfaces to reduce coupling between step 1 and step 2.
This found new bugs involving unintended persistent state.
Gotta catch that stuff early.
https://github.com/avajs/ava
Kent Beck is co-creator [1] of the JUnit Testing Framework and has most definitely heard of static and global variables.
[1] https://junit.org/junit4/project-info.html
Completely backward. This definition requires isolation, rather than granting it.
In reality, you (or your test framework) provides the isolation by hobbling along, executing only one test at a time.
The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation.
If I create a new in mem dB in the test there’s nothing stopping me from running it in parallel?
Nothing about that “requires” isolation. It is isolation.
In the cases where each unit test does have it's own individual Postgres or whatever, sure.
you can share some things and be isolated across other dimensions.
You could share a Postgres dB connection but just isolate the data logically.
You can isolate or share across individual tests, across suites, across envs, across runs, across time.
More isolation is generally better unless the complexity or cost is too high.
But I guess I was just confused as to how a test requires isolation rather than have it.
I'm not sure what you mean by that. All the frameworks I ever used were designed with test isolation as the primary design goal. Even when you set shared test fixtures and setup/teardown code, all they provide is a way to share code across tests, which are by themselves independent and isolated. BDD-style frameworks are the notable exception in breaking away from the pattern of having isolation as a fundamental design trait. In fact, the whole reason why BDD-style tools require special support from testing frameworks is that they need to work around test isolation in order to share context across steps.
But sometimes you do this kind of thing to avoid useless test brittleness: does your test check that `doSomething()` does what you expect, or do you have another test for that and this test only checks that `nowSomethingElse()` changes the object in a predictable way, e.g. updates a calculated field?
If it's the former, then it might be two tests masquerading as one, and this might make sense.
If it's the latter, you've changed a test that only checks what it cares about, and now you have a test that depends on unrelated implementation detail and will probably break unnecessarily in the future. Plus you've removed the assert that was documenting what it expects, so it's harder to tell if fixing it should mean updating both checks, or only the second one.
Either I'm not smart or disciplined enough to make it work, or my colleagues are not. Mostly both.
I can very much recommend his latest book ”Tidy first?”. It’s extremely short and concise, and is perfect for a very light book club within any tech team.
One term I can use is "defensive programming" (not sure if the term has been used before). People are going to do awful stuff with their code, it's up to you to draw boundary of correctness to stop it from spilling on the part that you're responsible for. It should be automated (with tests and static analysis) as well as documented. I think of those boundary as hazmat suits and I take extra care of maintaining their integrity.
It's not as much composition as compounding, and can work quite well.
Let's say something takes 20 steps, and you want to test all 20.
Instead of this:
you do this: This works well in certain situations, as it skips diplicated redundant testing. Requires some discipline so that tests don't drift away from each other.As most things, it depends on what those steps are. Perhaps you only need that one final (integration) test instead of 20 intermediate unit ones.
But it could also do things like skip test 3 if tests 2 or 1 failed - because it knows about the relationship.
In special when testing against a DB.
Forgive an old man some ruby:
You're not going to remember what tests "test 3" relies on. They aren't actually linear progressions 123. They will be "test this", "test that". If 3 fails you're going to want to immediately go and add all those asserts back to help you debug your assumptions.
The tests _will_ drift and that should be fine. Implicitly depending on other tests doesn't really get me anything.
The article mentions not to do this because "Deleting test1 loses us another property from the Test Desiderata—tests should be specific. That’s the property of tests where, when one fails, you know exactly where the problem is." but you'll know what line it failed on. And some test runners let you break a test into steps, where groups of lines are given a description.
Or put each step + assert in a helper function (e.g. `doStep1AndAssert()`), and each test only calls these helper functions?
Nothing is perfect, but copy/pasting chunks between tests like this isn't great when you want to refactor it's repetitive to read.
That depends.
Sometimes test 1 tests a combination of ways (e.g., property testing, or just going through a bunch of various inputs), and only a few of those are needed for test 2.
Sometimes you don't want your test 2 to be more complicated than it already is. Or the same things are needed checked in other tests. So you extract them into test 1.
And sometimes (and in some of code bases most of the time) test 1 is redundant and unnecessary. That's why I always advocate investing in integration tests (test 20) and skip all the intermediate tests.
My approach is to have acceptance tests documented for any feature. Like how it would be from the user point of view to actually use the software. Then do Integration tests for each part of that workflows. That's usually the most ROI you will get for testing. Then I invest into unit tests for particular elements that are very important. I start from the middle of the pyramid because an actual e2e is expensive to setup (easy to maintain afterwards) and having lots of unit tests (easy to setup) is expensive to maintain.
There's nothing wrong with hitting the same assertion multiple times, even if it doesn't sit nicely in your gut.
From a purely philosophical point of view: If I have testFoo(), testBar(), and testFooAndBar(), and my Foo is plain wrong, then both testFoo() and testFooAndBar() must fail. Anything less is misleading/dishonest.
From a practical side: Changes happen. Someone will remove testBar(), and then you're down to 0 assertions on Bar, even though you have a test claiming to testFooAndBar(). It's not even a crazy hypothetical. Someone with a different test philosophy will think (to quote TFA) "They are redundant! Something must be wrong." and delete testBar() because obviously testFooAndBar() already covers it.
Anyway, we all know how to deal with repetition. That's what programming is!
Contra Kent and, it seems, prevailing wisdom, I think simply deleting test1 is by far the simplest, clearest and best way. Provided that your testing framework tells you which specific assertion failed (e.g., by telling you the line number in a stack trace), you do know exactly where the problem is. The only thing you lose is that a test function or method may now cover several related checks (they are related by "setup dependence"), meaning their names may need to be somewhat broader. But you can still describe the specific semantics of each assert() check in a one-line comment beforehand if you want. There's no need to cram it into a legal method name.
ETA: Prefer to write tests whose "arrange" steps are as simple as possible, to minimise unnecessary overlaps. But if the simplest possible "arrange" step for a test is something that itself needs to be checked for correctness, just do that check right there, and nowhere else. Anything beyond that is ceremony that adds nothing useful.
and if you start simplifying the more complicated test later, you may not realize that you're accidentally deleting checks that don't exist anywhere else other than incidentally here. So it's less clear what's important without thinking it through each time.
now all that to say I don't think it's strictly that big of a deal either way - there's always tons of room for taste that can make one or the other way better in practice. But that was my gut-reaction when reading your comment.
The issue with your approach is that codepath execution is more of a graph than a linear timeline. With one test, you artificially constrains it to a linear timeline even if the assertions are correct. With multiples and independent you only assert one specific node. That lets you switch up how you do the preliminary steps. I much prefer an exhaustive test unit for step 1, and just a regular call in step 2 and step 3.
ETA: I'm assuming your objection to a "linear timeline" is that it reduces the potential for running tests in parallel -- have I got that right? If not, what do you see as being the problem with it?
Like if you were testing a drone stabilization software, testing the flying state can always assume that it has indeed taken off. No need to validate that it has done so for each scenarios, I can directly put it the correct value. It’s a contrived example, but that axiomatic aspect helps greatly when designing interfaces to reduce coupling between step 1 and step 2.