Should unit tests be stateless?

[aaaaalbert]

We don't have a policy for whether unit tests must "clean up after themselves" or not, i.e. may leave traces of the state / fact of their exceution or not. Some tests clean up, some don't despite causing significant artifacts that changes the behavior of later re-runs through the same test.

Pros of requiring cleanup

• If unit tests fail, this often means you will change code in response, and re-run the tests. Stateless tests speed this up a lot --- no re-cloning, re-building, etc.; Running the test again is like starting over anew.
• Forces you to think harder before/while you implement. You will see problems like this in production code for sure, so why not train yourself on writing tests beforehand?

Cons of requiring cleanup

• It's not always obvious that unit tests have side effects, and what these will be.
• Cleaning things up correctly is complex, and probably requires lots of duplicate code across tests.

---

A middle ground could be to somehow mark tests that require a re-build after they have run. Then at least it's clear that problems may arise.

---

Reaching out to @JustinCappos, @vladimir-v-diaz, @awwad in particular --- What do you think?

[JustinCappos]

I think you should remove state before running a test, not after. In
other words, put the system state in the correct state and then run. If
you clean up after a failure, then you may delete useful debugging
information.

On Fri, Oct 2, 2015 at 7:23 AM, aaaaalbert notifications@github.com wrote:

We don't have a policy for whether unit tests must "clean up after
themselves" or not, i.e. may leave traces of the state / fact of their
exceution or not. Some tests clean up
https://github.com/SeattleTestbed/seash/blob/master/tests/ut_seash_subprocess.py#L48-L67,
some don't despite causing significant artifacts that changes the
behavior of later re-runs through the same test
https://github.com/SeattleTestbed/seash/blob/master/tests/ut_seash_allmodules.py#L21-L23
.
Pros of requiring cleanup

• If unit tests fail, this often means you will change code in
  response, and re-run the tests. Stateless tests speed this up a lot --- no
  re-cloning, re-building, etc.; Running the test again is like starting over
  anew.
• Forces you to think harder before/while you implement. You will see
  problems like this in production code for sure, so why not train yourself
  on writing tests beforehand?

Cons of requiring cleanup

• It's not always obvious that unit tests have side effects, and what
  these will be.
• Cleaning things up correctly is complex, and probably requires lots
  of duplicate code across tests.

---

A middle ground could be to somehow mark tests that require a re-build

after they have run. Then at least it's clear that problems may arise.

Reaching out to @JustinCappos https://github.com/JustinCappos and
@vladimir-v-diaz https://github.com/vladimir-v-diaz in particular ---
What do you think?

—
Reply to this email directly or view it on GitHub
#66.

[aaaaalbert]

Sure, makes sense.

Any feelings on whether cleaning up is a MUST, SHOULD, or MAY in the sense of RFC 2119? 😉

(Currently, the decision is left to the implementor.)

[JustinCappos]

In the case of success I think it would be a SHOULD.

However, there are also pre and post scripts for certain test modules
(groups of tests) and the complexity of having per-test cleanups on success
while leaving state on failure, is a mess.

The tests are here to make our lives easier. If being very rigid about our
test behavior causes a lot of work, that is not helpful. However, needing
to debug some specific test in a different way may cause us a lot of work
over time. I would at a minimum expect any "outlier" tests to include
copious comments explaining how they work and why.

(We should already have this for some of the resource tests that may crash
the system if the repy VM does not handle them correctly.)

On Fri, Oct 2, 2015 at 12:17 PM, aaaaalbert notifications@github.com
wrote:

Sure, makes sense.

Any feelings on whether cleaning up is a MUST, SHOULD, or MAY in the sense
of RFC 2119? [image: 😉]

(Currently, the decision is left to the implementor.)

—
Reply to this email directly or view it on GitHub
#66 (comment).