Software Architecture Books Summary And Highlights -- Part 14 Testing

 

Highlights

Beyoncé Rule: If a product experiences outages or other problems as a result of infrastructure changes, but the issue wasn’t surfaced by tests in our Continuous Integration (CI) system, it is not the fault of the infrastructure change.

SWG ch 1 what is software engineering?

Test Goal

If a fault is present in a system, then we want it to fail during testing as quickly as possible. More emphasis should be placed on finding the most severe faults than on finding other faults.

Test writing guideline

• Control and observe system state
  • record and playback
  • store states in a single place so that it is easy to start a system, subsystem, or component in an arbitrary state for a test,
  • Preprocessor macros, when activated, can expand to state-reporting code or activate probe statements that return or display information, or return control to a testing console.
• limit nondeterminism
• Replace real implementation with test mocks using dependency injection or dynamically based on the context

SAP Ch 12 Testability

Failing to keep a test suite deterministic and fast ensures it will become roadblock to productivity.

In a test instead of setting timeout to wait for a result, it is better to poll every few micro seconds to improve determinism

SWG Ch 11 Testing Overview

Presenters and Humble Objects

The humble object pattern:

Split the behaviors into two modules or classes. One of those modules is humble; it contains all the hard-to-test behaviors stripped down to their barest essence. The other module contains all the testable behaviors that were stripped out of the humble object. e.g. Presenter and View pattern; ORM for db

CLA Part V: Architecture

Test api

The test should be considered part of the architecture.

Changes to common system components can cause hundreds, or even thousands, of tests to break. This is known as the Fragile Tests Problem.

Solution: develop test api. Test api should hide the structure of the application from the tests to decouple test from production code.

This API should have superpowers that allow the tests to avoid security constraints, bypass expensive resources (such as databases), and force the system into particular testable states.

CLA Part V: Architecture

Unit test writing guideline

the ideal test is unchanging: after it’s written, it never needs to change unless the requirements of the system under test change.

Test clarity is very important. To achieve that, we need completeness and conciseness. A test is complete when its body contains all of the information a reader needs in order to understand how it arrives at its result. A test is concise when it contains no other distracting or irrelevant information.

Test Behaviors, Not Methods

Structure tests to emphasize behaviors

Test state, not interactions.

Name tests after the behavior being tested

No logic in tests: The lesson is clear: in test code, stick to straight-line code over clever logic, and consider tolerating some duplication when it makes the test more descriptive and meaningful.

Write Clear Failure Messages

little bit of duplication is OK in tests so long as that duplication makes the test simpler and clearer.

Shared values in tests: Use helper to construct shared value allow each test to specify the param it only needs and without having to worry about specifying irrelevant information in the shared object

Shared setups: sometimes can be confusing when some values are buried in setup

Shared helpers and validations: The best validation helper methods assert a single conceptual fact about their inputs, in contrast to general-purpose validation methods that cover a range of conditions.

SWG Ch 12 Unit Testing

Test Doubles

A test double is an object or function that can stand in for a real implementation in a test

How to Decide When to Use a Real Implementation

1. Execution time:
   1. use real implementation until it is too slow
   2. parallel test
   3. use efficient build system
2. Determinism
   1. non determinism comes from external service dependency and system clock
3. Dependency construction
   1. the real imple’s dependency could be very complicated to setup
   2. use same production dependency construction code to reduce complexity

3 Techniques for Using Test Doubles

1. fake class

   the team that owns the real implementation should write and maintain a fake.

   before writing a fake, a trade-off needs to be made on whether the productivity improvements that will result from the use of the fake outweigh the costs of writing and maintaining it.

   To reduce the number of fakes that need to be maintained, a fake should typically be created only at the root api level

   If a fake’s clients are across multiple languages, then create a single fake service implementation and have tests configure the client libraries to send requests to this fake service.

   fake must have perfect fidelity to the real implementation, but only from the perspective of the test.

   Fake should fail fast for not covered features

2. Stubbing you specify to the function exactly what values to return. when(...).thenReturn(...)

   1. When to use: when you need a function to return a specific value to get the system under test into a certain state e.g. error

3. Interaction testing is a way to validate how a function is called without actually calling the implementation of the function.

real implementation > fake > stubbing > interaction testing

SWG Ch 13 Test Doubles

Large tests

Common gaps in unit tests

1. unfaithful doubles
2. configuration issues: bug in deployment configuration
3. Issues that arise under load
4. Unanticipated behaviors, inputs, and side effects
5. Emergent behaviors and isolated from real world mess

Large test challenges:

1. flaky, slow, not scalable
2. multiple ownership
3. lack of standardization. different from prod environment

unit test is valuable to short lived code

larger test is valuable to long running code for long-term healths

SWG Ch 14 Large tests

End to end test ownership

To guarantee the health and quality of the end to end tests, avoiding them to be too flaky or big, The best balance I have found is to treat the end-to-end test suite as a shared codebase, but with joint ownership.

MSV Ch 7 Testing

How to reduce the number of large tests

To reduce the number of large end to end test we should focus on a small number of core journeys to test for the whole system. Any functionality not covered in these core journeys needs to be covered in tests that analyze services in isolation from each other. These journeys need to be mutually agreed upon, and jointly owned.

MSV Ch 7 Testing

Who should define and write e2e test

Let service consumer define expectation and write tests for you to test their usage.

MSV Ch 7 Testing

Best usage of e2e test

end to end test is more used to measure the reliability of the production instead of blocking deployment and detect problem.

MSV Ch 7 Testing

Release remediation time vs more e2e test

Sometimes expending the same effort into getting better at remediation of a release can be significantly more beneficial than adding more automated functional tests. In the web operations world, this is often referred to as the trade-off between optimizing for mean time between failures (MTBF) and mean time to repair (MTTR).

MSV Ch 7 Testing

Related Chapters

SAP Ch 12 Testability

CLA Part V: Architecture

MSV Ch 7 Testing

SWG ch 1 what is software engineering?

SWG Ch 11 Testing Overview

SWG Ch 12 Unit Testing

SWG Ch 13 Test Doubles

SWG Ch 14 Large tests