Intellectual Control vs Statistical Control
Intellectual Control (IC)
When you understand the code at an intellectual level and you know what the code should do without having to execute it.
Essentially, intellectual control is the confidence you gain from being able to reason on the outcome of execution.
Statistical Control (SC)
Relying on testing to gain confidence in the correctness of the software.
Over-reliance on testing can lead developers to neglect intellectual control, which is the ability to reason about code and predict its behavior without executing it.
What happens when you don’t have IC
- People will resort to whack-a-mole development
- People start to perform “P hacking”
- Selective test execution
- Ignoring test failures
- Overfitting test cases
- Leads to the Inverted test pyramid
Test distribution without IC
Martini Pyramid
_________ /\
\ / / \
\ / Large / \ e2e tests
\___/ /______\
| | Medium / \ integration tests
|___| /__________\
/ \ Small / \ unit tests
/_______\ /______________\
- hard to write
- they are more likely to be flaky
- require more resources to:
- create (developer time)
- execute (machine time)
- and triage (developer time)
- might have a low signal to noise ratio
Focus on Intellectual Control
Maintaining intellectual control over complex systems can be challenging once it's lost. The complexity of these systems increases super-linearly, meaning that even a small increase in the number of modules can result in a significant jump in the number of interconnections that need to be understood. For example, if we have two modules that are not well understood, the complexity is 2. However, if we have ten modules, we have 55 paths that need to be understood.