Difference between revisions of "Test Triage and Intermittent Test techniques"
Line 38: | Line 38: | ||
** Not all cultures | ** Not all cultures | ||
* If still present in code 90 days later, may choose delete test framework | * If still present in code 90 days later, may choose delete test framework | ||
+ | |||
+ | |||
+ | Sidebar: If high defect rate, but also high code coverage, tests may not be testing the right kinds of things |
Revision as of 09:05, 24 August 2013
Attendance
- Tyler @macetw
- Emil
- @RobPark
Test Triage Techniques
Often tests appear without clear reason, many commits into a single build. Unreliable machines or frameworks, how do we respond?
Approaches:
- Not enough information in the log
- Limit smoke test time
1 team might be dedicated to triaging
- Build sheriff
- Assign person responsible
- File ticket
Assumed: Does everyone care that a build is red? Is a build/test cycle easy enough to reproduce?
Intermittent Failures
CI tool (TeamCity, Jenkins plugin) will handle assignment of an issue, a particular failing test, and there would be an owner of a certain issue, even if not always-failing.
Blame on load average, up-of-timeout-wait, but no real fix.
Approaches:
- Ticket made of while a build intermittently failed.
- Assign test to quarantine
- Management approval required
- Analysis of time-management
- Stuff moves gradually out of quarantine
Sidebar: Google test on the toilet mental note, for macetw, we need more code-literature in our bathroom
One approach:
- Remove from test running
- Would affect code coverage numbers
- Not all cultures
- If still present in code 90 days later, may choose delete test framework
Sidebar: If high defect rate, but also high code coverage, tests may not be testing the right kinds of things