Test Triage and Intermittent Test techniques

Attendance

• Tyler @macetw
• Emil
• @RobPark

Test Triage Techniques

Often tests appear without clear reason, many commits into a single build. Unreliable machines or frameworks, how do we respond?

Approaches:

• Not enough information in the log
• Limit smoke test time

1 team might be dedicated to triaging

• Build sheriff
• Assign person responsible
• File ticket

Assumed: Does everyone care that a build is red? Is a build/test cycle easy enough to reproduce?

Intermittent Failures

CI tool (TeamCity, Jenkins plugin) will handle assignment of an issue, a particular failing test, and there would be an owner of a certain issue, even if not always-failing.

Blame on load average, up-of-timeout-wait, but no real fix.

Approaches:

• Ticket made of while a build intermittently failed.
• Assign test to quarantine
  • Management approval required
  • Analysis of time-management
  • Stuff moves gradually out of quarantine

Sidebar: Google test on the toilet mental note, for macetw, we need more code-literature in our bathroom

One approach:

• Remove from test running
  • Would affect code coverage numbers
  • Not all cultures
• If still present in code 90 days later, may choose delete test scenario

Sidebar: If high defect rate, but also high code coverage, tests may not be testing the right kinds of things