Test Triage and Intermittent Test techniques

From CitconWiki
Revision as of 10:54, 24 August 2013 by Robpark (talk | contribs) (→‎Intermittent Failures)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigationJump to search

Attendance

  • Tyler @macetw
  • Emil
  • @RobPark

Test Triage Techniques

Often tests appear without clear reason, many commits into a single build. Unreliable machines or frameworks, how do we respond?

Approaches:

  • Not enough information in the log
  • Limit smoke test time

1 team might be dedicated to triaging

  • Build sheriff
  • Assign person responsible
  • File ticket

Assumed: Does everyone care that a build is red? Is a build/test cycle easy enough to reproduce?

Intermittent Failures

CI tool (TeamCity, Jenkins plugin) will handle assignment of an issue, a particular failing test, and there would be an owner of a certain issue, even if not always-failing.

Blame on load average, up-of-timeout-wait, but no real fix.

Approaches:

  • Ticket made of while a build intermittently failed.
  • Assign test to quarantine
    • Management approval required
    • Analysis of time-management
    • Stuff moves gradually out of quarantine

Sidebar: Google test on the toilet mental note, for macetw, we need more code-literature in our bathroom

One approach:

  • Remove from test running
    • Would affect code coverage numbers
    • Not all cultures
  • If still present in code 90 days later, may choose delete test scenario


Sidebar: If high defect rate, but also high code coverage, tests may not be testing the right kinds of things