Fragile Test

From CitconWiki
Revision as of 01:46, 30 July 2007 by 211.28.144.55 (Talk) (Bad Tests)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Less Fragile Tests

Tests

  • Asserts
    • Component
    • Integration
  • Functional
    • UAT
  • Non-Functional
    • Perf
    • Security
    • Usability
  • See ISO 9126

Good Tests

  • Pass Repeatedly
    • Timing (e.g. Selenium)
    • External Dependencies
      • External Systems
        • Reliability
        • Availability
      • Data
        • Set Values
        • Incorrect State
    • Heavy Mocking
      • Utility to build mocks
      • Easy switching mock to real
        • use same data, settings

Bad Tests

  • Hidden cause of failure
    • use screen capture logger, e.g. TimeSnapper
  • Visible Replay
  • Dump state data at failure

Tool advice

  • Time Snapper
  • EasyMock
  • Pretend Classes
  • DBUnit gets prod snapshot regularly (security issues: scrubbing) Table dependancies makes it a significant task

Data

  • Domain (name, addr, etc class pulled from DB
  • (abstracted data from test)
  • Data Driven Approach
  • Fitness Integration Testing

Strong correlation between importance of tests depending on the types of testing and what kind of thing you are doing (Product type development = high focus on test vs bespoke = low focus


Suggestion

  • Use a virtual machine, setup - refresh test machine using VMWARe to simulate MSG based, 20 mins business transactions

Test Data Repository

  • Used keyword substitution to create the real data

Typical Data

  • Edge / Border cases, Exception testing

Broken windows

  • Continually failing, low priority tests/defects cause a 'Broken windows' effect which degrades the value of CI.