Difference between revisions of "Functional tests take a long time"

From CitconWiki
Jump to navigationJump to search
(New page: Long Running Functional Tests These are the few notes I took from the "long running functional tests" discussions. Problems * Complete features take 1 day * Functional test takes 15 hou...)
 
(adding my rough draft notes from the session)
 
(One intermediate revision by one other user not shown)
Line 3: Line 3:
 
These are the few notes I took from the "long running functional tests" discussions.
 
These are the few notes I took from the "long running functional tests" discussions.
  
Problems
+
'''Problems'''
  
 
* Complete features take 1 day
 
* Complete features take 1 day
 
* Functional test takes 15 hours
 
* Functional test takes 15 hours
  
Alternatives, Risks, and Trade-offs
+
'''Alternatives, Risks, and Trade-offs'''
  
 
* Parallel testing vs. pipelined testing
 
* Parallel testing vs. pipelined testing
Line 19: Line 19:
 
** Show failures sooner, but
 
** Show failures sooner, but
 
** Does not typically lead to stopping the tests because we want to know all the results from that set of code
 
** Does not typically lead to stopping the tests because we want to know all the results from that set of code
 +
 +
 +
Alternate set of incomplete notes:
 +
 +
What do I do if my functional tests take 5 hours?
 +
 +
4 hour cycle vs. 5 minutes
 +
 +
(ci-machismo:  "mine is smaller than yours")
 +
 +
some tests we run all the time and some that we don't
 +
 +
order of magnitude & build pipeline tradeoff:
 +
5 minutes vs. 50
 +
20 minutes vs. 30
 +
 +
long running test suite but reporting of status after each suite  (separate emails?)
 +
 +
re-order decomposable steps
 +
 +
what tools are people using to do their functional tests?
 +
 +
new process: added qa, added release management.  developers aren't responsive to breaks in the QA builds. often there is disagreement about what the customer really wants, what the problems are. mini-waterfall labeled as Agile. requirements sprint, design sprint, coding sprint, testing sprint.

Latest revision as of 01:06, 10 April 2008

Long Running Functional Tests

These are the few notes I took from the "long running functional tests" discussions.

Problems

  • Complete features take 1 day
  • Functional test takes 15 hours

Alternatives, Risks, and Trade-offs

  • Parallel testing vs. pipelined testing
    • Compile
    • Fast unit tests
    • Slow unit tests
    • Functional tests
  • Incremental feedback during test runs
    • Show failures sooner, but
    • Does not typically lead to stopping the tests because we want to know all the results from that set of code


Alternate set of incomplete notes:

What do I do if my functional tests take 5 hours?

4 hour cycle vs. 5 minutes

(ci-machismo: "mine is smaller than yours")

some tests we run all the time and some that we don't

order of magnitude & build pipeline tradeoff: 5 minutes vs. 50 20 minutes vs. 30

long running test suite but reporting of status after each suite (separate emails?)

re-order decomposable steps

what tools are people using to do their functional tests?

new process: added qa, added release management. developers aren't responsive to breaks in the QA builds. often there is disagreement about what the customer really wants, what the problems are. mini-waterfall labeled as Agile. requirements sprint, design sprint, coding sprint, testing sprint.