WarStoriesAndSuccesses

Test strategies and flow and sharing of strategies

• Test pipelines and integration strategies across teams

• Architecture
• It’s so important to assume you will miss something .. so MTTR is critical

War stories

• Defeat: why can’t anyone deploy anything
• Victory: 3 months to change a web page —> changes in 1 day
  • Text link to 404 for the purpose of recording clicks of interest for house viewing
• PJ: Company daily release process but often would break and often take 3 people most of the day to get it out
  • Eventually, every time someone made a change of the code it would be deployed to prod and monitored
    • End to end test bankruptcy
  • Success
  • Monolithic system (with seams but tight coupling) was broken down into component based independently deployable systems
  • Be prepared, this process will take 3 years (or more)
  • How do you sell this idea?
    • The key to our success is shortening our feedback cycles
    • From Tony later: show other companies that are already past 3y to show what’s possible (and build trust)
• Tim: traditionally siloed functional org
  • Idea to cash was ~ 200d
  • 91% wait time
  • First response was to be mad at the person pointing at the problem
  • Improvement is slow, but it is improving
  • It was important to just get people talking about it even if they got upset initially
  • Value stream map made it all possible
• Success: Operations team and developer teams
  • Devs write run-books over a couple months, then throw it to the operators
  • Big bang release every couple of months
  • Build a more intimate relationship between Dev and Ops
    • Moved to central Jenkins instance from individual snowflakes
    • This integrated the dev workflow with the ops workflow (common environment)
  • Decoupled infra track from application track for deployments
  • Over a 6 month period — still monthly releases but far less stressful on everyone involved
  • Enjoyable pizza
• Frederick: Goal: faster release cycle — get to weekly
  • Current was monthly with bi-weekly patch releases
  • Visualized value stream maps with both
  • Took similarities and combined to 1 flow like the patch flow — currently at every 2w
  • Giant monolith
  • There were a lot of fears and resistance but enough trust was built to give it a try
  • Now trying to work towards TBD
• Small web team gets security scan result
  • Pen testers test quarterly — first report was very ugly
  • Introduced PR process, but devs weren’t able to spot security vulnerabilities
  • Secure Code Warrior gamefies this .. points for vulnerabilities found
  • Got some training (e.g. OWASP) and testers were more involved
  • Making progress
  • “Why should I train you (freelancer), what if you leave"
    • “What if you don’t train us and we stay” (response)
• Failure:
  • Cultural, hierarchical issues with a Dev silo, QA silo (which moved closer) and DevOps (eww) silo which was moved even further away
  • Improvements were made in delivering more often, but then there was some realization/questioning of this and so deliveries are now getting slowed/spaced out