'Wall Time' as a way of determining correctness

posted by  hjwin (10)  15 months ago (blogs.msdn.com)

Quite a few years back, I joined the Windows Server 2003 team as a developer.  Windows Server 2003 is a massive organization – I was part of the UDDI team which resided under Directory Services.

Our QA team had a suite of about 300 automated tests.  The collected the ‘wall time’ of a test run – so literally how many minutes/seconds it took for each test to run.  The ‘wall time’ amounts were aggregated into a test suite run time.

When I first joined, that suite of tests was taking about 2 hours to complete.  That was a concern because the test su