Unfortunately our data-driven management systems reward companies that pass the tests illegitimately, until they are finally caught. Like performance-enhanced athletes, the companies who skirt regulatory scrutiny can prosper and dominate.
Where does the ethic start? Who among us can cast the first stone, so to speak?
Having been involved in software quality, I know that there are conditions upon which the same product can be shown to pass, or to fail the test. The scope and breadth of the test is as important as the result, along with the conditions and potential interactions. Seasoned testers can break a system with very little effort, often by incorporating a few known failure modes into the equation. For example, adding a name with accents, hyphens, and apostrophes to a database operating concurrently with anti-virus software could cause a performance issue leading to data corruption.
I also know that having set up multiple product demonstrations for sales presentations, the end-to-end display of the features is not always reflective of actual operating conditions. User choices and options are often reduced to show only those working features, and results pages are often hard-coded to show the expected outcome, rather than take the chance of an error message or interrupted function appearing. This is best demonstrated in the movie Tucker, where the prototype automobile was being fixed on the stage immediately prior to its display.
Here is an administrative example related to the new ASQ budgeting format. The revenues and costs are structured to distinguish between General and Administrative (G&A) and Member Value (MV), with the target that ASQ Member Units should have 70% of their budget emphasizing Member Value items (known as the PAR Ration - Performance Awards and Recognition).
5500 Contract & Professional expense category is Member
Value if the item is used for Subject Matter Experts, Instructor Fees, or
Member Value events.
4600 tracks Contributions and Sponsorships as Member Value for
revenue from Contributions and Sponsorships.
Dan – eminent and illustrious ASQ Fellow – agrees to speak at an
ASQ section for 1 million dollars and a cup of coffee. In turn, Dan –
generous benefactor of quality – contributes 1 million dollars back to the
section (but keeps the coffee).
Revenue and Expenses for Member Value are sufficiently skewed to
show a PAR ratio of 99%, due to the million dollar line item in both revenues
and expenses attributed to Member Value.
1 million is absurd, but $5000 is a palatable speaker fee.
If I were to do the $5000 in-and-out swap, the books would balance but the
totals would be artificially inflated in order to get closer to the target PAR
ratio.
The root cause comes to the overall intent, which is to have a contrived status report based on artificially inflated levels of activity. When the status reports become the drivers of success or failure, the efforts will shift from legitimate quality pursuits (setting up processes and systems to consistently provide successful outcomes, delivering compliant solutions, satisfying customers, controlling outcomes) to contrived results and outcomes (i.e. "interpreted" statistics, "doctored" test results, inflated completion numbers, extrapolated estimates of total savings).
Also there is an obligation on the part of leadership and governance to not place reliance on a single set of measures. This was the flaw in EPA with respect to VW in that after the controlled emissions testing was done, their perceived obligation was met. It was only through independent after-market testing conducted at the University of Virginia that the excessive NOx emissions were revealed. The mitigation is for the testing and control agencies to adopt a more rigorous after-market surveillance program.
If quality is legitimate, the tests are simply a formality. If quality is contrived, then the testing will reveal the shortcomings. These increases in appraisal activities will make the systems harder to manipulate and falsify, thus increasing the assurance.
No comments:
Post a Comment