Juking the Stats


Metrics, Security and The Wire

An odd confluence recently
.
Sunday - I close out the Thanksgiving holiday with the Wire episode 48 - A New Day. The title refers to many developments, not the least is the change in the Baltimore Police Department from measuring performance through a set of statistics to a focus on major crimes. The performance and accountability of law enforcement was based on a set of periodically compiled crime statistics. These statistics were prone to "juking" - downgrading crimes or underreporting them to achieve the mandated reduction in the crime rate. Post-juke stats may not reflect the conditions in the neighborhood, but are the basis of promotions.
The Wire also portrays juked stats in the schools. The Baltimore schools, so past hope no politician even makes a promise regarding education, are also measured on stats - the standardized test scores which determine How Far the Child is Left Behind. Instead of addressing the problems of socialization and fundamental life skills required by the students at Tighlman Middle School, the focus is on teaching the test to raise the scores to get the funding. Juking the stats.
Monday- At lunch I drop by the book store and pick up a copy of The Restless Sleep by Stacy Horn. I get to page 14 that afternoon, where Ms. Horn outlines the career of Jack Maple and Edward Norris, including the establishment of CompStat in the NYPD. CompStat is the compilation of crime statistics, reported weekly and correllated to geographic data to determine trends in crime and to hold police department management accountable for criminal activity in their precincts. Horn readily credits the CompStat with the drop in crime in NYC in the 90s: "How many would be dead now without CompStat? ... (A) total of 9,598 people walking around who would have otherwise died horribly between 1994 and 2003." But even in 2003, there was significant juking and downgrading, and attribution of CompStat to the decrease in crime remains controversial.

Also on Monday, I read Mr. Walsh's commentary regarding security metrics (closer to my occupational home now than crime-fighting). Security folks have been going round and round regarding how and if the activities of information security folks should be measured.

Which leads me to muse upon this confluence. Did CompStat actually do any good? Do the skills in bubbling in ScanTron sheets appropriately reflect the performance of a school? Patches installed and viruses updated reflecting the actual security of data? There's a common problem here somewhere. If the people writing your paycheck, no matter if they are taxpayer or CEO, don't clearly understand what you do, only what happens in you don't do it, you can easily set up a system of facile statistics that will let you claim your effort has resulted in the status quo. When the status quo stinks hard enough, the stats get juked, and the system goes to hell.

Posted: Mon - November 27, 2006 at 06:45 PM           |

©