A U.S. risk assessment tool used by states across the country to predict crime is actually no better at predicting crimes than an untrained human being, according to a Dartmouth College study released Thursday.
The software, known as Correctional Offender Management Profiling for Alternative Sanctions (COMPAS), is supposed to predict a defendant’s likelihood of committing a misdemeanor or felony within two years of assessment. COMPAS uses 137 variables to predict a defendant’s risk level, but according to the Dartmouth study, it only predicts with 65.4 percent accuracy, the same level of accuracy achieved by a random person responding to an online survey using just seven variables.
“There was essentially no difference between people responding to an online survey for a buck and this commercial software being used in the courts,” researcher Hany Farid, who teaches computer science at Dartmouth, told WIRED. “If this software is only as accurate as untrained people responding to an online survey, I think the courts should consider that when trying to decide how much weight to put on them in making decisions.”
States across the country have adopted COMPAS and programs like it on the assumption that algorithms are inherently more accurate and less biased than human beings, but there seems to be very little evidence that this is true. Farid and a graduate student, Julia Dressel, developed their own assessment to test the accuracy of COMPAS, given that its creator, Equivant, is extremely secretive about its product.
Farid used Amazon Mechanical Turk, an online marketplace where individuals can sign up to take surveys for a small amount of cash. Each participant received the following information about 50 real-world defendants:
The defendant is a [SEX] aged [AGE]. They have been charged with: [CRIME CHARGE]. This crime is classified as a [CRIMINAL DEGREE]. They have been convicted of [NON-JUVENILE PRIOR COUNT] prior crimes. They have [JUVENILE- FELONY COUNT] juvenile felony charges and [JUVENILE-MISDEMEANOR COUNT] juvenile misdemeanor charges on their record.
Turk respondents were able to predict recidivism with 67 percent accuracy using the seven variables, beating out the COMPAS program. Equivant has taken notice of the study and isn’t happy, releasing a statement requesting that Farid hand over the data he used for the study, as well as evidence of peer review.
“Our Research division will review the data upon receipt. It is critical to ensure the appropriate methods, fair comparisons, and conclusions were made as the article is given wide circulation online,” the statement read.
Equivant asserts that Farid’s claim that COMPAS uses 137 inputs is a misnomer, and that the program really only uses six. The other 131 variables are “needs factors” which “are NOT used as predictors.” Regardless of the number of variables, however, Equivant did not contest the study’s conclusion that humans are as accurate as its program.
Farid did not respond to a request for comment from The Daily Caller News Foundation in time for publication.
Send Tips: anders@dailycallernewsfoundation.org
The Daily Caller News Foundation is working hard to balance out the biased American media. For as little as $3, you can help us. Make a one-time donation to support the quality, independent journalism of TheDCNF. We’re not dependent on commercial or political support and we do not accept any government funding.
All content created by the Daily Caller News Foundation, an independent and nonpartisan newswire service, is available without charge to any legitimate news publisher that can provide a large audience. All republished articles must include our logo, our reporter’s byline and their DCNF affiliation. For any questions about our guidelines or partnering with us, please contact licensing@dailycallernewsfoundation.org.