Pretrial risk assessment tool developed for Alaska
Beginning January 1, 2018, new information about defendants at their first pretrial bail hearing became available in all of Alaska’s courts. Judicial officers, defense, and prosecuting attorneys are receiving information from a new pretrial risk assessment tool that calculates whether a defendant is at low, moderate, or high risk for failure to appear at trial or to commit another crime if released. The tool, incorporated in Alaska’s new bail statute, aids in the judicial officer’s decision regarding pretrial bail conditions.
The turn to evidence-based pretrial practices is in response to the growing number of defendants who are remaining in custody through disposition of their cases. From 2004 to 2014, the number of pretrial inmates in Alaska’s prisons grew by 81 percent (Alaska Criminal Justice Commission (ACJC), 2017). “[I]n some cases, low-risk defendants who were unlikely to engage in new criminal activity remained behind bars because they couldn’t afford bail, while high-risk defendants who were likely to engage in new criminal activity and who paid bail were released” (ACJC, 2017: 17).
A review of defendants released pretrial from 2014 to 2015 in Alaska found that the likelihood that a person released from jail on bail would fail to appear (FTA) for their court hearings was 14 percent. The likelihood that they would be re-arrested on another offense while out on bail was 37 percent (Crime and Justice Institute, 2017).
Alaska’s new pretrial assessment tool will improve these numbers and public safety, according to Geri Fox. Fox leads the Alaska Department of Corrections’ Pretrial Enforcement Division. The division, created in 2016, is performing pretrial risk assessments on all defendants, as well as providing court reports and recommendations, monitoring individuals released pretrial, and providing other pretrial supervision services.
Risk assessment tools are being used throughout the country to aid in pretrial decisions as well as sentencing, probation, and parole. This article looks at risk assessment tools in general and the development of Alaska’s pretrial risk assessment tool.
History of assessment tools
The use of predictive models in criminal justice goes back to the 1920s and efforts to address crime by incapacitating “career criminals” (Kehl, Guo, & Kessler, 2017: 3).
Many early models relied on simple math and the assessment of correctional staff and clinical professionals. In the 1960s and early 1970s, studies questioned criteria being used by the models, their accuracy, and individual fairness (Kehl et al., 2017: 4–5).
Over time, risk assessment tools have evolved, with the largest shift accompanying a movement toward evidence-based practices. “Evidence-based risk/needs assessment instruments consider the interplay between static and dynamic risk factors,” according to Kehl at al. (2017: 8; emphases in original).
Static factors are those that do not change, including age at first arrest and current charge. Dynamic factors are those that can change over time, including current age, employment status, and whether a person has a substance use disorder.
Dynamic factors are often used to determine programming and treatment in addition to risk, since they provide a window into an offender’s criminogenic needs. These factors, which are collected in interviews, have the potential drawback of perpetuating gender and racial bias.
The drawback of static factors is that their immutability makes it more difficult for a defendant to show positive behavioral change (Bonta & Andrews, 2007). The latest generation of risk assessment tools use complex algorithms and large data sets that can be tweaked and adjusted over time to new data.
Alaska’s pretrial tool
Alaska worked with the Crime and Justice Institute (CJI), a division of the Boston-based nonprofit research and analysis organization Community Resources for Justice, to develop an Alaska-specific pretrial risk assessment tool for two reasons. First, while pre-existing open tools such as the Arnold Foundation’s Public Safety Assessment (PSA) are available, they have not been validated against Alaska populations. Second, many off-the-shelf commercial tools are proprietary — details of how they work are not made public, which has caused some challenges. (See “Proprietary and open risk assessment tools,” below.)
CJI used sample data from the Department of Corrections, Alaska Court System, and Department of Public Safety that was comprised of defendants who were either released from custody during the pretrial period (N=20,456) or who were detained and released on or after disposition of their case (N=8610). After cleaning and coding, 19,188 cases were identified to develop the pretrial risk assessment of failure to appear (FTA) and new criminal arrest (NCA).
Similar to PSA, Alaska decided to use only static risk factors. These factors are collected electronically without the need for an interview.
CJI found that not all potential risk factors had strong correlations with FTA or NCA or by gender and race (Table 1).
In addition, risk factors for FTA did not always predict well for NCA. For instance, total prior FTA warrants, FTA warrants in the past 3 years, and current FTA charge were all found to be predictive of future FTA, but not predictive of NCA. As a result, two scales were developed to contain the strongest predictors for each measure (Tables 2 and 3). (Judges will have to reconcile the two scales when using the new bail statute that only refers to one scale. Suggestions for reconciling this include using the highest on either scale to determine highest risk; see Table 4.)
Once the list of predictors was established, they were tested in terms of gender and race to make sure that they were equally predictive whether a defendant was male or female, White or Alaska Native (CJI, 2017).
The judge is still going to consider statutory guidelines such as the nature and circumstances of the offense, weight of the evidence, family ties, employment, length of residence, conviction record, FTA record, danger defendant poses to the victim, and reputation, character, and mental condition (AS 12.30.020 (i)).
Prosecutors and defense attorneys will receive information from the tool prior to a bail hearing and continue to play a critical role in assisting the court with relevant information, according to Fox.
“The judge has limited time to look at a case, try to understand it, and evaluate the risk. Alaska will now have an assessment to provide judges with some actuarial, statistical analysis of what we might be able to expect with defendants,” Fox said.
Although judges have discretion to make bail decisions, research shows that when presented with an algorithm, judges and prosecutors frequently give the actuarial analysis more weight. Rejection of the algorithm is often based on bias (Christin, Rosenblat, & Boyd, 2015: 7).
Studies also suggest that a well-designed algorithm may be far more accurate than a judge alone (Neufeld, 2017).
Transparency and oversight are two features of assessment tools that critics call essential to reducing inequities (Tashea, 2007).
Fox is committed to continuing to improve Alaska’s tool while providing information about how it is being used. (See “Limitations and quality assessment of Alaska pretrial screening tool” below.)
Pamela Cravez is editor of the Alaska Justice Forum.
Alaska Criminal Justice Commission (ACJC). (2017). Alaska Criminal Justice Commission Annual Report: October 22, 2017. Alaska Criminal Justice Commission.
Angwin, Julia; Larson, Jeff; Mattu, Surya; & Kirchner, Lauren. (2016). “Machine Bias: There’s Software Used across the Country to Predict Future Criminals. And It’s Biased against Blacks.” ProPublica (23 Mar 2016).
Bonta, James; & Andrews, D.A. (2007). Risk-Need-Responsivity Model for Offender Assessment and Rehabilitation. Public Safety Canada.
Christin, Angèle; Rosenblat, Alex; & Boyd, Danah. (2015). “Courts and Predictive Algorithms.” Presented at Data & Civil Rights: A New Era of Policing and Justice conference, Washington, DC, 27 Oct 2015.
Crime and Justice Institute (CJI). (2017). “Alaska Pretrial Risk Assessment” (webinar; 1 hr. 31 mins). Justice Reinvestment Initiative. Boston, MA: Crime and Justice Institute.
Holder, Eric. (2014). Remarks presented at the National Association of Criminal Defense Lawyers 57th Annual Meeting and 13th State Criminal Justice Network Conference, Philadelphia, PA, 1 Aug 2014.
Kehl, Danielle Leah; Guo, Priscilla; & Kessler, Samuel Ari. (2017). Algorithms in the Criminal Justice System: Assessing the Use of Risk Assessment in Sentencing. Cambridge, MA: Responsive Communities Initiative, Berkman Klein Center for Internet & Society, Harvard Law School.
Neufeld, Adam. (2017). "Commentary: In Defense of Risk-Assessment Tools — Algorithms Can Help the Criminal Justice System, but Only Alongside Thoughtful Humans." The Marshall Project (22 Oct 2017).
State v. Loomis, 881 N.W.2d 749. (Supreme Court of Wisconsin 2016), 13 Jul 2016.
Tashea, Jason. (2017). "Risk-Assessment Algorithms Challenged in Bail, Sentencing and Parole Decisions." ABA Journal (Mar 2017).
Alaska, Virginia, and Pennsylvania use risk assessment tools developed specifically for their state. Most, jurisdictions, though, use one of the commercial risk-assessment tools. The Level of Service Inventory – Revised (LSI-R), developed by Multi-Health Systems (the LSI-R isn’t used in pretrial), and COMPAS, created by the Northpointe company are two popular tools. These commercial tools employ both static and dynamic factors. COMPAS, which uses proprietary software and offers little transparency regarding its calculations, has been the subject of controversy. In a recent ProPublica investigative journalism piece on the use of COMPAS in Broward County, Florida, it was found that the tool predicted re-arrest at an accuracy rate of 61 percent, “somewhat more accurate than a coin flip.” ProPublica also found that the COMPAS algorithm predicted black offenders to be “future criminals” at twice the rate of white offenders (Angwin, Larson, Mattu, & Kirchner, 2016; see also State v. Loomis, 2016).
In 2014, U.S. Attorney General Eric Holder voiced concern about risk assessment tools. “Although these [risk assessment] measures were crafted with the best intentions, I am concerned that they may inadvertently undermine our efforts to ensure individualized and equal justice.” Speaking at the annual meeting of the National Association of Criminal Defense Lawyers, Holder added that the tools “may exacerbate unwarranted and unjust disparities that are already far too common in our criminal justice system and our society.”
Risk assessment tools used for pretrial decisions generally focus on static risk factors. The Public Safety Assessment (PSA), developed by the Laura and John Arnold Foundation, is used by 29 jurisdictions in the country including all of Arizona, Kentucky, and New Jersey (Kehl et al., 2017: 10). PSA uses a narrow group of static risk factors — offender’s age at time of arrest, criminal history, prior FTA’s — and is based on data from 1.5 million crimes spanning 300 U.S. jurisdictions. Unlike proprietary, blackboxed commercial tools such as COMPAS, PSA makes all factors open to public scrutiny.
Lucas County, Ohio adopted the PSA tool in January 2015. A study funded by the Arnold Foundation found no race or gender bias in outcomes. Those released without bail increased from 14 percent to about 28 percent. Those out on release who were arrested for another crime was cut from 20 percent to 10 percent (Tashea, 2017).
Some of the strategies the Pretrial Division team will use to ensure quality pretrial assessment is a process they refer to as Inner-Rater Reliability (IRR), according to Pretrial Division Director Geri Fox. Every month, approximately six percent of all assessments will be scored by another officer who is unaware that the assessment was previously scored. When errors are detected, officers will receive coaching to assist them with future assessment. Officers also receive initial training and follow up training to ensure quality assessment. Finally, the software application has internal checks to reduce potential errors, according to Fox.
Juvenile convictions are not generally part of pretrial assessment tools, Fox pointed out.
The current Alaska pretrial assessment tool lacks out-of-state criminal history information due to FBI security rules for criminal justice data. However, over the next year, Fox’s team will collect information about out-of-state convictions. A new validation study will be completed to include out of state criminal history as part of future pretrial assessments. In the meantime, judges have discretion in most cases to factor any out-of-state criminal history into release decisions. Multiple data points will be tracked over the next few years and outcomes of the new pretrial functions monitored, according to Fox.
The tool will change over time, Fox says, as information is collected about its effectiveness. It will continue to improve. “This is part of the reason criminal justice systems have adopted evidence based practices. Information and quality data can assist with future policy making to enhance public safety.”
The Crime and Justice Institute webinar “Alaska Pretrial Risk Assessment” describes the risk assessment tool, and can be viewed by registering name and email address at https://attendee.gotowebinar.com/recording/1467307448127263490.