(Police) Early Intervention System for Reducing Adverse Police Incidents

Early Intervention System for Reducing Adverse Police Incidents

Partner(s): White House, Charlotte Mecklenburg Police Department, Metro Nashville Police Department, San Francisco Police Department, and others.
Status: In Progress
Github Repo: https://github.com/dssg/police-eis

Background
Recent high profile cases of law enforcement officers using deadly force against civilians and other instances of police misconduct have caused a growing political and public uproar. Adverse incidents between police and the public such as deadly shootings, unnecessary use of force, or instances of racial profiling can cause serious or deadly harm. They can also damage police legitimacy and increase mistrust between citizens and the police, leading to further negative interactions. Evidence suggests these events can be prevented by targeting interventions based on an Early Intervention System (EIS) that flags police officers who are at high risk for involvement in such adverse events.

Existing Solutions are not effective
Many large police departments have developed “early intervention systems” to try to identify officers most likely to have adverse interactions with the public (adverse interactions include violations of department rules of conduct, unjustified use of force, civilian or officer injury, and discourteous officer behavior). These systems are developed using two approaches:

Threshold systems flag officers as high risk if the officers have a minimum count of certain events in a given period of time, such as 3 complaints in 180 days.
Outlier systems flag officers as high risk if the officers have an unusual number of certain events compared to other officers.

Unfortunately, both of these systems have several problems:

These systems are neither accurate at predicting future risk nor identifying preventable incidents.
Because they are so simple, threshold systems eliminate important context. For example, CMPD’s system used to have the same thresholds for all officers, whether they work the midnight shift in a high-crime area or the business district at noon. Outlier systems are better because they compare officers in similar assignments, but they also fail to account for important factors, such as the level of risk the group faces (e.g. the lowest-risk officer in a high-risk unit may be at higher risk than the highest-risk officer in a low-risk unit.) Most thresholds and outliers are chosen through expert intuition rather than predictive accuracy.
They are reactive and only support punitive interventions, rather than proactive and supporting preventative interventions.
They either classify an officer as at-risk or not without any prioritization, making police departments unable to prioritize officers in need of interventions.
They are difficult to customize and maintain. Threshold and outlier systems give binary, yes/no flags, and intuition-based systems require experts to continually monitor and modify the thresholds and outliers they use. At least one vendor hard-codes thresholds into their systems, making changes difficult and costly — which is good for the vendor but bad for the department.
They are easily gamed. Multiple departments have raised this concern. Because the thresholds and outliers are so easy to see and understand, officers can modify their bad behaviors slightly to avoid detection.

Our Solution
Our work was initiated in March 2015 as part of the White House’s Police Data Initiative launched out of President Obama’s Task Force on 21st Century Policing. As part of this effort, we discussed data-related challenges with a large number of police departments across the US, and they noted that existing EISs were ineffective. In collaboration with the Charlotte-Mecklenburg Police Department (CMPD) and the Metropolitan Nashville Police Department (MNPD), The Center for Data Science and Public Policy built the first data-driven, machine learning-based Early Intervention System (EIS). It is now deployed and operational at several police departments.

How does it work?

We use machine learning to build a better EIS. Machine learning is the ability of a computer to learn patterns in data and to use those patterns to make accurate predictions. It has been used over the past 30 years to solve thousands of problems, from self-driving cars to providing search results on Google. Machine learning can handle complex data from disparate sources, including dispatches, arrests, field interviews, training, demographics, crime, neighborhood features, and police and media narratives. This enables not only more accurate predictions but also a better understanding of what puts officers at risk.

Our system offers more flexibility. Our system provides continuous risk scores rather than binary flags. Risk scores enable the department to rank all officers by risk, from top to bottom; to explicitly choose tradeoffs (more correct flags versus more incorrect flags); and to allocate by available resources (for example, deciding which officers should receive department-wide training first versus deciding which officers should receive training).

Unlike threshold and outlier systems, our machine learning system allows the department how far ahead the system should predict, down to the next dispatch. This gives the department the ability to decide if it should send lower-risk officers to a call even if they’re farther away.

By combining comprehensive data from various sources inside a police department (HR, internal affairs, dispatches, arrests, stops, etc.), our data-driven, predictive Early Intervention System can be used to focus interventions (training, supervisor counseling, psychological counseling and Employee Assistance Program referrals) on officers who would most benefit in terms of reducing adverse incidents. This system will allow police departments to:

Decrease adverse police incidents with the public – We can improve over the threshold-based systems used in many police departments by identifying and using only the factors that best predict whether an officer will have an adverse incident.
Target limited resources on the officers that most need additional support – Supervisors will be able to use this system to see which officers are most in need of counseling, training, or other assistance to best prepare them to deal safely and positively with individuals in their communities and the high-stress nature of their job.
Reward low-risk officers and model their behavior – Our EIS produces a ranked list of every officer in the department based on their risk of an adverse incident. This allows the department to identify not only officers at high risk but also those with exceptionally low risk.
Evaluate the effectiveness of existing interventions – and develop new ones based on the risk factors for their officers.
Identify the “risk” of other types of incidents – Our EIS can be customized to predict the “risk” of any type of event concerning an officer, such as injuries, complaints, or even compliments. The department can determine the specific outcomes that are most important to them, that they want to increase or decrease, and can adjust the system based on the number and type of interventions available.

Data
Our system is able to take data from multiple sources and use it to build predictors that can generate a risk score for an individual officer. In most departments we’ve worked with, we use any data we can get about an officer including officer demographics, training, payroll, on-the-job actions, internal affairs data (complaints, investigations, reviews of incidents, etc.), dispatch data, stops, arrests, adverse incident reports, as well as publicly available data (such as the American Community Survey and open data portals including 311 and 911 calls).

Analysis
Our system uses historical data about officers to build predictive models that can generate accurate risk scores for an officer being involved in an adverse incident (of various types) in the next 3, 6, or 12 months. We explored different machine learning methods and currently use random forests. More details of our methodology are available in the publications listed below.

Results
Compared to existing Early Intervention Systems, our data-driven system can correctly identify 10-20% more officers who go on to have adverse incidents over a 12-month period while reducing the false positives by 50-60%. This improved capability allows police departments to effectively and efficiently identify at-risk officers and provide them the necessary preventative interventions before an adverse incident occurs.

We have demonstrated our system’s performance in two departments. The Charlotte-Mecklenburg Police Department (CMPD) and Metropolitan Nashville Police Department (MNPD) gave us data on officer attributes (e.g. demographics, join date), officer activities (e.g. arrests, dispatches, training), and internal affairs investigations (the case and outcome). We supplemented their data with publicly available sources, such as American Community Survey data, weather data, shapefiles, and quality-of-life surveys. We then simulated history by showing how our system would have done if each department had used it before running evaluation pilots.

We used CMPD’s EIS thresholds as the baseline for both departments (MNPD doesn’t have an EIS.) Our system correctly flags 10-20% more officers who go on to have adverse incidents while reducing incorrect flags by 50% or more.

The work for each department focused on slightly different tasks. With CMPD, we predicted all major adverse incidents, ignoring more minor violations such as uniform issues. With MNPD, we predicted sustained complaints and disciplinary actions.

We have updated the system to automatically incorporate supervisor feedback. Supervisors have expertise and information about officers that the EIS does not, so we are collecting their feedback on the EIS’s risk assessments and weighing their feedback based on their past accuracy.

Future Plans and Areas for Improvement
Our EIS was deployed and went live at Charlotte-Mecklenburg Police Department in November 2017, becoming the first data-driven Early Intervention System in use. CMPD built a web interface for the EIS that not only incorporates the risk score with an officer’s history but also collects supervisor feedback on the officer risk as well as intervention information. The EIS will improve based on the feedback from the department supervisors and the IA team, and we will conduct intervention analysis to understand which interventions work, which of them are effective for what types of risks and officers, and the most effective ways to reduce future risk and improve policing. The Metropolitan Nashville Police Department also uses our EIS to notify supervisors across the department of officers at risk of adverse incidents.

Awards
The International Association of Chiefs of Police and Laura and John Arnold Foundation recognized our work with the Charlotte-Mecklenburg Police Department with the 2018 Leadership in Law Enforcement Research Award.

Publications
Early Intervention Systems: Predicting Adverse Interactions Between Police and the Public. Jennifer Helsby, Samuel Carton, Kenneth Joseph, Ayesha Mahmud, Youngsoo Park, Andrea Navarrete, Klaus Ackermann, Joe Walsh, Lauren Haynes, Crystal Cody, Major Estella Patterson, and Rayid Ghani. Criminal Justice Policy Review . Vol 29, Issue 2, pp. 190 – 209. March 9, 2017.

Identifying Police Officers at Risk of Adverse Events. Carton, S., Helsby, J., Joseph, K. , Mahmud, A., Park, Y., Walsh, J., Haynes, L., Cody, C., Patterson, E., Ghani, R. 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2016).

Building Better Early Intervention Systems. Crystal Cody, Estella Patterson, Kerr Putney, Jennifer Helsby, Joe Walsh, Lauren Haynes, Rayid Ghani, Samuel Carton, Kenneth Joseph, Ayesha Mahmud, Youngsoo Park. Police Chief Magazine. 2016.

Deploying Machine Learning Models for Public Policy: A Framework. Klaus Ackermann, Joe Walsh, Adolfo De Unánue, Hareem Naveed, Andrea Navarrete Rivera, Sun-Joo Lee, Jason Bennett, Michael Defoe, Crystal Cody, Lauren Haynes and Rayid Ghani. 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2018).

Media Coverage
Can Police Use Data Science to Prevent Deadly Encounters? Scientific American
How Science Could Help Prevent Police Shootings. Mother Jones.
U. of C. researchers use data to predict police misconduct. Chicago Tribune
We Now Have Algorithms To Predict Police Misconduct. FiveThirtyEight
US police use machine learning to curb their own violence. New Scientist

Videos
DSSG 2015 https://www.youtube.com/watch?v=eiA74XXGSF8
DSSG 2016 https://www.youtube.com/watch?v=nGoe2dLHogs
DSSG 2018 https://www.youtube.com/watch?v=GZ4MzxdaM_g
KDD 2016 https://www.youtube.com/watch?v=SFGvmrJ5rjM

University of Chicago https://www.youtube.com/watch?v=_p9Cz8MJ0Mw