SFPD Early Intervention System: DataSF

Partner(s): San Francisco Police Department and DataSF
Status: Completed in January 2018
Github Repo: https://github.com/dssg/sfpd-eis/
Team:  Hareem Naveed, Joe Walsh, Adolfo De Unanue, Lauren Haynes, Rayid Ghani

Summary
DSaPP partnered with the San Francisco Police Department (SFPD) and DataSF, San Francisco’s open data initiative, to test the replicability of DSaPP’s Early Intervention System (EIS) that identifies police officers at risk of adverse incidents. Like many police departments across the country, SFPD has an EIS that flags officers using simple thresholds, such as three citizen complaints within a six-month period. The existing system’s high rate of false flags has not only overloaded the EIS team; it has also led to a legitimacy crisis in the department, where EIS flags may not be not acted upon. DSaPP’s EIS proved to be more accurate and flexible than SFPD’s existing system, giving the department more control over false flags and helping them target interventions more efficiently.

Background
Like many departments across the country, the San Francisco Police Department has an Early Intervention System (EIS) that flags officers using simple thresholds, such as three citizen complaints in six months. EISs try to identify officers with patterns of problematic performance or signs of job and personal stress in order to prevent adverse incidents through targeted training, counseling, and other interventions. The SFPD system’s high rate of false flags has overloaded their EIS team and led to a legitimacy crisis in the department, where EIS flags are not acted upon or are seen as a point of pride. SFPD wants an EIS that provides more accurate warnings earlier so their interventions can be applied effectively.

Goals
With support from the Civic Analytics Network and the Laura and John Arnold Foundation, DSaPP partnered with SFPD and DataSF to replicate DSaPP’s EIS for identifying police officers at risk of adverse incidents. DSaPP’s system is the first data-driven EIS to be built and deployed, with implementations running in the Charlotte-Mecklenburg and Metropolitan Nashville Police Departments. DSaPP’s system has shown to be more accurate, insightful, and actionable than threshold-based systems, such as SFPD’s. This project aimed to quantify the historical performance of SFPD’s existing system and to estimate the performance improvements SFPD could expect if they were to implement DSaPP’s system.

Data
DSaPP used SFPD’s internal data to build the prototype EIS. The data included officer characteristics such as demographics, historical job activities (e.g. traffic stops, dispatches, arrests, use of force, vehicle pursuits), previous job feedback (e.g. civil suits, compliments, tort claims), training, secondary jobs, and  days taken off. Most of the data covered the last eight years. The department also provided a definition of “adverse incident” (where an Internal Affairs investigation leads to a sustained, unjustified, or preventable finding) and the department’s intervention capacity. Much of the department’s data were of limited use due to aggregations of time (e.g. data aggregated by month) and officers (e.g. data not stored at an individual officer level).

Analysis
DSaPP’s data-driven EIS takes all available data and uses machine learning to detect patterns that precede adverse incidents, making the EIS predictive and enabling prioritized and preventative interventions. Our prototype analyzed thousands of variable combinations (e.g. traffic stops, arrests, uses of force, compliments, dispatches) over time to determine which best identify officers at risk. The prototype generated a risk score for every officer for a given day, which the department can use to prioritize officers for interventions.

DSaPP evaluated the prototype and SFPD’s existing system on historical predictive performance. Both systems made officer predictions only using data available at the time of prediction. The main performance metric was efficiency: the proportion of flagged officers who went on to have an adverse incident. Both systems were repeatedly tested over time, from 2010 to 2017.

Results
Historical data showed DSaPP’s EIS to be twice as efficient as SFPD’s existing EIS. Unlike the SFPD EIS, DSaPP’s system is more flexible because it allows the department to choose how many officers are flagged. This means the department can choose more coverage (e.g. for the same total number of officers flagged, the system will identify 2-3 times as many officers who go on to have adverse incidents in the next year) or more efficiency (identify the same number of officers who go on to have adverse incidents while the total number of officers flagged is 2-3 times fewer), depending on its needs.

The DSaPP EIS also flags different types of officers. The existing EIS flags officers with many historical EIS alerts, Use-of-Force incidents, and more activity, in general. Because it draws on a wider range of data, the DSaPP EIS generates flags with more accuracy and nuance, without relying purely on activity or historical threshold-based EIS alerts.

Future Plans and Areas for Improvement
Based on the results, we recommended that the SFPD explore the development and implementation of a data-driven EIS. DSaPP suggested two steps: 1) Conduct a pilot to validate our findings and 2) Improve the department’s data collection and storage.