Types of Evaluations in Policing, Limits of RCTs and the Maryland Scientific Methods Scale
Griffith Discussion: Types of Evaluations in Policing, Limits of RCTs and the Maryland Scientific Methods Scale
Explain the different types of evaluations, including the method and strengths and weaknesses of each. What questions about policing cannot be evaluated by an RCT, and what type of evaluation would fit better? Give examples of police research questions best suited to each of the five levels of the Maryland Scientific Methods Scale.
Was it delivered as planned?
Did it reduce crime?
Was it worth the cost?
Ethics, rare events, whole-city policies
Correlation to randomization
Model response
Original example for study purposes (about 420 words). Sample discussion post; cite your Griffith course readings alongside these sources.
Types of evaluation. Needs assessments identify the nature and size of a problem, using crime data, surveys and community input; they are strong for targeting resources but say nothing about effectiveness. Process (implementation) evaluations examine whether a program was delivered as intended, using observation, interviews, records and fidelity checks; they explain why programs succeed or fail but cannot show impact. Outcome or impact evaluations test whether a program changed results such as crime, victimization or public trust, using experimental or quasi-experimental designs; they are the strongest test of effectiveness but are costly and sensitive to design quality. Economic evaluations, such as cost-benefit or cost-effectiveness analyses, compare costs with outcomes; they inform funding decisions but depend on assumptions about how to value outcomes (Rossi et al., 2019).
Questions an RCT cannot easily answer. Randomized controlled trials are the strongest design for causal claims, and stronger designs tend to report less favorable results than weaker ones (Weisburd et al., 2001). But many policing questions cannot be randomized ethically or practically. For example, “Did a statewide use-of-force law reduce police shootings?” cannot be randomized because the law applies to everyone at once; a time-series or difference-in-differences design comparing states would fit better. “How do residents experience a new community policing strategy?” is better answered by qualitative or mixed-methods process evaluation. Rare events, such as officer-involved shootings, may also need long time series rather than trials.
Maryland Scientific Methods Scale examples. The scale ranks study designs from 1 (weakest) to 5 (strongest) according to their ability to rule out alternative explanations (Farrington et al., 2002; Sherman et al., 1997).
- Level 1, correlation at one point in time: “Are neighborhoods with more foot patrols associated with lower fear of crime?”
- Level 2, before-and-after measures without a comparison group: “Did burglaries in one district fall after a new burglary task force began?”
- Level 3, before-and-after with a comparable control area: “Did robberies fall in streets with new CCTV cameras compared with similar streets without cameras?”
- Level 4, before-and-after with multiple experimental and control units, controlling for other variables: “Did body-worn cameras reduce complaints across several precincts compared with matched precincts, adjusting for staffing and call volume?”
- Level 5, random assignment of units to treatment and control: “Does directed patrol at randomly assigned crime hot spots reduce calls for service compared with control hot spots?” Hot spots policing experiments are classic examples of Level 5 research.
Choosing the strongest feasible design, and pairing impact evaluation with process evaluation, gives police leaders both evidence of what works and an understanding of why.
References
Farrington, D. P., Gottfredson, D. C., Sherman, L. W., & Welsh, B. C. (2002). The Maryland Scientific Methods Scale. In L. W. Sherman, D. P. Farrington, B. C. Welsh, & D. L. MacKenzie (Eds.), Evidence-based crime prevention (pp. 13–21). Routledge.
Rossi, P. H., Lipsey, M. W., & Henry, G. T. (2019). Evaluation: A systematic approach (8th ed.). SAGE.
Sherman, L. W., Gottfredson, D., MacKenzie, D., Eck, J., Reuter, P., & Bushway, S. (1997). Preventing crime: What works, what doesn’t, what’s promising. National Institute of Justice.
Weisburd, D., Lum, C. M., & Petrosino, A. (2001). Does research design affect study outcomes in criminal justice? The Annals of the American Academy of Political and Social Science, 578(1), 50–70. https://doi.org/10.1177/000271620157800104
Tip: give each SMS level a distinct policing example and say briefly why the design fits; instructors look for understanding of what each level controls for.
Frequently asked questions
What is the Maryland Scientific Methods Scale?
A five-level scale developed to rate the methodological rigor of crime prevention evaluations, from correlational studies (Level 1) to randomized experiments (Level 5).
Why not always use an RCT?
Some interventions cannot be randomized for ethical, legal or practical reasons, such as laws that apply to everyone.
More model assignments posted today
- Organizational culture analysis
- Social justice leadership slides
- Virtual MD mental health triage list
- DNP880 leadership self-assessment paper
- NU673 PTSD in older adults journal
- Family conflict situation analysis
- Senge laws leadership journal
- System archetypes and leadership
- Networked leadership discussion