Lesson Report:
Title
Good Intentions, Bad Outcomes: Algorithmic Bias, Proxies, and Predictive Policing
Synopsis: The session examined how algorithmic systems can produce unequal, harmful outcomes even when designed for socially valuable goals. Using the Allegheny child welfare risk-scoring system as a case study, students analyzed flawed proxies and biased datasets, then extended the analysis to a predictive policing scenario to surface feedback loops, representation problems, and real-world harms. The class closed by reframing the course toward solution-oriented policy design for responsible AI governance.
Attendance
– Absent: 0 students explicitly mentioned
– Notes: 1 student disconnected and rejoined; 1 student arrived late
Topics Covered (chronological, with activity/topic labels)
1) Conceptual Shift and Review: From malicious intent to systemic harm
– Framed the week’s shift: earlier focus on intentional misuse of algorithms versus today’s focus on how system design, proxies, and data bias can create social inequalities even with good intentions.
– Revisited “biasâ€� in this context: bias as judgment clouded by outside factors, leading to systematically skewed outcomes. In algorithmic bias, overrepresented patterns in training data push outputs toward majority/Western contexts.
– Illustrative examples:
– LLM advice skew: When asked about Central Asian social/family issues, models default to Western-centric advice unless carefully constrained.
– Image generation skew: “Bishkekâ€� stylized to look like other more represented cities; reflects data overrepresentation rather than malicious instruction.
2) Reading Debrief: Allegheny child welfare risk algorithm (AFST-like system)
– Purpose: Predict likelihood of child abuse/neglect to support caseworker screening decisions via a risk score. Intended as one input but became over-relied upon despite acknowledged flaws.
– Key concepts reviewed and applied:
– Proxies (flawed proxies):
– Identified proxies from the reading: community referrals; child placement (e.g., foster care).
– Why flawed:
– These measure system activity/administrative events rather than the underlying harm. They cannot directly observe abuse/neglect.
– “Foster careâ€� proxy structurally biases against adoptive/foster parents by elevating risk score solely due to placement, increasing the chance of unwarranted scrutiny or intervention.
– Rationale acknowledged: some harm does occur in foster contexts, but proxy use overgeneralizes and mislabels many safe placements.
– Biased datasets:
– Data sources: government databases (state-run services, therapists employed by the state, and other public systems).
– Invisibility problem: Families using private services (therapy, child welfare supports) leave fewer/no records in government data; they are “invisibleâ€� to the algorithm.
– Over-/under-sampling:
– Poor families: overrepresented due to heavier interaction with public services; more data trails increase likelihood of being flagged.
– Wealthy families: underrepresented because private services don’t feed the same databases, reducing algorithmic visibility of comparable issues.
– Concrete example discussed: A parent using a state therapist was flagged because the report entered a public database; a similarly situated wealthy parent using private therapy would likely not be flagged.
– Human discretion vs. algorithmic inflexibility:
– Pamela Simmons’ quote: preference for a person over “a number you can’t fix.â€�
– Discussion takeaway: Algorithmic scores lack contextual flexibility; human decision-makers can weigh nuance, hear explanations, and adapt decisions.
3) Framing the Broader Governance Question
– Introduced the policy debate: Should we cede more political/administrative decisions to “objectiveâ€� computational systems?
– Set the analytical lens: Even “cold, hard dataâ€� decisions embed proxy choices and data biases that can launder past inequities as neutral.
4) Activity 2: Predictive Policing Case—“Crime Forecast�
– System brief:
– Private AI tool promises to predict streets at risk of street robberies to better deploy patrols.
– Trained on 10 years of arrest data for street robbery.
– Operational rule: If a street exceeds a threshold of robberies in a year, it is auto-flagged “at risk,â€� triggering increased patrols.
– Breakout prompt (4 guiding questions):
1) What problem does this project claim to solve?
2) Who will be over-/under-represented and why?
3) Are arrests a suitable proxy for crime? What’s the difference?
4) What are the real-world harms of false positives?
5) Plenary Debrief: Findings on Crime Forecast
– Claimed problem solved:
– Resource allocation: With limited officers and budgets, concentrate patrols where risk is predicted instead of blanket coverage.
– Practical goals surfaced: Identify “at riskâ€� vs. “saferâ€� areas, reduce robberies, and improve efficiency.
– Representation dynamics:
– Overrepresentation: Poor and minority neighborhoods with historically heavier police presence generate more arrests and thus more training data; they are more likely to be flagged again.
– Underrepresentation: Wealthier/low-police areas seem “saferâ€� in the data because crimes there are less visible, less reported, or handled privately, yielding fewer arrests.
– Core insight: The dataset reflects historical policing patterns more than an unbiased map of crime itself.
– Proxy critique (arrest vs. crime):
– Arrest is a system event dependent on police presence, discretion, and visibility; crime is a legal violation, much of which goes undetected, unreported, or not pursued.
– Examples raised: Financial crimes, domestic violence, and some drug offenses may occur at high rates in affluent areas but result in fewer arrests due to lower visibility/reporting.
– Resulting dynamic: The model predicts future policing intensity, not necessarily future crime—creating a feedback loop that recycles older biases under an “objectiveâ€� veneer.
– Harms of false positives (flagging an area that isn’t actually at elevated risk):
– Misallocation: Wastes patrol resources; opportunity costs for areas with genuine need.
– Community impacts: Heightened anxiety, perceived stigmatization, potential erosion of trust, and possible protests.
– Enforcement bias: Increased stops and arrests of innocents; behaviors overlooked in “safeâ€� areas may be penalized in “at riskâ€� zones, entrenching unequal enforcement.
– Crime displacement: Offenders may shift to under-policed areas if deployment patterns become predictable.
– Synthesis: Arrest-as-proxy plus biased historical data yields a “closed loopâ€� where past policing decisions drive future deployments, amplifying inequities while appearing data-driven.
6) Closing, Assignments, and Scheduling Notes
– Policy memo assignment:
– Due: Saturday, November 1 (about three weeks from class date).
– Length: 4 pages.
– Task: Select a jurisdiction (country/state/region) with a concrete AI-related problem (e.g., Allegheny-like screening, predictive policing, deepfakes). Define the problem and propose actionable policy solutions.
– Sources requirement: At least two in-class sources and at least two outside sources.
– Instructor to post expanded instructions on eCourse and email students lacking access.
– Video journals:
– Due: Tonight.
– Schedule:
– Class meets next week (AUCA fall break is the week after: Oct 20–24). Note that another institution’s break does not affect this class.
Actionable Items
Urgent (before next class)
– Post detailed policy memo instructions and rubric to eCourse; include exemplar outlines and acceptable topic scope.
– Email full assignment brief to the student without eCourse access; coordinate with IT/registrar to expedite their eCourse enrollment.
– Collect and acknowledge receipt of video journals submitted tonight.
– Clarify and confirm the policy memo due date (Saturday, Nov 1) across all platforms to avoid confusion.
High priority (this week)
– Provide citation details for the Allegheny reading (title/author/link) so students can reference it properly in memos.
– Share a short guide on evaluating proxies vs. targets (e.g., arrest vs. crime) and on identifying representation bias in datasets; include 2–3 recommended external readings.
– Outline next session’s plan on AI in governance and predictive systems, highlighting mitigation strategies (e.g., auditing datasets, proxy validation, fairness constraints, human-in-the-loop).
Medium priority
– Follow up with IT to resolve the lingering eCourse access issue for affected student(s).
– Prepare a brief mini-lecture on feedback loops in predictive systems with concrete examples and visual diagrams to support the Crime Forecast discussion.
– Compile a list of acceptable “places and problemsâ€� for memos (e.g., EU AI Act context, state-level child welfare tools, city-level predictive policing pilots, platform content moderation systems).
Longer-term
– Design an in-class activity on alternative metrics and safeguards (e.g., measuring service need, randomization to break loops, auditing for disparate impact).
– Plan a peer-review workshop for memo drafts one week before the due date to improve solution quality and sourcing.
Homework Instructions:
ASSIGNMENT #1: Weekly Video Journal — Bias, Proxies, and Predictive Policing
You will synthesize today’s discussion and reading by reflecting on how good intentions in algorithm design can still produce harmful outcomes. Use the Allegheny child welfare algorithm and our “Crime Forecast� exercise to demonstrate your understanding of biased data, flawed proxies, over/under-representation, and the real-world harms of false positives.
Instructions:
1) Select your focus: Choose one theme from today’s session to center your reflection on:
– Flawed proxies (e.g., arrest as a proxy for crime; foster care placement as a proxy for harm)
– Biased/imbalanced datasets (e.g., poor families over-represented, wealthy families under-represented)
– Feedback loops (e.g., predictive policing reinforcing past policing patterns)
– Human vs algorithmic judgment (e.g., Pamela Simmons’ “you can’t fix that numberâ€� insight)
2) Describe the concept in your own words: Briefly explain what the concept means and why it matters for fair decision-making.
3) Ground it in today’s material: Use a concrete example from class:
– Allegheny example: how reliance on government databases made families using private services “invisible,â€� and how foster care placement unfairly increased risk scores for adoptive parents.
– Crime Forecast example: how using arrest data predicts policing patterns rather than true crime risk, potentially over-policing poorer neighborhoods and under-detecting harm elsewhere.
4) Analyze the harm: Explain at least one potential real-world consequence (e.g., unnecessary investigations, wasted resources, stigma, wrongful arrests, or unequal treatment).
5) Connect to solutions or safeguards: Offer one practical step that could reduce harm (e.g., audit datasets for representation, replace/validate proxies, limit automated triggers, require human-in-the-loop review, create an appeal process).
6) Close with a question: End by posing one question you want to explore next (e.g., “How can we measure harm directly rather than via flawed proxies?�).
7) Submit your video: Follow the usual course guidelines for video journals and submit it tonight. Make sure your audio is clear and include your name and topic in the filename.
ASSIGNMENT #2: Policy Memo (4 pages) on an AI Governance Problem — Due Saturday, November 1
You will write a four-page policy memo that identifies an AI-related problem in a specific place and proposes concrete solutions. This assignment builds directly on today’s themes: proxies vs. direct measures, biased datasets, over/under-representation, feedback loops in deployment, human vs. algorithmic discretion, and the tangible harms of false positives.
Instructions:
1) Choose your case:
– Select a specific jurisdiction (e.g., a country, state, city, or regional body like the EU).
– Select one AI issue occurring or likely to occur there (examples discussed: Allegheny-style child welfare risk scoring; predictive policing like our “Crime Forecastâ€� scenario; deepfakes/synthetic media harms; student dropout prediction systems; etc.).
2) Define the problem clearly:
– Briefly explain how the system works and its intended goal.
– Identify the proxies it relies on (e.g., arrests as a stand-in for crime; foster care placement as a stand-in for harm) and why those proxies might be flawed.
– Describe the data sources used and any representation issues (who is oversampled/undersampled or invisible and why).
3) Diagnose risks and harms:
– Explain how biased data and flawed proxies can produce feedback loops (e.g., policing patterns re-labeled as “objectiveâ€� predictions).
– Identify concrete harms of false positives and false negatives in your context (e.g., wrongful arrests or investigations, wasted resources, inequities in who is scrutinized or protected).
4) Stakeholder analysis:
– List key stakeholders (e.g., affected families/communities, agencies, service providers, law enforcement, courts, NGOs) and how each is impacted.
5) Develop policy options (at least three):
– Examples to consider: replace or validate proxies; require regular bias and performance audits; cap the algorithm’s influence in decisions; mandate human-in-the-loop review and a clear appeal pathway; improve data quality/coverage (e.g., incorporate non-governmental service data where lawful and appropriate); increase transparency about model inputs/limitations; pause or restrict deployment until validation targets are met.
– For each option, note at least one likely benefit and one trade-off/limitation.
6) Make a recommendation:
– Choose the option (or combination) you believe best addresses the problem in your jurisdiction and justify it using today’s core concepts (bias, proxies, representation, feedback loops, and real-world harm).
7) Implementation plan:
– Outline concrete next steps: responsible bodies, needed resources/skills, timelines for audits/validation, and how you’ll monitor for unintended consequences after rollout.
8) Use evidence:
– Cite at least two in-class sources (e.g., the Allegheny reading and today’s discussion) and at least two credible outside sources to support your analysis and recommendations.
9) Deliverable:
– A four-page memo (not including references). Use a consistent citation style. Submit by Saturday, November 1.