Case Studies Index
Every Real-World Story from This Course — Plus Deep Dives
Each module opens with a real-world case. This index collects them for quick reference and includes three expanded case studies for deeper discussion.
Quick Reference: All 13 Cases
| # | Case | Module | Key Lesson |
|---|---|---|---|
| 1 | Mata v. Avianca — Lawyer submitted 6 fake AI-generated court citations | M1 | AI hallucinates confidently. Always verify. |
| 2 | Amazon’s Hiring Algorithm — Penalized resumes containing “women’s” | M2 | ML inherits bias from historical data. |
| 3 | The Strawberry Problem — LLMs can’t count R’s in “strawberry” | M3 | LLMs predict tokens, not letters. Pattern-match, not understand. |
| 4 | The Whisperers Study — 26 models across 5 providers agreed: intent is #1 | M4 | BRIEFS / The 3 Questions built on what AI itself says it needs. |
| 5 | 47 Citations, 23 Fake — Student’s lit review mixed real and fabricated sources | M5 | AI blends real and fake seamlessly. Verify every citation. |
| 6 | Two Tuesdays — Same job, same person, 90 minutes saved | M6 | Productivity comes from workflow design, not raw tool power. |
| 7 | Pope in a Puffer Jacket — AI image fooled millions | M7 | Photorealistic fakes in seconds. Verification is a survival skill. |
| 8 | The Professor Who Deleted His Slides — AI section 72.20 vs traditional 68.20 | M8 | Well-designed AI integration can outperform traditional instruction. |
| 9 | The $0.002 Experiment — SLM-RAG matched GPT-4 at 92% lower cost | M9 | Small models + RAG = big model performance. Infrastructure is equity. |
| 10 | COMPAS — Algorithm with racially disparate error rates | M10 | Multiple definitions of “fair.” Conscious choices required. |
| 11 | ELEVATE Survey — 92% said AI tutor helps recall prior concepts | M11 | AI in education works when constrained, guided, and measured. |
| 12 | The 28% Premium — AI-skilled professionals earn 28% more | M12 | AI literacy is career capital across every industry. |
| 13 | Every Prediction Is Wrong — “The Internet? Bah!” | M13 | We overestimate short term, underestimate long term. |
Deep Dive #1: Mata v. Avianca — The Lawyer Who Trusted AI
What happened: In 2023, New York attorney Steven Schwartz used ChatGPT to research case law for a federal lawsuit against Avianca Airlines. The AI generated six court decisions as supporting precedents. Schwartz cited them in his brief, filed it with the court, and only discovered a problem when opposing counsel couldn’t find any of the cases. They didn’t exist. ChatGPT had fabricated case names, docket numbers, judicial opinions, and legal reasoning — all completely fictional but entirely convincing.
What went wrong: Schwartz later told the court he had “never used ChatGPT before” and “did not know it could generate false content.” He had asked the AI whether the cases were real, and it confirmed they were. He never checked a legal database. He trusted the tool because the output looked authoritative — proper formatting, realistic case numbers, plausible legal reasoning.
The consequences: Judge Kevin Castel sanctioned Schwartz and his colleague, calling the citations “fake judicial decisions with fake quotes and fake internal citations.” Both attorneys were fined $5,000 each. The case became international news and triggered a wave of AI use policies across law firms and courts worldwide.
Why it matters for you: This isn’t a story about bad lawyers or dangerous technology. It’s about the alignment gap between what AI produces (confident, well-formatted text) and what it guarantees (nothing). ChatGPT doesn’t search legal databases. It predicts what legal text should look like based on its training data. Sometimes those predictions correspond to real cases. Sometimes they don’t. The model itself cannot tell the difference — and neither can you, unless you verify.
Discussion questions: 1. Should Schwartz have been sanctioned? He didn’t intend to deceive — he genuinely believed the cases were real. 2. What verification steps would have caught this in under 10 minutes? 3. Your profession has its own version of “fake citations” — what does it look like in your field, and how would you catch it?
Deep Dive #2: COMPAS and the Impossibility of Fair Algorithms
What happened: COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) is a risk assessment algorithm used in U.S. courts to predict whether a defendant is likely to reoffend. Judges use the scores when making bail, sentencing, and parole decisions. In 2016, ProPublica analyzed COMPAS predictions against actual outcomes for over 7,000 defendants in Broward County, Florida.
What they found: The algorithm’s error rates were starkly different by race. Black defendants who did NOT go on to reoffend were incorrectly flagged as high risk at nearly twice the rate of white defendants (45% vs 23%). Conversely, white defendants who DID reoffend were incorrectly flagged as low risk more often than Black defendants (48% vs 28%). The algorithm was equally accurate overall for both groups — but its mistakes fell in opposite directions.
The mathematical impossibility: Northpointe (COMPAS’s creator) responded that their tool satisfied a different definition of fairness: among people who received the same risk score, the actual reoffense rate was the same regardless of race. Both ProPublica and Northpointe were correct — about different things. Mathematician Alexandra Chouldechova formally proved in 2017 that when base rates differ between groups (as they do in the criminal justice system due to systemic factors), you cannot simultaneously satisfy equal false positive rates, equal false negative rates, and equal predictive accuracy across groups. At least one must be sacrificed.
Why it matters for you: Every AI system that makes predictions about people faces this dilemma. A hospital’s patient risk model. A bank’s loan approval algorithm. A school’s early warning system. There is no single definition of “fairness” that satisfies everyone. The choice of which fairness metric to prioritize is a values decision — and it should be made transparently, by humans, with input from affected communities. Not buried in a technical specification that nobody reads.
Discussion questions: 1. If you had to choose between equal false positive rates and equal predictive accuracy, which would you prioritize? Why? Does your answer change depending on the context (criminal justice vs healthcare vs education)? 2. The algorithm doesn’t use race as an input. But it uses zip code, employment history, and prior arrests — all correlated with race due to systemic factors. Is this “using race” or not? 3. Who should decide which definition of fairness an algorithm uses? The developers? The judges? The affected communities? Legislators?
Deep Dive #3: The SLM-RAG Experiment — Small AI, Big Impact
What happened: In 2025, a research team asked a simple question: could a small, cheap AI model running on a single campus GPU match the performance of GPT-4 for domain-specific educational tasks? They set up a 7-billion parameter open-source model (Mistral 7B) on university hardware, connected it to a Retrieval-Augmented Generation (RAG) pipeline loaded with undergraduate statistics textbook content, and compared its responses to GPT-4’s on 100 student questions.
The results: The small model matched GPT-4 on accuracy (no statistically significant difference, p = 0.41). It cost 92% less per query ($0.002 vs $0.025). Response latency dropped 45%. And critically: all student data stayed on campus hardware — no data left the institution.
Why it worked: RAG closes the knowledge gap. A small model “knows” less than GPT-4, but when you give it the right documents at query time, it doesn’t need to know everything — it just needs to read the relevant passage and answer from it. For domain-specific tasks (answering statistics questions from a specific textbook), this is sufficient. The model doesn’t need to know about Shakespeare or cooking recipes — it needs to know Chapter 7.
The equity argument: If the only path to quality AI is a $20/month subscription per user, AI’s benefits flow disproportionately to institutions and individuals who can afford it. But if a $2,000 GPU can serve an entire department, the economics change fundamentally. Community colleges, rural clinics, small nonprofits, and developing-world institutions gain access to AI that performs comparably to expensive cloud services — on their terms, with their data, under their control.
Discussion questions: 1. Under what conditions does “small model + relevant context” outperform “big model + general knowledge”? When does it fail? 2. If your institution could run its own AI for $2,000 in hardware plus electricity, what would you use it for first? 3. The experiment showed no statistical difference in accuracy. But are there other dimensions (creativity, nuance, handling edge cases) where the small model might fall short?
Module Discussion Questions
Case 1 (Mata v. Avianca): Should lawyers face sanctions for unverified AI citations? What standard is reasonable?
Case 2 (Amazon’s Hiring): If training data reflects real hiring patterns, is the algorithm biased or accurately reflecting a biased world?
Case 3 (Strawberry Problem): If LLMs write poetry but can’t count letters, what’s the difference between “intelligence” and “pattern prediction”?
Case 4 (Whisperers): Why is cross-provider convergence (26/26) more convincing than one company’s recommendation?
Case 5 (Fake Citations): The student didn’t intend to deceive. Should the penalty differ from intentional fabrication?
Case 6 (Two Tuesdays): What’s the BEST use of reclaimed time? What’s the worst?
Case 7 (Pope Image): The Pope image was humor. The same tech enables deception. Should tools restrict what they generate?
Case 8 (ELEVATE): If AI tutoring outperforms lectures, should universities change how they teach? What would be lost?
Case 9 (SLM-RAG): If local AI matches cloud AI for domain tasks, what are the equity implications for institutions that can’t afford subscriptions?
Case 10 (COMPAS): If mathematical fairness is impossible (Chouldechova), who decides which definition to prioritize?
Case 11 (ELEVATE Survey): Students said the biggest barrier was prompting skill. Should it be taught standalone or embedded in every course?
Case 12 (28% Premium): If AI skills command a wage premium, whose responsibility is the training?
Case 13 (Predictions): If experts are consistently wrong about technology timelines, how should organizations plan for AI?
No single right answers — that’s the point.
Back to Modules