AI Infrastructure — Under the Hood

Module 9 · Cloud, Local, RAG, Fine-Tuning, Open Source

Content current as of April 2026. Model names and prices change every few months; the infrastructure principles — where your data goes, what closes the knowledge gap, when to prompt vs. retrieve vs. fine-tune — are what’s durable.

Learning Objectives

Builds on: Module 8 (custom GPTs and agents)

After completing this module, you will be able to:

  1. Explain the tradeoffs between cloud, local, and hybrid AI deployment, including what data crosses which boundary
  2. Describe what small language models (SLMs) are, why 4-bit quantization matters, and when a 7B model is enough
  3. Explain RAG (Retrieval-Augmented Generation) as the single most important applied-AI technique in 2026
  4. Decide — on evidence, not vibes — when to prompt, when to RAG, when to fine-tune, and when to do nothing
  5. Navigate the open-source ecosystem (Hugging Face, Ollama, LangChain/LlamaIndex) without drowning in it
  6. Apply a decision framework that scales from a nonprofit with 3 users to a hospital with 3,000

The $0.002 Experiment

In 2025, a research team at a public university asked a simple question: could a small, cheap AI running on a single campus GPU match the accuracy of GPT-4?

The answer surprised even the researchers.

They took two small language models — Mistral-7B and DeepSeek-7B, each with 7 billion parameters (compare GPT-4’s estimated 1.8 trillion) — fine-tuned them on university course materials using a technique called QLoRA, and paired them with a local vector database that could retrieve relevant textbook passages on demand. The system is called SLM-RAG (Small Language Model with Retrieval-Augmented Generation).

Across 186 graded prompts from two undergraduate statistics courses:

Model Median Score (0–5) Mean Latency Cost per 1K Tokens
SLM-RAG (Mistral-7B) 4.3 6.8 seconds $0.002
SLM-RAG (DeepSeek-7B) 4.2 7.1 seconds $0.002
GPT-4 4.4 12.4 seconds $0.025

The difference in accuracy was not statistically significant (p = 0.41). The SLM-RAG system matched GPT-4 while cutting cost by 92% and latency by 45%. All traffic stayed on a single NVIDIA A6000 GPU on campus. No student data ever left the building.

This experiment proved something that the industry’s marketing mostly obscures: for domain-specific work where you can provide relevant context, you do not need a frontier model. A small, well-configured model running on your own hardware can match the giants — at a cost and privacy profile that changes what’s possible for schools, clinics, and small organizations (Yatawara et al., 2025, Campus-Hosted Small Language Models with Retrieval-Augmented Generation, AI Journal, MDPI).

This module teaches you the infrastructure landscape as it actually stands in April 2026 — what runs where, what costs what, and how to make the right choice for your situation.

Cloud vs Local: Where AI Runs (and Where Your Data Goes)

Every AI tool you use runs on one of two kinds of infrastructure. The distinction matters far more than most people realize, because where the computation happens determines where your data goes.

Cloud AI (APIs and chat platforms)

When you use ChatGPT, Claude, Gemini, or Perplexity, your prompt is sent to remote servers owned by the AI company. The model runs there, and the response comes back. Most people never think about this.

Advantages. No hardware. Always the newest model. Trivial to use.

Disadvantages. Your data travels to — and is at least briefly processed on — someone else’s servers. Ongoing subscription or per-token costs. You depend on the provider’s pricing, policies, and continued existence. No internet, no AI.

Local AI (on your own hardware)

With tools like Ollama or LM Studio, you download and run AI models on your own machine. The weights live on your SSD, the computation happens on your GPU or CPU, and your data never leaves the box.

Advantages. Complete privacy. Zero marginal cost after hardware. Works offline. No rate limits.

Disadvantages. Hardware required — though less than you’d think (see below). Local models are usually a half-step behind the frontier on pure reasoning benchmarks. You handle updates. Initial setup is a Saturday project.

Hybrid: the pattern organizations actually use

Almost every serious organization ends up with a hybrid stack:

  • Cloud for general-purpose, low-sensitivity work — drafting, brainstorming, summarizing public documents
  • Local or on-premises for anything touching regulated data — student records (FERPA), patient data (HIPAA), client files, proprietary IP, financial detail

The SLM-RAG experiment is this philosophy: when student data matters, bring the AI to the data instead of sending the data to the AI.

Common Misconceptions About AI Infrastructure
  • “Local AI is always worse than cloud AI.” For domain-specific work with RAG, a 7B local model matched GPT-4 at p = 0.41. Size isn’t everything. Context is.
  • “Cloud AI is private enough.” Free tiers may use your conversations for training. Enterprise tiers contractually don’t, but data still transits external servers. For HIPAA/FERPA, local is often the only genuinely compliant option.
  • “You need a $5,000 GPU to run local AI.” Phi-4-mini (3.8B), Gemma 3 4B, and Llama 3.2 3B all run well on an 8 GB laptop. An M-series MacBook runs 8B models at 40+ tokens/second. The entry bar is low.
  • “Fine-tuning is always worth it.” For most tasks, RAG is cheaper, faster, and good enough. Fine-tune only when you need the model to adopt a behavior pattern, not reference specific content.
  • “Open-source means second-tier.” Llama 4, DeepSeek R1/V3, Qwen 3, and GLM-4.7 post competitive numbers against proprietary frontier models on major benchmarks as of Q1 2026. Several are released under permissive licenses (MIT, Apache 2.0). The open-source stack is world-class.

Which infrastructure for YOUR situation?

Recommended. Hybrid. HIPAA-compliant cloud for administrative and educational content; local or on-premises for anything touching PHI. RAG over clinical guidelines for decision support. Ambient AI scribes (Abridge, Nuance DAX Copilot, Suki, Freed) run through vendor BAAs and are now common — a recent JAMA multi-site study reported they cut clinician EHR time by ~13 minutes per day and documentation time by ~16 minutes.

Key constraint. HIPAA is non-negotiable, and a Business Associate Agreement (BAA) is the minimum bar for any vendor. Ambient scribes also raise a second-order question: if the AI documents more diagnoses, billing goes up — make sure the clinical motivation, not the coding motivation, is driving adoption.

Recommended. Cloud AI (enterprise tiers with FERPA language in the contract) for teaching support and content creation. Institutional or local hosting for anything involving student records. Custom GPTs (ChatGPT Edu) or Claude Projects for student-facing tutors that stay inside the course corpus.

Key constraint. FERPA governs student data — and free-tier cloud tools do not satisfy it. CSU’s ChatGPT Edu deployment is the template most public universities are converging on.

Recommended. Start with free cloud tiers. Upgrade to paid ($20/user) when you hit message limits. Move to Team plans at ~5–20 users. Consider local only when (a) you handle sensitive client data or (b) your token bill exceeds the amortized cost of a $2,000 GPU.

Key constraint. Budget — but don’t over-optimize prematurely. The first $20/month buys 95% of the value. The hard cost decision comes at the $200–$2,000/month level.

Recommended. On-premises or government-cloud (FedRAMP High / IL-5) for any data involving citizens. Local RAG over policy documents and procedures. Avoid generic SaaS chatbots for anything beyond public-facing FAQ.

Key constraint. Data sovereignty, public accountability, and — as the COMPAS case in Module 10 showed — fairness testing before any deployment that affects a person’s rights, liberty, or benefits.

The Real Costs: A 2026 Comparison

What does AI actually cost to run? Here is the landscape as of April 2026, with prices verified against provider pricing pages this month:

Approach Typical cost Best for Real trade-off
Free tiers (ChatGPT, Claude, Gemini, Grok) $0 Getting started, light personal use Daily message caps; free-tier data may be used to train
Paid individual (ChatGPT Plus, Claude Pro) ~$20/user/mo Individual power users Per-user cost compounds fast for teams
Team plans (ChatGPT Team, Claude Team) $25–$30/user/mo Small teams (5–20) Still per-seat; data policies vary
API access — budget tier (GPT-5.4 Nano, Gemini 3 Flash, Haiku 4.5, Grok 4.1) ~$0.10–$1 per M input tokens Developers, high-volume automation Quality gap vs. flagship on hard reasoning
API — flagship (GPT-5.4, Gemini 3.1 Pro, Claude Opus 4.7, Grok 4) $2–$5 per M input, $12–$25 per M output Complex reasoning, agents, long context Bills can run thousands a month at scale
Local model (Ollama + open-source) $0 after hardware (~$1–3K) Privacy-critical, heavy steady load Setup effort; slightly behind on reasoning benchmarks
Enterprise (Azure OpenAI, AWS Bedrock, Vertex AI) $1,000–$50,000+/mo Regulated, large-scale deployments Complex contracts; long onboarding

Practical levers that most teams miss. The list prices above are a starting point. Two mechanisms can cut real API spend by 80–95%:

  • Prompt caching. Repeated context (system prompts, long documents, a whole codebase) is cached and billed at ~10% of the input rate on cache hits. For RAG and agent workloads, this is the single biggest cost lever.
  • Batch processing. Asynchronous batch APIs ship results within 24 hours at a ~50% discount across providers. If your task can wait, you pay half.

The SLM-RAG sweet spot. For a university department or small clinic handling ~200 queries/day on domain-specific content, a $2,000 GPU running a local model with RAG costs roughly $50–$80/month (electricity plus amortized hardware over 5 years) vs $500–$1,500/month for equivalent flagship API usage. Break-even sits at 3–4 months. That math is what made the original SLM-RAG experiment viable — and what makes it replicable by almost any public institution.

Small Language Models: Why Smaller Can Be Better

The AI industry’s loudest marketing message is “bigger is better.” The numbers tell a more interesting story. For domain-specific tasks with appropriate context, small models match or beat their giant counterparts — and do so at 50× less compute.

The key technique is quantization. A 7B parameter model at full 16-bit precision needs ~14 GB of VRAM. Quantized to 4 bits (Q4_K_M) — the current community default — it needs ~5 GB with negligible quality loss. That’s the reason the same model that required an $8,000 server GPU two years ago now runs on a MacBook Air.

The SLMs worth knowing in 2026

Model Params Creator License What it’s best at
Phi-4 / Phi-4-mini 14B / 3.8B Microsoft MIT Reasoning and math — Phi-4 beats many 70B models on MATH
Llama 3.3 / Llama 4 Scout 8B / 17B active (MoE) Meta Llama Community License Best all-around general model; huge fine-tuning ecosystem
Gemma 3 4B / 12B / 27B Google Gemma License Multimodal, polished prose, excellent instruction-following
Qwen 3 7B / 14B / 32B / 235B Alibaba Apache 2.0 Strong coding and multilingual; Qwen2.5-Coder is the de facto coding SLM
DeepSeek-R1 (distilled) 1.5B – 70B DeepSeek MIT Best open reasoning; distilled variants bring chain-of-thought to consumer hardware
Mistral 7B / Small 3 7B / 22B Mistral Apache 2.0 Fast, efficient, strong at structured extraction

VRAM rule of thumb (Q4_K_M quantization)

A quick planning heuristic you can actually use:

  • ~0.6 GB of VRAM per billion parameters at Q4_K_M
  • 8 GB VRAM → comfortable for 7–8B models (Llama 3.3 8B, Qwen 3 8B, Phi-4-mini)
  • 12 GB VRAM (RTX 3060/4060) → 12–14B models (Phi-4, Gemma 3 12B, Qwen 3 14B)
  • 24 GB VRAM (RTX 4090 / A5000) → 22–32B models (Gemma 3 27B, Qwen 3 32B, Mistral Small)
  • 48 GB+ VRAM → 70B models at Q4 (Llama 3.3 70B) or 2×24 GB cards
  • Apple Silicon unified memory counts as VRAM — an M-series Mac with 32 GB runs 12–14B models comfortably

When SLMs beat big models

  • Domain-specific Q&A where RAG can supply the context
  • Privacy-sensitive settings where data cannot leave the building
  • High-volume simple tasks (classification, extraction, routing) where $0.002 vs $0.025 per 1K tokens compounds into real money
  • Latency-sensitive apps — smaller models respond faster
  • Edge deployment — on-device on phones, laptops, or field equipment

When you still want the frontier

  • Multi-step agentic reasoning where the model must plan, act, check, and revise
  • Broad world knowledge where no RAG corpus is available
  • Novel problems that require the strongest generalist available
  • Tight production deadlines where “mostly works” beats “works once we’ve tuned it”

Retrieval-Augmented Generation (RAG)

RAG is the single most important applied-AI technique to understand in 2026. It solves the one problem every LLM has: it doesn’t know your data.

The problem

A trained LLM has read billions of web pages, books, and articles. It has not read your organization’s policy manual, your course textbook, this quarter’s sales reports, or the patient intake forms for a specific clinic. Ask it a question about any of those and it does one of two things: (1) it admits ignorance, or (2) it hallucinates a fluent-sounding answer based on patterns from similar-looking documents it did read. Option 2 is the dangerous one — and the one behind most of the LLM failures in Module 1.

The solution

RAG inserts a retrieval step between the user’s question and the model’s answer:

┌────────────────────────────────────────────────────────┐
│ 1. CHUNK  — split your documents into small passages   │
│            (typically 300–1,000 characters each)       │
├────────────────────────────────────────────────────────┤
│ 2. EMBED  — convert each chunk into a vector           │
│            (a list of numbers encoding meaning)        │
├────────────────────────────────────────────────────────┤
│ 3. STORE  — save the vectors in a vector DB            │
│            (Qdrant, Pinecone, ChromaDB, FAISS, pgvector) │
├────────────────────────────────────────────────────────┤
│ 4. RETRIEVE — when a question comes in, find the top-K │
│              most semantically similar chunks          │
├────────────────────────────────────────────────────────┤
│ 5. AUGMENT — inject those chunks into the prompt:      │
│              "Based on these passages from our manual…"│
├────────────────────────────────────────────────────────┤
│ 6. GENERATE — the LLM answers using YOUR text as       │
│              context, dramatically reducing hallucination │
└────────────────────────────────────────────────────────┘

Analogy. RAG is like handing an expert consultant your company’s manual before the meeting. They’re still the expert — but now their advice is grounded in your specific situation, not the generic one from their training.

Why this closes the knowledge gap. Modern LLMs are excellent at reading text and synthesizing it; they’re worse at recalling specific text from their training. RAG converts a recall problem into a reading problem — which is the one the model is good at. That is why a 7B model plus relevant context competes with a 1.8T model without it: small model + right context ≈ big model without context.

RAG in the SLM-RAG experiment

In SLM-RAG, course PDFs were chunked into ~500-character passages, embedded with the BAAI/bge-small-en-v1.5 embedding model, and stored in a local Qdrant vector database. When a student asked a question, the top-4 most similar passages were retrieved and prepended to the prompt with explicit “Based on these passages…” instructions. That single architectural choice — not a bigger model, not more training data — is what closed the gap with GPT-4.

What actually makes RAG work in 2026

The six-step pipeline above is the shape of RAG. The engineering — what separates a demo that looks good on three questions from a system that holds up in production — happens in three places. These are the decisions that determine whether your RAG system is actually useful.

1. The embedding model is the single most important choice. Vector search retrieves whatever the embedder says is “similar.” If the embedder doesn’t capture domain-specific meaning, nothing downstream saves you. Current defaults:

  • text-embedding-3-small (OpenAI) — 1536 dims, cheap, strong general baseline, configurable dimensions
  • BAAI/bge-small-en-v1.5 — 384 dims, free, open-source, excellent for English. What SLM-RAG used.
  • BAAI/bge-large-en-v1.5 / BAAI/bge-m3 — larger, better, still free; bge-m3 supports 100+ languages
  • jina-embeddings-v3 — strong multilingual performance
  • Domain-adapted — if you have labeled query/document pairs from real use, fine-tuning a bi-encoder on your corpus beats any off-the-shelf model for your task

For any RAG system that matters, evaluate at least two embedders on a held-out set of ~50 query–passage pairs from your domain before committing.

2. Hybrid retrieval beats dense alone. Pure vector search has a well-documented failure mode: it retrieves things that are semantically close but miss specific terms. Ask about “phi regulatory compliance” and dense search may return passages about HIPAA generally while skipping the one document that literally says “phi disclosure requirements.” The production-grade pattern:

Query
  ├──→ Sparse (BM25): top-50 by keyword/lexical match
  ├──→ Dense (embeddings): top-50 by semantic similarity
  │
  └──→ Merge with Reciprocal Rank Fusion (RRF)
         │
         └──→ Rerank top-50 with cross-encoder → top-5 → LLM

BM25 is the 1990s keyword-search algorithm. It finds exact term matches, handles entity names, product codes, and technical jargon that dense embeddings often fuzz over. RRF is a one-line fusion algorithm: each document gets a score of sum(1 / (k + rank)) across the two lists, with k typically 60. Documents that rank well in both lists win. LangChain’s EnsembleRetriever, LlamaIndex’s hybrid retriever, and every serious vector DB (Qdrant, Weaviate, Milvus, pgvector, Elasticsearch) support hybrid out of the box.

3. Reranking is where precision comes from. The retriever’s job is recall — casting a wide net. The reranker’s job is precision — carefully reordering the net’s contents. A cross-encoder reranker (Cohere Rerank, BAAI/bge-reranker-v2, Jina Reranker) takes the query and each candidate together as a concatenated input and produces a relevance score. Unlike bi-encoder embeddings, it sees the interaction between query and document tokens, which is substantially more accurate. Cost: 50–200 ms to rerank 50 candidates. Payoff: typically 15–30% improvement in top-5 precision on real RAG benchmarks.

Standard production pattern: hybrid retrieve top-50 → rerank with cross-encoder → pass top-5 to the LLM. If your RAG system doesn’t follow this pattern, it’s almost certainly leaving quality on the table.

GraphRAG: when vector search isn’t enough

Classic RAG retrieves passages based on similarity to the query. That’s excellent for “find me the fact” questions. It fails on “global” questions: “What themes appear across this corpus?”, “Summarize the main arguments in my 200 annual reports,” “Which authors disagree with each other?” No single passage contains the answer — the answer emerges from the structure of the corpus.

GraphRAG (Microsoft’s 2024 open-source release) addresses this by pre-building a knowledge graph during ingestion. An LLM extracts entities and relationships from each document, builds a graph, then clusters the graph into hierarchical communities. At query time, the system traverses the graph instead of just retrieving nearest-neighbor chunks. On thematic and summary-level questions, GraphRAG consistently outperforms classic RAG on benchmarks — often substantially. The cost: graph construction is expensive (you run an LLM over the whole corpus once, upfront), and you need a graph database and schema.

Practical decision rule. Start with hybrid + rerank. If your users are asking questions like “what are the main findings across these papers” rather than “what does section 4.2 say about X”, GraphRAG is worth the engineering lift. Otherwise, hybrid + rerank covers 95% of real use cases.

Long context windows: does RAG still matter?

A real 2025–2026 question. Gemini 2.5 Pro offers 2M tokens of context; Claude Opus 4.7 is at 200K–1M depending on tier; Llama 4 Scout ships with a 10M-token context window. You can fit a whole textbook in the prompt. So why bother with retrieval?

Three reasons RAG still matters even in the long-context era:

  • Cost. A 500K-token prompt at flagship rates costs $1–3 per query. RAG retrieves the 2,000 tokens that actually matter and bills you for those. Over a million queries that’s the difference between a $2M and a $20K bill.
  • Latency. Processing 500K tokens takes 5–30 seconds depending on model. RAG retrieves in <100 ms and generates the answer in 2–5 seconds.
  • “Lost in the middle.” Well-documented effect: LLMs attend less reliably to information in the middle of long contexts than at the beginning or end. A 200-page PDF dumped into the prompt gets worse retrieval than a surgically selected 4-passage excerpt.

The pragmatic synthesis: long context changes the constraint, not the principle. For large corpora (anything more than a single book), RAG wins on cost, speed, and accuracy. For a single long document the user is actively working with, long context is cleaner. Real systems use both: RAG over the large corpus, long context for the working document.

✅ Test Yourself: RAG Basics
  1. What problem does RAG solve? (LLMs don’t know your specific data; RAG retrieves and injects it as context.)
  2. List the 6 steps of the pipeline. (Chunk → Embed → Store → Retrieve → Augment → Generate.)
  3. Why does RAG reduce hallucinations? (The model answers from text that is present in the prompt instead of reconstructing from training memory.)
  4. What’s the difference between retrieval and reranking? (Retrieval = recall, cast a wide net for top-50 candidates; reranking = precision, carefully reorder to put the 5 most relevant at the top using a cross-encoder that sees query–document interactions.)
  5. When is RAG the wrong choice? (When you need a behavior pattern or style — that calls for fine-tuning — or when the task requires no domain grounding at all. For “global” questions over a corpus, consider GraphRAG instead.)

Fine-Tuning: Teaching the Model Your Voice

Where RAG provides context at query time, fine-tuning permanently nudges the model’s weights so its default behavior changes.

Prompt vs. RAG vs. fine-tune — when each wins

Approach What it does Best for Effort
Prompting Instructions + examples in the conversation One-offs, general questions, prototyping, “is this even worth automating?” Minutes
RAG Retrieve YOUR docs, add them as context Domain-specific Q&A, grounding in proprietary content, reducing hallucinations Hours to a few days
Fine-tuning Permanently adjust model weights on your examples Adopting a specific style, vocabulary, tone, or output schema Days; GPU required

The decision rule: if the answer is in a document, use RAG. If the way you want the model to answer is what’s special — your house style, a specific rubric, a structured output schema — fine-tune. If you don’t know which, start with RAG. RAG is reversible in an hour; fine-tuning is a weekend you can’t get back.

QLoRA: why fine-tuning is now cheap

Traditional fine-tuning of a 7B model required multiple server GPUs and hundreds of GB of memory. QLoRA (Quantized Low-Rank Adaptation) changed that:

  1. Load the base model in 4-bit quantized form (14 GB → ~5 GB for a 7B model)
  2. Freeze the base weights entirely
  3. Attach small LoRA adapters — tiny trainable matrices that learn the delta between the base behavior and the behavior you want
  4. Train for a few epochs on your Q&A pairs

For SLM-RAG, QLoRA fine-tuning of Mistral-7B and DeepSeek-7B ran in hours — not days — on a single A6000. The result was a model that used the course’s vocabulary, followed the course’s problem-solving approach, and referenced concepts in the order the course taught them.

The Open-Source AI Ecosystem

A vibrant ecosystem of free, open-source tools makes all of this accessible to individuals and small organizations.

Hugging Face — the GitHub of AI

Hugging Face hosts hundreds of thousands of models, datasets, and evaluation benchmarks, almost all freely downloadable. If you need a model for text generation, embeddings, speech, translation, vision, or almost any task, Hugging Face has it. The transformers and datasets libraries are the lingua franca of applied ML.

Ollama — run models locally with one command

Ollama makes local inference embarrassingly easy:

# Install (macOS/Linux/Windows)
curl -fsSL https://ollama.com/install.sh | sh

# Pull and run a model
ollama run llama3.3:8b

# Or try a reasoning model
ollama run deepseek-r1:7b

No cloud. No API keys. No subscription. Your prompts never touch the internet. Ollama also exposes an OpenAI-compatible API on localhost:11434, which means any app built for OpenAI (or Claude, via proxies) can point at a local model with a one-line config change.

LM Studio / Jan — GUI alternatives

If the command line isn’t your thing, LM Studio and Jan.ai provide polished desktop GUIs for downloading, configuring, and chatting with local models. LM Studio in particular is how most non-technical people first experience local AI.

LangChain / LlamaIndex — build real applications

Once you’ve moved past chat, these frameworks are how you connect a model to tools, databases, documents, and workflows. LlamaIndex is specifically optimized for RAG; LangChain is more general. Both support local models, cloud APIs, and most vector databases out of the box. They are the right level of abstraction for building a RAG system, a multi-step agent, or any workflow that chains an LLM with other systems.

The Infrastructure Decision Framework

Choosing the Right Approach

Ask these questions in order, and stop at the first “yes”:

  1. Is the data sensitive or regulated? (HIPAA/FERPA/PII/client-confidential?) → Consider local or on-premises.
  2. Do I need domain-specific accuracy over my own documents? → Add RAG.
  3. Do I need the model to adopt a specific voice, style, or output schema? → Consider fine-tuning.
  4. Is per-token cost a binding constraint? → Consider SLMs for the hot path; reserve flagship models for the genuinely hard 5% of requests.
  5. Do I need cutting-edge capability I can’t get locally? → Use the best cloud model available — with caching and batching.
  6. Am I still exploring? → Start with a free cloud tier. Move to local/custom only once you know what actually works.

Most people should start with cloud AI, add RAG when they hit the “it doesn’t know our stuff” wall, and only fine-tune when prompting and RAG visibly can’t close the gap.

Guided Practices

Guided Practice 9.1 — Cost Calculator

You’re advising a small nonprofit that processes 200 document queries per day. Compare the options:

  1. Option A. ChatGPT Team at $25/user/month × 3 staff users.
  2. Option B. Claude API — estimate 500 input tokens + 200 output tokens per query, Haiku 4.5 at $1/M input, $5/M output.
  3. Option C. Local Llama 3.3 8B on a $2,000 desktop (Ollama, electricity only).

For each: compute monthly cost, identify the main advantage, and identify the main risk. Which do you recommend and why?

Hints

Option A. 3 × $25 = $75/month. Simple, no setup. Risk: staff may paste sensitive data into Team tier without reading the data policy.

Option B. 200 queries × 30 days = 6,000 queries/month. Input: 3M tokens × $1/M = $3. Output: 1.2M × $5/M = $6. ≈ $9/month. Cheaper, but requires a developer to wrap it in a usable interface. Cache-heavy RAG workloads could drop this to pennies.

Option C. ≈ $30/month electricity + amortized hardware (~$33/month over 5 years) = ≈ $63/month. Data stays local. Risk: requires someone competent to set it up and update models quarterly.

Recommendation depends on technical capacity. If the nonprofit has no developer: A. If it has one who can spend a day wrapping the API: B (by far the best cost-per-query). If it handles sensitive client data: C.
Guided Practice 9.2 — RAG vs Fine-Tuning Decision

For each scenario, decide whether RAG, fine-tuning, both, or neither is appropriate. Justify in one sentence.

  1. A hospital wants an AI that answers patient questions using its post-surgery care protocols.
  2. A marketing agency wants an AI that writes copy in their distinctive brand voice across all clients.
  3. A student wants help studying for a chemistry exam.
  4. A law firm wants an AI that can reference their 10,000-page internal policy manual.
  5. A restaurant wants a chatbot that takes reservations.
  6. A publisher wants an AI that can both match their house style and cite their own back catalog accurately.
Answers
  1. RAG. Ground responses in the actual protocols; fine-tuning adds risk and no clear benefit for a medical content lookup.
  2. Fine-tuning. Brand voice is a behavior pattern, not a retrievable document — and the pattern must apply to new topics, so RAG alone can’t carry it.
  3. Neither. Use a good prompt with SMARTS. A general-purpose model already knows high-school chemistry.
  4. RAG. Classic use case. 10K pages vastly exceeds any context window, but vector search handles it in milliseconds.
  5. Neither. This is a structured booking workflow, not a language task. Use a proper booking system with an LLM wrapper only if you want one.
  6. Both. Fine-tune on past published pieces for voice, then add RAG over the back catalog for accurate citations. This is the only scenario in the list where combining is the right answer.

Module 9 Summary

Key Takeaways

  1. Cloud AI is convenient but sends your data to external servers. Local AI keeps everything on your machine but requires hardware and setup. Most real organizations run a hybrid — cloud for general, local for regulated.

  2. Small language models (3B–14B parameters at Q4_K_M quantization) handle the vast majority of production tasks, especially when paired with RAG. SLM-RAG proved it: a 7B local model matched GPT-4 accuracy at 92% lower cost and half the latency.

  3. RAG — retrieve your documents, inject them as context, generate — is the single most effective technique for grounding AI in your data and suppressing hallucinations. If you learn one infrastructure pattern in 2026, make it this one.

  4. Fine-tuning permanently adjusts model weights. QLoRA makes it affordable on a single GPU. Use it when you need the model to adopt a specific behavior pattern, not when you just need it to look things up.

  5. Decision order. Prompting (fast, free) → RAG (domain grounding) → Fine-tuning (behavioral change) → Custom infrastructure (full control). Don’t jump to step 3 when step 2 will do.

  6. Ollama runs models locally with one command. Hugging Face hosts hundreds of thousands of free models. LangChain/LlamaIndex connects them to your data and workflows. The open-source stack is world-class in 2026.

  7. The equity argument. If only a $200/month subscription buys you good AI, AI’s benefits flow to whoever can afford it. If a single campus GPU matches the frontier for your actual work — every school, every clinic, every small business can participate on its own terms.

💭 Think About It

The SLM-RAG experiment wasn’t just a technical achievement — it was a statement about equity. If the only way to access good AI is a $200/month cloud subscription routed through someone else’s servers, then AI’s benefits flow to institutions and individuals who can afford it, and the data flows somewhere else. But if a single GPU on a campus server can match a frontier model for domain-specific work, then every school, every clinic, every small business can have AI that works for them — on their terms, with their data, at a cost they can sustain. The infrastructure choice isn’t merely technical. It’s political. It determines who gets to participate in the AI revolution and who gets left behind.

🔬 Lab 9: The RAG Experiment

Lab 9: The RAG Experiment

Time: 60 minutes | Tools needed: NotebookLM (as a simplified RAG system) + any general chatbot + 3–5 documents from your field

Purpose. Experience the difference between grounded and ungrounded AI responses, and see hallucinations appear in real time.

Part A — Prepare Your Documents (10 min)

  1. Gather 3–5 documents from your domain (course materials, policy manuals, research papers, technical guides).
  2. Upload them to NotebookLM. NotebookLM is a simplified RAG system — it retrieves from your documents before generating.

Part B — Grounded vs Ungrounded Comparison (25 min)

  1. Write 5 domain-specific questions that require knowledge from your documents.
  2. Ask each question to BOTH NotebookLM (grounded in your docs) AND a general chatbot (ChatGPT, Claude, Gemini — ungrounded).
  3. Rate each pair:
Question NotebookLM Accuracy (1–5) Chatbot Accuracy (1–5) Which was more useful? Did the chatbot hallucinate?
Q1
Q2
…

Part C — Boundary Testing (15 min)

  1. Ask NotebookLM 2 questions that cannot be answered from your documents.
  2. Does it acknowledge the limitation, or does it confabulate?
  3. Ask the same questions to the general chatbot. Compare. Note: a good RAG system saying “I don’t know” is a feature, not a failure.

Part D — Reflection (10 min)

  1. Write a short paragraph: what’s the value of grounding AI in your own documents? When would you use NotebookLM vs a general chatbot? How does this connect to SLM-RAG?

Submit. Your comparison table, boundary-test results, and reflection.

🎮 Module 9 Quiz

Previous: Module 08 — Agents and GPTs | Next: Module 10 — AI Ethics and Responsibility