🚨 The Hidden Risks of Using AI in Schools (And What Districts Often Miss)

Jim Serpe
Jun 15
3 min read

Updated: Aug 1

AI is rapidly transforming how teachers work — grading, drafting feedback, summarizing data, generating plans, and more. But there’s a growing issue most districts don’t see until it’s too late:

Not all AI tools are built for K‑12, and many quietly put districts at risk.

Here are the biggest dangers I’m seeing right now — and what every district leader should understand.

1️⃣ AI Tools That Create New Student Records

When an AI system generates grades, feedback, goals, or performance predictions, it’s not just “helping.” It’s creating new education records, which means FERPA applies instantly. Most districts don’t realize this is happening behind the scenes.

2️⃣ Student Work Sent to “Outside AI Providers”

This is the most misunderstood risk in K‑12 AI. Many edtech vendors don’t run their own AI models. Instead, they send student data to outside AI providers — companies the district has no contract with.

Examples of outside AI providers include:

OpenAI (ChatGPT, GPT‑4, GPT‑4o)
Anthropic (Claude 3, Claude 3.5)
Google Gemini / Vertex AI
Microsoft Azure OpenAI
Amazon Bedrock (Anthropic, Cohere, Meta Llama, etc.)
Foreign or smaller LLMs like Mistral, AI21, Alibaba Qwen, Baidu ERNIE

If a vendor sends student writing, assessments, or IEP‑related content to any of these providers, the district must have a contract with each of them. If not, it’s considered redisclosure of education records — a direct FERPA violation. This is where most AI tools fail in compliance.

3️⃣ “Training on Student Data” (The Silent Deal‑Breaker)

If an AI model improves itself using student submissions, that’s a hard stop. FERPA does not allow vendors to reuse or learn from student data — ever. Yet many AI companies quietly do this unless districts ask the right questions.

4️⃣ AI That Stores or Retains Student Content

Some platforms keep student work indefinitely to “enhance accuracy.” Districts must control retention, not vendors. If the vendor decides how long student data is stored, that’s a compliance red flag.

5️⃣ Teachers Pasting PII Into Public AI Tools

This is the most common risk — and it’s not the teacher’s fault. Without guardrails, overwhelmed staff will paste:

Student writing
Parent emails
IEP‑related notes
Behavior logs
Assessment data

This is a workflow problem, not a teacher problem. Districts need safe, approved AI pathways — or unsafe ones will fill the gap.

6️⃣ “Automated Grading” Without Human Review

If AI assigns grades or feedback without teacher oversight, the district is responsible for the accuracy and the record it creates. That’s both a compliance risk and an equity risk.

7️⃣ The Importance of Training and Awareness

Educators must be trained on the risks associated with AI tools. Understanding the implications of using these technologies is crucial. Regular training sessions can help staff recognize potential pitfalls.

8️⃣ Building a Framework for Safe AI Use

Districts should establish a framework for safe AI use. This includes setting guidelines for what data can be shared and with whom. A clear policy can help mitigate risks and ensure compliance.

9️⃣ Engaging with AI Vendors

When selecting AI tools, districts should engage with vendors. Ask questions about data handling practices and compliance measures. Ensure that vendors understand FERPA and are committed to protecting student data.

10️⃣ The Role of Parents and Guardians

Parents and guardians should be informed about how AI tools are used in schools. Transparency builds trust and allows families to understand the technology's role in their children's education.