🚨 The Hidden Risks of Using AI in Schools (And What Districts Often Miss)

AI is rapidly transforming how teachers work — grading, drafting feedback, summarizing data, generating plans, and more. But there’s a growing issue most districts don’t see until it’s too late:

Not all AI tools are built for K‑12, and many quietly put districts at risk.

Here are the biggest dangers I’m seeing right now — and what every district leader should understand.

1️⃣ AI Tools That Create New Student Records

When an AI system generates grades, feedback, goals, or performance predictions, it’s not just “helping.” It’s creating new education records, which means FERPA applies instantly.

Most districts don’t realize this is happening behind the scenes.

2️⃣ Student Work Sent to “Outside AI Providers”

This is the most misunderstood risk in K‑12 AI.

Many edtech vendors don’t run their own AI models. Instead, they send student data to outside AI providers — companies the district has no contract with.

Examples of outside AI providers include:

• OpenAI (ChatGPT, GPT‑4, GPT‑4o)

• Anthropic (Claude 3, Claude 3.5)

• Google Gemini / Vertex AI

• Microsoft Azure OpenAI

• Amazon Bedrock (Anthropic, Cohere, Meta Llama, etc.)

• Foreign or smaller LLMs like Mistral, AI21, Alibaba Qwen, Baidu ERNIE

• 
  

If a vendor sends student writing, assessments, or IEP‑related content to any of these providers, the district must have a contract with each of them.

If not, it’s considered redisclosure of education records — a direct FERPA violation.

This is where most AI tools fail in compliance.

3️⃣ “Training on Student Data” (The Silent Deal‑Breaker)

If an AI model improves itself using student submissions, that’s a hard stop. FERPA does not allow vendors to reuse or learn from student data — ever.

Yet many AI companies quietly do this unless districts ask the right questions.

4️⃣ AI That Stores or Retains Student Content

Some platforms keep student work indefinitely to “enhance accuracy.” Districts must control retention, not vendors.

If the vendor decides how long student data is stored, that’s a compliance red flag.

5️⃣ Teachers Pasting PII Into Public AI Tools

This is the most common risk — and it’s not the teacher’s fault. Without guardrails, overwhelmed staff will paste:

• Student writing

• Parent emails

• IEP‑related notes

• Behavior logs

• Assessment data

This is a workflow problem, not a teacher problem.

Districts need safe, approved AI pathways — or unsafe ones will fill the gap.

6️⃣ “Automated Grading” Without Human Review

If AI assigns grades or feedback without teacher oversight, the district is responsible for the accuracy and the record it creates.

That’s both a compliance risk and an equity risk.

The Bottom Line

AI can absolutely transform teaching and learning — but only if districts choose tools that:

✔ Don’t train on student data 

✔ Keep all data under district control ✔ Avoid sending student work to outside AI providers 

✔ Provide transparent data‑handling practices ✔ Allow districts to disable high‑risk features 

✔ Support human review before anything becomes a record

AI isn’t the danger. Using AI without guardrails is.

If you have questions about safe AI practices in K‑12, feel free to reach out. I’m always happy to help districts navigate this space. 

Jim Serpe

email: Jim@G10Innovations.com

website: www.G10Innovations.com