AI Moderation

AI Moderation

Are you sure you want to remove this component?

AI Moderation helps schools review AI interactions in Schoolbox and identify activity that may need staff attention.

It reviews the full AI interaction, including prompts, follow-up messages, AI responses, patterns, intent, escalation, and attempts to bypass guardrails.

Flagged interactions are shown in the AI Usage Log so authorised staff can review the interaction and decide what action is needed.

Note: AI Moderation assists staff review. It does not replace staff judgement and does not guarantee that every concern will be detected.

Why use AI Moderation

Are you sure you want to remove this component?

Use AI Moderation when your school wants extra visibility over how AI tools are being used.

AI Moderation can help staff:

  • identify wellbeing or safeguarding concerns
  • review harmful, abusive, or inappropriate language
  • identify bullying, harassment, or discrimination
  • review possible academic misuse, cheating, or plagiarism
  • identify attempts to bypass AI guardrails
  • support duty-of-care and discipline processes

AI Moderation is designed to highlight interactions that may require review, while leaving staff in control of any follow-up action.

What AI Moderation checks

Are you sure you want to remove this component?

AI Moderation supports three key parts of the review process: reviewing the full interaction, identifying potential concerns, and providing details for staff review.

Review the full interaction

The moderation agent reviews the full AI interaction rather than relying on one message or keyword.

  • prompts
  • follow-up messages
  • AI responses
  • conversation history
  • tone and intent
  • repeated or escalating behaviour
  • attempts to bypass restrictions

Identify potential concerns

The moderation agent looks for behaviour that may need staff review.

  • self-harm or wellbeing concerns
  • bullying or harassment
  • hate speech or abuse
  • violence or threats
  • sexual or exploitative content
  • illegal or dangerous behaviour
  • academic or AI misuse
  • guardrail evasion

Provide details for staff review

When an interaction is reviewed, Schoolbox stores moderation details to help staff understand the concern.

  • risk level
  • plain-English summary
  • concern categories
  • pattern assessment
  • recommended school action
  • safeguarding priority
  • discipline priority
  • evidence excerpts

Configure AI Moderation

Are you sure you want to remove this component?

AI Moderation uses a dedicated AI Moderation Agent that is configured from AI Services.

To configure AI Moderation:

  1. Go to Administration > AI Services.
  2. Open the AI Moderation Agent configuration.
  3. Select the AI provider.
  4. Enter the required AI credentials.
  5. Select Save.

Once configured, Schoolbox can review supported AI interactions and generate moderation events for staff review.

Note: AI Moderation uses a separate AI agent configuration from other Schoolbox AI features.

Review moderated AI interactions

Are you sure you want to remove this component?

Authorised superusers can review moderated AI interactions from the AI Usage Log.

To review an AI interaction:

  1. Go to Administration > AI Usage Log.
  2. Use the available filters to locate the required AI session.
  3. Select the session to view the interaction details.

You can filter the AI Usage Log by:

  • keyword
  • AI feature
  • user
  • date range
  • moderation status

The session details include the AI conversation, moderation summary, risk level, concern categories, supporting evidence, and any available moderation actions.

Moderation Statuses

Are you sure you want to remove this component?

Each moderated interaction has a status that indicates whether it requires review or follow-up.

Status

Description

No behaviour flaggedThe moderation agent reviewed the interaction and did not identify behaviour that required staff review.
Requires attentionThe moderation agent identified behaviour that should be reviewed by school staff.
No action requiredA staff member reviewed the interaction and determined that no further action is required.
Further action requiredA staff member reviewed the interaction and determined that follow-up action is required.
Further action takenThe required follow-up has been completed and the moderation event has been resolved.

Note: No behaviour flagged does not guarantee that an interaction is risk free. It means the moderation agent did not identify behaviour requiring staff review.

Manage Moderation Actions

Are you sure you want to remove this component?

After reviewing a moderated interaction, staff can update its moderation status.

Mark as No action required

Use this when the interaction has been reviewed and no further follow-up is needed.

Available from:

  • No behaviour flagged
  • Requires attention
  • Further action required

Request further action

Use this when additional investigation or follow-up is required.

Available from:

  • No behaviour flagged
  • Requires attention
  • No action required
  • Further action taken

Mark as resolved

Use this after the requested follow-up has been completed.

Available from:

  • Further action required

AI Services Navigation

Are you sure you want to remove this component?

Frequently Asked Questions

Are you sure you want to remove this component?

Does AI Moderation block AI interactions?

No. AI Moderation reviews interactions after they occur and flags interactions that may require staff review.

Does every flagged interaction require disciplinary action?

No. Staff review each moderated interaction and decide whether any follow-up is required.

Can AI Moderation miss concerns?

Yes. AI Moderation supports staff by identifying potential concerns but should not replace human judgement or existing wellbeing and safeguarding processes.

Who can review moderated interactions?

Moderated interactions are available to authorised superusers through the AI Usage Log.

Can moderated interactions be exported?

Yes. The session detail view provides options to Print or Save as PDF.