Posting a photo of the school swimming carnival or the Year 12 formal used to be a straightforward way to celebrate a community's wins. Increasingly, it's a decision that carries risk school leaders never had to weigh before.
That's the warning from eSafety, which has published a new Online Safety Advisory after seeing a sharp rise in reports over the past 18 months involving the misuse of school-related images.
Worryingly, much of the harmful content has been traced back to imagery pulled straight from school social media accounts and websites, raising serious implications
The regulator says powerful AI tools are now widely accessible and prone to weaponisation, available as stand-alone apps and increasingly built into social media, messaging platforms and search engines.
If a school student or staff member is targeted, taking a clear four-step response within the first 24 hours is critical, said eSafety.
- Collect evidence such as screenshots, URLs, timestamps and messages, keeping access tightly restricted to avoid re-exposing the affected person.
- Report the content directly to the platform using the eSafety Guide, or escalate to eSafety itself.
- Communicate carefully with families in broad terms, outlining the school's response and urging them not to circulate the material further.
- Schools should plan for content resurfacing through reposts or new accounts, staying alert and reassessing their response if the material recirculates.
Practical care, not just policy
eSafety Commissioner Julie Inman Grant said a person targeted by fake or humiliating content may feel exposed and powerless, so schools should explain how they are responding and provide practical help.
“Affected staff and students may need privacy protection or less exposure to the material. Staff may also benefit from an employee assistance program,” Inman Grant told The Educator. “Treat staff-targeted abuse as a wellbeing and workplace issue within the school response.”
Students affected by image misuse need support suited to their age and circumstances, she added.
“Schools should involve parents or carers where appropriate and avoid actions that increase shame or draw more attention to the content.”
Over the past 18 months, eSafety has seen a rise in reports from teachers, staff and students involving school-related image misuse. Much of the harmful content the organisation is seeing has been traced back to imagery pulled straight from school social media accounts and websites.
To help leaders address this, Inman Grant said eSafety has alerted schools to the escalated safety risks associated with sharing photos and videos online, amid the rapid evolution, deployment and accessibility of AI tools.
“Powerful AI tools are now widely accessible and prone to weaponisation,” Inman Grant told The Educator. “They are readily available as stand-alone apps and services with newer AI tools increasingly being offered through social media and messaging platforms and integrated into search engines.”
Help is at hand
Inman Grant said the emerging threat has prompted eSafety to publish a new Online Safety Advisory, to help schools make safer posting decisions and to have action plans in place to respond when images are misused.
“Schools should review how they collect, approve, post and store images,” she said. “A risk-based approach helps schools decide which images belong on public channels and which are better shared in a closed or restricted space.”
Inman Grant said eSafety is keeping the pressure on technology companies to put barriers in place to disrupt poor behaviour, but schools can also play an important role in keeping their staff and students safe by taking a risk-based approach to considering which images to post publicly.
“School communities should be aware that not all harmful material can be addressed under the Online Safety Act, especially material that targets adults,” she said.
“This is one of the reasons why it is so important to carefully assess the risks when posting imagery online – imagery that may later be digitally altered without consent.”
The standards behind the shift
Enforcement action is already reshaping the nudify landscape, with several major services pulling back or falling in line. But with AI tools becoming more accessible by the day, Inman Grant explains what schools need to be telling their communities right now.
“As a result of eSafety action over the past year, seven of most widely used nudify services in Australia, previously visited hundreds of thousands of times a month, have now either withdrawn access, or taken measures to comply with their obligations under the Online Safety Act,” she said.
“eSafety’s world-first industry standards address the most harmful online content, including AI-generated deepfakes and de-clothing apps that can be used to create explicit material depicting children.”
Inman Grant said these standards place clear obligations on technology companies to detect and remove this material quickly, while also taking stronger steps to prevent their platforms from being misused in the first place.
“Companies can face penalties of up to $54.6 million for non-compliance,” she said.
“These new requirements are part of eSafety’s ongoing work to ensure platforms are designing and operating with Safety by Design at their core, including addressing AI-generated content that could be used to harm or exploit children.”