AI could be used to spot grooming from players’ in-game behaviour

Artificial intelligence could help detect child grooming in online games by analysing player behaviour, a University of Malta professor has said.

Georgios Yannakakis, a researcher on digital games, said platforms with data on how players engage could, in principle, train systems to flag predators. “You could build AI that protects you against these people,” he said.

His comments follow a MaltaToday report on 5 October, in which a police inspector from the Counter-terrorism Unit described how violent online networks make contact with children inside games.

However, Yannakakis warned that AI is also a risk. He said AI can act with full agency. “AI can be a virus in a game,” he said, adding the danger comes not only from toxic or predatory people but also from AI. Generative AI can create “convincing people or avatars” that “move around, play your game and talk to children”, he said. Such agents could be exploited by people who want to be in contact with children.

But AI could also serve as a counter-agent to protect children. “We need the vaccine that comes in the form of AI,” Yannakakis said. Studios that want generative AI to make games “more creative, more fun to play” should “equally invest money and afford resources to make AI that is protecting its users from malicious AI”, he said.

He suggested an “AI companion” that would warn a player before they start chatting with a human or an AI agent. It would sit “in addition to your parents, who might not be there when you need them”, Yannakakis said.

His argument draws on a 2021 study, For Honour, co-authored with Ubisoft researchers. Random forest models trained on gameplay from 1,793 players sanctioned by Ubisoft community managers distinguished sanctioned players from others with at least 82% accuracy.

He said the team could “easily” identify toxic players just by looking at particular playing patterns. Players who preferred fighting AI opponents tended to be more toxic when matched against humans.

Chat is where the risk sits

Police Inspector Zachary Zammit had told MaltaToday that chat is one of the biggest factors in how children are approached in games. Yannakakis said playing is “a safe space”, but chatting “can be safe only if it is somehow regulated”.

One of the 2021 study’s predictors of toxic behaviour was chatting patterns within the game. Chat is “a very powerful medium” for allowing social interaction among players, Yannakakis said. However, he warned, it can also enable grooming and toxic behaviours.

Studios like Roblox or Ubisoft could monitor behaviour centrally and send alarms when predictions suggest grooming may be happening, he said.

Protection models would need data on reported cases to learn from, but such cases are rare. Still, algorithms can learn from a few examples and make highly accurate predictions, he noted. “As long as we have data of reported grooming behaviours, then we can build AI models that learn to predict these patterns.”

What Roblox says it does

Roblox now requires a face scan age check to chat, assigning users to age groups that can communicate with their own and neighbouring groups. Chief Safety Officer Matt Kaufman said it estimates ages for users under 18 “within about 1.4 years, plus or minus”. Parents told the BBC children had been wrongly identified as adults.

Yannakakis said the key question is “how bulletproof it is”. If data showed six-year-olds interacting with 30-year-olds, “then clearly the system is not safe”, he said. “Roblox needs to redesign this regulatory framework, and they need to improve on their safety.”

Roblox also uses Sentinel, an AI tool to detect early signs of child endangerment. The company says Sentinel identified nearly 70% of cases detected in the 12 months to 7 August 2026.

The industry's AI problem

Yannakakis said studios are trying to use AI to create new worlds and games. He expects smarter agents and non-player characters and more open worlds and open narratives in games.

“But there is a catch at the same time,” he said. Some major studios’ aggressive AI adoption has “backfired” and developers are not entirely happy with this aggressive AI adoption in some studios.

Yannakakis, who is also CEO and co-founder of Human Feedback AI, said its tools measure player experience using video, sound and voice. The project was funded by Xjenza Malta, a government agency, through the Label.me TDP Lite project.

Yannakakis sees no reason the tools could not detect toxicity with access to chat behaviour. Videos, he said, are very powerful high-resolution signals that reveal a lot of information. “Human reviewers identify toxic behaviour in gameplay footage so then we can learn to predict it,” he said.