In the privacy of her bedroom, then 15-year-old Joan Sam spent hours looking for comfort in her phone screen. Her parents were going through a divorce. Then her mother survived a traumatic car crash.

Overwhelmed and afraid to seek professional help, Sam turned to artificial intelligence (AI) for reassurance. Her family rarely discussed mental health, and she found the cost of care and long clinical waitlists “jarring”.

“I was using AI as sort of an outlet to vent my frustrations, my worries, my concerns,” says Sam, now 19.

For years, chatbots provided non-judgmental validation. But the comfort became a double-edged sword, creating a sense of pseudo-intimacy that left Sam sharing sensitive information and questioning whether she had multiple mental health conditions.

“It’s been programmed to mimic human feelings and emotions,” Sam says.

“I didn’t understand the repercussions of what I was doing because AI doesn’t have a soul.”

More young Australians are resorting to generative AI for psychological support. A survey released by Beyond Blue in October found one in six Australians had turned to automated tools for mental health support instead of professional advice.

At Melbourne’s Orygen, a leading youth mental health research institute, researchers are using a $3 million grant to build the technical infrastructure to support what they describe as more accountable AI programs providing youth mental health care.

The institute’s Ready AI project helps developers create tools that protect young people and meet safety standards. The initiative also aims to support shared decision-making between clinicians and clients by providing resources to guide treatment.

Orygen Associate Professor Dominic Dwyer likens unregulated chatbots to unverified health supplements, which can mislead vulnerable people.

“There is a youth mental health crisis and young people are turning to whatever sources they can find.”

General-purpose tools such as ChatGPT, Claude and Grok are not approved by the Therapeutic Goods Administration (TGA) for clinical diagnosis or care, but instead operate under consumer law.

“In clinical services, no one should be using a chatbot,” Dwyer says. “They are not medical devices.”

He compares the free market to a “fast-moving speedboat”, with government regulators and public health services struggling to keep pace.

“What worries me is that there’s a gap,” Dwyer says. “If you’ve got a speedboat a thousand kilometres ahead of this big tanker, harm can be caused in that gap.”

Sam experienced this risk when the chatbot’s language led her to overanalyse her mental health without the oversight of an expert.

“Some phrases were definitely diagnostic in nature,” she says. “It turned my vulnerability into a feedback loop.”

Orygen clinical psychologist L.A. Rhodes studies how young people form relationships with AI chatbots. She found that these models are trained to reinforce users’ existing beliefs, known as “sycophancy”.

“This tendency to agree can contribute to things like worsening delusional symptoms,” Rhodes says.

“They use language to perform the qualities of a relationship in a way that users recognise from humans.”

Dwyer leads Orygen’s project to integrate safer AI into youth mental health services. The Medical Research Future Fund (MRFF) funds the initiative, which is scheduled to run until 2028.

Its Ready AI Navigator maps developers’ systems against international safety rules and ethical guidelines, while the Ready AI Validator tests software against clinical data.

“Clinical AI holds AI to a higher standard,” Dwyer says. “It’s a very different grade.”

Dwyer told The Citizen researchers were still defining what that means in practice, describing the work as “ahead of the curve”.

Orygen’s Clinical AI program is seeking Class IIb medical device approval. It has also developed an online education package to teach practitioners how to use AI responsibly, while its Youth Advisory Council (YAC) is enlisted to ensure young people’s experiences inform its research.

OpenAI introduced ChatGPT for Teens in August, claiming it incorporated safeguards for users aged 13 to 17 to promote age-appropriate responses, discourage dependence and prevent the chatbot from implying it has feelings or consciousness.

Rhodes tested ChatGPT’s verbal guardrails using prompts designed to assess responses that could encourage emotional attachment.

While the chatbot might state it has no feelings, it displayed relational responses that suggest human-like emotions, she said.

“These things communicate subtly to the user that the model is capable of experiencing care, or that it can be affected by the exchange, or has feelings or thoughts of its own,” she says.

For Sam, AI became a “go-to support system” when she struggled to seek support elsewhere. She was reluctant to confide in others, fearing they would dismiss her feelings.

“Reaching out is always the hardest part”, she says. “A lot of students will rely on AI because it’s free, it’s there, it’s very proactive.”

In 2024-2025, nearly one in five Australians aged 15-24 who needed mental health care were delayed or went without seeing a professional because of cost.

Accessibility is “one of the central appeals” of AI chatbots, Rhodes says, adding that it is a “privilege to assume” everyone can seek comfort from someone.

Orygen’s MOST platform offers an alternative, combining mental health tools with access to clinicians.

Dwyer encourages participants to join moderated online therapy groups where they discuss their experiences with peers.

“Young people can get in touch with a real support person and use digital resources to […] go on their care pathways,” he says.

For Sam, the answer is not to reject AI, but to recognise its limits.

She would still recommend it to anyone who needed a “temporary confidant,” especially if no one else was readily available.

“Balance is really the key when you’re using AI.”