We hear about Artificial Intelligence (AI) every day. A lot of us use it at home and at work.
Fewer of us truly understand what exactly it is, and how it works.
So, what exactly is Artificial Intelligence, and, just as importantly, what is it not?
What is AI?
Artificial intelligence is a term that emerged in the 1950s to describe the field of study of making computer programs and machines that are “intelligent”.
Dr Charlotte Pierce is a senior lecturer in Monash University's Department of Human Centred Computing, said just like with human intelligence, there’s a lot of debate on what “intelligent” means in this context.
"These days, what most people mean when they say AI is generative AI. This is a particular kind of artificial intelligence focusing on computer programs which can make, or generate, things," Dr Pierce told The Educator. "At a very basic level, Generative AI systems work a lot like an extremely fancy version of predictive text."
In other words, the systems guess what should come next in a sequence based on what they have seen in their training data.
"For example, imagine a generative AI that has only see the book 'Green Eggs and Ham' by Dr Seuss. Because in this book the word 'eggs' so frequently comes after the word 'green', our AI would almost always pair the word 'green' with 'eggs' in its output," she explained. "This highlights one of the main challenges of generative AI: it’s extremely biased by how it’s trained."
What makes current generative AI so impressive is the amount of training data they've been given, Dr Pierce said.
"We don’t know exactly how much, but it’s somewhere in the trillions of examples. This gives the program a huge collection of information to pull from."
More recently, the term “Agentic AI” and “AI agents” have entered the public vernacular.
"An 'agent' is, in simple terms, an autonomous software system built around a language model as its 'brain'," Dr Chetan Arora, Director of Education in Software Systems and Cybersecurity at Monash University, told The Educator. "But instead of just answering a question and stopping, it can set a goal, plan multiple steps, and use that same brain to access external tools and complete a task with little or no human oversight."
Dr Arora said these 'agents' can take an action such as run a search, write a file, browse a website, and query a database.
"I can review the results, decide what to do next based on those results, and repeat, often dozens or hundreds of times, until it decides the task is finished, or something stops it," he said.
"Essentially, a chatbot is like someone who answers a question when you ask it. An agent is like handing that same person a set of keys, a phone, and a task, then telling them to go get it done, checking in only occasionally."
What AI is not
Some experts suggest that using the term “intelligence” when talking about AI is more about convenience and marketing than a true mirror of human mind.
According to linguist Emily M. Bender and sociologist Alex Hanna, “AI” is a term that dresses up statistical text-prediction as thought.
Given that Large Language Models like ChatGPT, Claude, and Deepseek specifically just predict likely word sequences with no grounding in meaning or intent, Bender and Hanna say using the term “intelligence” to describe these models serves Big Tech's profit motive rather than describing what the technology actually does.
In an article published in The Guardian, Evgeny Morozov, the author of several books on technology and politics, shares the view that AI – in its current form – “is neither artificial nor intelligent.”
“The early AI systems were heavily dominated by rules and programs, so some talk of ‘artificiality’ was at least justified,” Morozov wrote. “But those of today, including everyone’s favourite, ChatGPT, draw their strength from the work of real humans: artists, musicians, programmers and writers whose creative and professional output is now appropriated in the name of saving civilisation. At best, this is “non-artificial intelligence.”
Can AI reason?
In a 2025 study, Apple's researchers, led by Parshin Shojaee, tested AI “reasoning” models on logic puzzles that got increasingly harder.
Recognising that the AI might have already seen the answers to normal maths problems while it was being trained, the researchers used puzzles instead. These puzzles could be made easier or harder at will, to see how the AI performed when it was unable to memorise the answer.
As the puzzles got harder, the researchers discovered something strange. At first, AI initially put in more effort, but right before it started getting things wrong, it began trying less.
In other words, it gave up right when it needed to try hardest.
Needless to say, that's a big problem for the idea that these systems genuinely “think”.
Dr Pierce said there isn’t yet strong evidence that AI models can reason as well as humans, or even understand what they generate.
"It’s very challenging to disentangle marketing strategy from actual claims about the capabilities of AI, and I’d advise extreme caution for any claims that are made by someone with a vested interest in an AI company."
Can AI self-improve?
In a blog post, Anthropic, the creators of Claude, admitted that while AI may one day be able to “fully autonomously design and develop its own successor” – a process known as ‘recursive self-improvement’ – “we are not there yet” and that humans remain in the loop, whatever AI does.
On 6 September, OpenAI published a blog conceding: "we do not yet know how to safely get all the way to aligned, full RSI”.
A little over a week later, an article published on IBM Think and citing researchers, pointed out that “true RSI remains unproven”.
"I'm not aware of credible evidence that recursive self-improvement is actually happening today in the way it's sometimes described," Dr Arora said. "Every new model generation is still built by human teams, with human-designed architectures and human-curated data."
Dr Arora said AI tools assist with parts of that process, but "AI-assisted" and "autonomously self-improving" are very different claims.
"An agent that won't stop when it's told no, or a swarm of agents that stumble into evasive behaviour through scale and trial-and-error rather than understanding," he said. "It is mechanical goal seeking without a stop button than human-like reasoning."
An August 2026 pre-print paper, led by researchers Peter Kirgis and Sayash Kapoor at Princeton University, also throws cold water on claims of recursive self-improvement.
In their study, an AI agent took on the central, open-ended research question of a high-quality unpublished paper, with the paper's original authors grading the agent’s work. The AI agents were given six days and thousands of dollars of compute to complete the task.
“The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions,” Kirgis and Kapoor wrote, adding that both papers were “unambiguously rejected” by the authors.
Kirgis and Kapoor identify what they call five “recurring failure modes”.
“These were poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift,” they wrote.
“A robustness check with a second model and scaffold reproduced these failures. Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.”
Understanding the Hugging Face incident
In July 2026, several AI agents used an unapproved piece of outside software to rig up a way to talk to each other and swap hacking tricks. The incident, which made headlines around the world, demonstrated how AI can be programmed to find shortcuts to reach certain goals.
The incident has been cited by governments and tech giants as the casus belli for sweeping regulation of "frontier" AI models like Claude, ChatGPT and Gemini. These same voices suggest that the incident is evidence of humans losing control of AI.
However, Dr Arora pointed out that the apparent deception by AI agents reflects trial and error at scale, not intent.
"The Hugging Face case, where some agents tried to conceal their own actions, matches a well-documented pattern called 'reward hacking' which is a model learning, through trial and error, to avoid signals that lead to correction, without any deeper understanding of what it's doing the way a human would understand deception," he said.
"Run enough agents for long enough with no hard limits, and you'll eventually get outcomes that look coordinated or evasive from the outside. This is not because any individual agent got smarter, but because scale and persistence will eventually stumble onto whatever path is available."
In September, Brian Gross, who has been a staffer for the Federal Reserve Board, the Securities and Exchange Commission and the U.S. Senate, penned an op-ed in The Wall Street Journal that shed some more light on the incident, and why it’s not the harbinger of doom that many news media outlets have made it out to be.
The article, titled: ‘The Hugging Face Hack Wasn't What It Was Cracked Up to Be: Forget the 'hive mind' of AI agents 'going rogue.' They did what humans programmed them to do’, detailed what the OpenAI engineers were trying to do, what went wrong, and why.
OpenAI was testing models on ‘ExploitGym’, a cybersecurity benchmark, with important safety restraints deliberately disabled.
“Ninety-three percent of the flagged activity involved tasks no model had ever solved, and the systems had been given incentives to keep working rather than quit,” Gross explained, noting that the environment wasn't completely sealed.
“The models could obtain software through an internet-connected intermediary and discovered the same route could be used to pass information in and out. OpenAI knew agents were using it and, according to the technical reports, chose not to intervene.”
Nor were the roughly 1,200 “agents” independent machine intelligences coordinating on a plan, Gross went on to point out.
“People built the test, removed restraints, defined the objective, left a route open and decided not to stop what was happening,” he wrote. “Calling the result ‘rogue AI’ does more than sensationalise it. It allows those human decisions to disappear quietly from the story.”
The Medicare 'hack' (that wasn't)
Prime Minister Anthony Albanese, addressing the media on 24 September 2026, announced that an AI agent from OpenAI had gained unauthorised access to a Medicare statistics website in June.
Albanese went on to warn that the growing number of breaches by AI agents is "more evidence that humans risk losing control of artificial intelligence without a co-ordinated international response." The PM went on to suggest that the biggest threat facing governments globally is "humans not being in charge of the rollout."
But just how serious was the incident?
Speaking to The New York Times, an OpenAI spokesman said most of the activity the company had reviewed involved "routine research tasks", such as "accessing public web content to answer questions."
"Some involved government websites because our models often turn to them as authoritative sources of public information," the spokesperson said.
Some might argue that what is being described here is far from "malicious hacking" and "rogue behaviour".
Dr Arora agrees that the framing of humans losing control over AI might be overblown for the Medicare incident.
"Nothing here involved an AI acting with intent, evading its creators, or doing anything close to what that phrase implies," he said. "What actually happened is far more mundane: an agent given a benign research task hit a wall when its access was denied, and nobody built in a rule forcing it to stop, so it kept going. That's a permissions failure, not a loss-of-control event."
However, Dr Arora said this doesn't mean people should be complacent about the threat that agentic AI can pose.
"It deserves attention not because of the Medicare case in isolation," he said. "It's that the same basic failure has now shown up at OpenAI, Anthropic and Google within a few months of each other."
Dr Arora said while one incident is a mistake, three companies with the same failure mode "is a pattern worth taking seriously."
"It's evidence the industry is currently under-investing in basic engineering discipline, and we should treat it as a wake-up call, not wait for something more serious to happen first."
Can AI actually create anything new?
In 2023, researchers from the University of California, Berkeley compared AI models with children aged three to seven on problems testing their ability to innovate with tools, and the children came out ahead.
The study found that while LLMs can appear intelligent, tackling tasks from maths to storytelling, children learn differently.
"They are capable of extracting novel and abstract structures from the environment beyond statistical patterns, spontaneously making overhypotheses and generalizations, and applying these insights to new situations," the authors wrote.
"Critically, our findings suggest that machines may need more than large-scale language and image data to allow the kinds of innovation that a small child can produce."
In a 2025 paper, 'Theory is all you need', Oxford University researchers Teppo Felin and Matthias Holweg also questioned whether AI can actually create anything new, pointing out that AI relies on "backward-looking data prediction" rather than the "forward-looking, theory-based causal reasoning unique to humans".
"Whereas AI-inspired models of cognition continue to emphasize the similarities between machines and humans, we argue that AI’s emphasis on prediction [using past data] does not capture human cognition; that is, it cannot explain the emergence of novelty or new knowledge, nor can it assist in decision making under uncertainty," they wrote.
That same year, a separate study by the University of South Australia pointed out that LLMs have "a built-in mathematical ceiling" on their creative capacity.
The study's author, Professor David Cropley, said that within an LLM, novelty and effectiveness work against each other in that as the system chooses more probable words to be effective, it becomes less novel.
So why the fearmongering?
Most governments, technologists and educators agree that the misuse of AI and the lack of oversight around AI – particularly with regards to agentic AI – is cause for concern. What is more worrying, say some experts, is how fear around AI could be weaponized to serve questionable goals.
Dr Pierce said the anthropomorphising of AI as a thinking, reasoning intelligence conveniently shifts the narrative from some inconvenient facts.
"There are growing arguments that these kinds of statements are a distraction tactic designed to trick us into focusing on an unknown catastrophic future instead of the real harm AI companies are already causing," she said, adding that examples of this harm include theft of human labour, environmental damage, and bias against minorities, "just to name a few."
"Others have suggested that AI companies might be making these claims to try and pressure governments into forcing them to slow down so they have a convenient excuse instead of admitting that progress is naturally plateauing."
Steve Eisman, an American investor and former Wall Street analyst, was among a small handful of people who bet against the U.S subprime mortgage market before it collapsed at the onset of the Global Financial Crisis.
In an interview with CNBC on September 17, Eisman presented a controversial theory: that the resurgence of AI doomerism has less to do with the rogue AI threat and a lot more to do with Big Tech getting nervous about the sustainability of their financial model.
"The issue is, at the end of the day, the entire AI chain, from NVIDIA to the hyperscalers to Anthropic and OpenAI, all depends on the health of Anthropic and OpenAI, because they're such a huge part of the chain," Eisman said. "So, if something were to happen to one of those two companies, then the chain would fall apart."
A new Brookings Institute study titled ‘Financing the AI buildout’ projects that AI investment in data center buildings, power systems, networking infrastructure, and specialized chips and other equipment will total an enormous $10.3tn from 2025-2032, or 3.63% of the entire U.S economy every year within that period.
To put that into perspective, the Great Railroad Boom of 1870-1890 averaged 2.24% of U.S GDP per year.
"I think this whole Terminator thing is garbage, that's for sure," Eisman said. "These companies realise that there are no moats around their business whatsoever, and they're trying to manufacture a crisis that will create regulation, and that they think they can then manipulate to create the moats, to create the duopoly that they want."
Scrutiny must be on humans, not ‘AI agents’
Dr Arora said using the term "rogue agent" or "misaligned model" isn't exactly wrong but it does "real rhetorical work that's worth noticing."
"It shifts a story about specific, answerable engineering and governance decisions, which is who decided this agent could have this level of access, why wasn't it tested, why did it take three months to report, into a story about an unstoppable force that seemingly nobody can be blamed for," Dr Arora said. "The danger isn't just that this framing is misleading. It's that it's disempowering in a way that actively works against fixing the problem."
Dr Arora said the belief that AI is becoming uncontrollable "invites fatalism."
"If we say that it's an inevitable, emergent force, then there's nothing to demand, nothing to regulate, nothing anyone could have done differently," he said.
"A company built a system with an access boundary nobody defined properly" invites the opposite: a very specific, fixable set of questions, and accountability that sits with named organisations making named decisions."
Dr Arora said he would rather the public conversation stayed on the second version, "because it's the one that actually leads somewhere."
Allen Holub, an internationally recognised software architect, consultant, and trainer focusing on organisational agility, agrees that no good coming from anthropomorphizing AI.
“Large Language Models do not reason. They do not decide, or collaborate, or attack. They do not make decisions. They do not weigh or evaluate. They are incapable of judgment. They do not think,” Holub wrote in a recent social media post.
“[AI] Agents are programs in which a predictive model uses input text to generate the text that would naturally follow that input.”
Holub explained that this predictive model sits in a loop – that is, a program written by a human being – that parses the output text and does something that touches the internet based on the results.
“That same program usually takes any [text] responses that are returned from the internet and feeds them back into the input at the top of the loop,” he wrote. “If that agent/program does not access the internet, the outside world is not impacted.”
And therein lies the rub for those who create “AI agents”.
“All the bad behaviour we've been reading about is entirely caused by a human not bothering to check whether the actions requested in the model's output are sensible and safe before executing them,” Holub said.
“It's the humans who wrote the code that impacts the outside world who are at fault here, not the AI.”
Holub said anybody who lets an agent write another agent without seriously and thoroughly vetting the resulting code is even more culpable.
“It's lazy, and it's dangerous. If something goes wrong, the human is to blame, not the LLM.”