Tech
How do ‘AI detection’ tools actually work? And are they effective?
As nearly half of all Australians say they have recently used artificial intelligence (AI) tools, knowing when and how they’re being used is becoming more important.
Consultancy firm Deloitte recently partially refunded the Australian government after a report they published had AI-generated errors in it.
A lawyer also recently faced disciplinary action after false AI-generated citations were discovered in a formal court document. And many universities are concerned about how their students use AI.
Amid these examples, a range of “AI detection” tools have emerged to try to address people’s need for identifying accurate, trustworthy and verified content.
But how do these tools actually work? And are they effective at spotting AI-generated material?
How do AI detectors work?
Several approaches exist, and their effectiveness can depend on which types of content are involved.
Detectors for text often try to infer AI involvement by looking for “signature” patterns in sentence structure, writing style, and the predictability of certain words or phrases being used. For example, the use of “delves” and “showcasing” has skyrocketed since AI writing tools became more available.
However the difference between AI and human patterns is getting smaller and smaller. This means signature-based tools can be highly unreliable.
Detectors for images sometimes work by analyzing embedded metadata which some AI tools add to the image file.
For example, the Content Credentials inspect tool allows people to view how a user has edited a piece of content, provided it was created and edited with compatible software. Like text, images can also be compared against verified datasets of AI-generated content (such as deepfakes).
Finally, some AI developers have started adding watermarks to the outputs of their AI systems. These are hidden patterns in any kind of content which are imperceptible to humans but can be detected by the AI developer. None of the large developers have shared their detection tools with the public yet, though.
Each of these methods has its drawbacks and limitations.
How effective are AI detectors?
The effectiveness of AI detectors can depend on several factors. These include which tools were used to make the content and whether the content was edited or modified after generation.
The tools’ training data can also affect results.
For example, key datasets used to detect AI-generated pictures do not have enough full-body pictures of people or images from people of certain cultures. This means successful detection is already limited in many ways.
Watermark-based detection can be quite good at detecting content made by AI tools from the same company. For example, if you use one of Google’s AI models such as Imagen, Google’s SynthID watermark tool claims to be able to spot the resulting outputs.
But SynthID is not publicly available yet. It also doesn’t work if, for example, you generate content using ChatGPT, which isn’t made by Google. Interoperability across AI developers is a major issue.
AI detectors can also be fooled when the output is edited. For example, if you use a voice cloning app and then add noise or reduce the quality (by making it smaller), this can trip up voice AI detectors. The same is true with AI image detectors.
Explainability is another major issue. Many AI detectors will give the user a “confidence estimate” of how certain it is that something is AI-generated. But they usually don’t explain their reasoning or why they think something is AI-generated.
It is important to realize that it is still early days for AI detection, especially when it comes to automatic detection.
A good example of this can be seen in recent attempts to detect deepfakes. The winner of Meta’s Deepfake Detection Challenge identified four out of five deepfakes. However, the model was trained on the same data it was tested on—a bit like having seen the answers before it took the quiz.
When tested against new content, the model’s success rate dropped. It only correctly identified three out of five deepfakes in the new dataset.
All this means AI detectors can and do get things wrong. They can result in false positives (claiming something is AI generated when it’s not) and false negatives (claiming something is human-generated when it’s not).
For the users involved, these mistakes can be devastating—such as a student whose essay is dismissed as AI-generated when they wrote it themselves, or someone who mistakenly believes an AI-written email came from a real human.
It’s an arms race as new technologies are developed or refined, and detectors are struggling to keep up.
Where to from here?
Relying on a single tool is problematic and risky. It’s generally safer and better to use a variety of methods to assess the authenticity of a piece of content.
You can do so by cross-referencing sources and double-checking facts in written content. Or for visual content, you might compare suspect images to other images purported to be taken during the same time or place. You might also ask for additional evidence or explanation if something looks or sounds dodgy.
But ultimately, trusted relationships with individuals and institutions will remain one of the most important factors when detection tools fall short or other options aren’t available.
This article is republished from The Conversation under a Creative Commons license. Read the original article.
Citation:
How do ‘AI detection’ tools actually work? And are they effective? (2025, November 16)
retrieved 16 November 2025
from https://techxplore.com/news/2025-11-ai-tools-effective.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.
OpenAI, Anthropic, and Block have cofounded a new open source organization—the Agentic AI Foundation—to promote standards for artificial intelligence agents.
The three companies are also transferring ownership of some widely used agentic technologies over to the foundation. This includes Anthropic’s Model Context Protocol (MCP), which allows agents to connect and interact; OpenAI’s Agents.md, which lets programs and websites specify rules for coding agents; and Goose, a framework for building agents developed by Block. These technologies were already free to use, but through the new foundation it will be possible for others to contribute to their development.
“MCP is used by many companies, but there are others [who don’t use it],” says Nick Cooper, who leads work on the protocol at OpenAI. Cooper says that making MCP an open standard should encourage developers and companies to embrace it and build systems that integrate agentic AI. “That open interoperability—that open standard—really means that companies can talk across providers, and across agentic systems.”
The Agentic AI Foundation is being created under the Linux Foundation, which oversees development of the widely used open source Linux operating system as well as other projects. The foundation provides legal and technological support for the creation of open source foundations. Other companies who have signed on to the AAIF, beyond the three founding members, include Google, Microsoft, AWS, Bloomberg, and Cloudflare.
The new foundation reflects a nascent shift from chat-based AI systems to greater use of programs that take actions on behalf of users. This kind of agentic AI promises a potentially lucrative new paradigm in which AI agents use the web and negotiate with one another to power all sorts of applications. Consumers may, for example, use AI assistants to buy and book things, while businesses use AI agents to manage transactions and customer interactions.
Srinivas Narayanan, chief technology officer of B2B applications at OpenAI, envisions a time when large numbers of AI agents routinely communicate with one another in the course of business. The AI industry working across the same open standards should help ensure that those interactions happen seamlessly. “Open source is going to play a very big role in how AI is shaped and adopted in the real world,” Narayanan says.
The question of openness seems crucial to AI right now. US companies mostly make money by offering access to powerful closed models through application programming interfaces, or APIs. Meta previously released the weights for its best model, Llama, so that anyone could download and run it, although the company has recently signaled a shift to a more closed approach. A number of Chinese AI companies, including DeepSeek, Alibaba, Moonshot AI, and Z.ai, provide strong open source models that have become popular with developers, startups, and AI researchers. Some worry that this picture could give Chinese firms a big strategic advantage over time.
There’s no way to recharge the ring. Migicovsky says he didn’t want yet another gadget to charge every day, so instead, the Pebble Index has non-rechargeable silver oxide hearing aid batteries designed to last 2 years with average use. Once the device’s battery is nearly dead, users will receive a notification in the app, and the idea is you’ll buy a new Pebble Index—an idea that’s easier to get behind knowing the ring costs just $75, though the price will jump to $99 after the first batch. (You’ll also be able to send your old Index to the company for recycling.)
When your audio is sent to your phone, an open source speech-to-text AI model processes it locally to convert your voice notes to text. Then, an on-device large language model will categorize the audio, deciding whether it’s a reminder, a timer, or a general note. A feed shows all your memory logs, and you scroll through it to find and listen to each clip. None of this data is ever sent to the cloud; it all stays on your phone. “These are your innermost thoughts,” Migicovsky says. “You don’t want to send them anywhere.”
By default, all of your musings with the ring are handled by the Pebble app. So if you had it set a reminder, you’ll get one from the Pebble app. However, you can customize the destination if you prefer to use your own service. If you use the Notion app for notes and tasks, for example, you can set it up so that your reminders and thoughts will be sent there.
Broad Strokes
The open source nature of the Pebble app means there’s no limit to customization. You press and hold the button to log a note, but you can have a single press trigger an action. Migicovsky says he set his to play or pause music, and a double-press switches tracks. But you can set it to take a photo remotely or activate a smart home routine. There will be an actions category in the Pebble app store where folks can publish their custom actions.
Governments should make software companies liable for developing insecure computer code. So says Katie Moussouris, the white hat hacker and security expert who first persuaded Microsoft and the Pentagon to offer financial rewards to security researchers who found and reported serious security vulnerabilities.
Bug bounty schemes have since proliferated and have now become the norm for software companies, with some, such as Apple, offering awards of $2m or more to those who find critical security vulnerabilities.
Moussouris likens security vulnerability research to working for Uber, only with lower pay and less job security. The catch is that people only get paid if they are the first to find and report a vulnerability. Those who put in the work but get results second or third get nothing.
“Intrinsically, it is exploitative of the labour market. You are asking them to do speculative labour, and you are getting something quite valuable out of them,” she says.
Some white hat hackers, motivated by helping people fix security problems, have managed to make a living by specialising in finding medium-risk vulnerabilities that may not pay as well as the high-risk bugs, but are easier to find.
But most security researchers struggle to make a living as bug bounty hunters.
“Very few researchers are capable of finding those elite-level vulnerabilities, and very few of the ones that are capable think it is worth their while to chase a bug bounty. They would rather have a nice contract or a full-time role,” she says.
Ethical hacking comes with legal risks
Its not just the lack of a steady income. Security researchers also face legal risks from anti-hacking laws, such as the UK’s Computer Misuse Act and the US’s draconian Computer Fraud and Abuse Act.
When Moussouris joined Microsoft in 2007, she persuaded the company to announce that it would not prosecute bounty hunters if they found online vulnerabilities in Microsoft products and reported them responsibly. Other software companies have since followed suit.
The UK government has now recognised the problem and promised to introduce a statutory defence for cyber security researchers who spot and share vulnerabilities to protect them from prosecution.
Another issue is that many software companies insist on security researchers signing a non-disclosure agreement (NDA) before paying them for their vulnerability disclosures.
This flies against the best practices for security disclosures, which Moussouris has championed through the International Standards Organisation (ISO).
When software companies pay the first person to discover a vulnerability a bounty in return for signing an NDA, that creates an incentive for those who find the same vulnerability to publicly disclose it, increasing the risk that a bad actor will exploit it for criminal purposes.
Worse, some companies use NDAs to keep vulnerabilities hidden but don’t take steps to fix them, says Moussouris, whose company, Luta Security, manages and advises on bug bounty and vulnerability disclosure programmes.
“We often see a big pile of unfixed bugs,” she says. “And some of these programmes are well funded by publicly traded companies that have plenty of cyber security employees, application security engineers and funding.”
Some companies appear to regard bug bounties as a replacement for secure coding and proper investment in software testing.
“We are using bug bounties as a stop-gap, as a way to potentially control the public disclosure of bugs, and we are not using them to identify symptoms that can diagnose our deeper lack of security controls,” she adds.
Ultimately, Moussouris says, governments will have to step in and change laws to make software companies liable for errors in their software, in much the same way car manufacturers are responsible for safety flaws in their vehicles.
“All governments have pretty much held off on holding software companies responsible and legally liable, because they wanted to encourage the growth of their industry,” she says. “But that has to change at a certain point, like automobiles were not highly regulated, and then seatbelts were required by law.”
AI could lead to less secure code
The rise of artificial intelligence (AI) could make white hat hackers redundant altogether, but perhaps not in a way that leads to better software security.
All of the major bug bounty platforms in the US are using AI to help with the triage of vulnerabilities and to augment penetration testing.
An AI-powered penetration testing platform, XBow, recently topped the bug bounty leaderboard by using AI to focus on relatively easy-to-find vulnerabilities and testing likely candidates in a systematic way to harvest security bugs.
“Once we create the tools to train AI to make it appear to be as good, or better in a lot of cases, than humans, you are pulling the rug out of the market. And then where are we going to get the next bug bounty expert?” she asks.
The current generation of experts with the skills to spot when AI systems are missing something important is in danger of disappearing.
“Bug bounty platforms are moving towards an automated, driverless version of bug bounties, where AI agents are going to take the place of human bug hunters,” she says.
Unfortunately, it’s far easier for AI to find software bugs than it is to use AI to fix them. And companies are not investing as much as they should in using AI to mitigate security risks.
“We have to figure out how to change that equation very quickly. It is easier to find and report a bug than it is for AI to write and test a patch,” she says.
Bug bounties have failed
Moussouris, a passionate and enthusiastic advocate of bug bounty schemes, is the first to acknowledge that bug bounty schemes have, in one sense, failed.
Some things have improved. Software developers have shifted to better programming languages and frameworks that make it harder to introduce particular classes of vulnerability, such as cross-site scripting errors.
But there is, she suggests, too much security theatre. Companies still address faults because they are visible, but hold off fixing things that the public can’t see, or use non-disclosure agreements to buy silence from researchers to keep vulnerabilities from the public.
Moussouris believes that AI will ultimately take over from human bug researchers, but says the loss of expertise will damage security.
The world is on the verge of another industrial revolution, but it will be bigger and faster than the last industrial revolution. In the 19th century, people left agriculture to work long hours in factories, often in dangerous conditions for poor wages.
As AI takes over more tasks currently carried out by people, unemployment will rise, incomes will fall and economies risk stagnation, Moussouris predicts.
The only answer, she believes, is for governments to tax AI companies and use the proceeds to provide the population with a universal basic income (UBI). “I think it has to, or literally there will be no way for capitalism to survive,” she says. “The good news is that human engineering ingenuity is still intact for now. I still believe in our ability to hack our way out of this problem.”
Growing tensions between governments and bug bounty hunters
The work of bug bounty hunters has also been impacted by moves to require software technology companies to report vulnerabilities to governments before they fix them.
It began with China in 2021, which required tech companies to disclose new vulnerabilities within 48 hours of discovery.
“It was very clear that they were going to evaluate whether or not they were going to use vulnerabilities for offensive purposes,” says Moussouris.
In 2020, the European Union (EU) introduced the Cyber Resilience Act (CRA), which introduced similar disclosure obligations, ostensibly to allow European government to prepare their cyber defences.
Moussouris is a co-author of the ISO standard on vulnerability disclosure. One of its principles is to limit the knowledge of security bugs to the smallest number of people before they are fixed.
The EU argues that its approach will be safe because it is not asking for a deep technical explanation of the vulnerabilities, nor is it asking for proof-of-concept code to show how vulnerabilities can be exploited.
But that misses the point, says Moussouris. Widening the pool of people with access to information about vulnerabilities will make leaks more likely and raises the risk that criminal hackers or hostile nation-states will exploit them for crime or espionage.
Risk from hostile nations
Moussouris does not doubt that hostile nations will exploit the weakest links in government bug notification schemes to learn new security exploits. If they are already using those vulnerabilities for offensive hacking, they will be able to cover their tracks.
“I anticipate there will be an upheaval in the threat intelligence landscape because our adversaries absolutely know this law is going to take effect. They are certainly positioning themselves to learn about these things through the leakiest party that gets notified,” she says.
“And they will either start targeting that particular software, if they weren’t already, or start pulling back their operations or hiding their tracks if they were the ones using it. It’s counterproductive,” she adds.
Moussouris is concerned that the US will likely follow the EU by introducing its own bug reporting scheme. “I am just holding my breath, anticipating that the US is going to follow, but I have been warning them against it.”
The UK’s equities programme
In the UK, GCHQ regulates government use of security vulnerabilities for spying through a process known as the equities scheme.
That involves security experts weighing up whether the UK would place its own critical systems at risk if it failed to notify software suppliers of potential exploits against the potential value of the exploit for gathering intelligence.
The process has a veneer of rationality, but it falls down because, in practice, government experts can have no idea how widespread vulnerabilities are in the critical national infrastructure. Even large suppliers like Microsoft have trouble tracking where their own products are used.
“When I was working at Microsoft, it was very clear that while Microsoft had a lot of visibility into what was deployed in the world, there were tonnes of things out there that they wouldn’t know about until they were exploited,” she says.
“The fact that Microsoft, with all its telemetry ability to know where its customers are, struggled means there is absolutely no way to gauge in a reliable way how vulnerable we are,” she adds.
Kate Moussouris spoke to Computer Weekly at the SANS CyberThreat Summit.
Swinger to make 70 staff redundant, loss of Versace orders proves decisive
OpenAI, Anthropic, and Block Are Teaming Up to Make AI Agents Play Nice
Ayeza Khan stuns fans with vibrant saree looks
-
Sports1 week agoIndia Triumphs Over South Africa in First ODI Thanks to Kohli’s Heroics – SUCH TV
-
Entertainment1 week agoSadie Sink talks about the future of Max in ‘Stranger Things’
-
Fashion1 week agoResults are in: US Black Friday store visits down, e-visits up, apparel shines
-
Politics1 week agoElon Musk reveals partner’s half-Indian roots, son’s middle name ‘Sekhar’
-
Tech1 week agoPrague’s City Center Sparkles, Buzzes, and Burns at the Signal Festival
-
Sports1 week agoBroncos secure thrilling OT victory over Commanders behind clutch performances
-
Business1 week agoCredit Card Spends Ease In October As Point‑Of‑Sale Transactions Grow 22%
-
Entertainment1 week agoNatalia Dyer explains Nancy Wheeler’s key blunder in Stranger Things 5