Connect with us

Tech

A new way to test how well AI systems classify text

Published

on

A new way to test how well AI systems classify text


SP-Attack pipeline. High-flip-capacity words are used to conduct low-cost single-word adversarial attacks. Credit: Expert Systems (2025). DOI: 10.1111/exsy.70079

Is this movie review a rave or a pan? Is this news story about business or technology? Is this online chatbot conversation veering off into giving financial advice? Is this online medical information site giving out misinformation?

These kinds of automated conversations, whether they involve seeking a movie or restaurant review or getting information about your or health records, are becoming increasingly prevalent. More than ever, such evaluations are being made by highly sophisticated algorithms, known as text classifiers, rather than by human beings. But how can we tell how accurate these classifications really are?

Now, a team at MIT’s Laboratory for Information and Decision Systems (LIDS) has come up with an innovative approach to not only measure how well these classifiers are doing their job, but then go one step further and show how to make them more accurate.

The new evaluation and remediation software was developed by Kalyan Veeramachaneni, a principal research scientist at LIDS, his students Lei Xu and Sarah Alnegheimish, and two others. The is being made freely available for download by anyone who wants to use it.

The team’s results were published on July 7 in the journal Expert Systems in a paper by Xu, Veeramachaneni, and Alnegheimish of LIDS, along with Laure Berti-Equille at IRD in Marseille, France, and Alfredo Cuesta-Infante at the Universidad Rey Juan Carlos, in Spain.

A standard method for testing these classification systems is to create what are known as synthetic examples—sentences that closely resemble ones that have already been classified. For example, researchers might take a sentence that has already been tagged by a classifier program as being a rave review, and see if changing a word or a few words while retaining the same meaning could fool the classifier into deeming it a pan. Or a sentence that was determined to be misinformation might get misclassified as accurate. This ability to fool the classifiers makes these .

People have tried various ways to find the vulnerabilities in these classifiers, Veeramachaneni says. But existing methods of finding these vulnerabilities have a hard time with this task and miss many examples that they should catch, he says.

Increasingly, companies are trying to use such evaluation tools in real time, monitoring the output of chatbots used for various purposes to try to make sure they are not putting out improper responses. For example, a bank might use a chatbot to respond to routine customer queries such as checking account balances or applying for a credit card, but it wants to ensure that its responses could never be interpreted as financial advice, which could expose the company to liability.

“Before showing the chatbot’s response to the end user, they want to use the text classifier to detect whether it’s giving financial advice or not,” Veeramachaneni says. But then it’s important to test that classifier to see how reliable its evaluations are.

“These chatbots, or summarization engines or whatnot, are being set up across the board,” he says, to deal with external customers and within an organization as well, for example providing information about HR issues. It’s important to put these text classifiers into the loop to detect things that they are not supposed to say, and filter those out before the output gets transmitted to the user.

That’s where the use of adversarial examples comes in—those sentences that have already been classified but then produce a different response when they are slightly modified while retaining the same meaning. How can people confirm that the meaning is the same? By using another large language model (LLM) that interprets and compares meanings.

So, if the LLM says the two sentences mean the same thing, but the classifier labels them differently, “that is a sentence that is adversarial—it can fool the classifier,” Veeramachaneni says. And when the researchers examined these adversarial sentences, “we found that most of the time, this was just a one-word change,” although the people using LLMs to generate these alternate sentences often didn’t realize that.

Further investigation, using LLMs to analyze many thousands of examples, showed that certain specific words had an outsized influence in changing the classifications, and therefore the testing of a classifier’s accuracy could focus on this small subset of words that seem to make the most difference. They found that one-tenth of 1% of all the 30,000 words in the system’s vocabulary could account for almost half of all these reversals of classification, in some specific applications.

Lei Xu Ph.D. ’23, a recent graduate from LIDS who performed much of the analysis as part of his thesis work, “used a lot of interesting estimation techniques to figure out what are the most powerful words that can change the overall classification, that can fool the classifier,” Veeramachaneni says.

The goal is to make it possible to do much more narrowly targeted searches, rather than combing through all possible word substitutions, thus making the computational task of generating adversarial examples much more manageable. “He’s using large language models, interestingly enough, as a way to understand the power of a single word.”

Then, also using LLMs, he searches for other words that are closely related to these powerful words, and so on, allowing for an overall ranking of words according to their influence on the outcomes. Once these adversarial sentences have been found, they can be used in turn to retrain the classifier to take them into account, increasing the robustness of the classifier against those mistakes.

Making classifiers more accurate may not sound like a big deal if it’s just a matter of classifying news articles into categories, or deciding whether reviews of anything from movies to restaurants are positive or negative. But increasingly, classifiers are being used in settings where the outcomes really do matter, whether preventing the inadvertent release of sensitive medical, financial, or security information, or helping to guide important research, such as into properties of chemical compounds or the folding of proteins for biomedical applications, or in identifying and blocking hate speech or known misinformation.

As a result of this research, the team introduced a new metric, which they call p, which provides a measure of how robust a given classifier is against single-word attacks. And because of the importance of such misclassifications, the research team has made its products available as open access for anyone to use. The package consists of two components: SP-Attack, which generates adversarial sentences to test classifiers in any particular application, and SP-Defense, which aims to improve the robustness of the classifier by generating and using adversarial sentences to retrain the model.

In some tests, where competing methods of testing classifier outputs allowed a 66% success rate by adversarial attacks, this team’s system cut that attack success rate almost in half, to 33.7%. In other applications, the improvement was as little as a 2% difference, but even that can be quite important, Veeramachaneni says, since these systems are being used for so many billions of interactions that even a small percentage can affect millions of transactions.

More information:
Lei Xu et al, Single Word Change Is All You Need: Using LLMs to Create Synthetic Training Examples for Text Classifiers, Expert Systems (2025). DOI: 10.1111/exsy.70079

This story is republished courtesy of MIT News (web.mit.edu/newsoffice/), a popular site that covers news about MIT research, innovation and teaching.

Citation:
A new way to test how well AI systems classify text (2025, August 14)
retrieved 14 August 2025
from https://techxplore.com/news/2025-08-ai-text.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.





Source link

Tech

Study examines whether policy intervention could combat ransomware

Published

on

Study examines whether policy intervention could combat ransomware


Credit: Pixabay/CC0 Public Domain

As ransomware attacks become more common and complex—and costly to the crimes’ targets—a University of Texas at Dallas researcher is examining how policymakers might combat cybercriminals.

Dr. Atanu Lahiri, an associate professor of information systems at the Naveen Jindal School of Management, said ransomware has become one of the top cybersecurity threats facing organizations worldwide. Spread primarily through email phishing scams and exploitation of unpatched software bugs, ransomware robs a user’s access to computer files until a ransom is paid.

“The data is still on your computer,” he said. “It’s locked up, and the criminals have the key.”

In a study published in Information Systems Research, Lahiri and a colleague examined whether and under what circumstances policy intervention could help deter this type of cyberattack. He found that effective response solutions might depend on factors such as the value of compromised information, the nature of the ransom demand, and who or what organization is most affected.

Although paying ransom often seems preferable to facing business disruptions, payments also embolden the attackers and encourage them to come back for more. This ripple effect, or externality, which is driven by extortion, creates a unique problem dubbed “extortionality” by the authors.

“There are two questions: When do we care, and what do we do?” Lahiri said. “Should ransom payments be banned or even penalized?”

The disruptions caused by can be crippling for businesses. In 2024, the FBI’s Internet Crime Complaint Center received more than 3,000 ransomware complaints. Victims paid over $800 million to attackers, according to research by Chainalysis, although the impact is likely much higher because many incidents and payments go unreported.

The illegal breaches have hit targets ranging from Fortune 500 companies to police departments to government and university systems.

Lahiri was inspired to explore potential solutions as federal and state lawmakers grapple with laws to restrict government entities and other companies from paying ransoms to regain access to their data. He found that fighting these threats through legislation is tricky because a ban on ransom payments or other penalties could negatively affect the victim, whose goal is simply to recover compromised information quickly and with minimal disruption.

For example, outright bans on ransom payment are particularly problematic for hospitals, where lives are at stake and critical lifesaving information can’t be accessed.

On the other hand, paying ransom rewards criminal behavior, encourages more breaches and elevates the risk of additional attacks, the researchers found.

Through mathematical models and simulations, Lahiri determined that an ideal scenario in many cases would be for companies not to give in to an attacker’s ransom demand. In practice, however, this solution is not so clear-cut.

“It relies on you trusting the other guy, in this case other organizations, not to pay up either,” he said. “It would be better if nobody paid, but if someone does, it would raise the risk for everybody.”

“You have to be careful when you impose a ban, though,” said Lahiri, who teaches the graduate class Cybersecurity Fundamentals at UT Dallas, serves as director of the cybersecurity systems certificate program, and chairs the University Information Security Advisory Committee. “A more reasoned approach might be to first try incentives or a penalty to deter ransom payments.”

If the attackers are not strategic in choosing their ransom asks—and do not demand different sums from the victims depending on their ability to pay—Lahiri recommends that policymakers impose fines or taxes on companies that pay ransoms.

“When imposing a ban, policymakers should be mindful,” he said. “In particular, hospitals and critical infrastructure firms should be exempted to avoid excessive collateral damage from business disruption.

“In some cases, you wouldn’t even have to impose the ban, but if you talk a lot about a ban, ransom payers would take notice. Even the specter of a ban might do the trick and make organizations invest in backup technologies that can help them recover without having to pay the attackers.”

The best offense, Lahiri said, is a good defense, and the is simply more redundancy. Backing up data and practicing drills on recovering information is a strong way to avoid paying the attacker. Policymakers could incentivize redundancy measures, he said, by subsidizing backup technology, practice drills and awareness campaigns.

“One of the biggest problems is that people don’t invest in backups,” Lahiri said. “They don’t conduct drills, like fire drills. Security is always seen as a hassle.

“If we had great backups and we could recover from the attacks, we would not be paying the ransom in the first place. And we would not be talking about extortionality.”

Dr. Debabrata Dey, Davis Professor and area director of analytics, information and operations at the University of Kansas, is a co-author of the study.

More information:
Debabrata Dey et al, “Extortionality” in Ransomware Attacks: A Microeconomic Study of Extortion and Externality, Information Systems Research (2025). DOI: 10.1287/isre.2024.1160

Citation:
Study examines whether policy intervention could combat ransomware (2025, August 28)
retrieved 28 August 2025
from https://techxplore.com/news/2025-08-policy-intervention-combat-ransomware.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.





Source link

Continue Reading

Tech

Manufacturas Eliot boosts digital shift with Coats Digital’s VisionPLM

Published

on

Manufacturas Eliot boosts digital shift with Coats Digital’s VisionPLM



Coats Digital is pleased to announce that that Manufacturas Eliot, one of Colombia’s leading fashion textile groups, has selected VisionPLM to advance its digital transformation strategy. The solution will optimise product lifecycle management across its portfolio of brands—Patprimo, Seven Seven, Ostu, and Atmos—enhancing collaboration, streamlining operations, and enabling greater speed to market.

Manufacturas Eliot, a Colombian fashion group, has selected Coats Digital’s VisionPLM to boost digital transformation across its brands.
The platform will enhance collaboration, speed up product development, and streamline operations.
VisionPLM aims to improve agility, traceability, and decision-making, supporting Eliot’s drive for innovation and sustainable growth.

Founded in 1957, Manufacturas Eliot is a vertically integrated manufacturer producing over 20 million garments annually. Renowned for delivering high-quality, accessible fashion, the group continues to invest in technologies that support sustainable growth and operational excellence.

The implementation of VisionPLM demonstrates Elliot’s strong commitment to end-to-end digitalisation across the value chain. By introducing VisionPLM, Eliot aims to improve product development agility, reduce time-to-market, and ensure seamless communication across cross-functional teams.

Juliana Pérez, Design Director, Seven Seven, commented: “From the design team’s point of view, we’re really excited about implementing VisionPLM, as it will allow us to manage our collections in a more structured way and collaborate efficiently with other departments.”

Angela Quevedo, Planning Director,  Manufacturas Eliot, added: “VisionPLM will significantly improve the planning and coordination of our operations by enabling a more accurate flow of information and reducing response times across the supply chain. It will also help us optimise processes and accelerate decision-making.”

Tailored specifically for the fashion industry, VisionPLM integrates tools that boost development speed, improve traceability, and enhance decision-making. By centralising design, sourcing, and supplier collaboration in one digital platform, the solution enables a streamlined, transparent, and responsive approach to managing collections.

Oscar González, Coats Digital – LATAM, said: “We’re proud to continue supporting Manufacturas Eliot on its digital transformation journey. The adoption of VisionPLM marks a key milestone in advancing its fashion innovation strategy—enabling faster, smarter decision-making and more agile collaboration across teams and suppliers. Its helping to build a future-ready, connected operation that’s fully aligned to the demands of today’s fashion market.”

Note: The headline, insights, and image of this press release may have been refined by the Fibre2Fashion staff; the rest of the content remains unchanged.

Fibre2Fashion News Desk (HU)



Source link

Continue Reading

Tech

Top CDC Officials Resign After Director Is Pushed Out

Published

on

Top CDC Officials Resign After Director Is Pushed Out


Susan Monarez is no longer the director of the US Centers for Disease Control and Prevention, according to a post by the official Department of Health and Human Services X account. She had been in the position for just a month. In the wake of her apparent ouster, several other CDC leaders have resigned.

Named acting CDC director in January, Monarez was officially confirmed to the position by the Senate on July 29 and sworn in two days later. During her brief tenure, the CDC’s main campus in Atlanta was attacked by a gunman who blamed the Covid-19 vaccine for making him sick and depressed. A local police officer, David Rose, was killed by the suspect when responding to the shooting.

In a statement Wednesday evening Mark Zaid and Abbe David Lowell, Monarez’s lawyers, alleged that she had been “targeted” for refusing “to rubber-stamp unscientific, reckless directives and fire dedicated health experts.” The statement further says that Monarez has not resigned and does not plan to, and claims that she has not received notification that she’s been fired.

According to emails obtained by WIRED, at least three other senior CDC officials resigned Wednesday evening: Demetre Daskalakis, director of the National Center for Immunization and Respiratory Diseases; Debra Houry, chief medical officer and deputy director for program and science; and Daniel Jernigan, director of the National Center for Emerging and Zoonotic Infectious Diseases.

More resignations are expected to become public soon, say CDC with knowledge of the departures.

“I worry that political appointees will not make decisions on the science, but instead focus on supporting the administration’s agenda,” says one CDC employee, who was granted anonymity out of concerns over retribution. “I worry that the next directors will not support and protect staff.”

President Donald Trump’s original pick to lead the CDC was David Weldon, a physician and previous Republican congressman from Florida who had a history of making statements questioning the safety of vaccines. But hours before his Senate confirmation hearing in March, the White House withdrew Weldon’s nomination. The administration then nominated Monarez.

The CDC leadership exits come amid recent vaccine policy upheaval by HHS secretary Robert F. Kennedy Jr., who in May removed the Covid-19 vaccine from the list CDC’s recommended vaccines for healthy children and pregnant women. The following month, he fired all 17 sitting members of the CDC’s Advisory Committee on Immunization Practices, a group of independent experts that makes science-based recommendations on vaccines.

In their place, he installed eight new members, including several longtime vaccine critics. “A clean sweep is necessary to reestablish public confidence in vaccine science,” Kennedy said in a statement at the time.

Earlier this month under Kennedy’s leadership, HHS canceled a half billion dollars in funding for research on mRNA vaccines. This month HHS also announced the reinstatement of the Task Force on Safer Childhood Vaccines, a federal advisory panel created by Congress in 1986 to improve vaccine safety and oversight for children in the US. The panel was disbanded in 1998, when it issued its final report. Public health experts worry that the panel is a move to further undermine established vaccine science.



Source link

Continue Reading

Trending