Imagine asking someone to sort thousands of messages into different boxes, then asking them to sort the next thousand on their own. You would not expect them to get it right after seeing only one example. You would need to show them many examples, explain why each one belongs in a particular box, and include examples that look similar but actually belong somewhere else.
Training a machine-learning classifier works in a similar way. Researchers label real-world content, and the model learns patterns from those labelled examples before applying what it has learned to new content. The difficulty is that extremist narratives do not always arrive with clear labels. The same words can mean different things depending on who uses them, who they are directed at, and what is happening around them, raising a crucial question, “How do we teach AI to recognise the difference?”
This question was at the heart of the third SEAN-CSO 2026 webinar, “Detecting Extremist Narratives Using AI,” held on 29 September 2026. The session featured Matteo Vergani, Associate Professor at Deakin University, with Aughya Shandriasti, Curriculum Director at Yayasan Ludere Nusantara Gemilang (Ludenara), as moderator. The discussion explored how machine-learning classifiers can support P/CVE research while examining the human work and judgement behind the technology.
Detecting Patterns at Scale
During the session, Matteo introduced participants to possible uses of machine-learning classifiers in CVE. Through a Padlet activity, participants suggested applications ranging from dashboards for monitoring extremism on social media and identifying radicalisation patterns to detecting potential early signs of radicalisation through online engagement. These applications show where AI can be useful because, instead of asking researchers to manually examine every piece of online content, classifiers can help surface patterns across large datasets and identify material that may require closer attention.
But the moment we ask a machine to classify something, another question appears: what exactly are we asking it to classify? Participants explored this through a second Padlet activity focused on the ethical implications of using machine-learning classifiers in CVE. Their responses raised concerns about false positives, AI bias, privacy, access to sensitive information, and the risk of reinforcing an “us versus them” distinction, while also highlighting the importance of treating classification as a secondary source rather than relying on it as the sole basis for research or reporting. These concerns point to a fundamental distinction: a classification is not the same as an understanding of the content.
The Human Work Behind the Machine
The work behind a classifier begins long before the model produces an output. Matteo explained that a useful rule of thumb is to have at least 1,000 annotations per category, so a classifier distinguishing between hateful and non-hateful content would require at least 1,000 examples for each category. When more categories are introduced, each category requires its own set of examples, and reaching high detection accuracy can require several thousand annotations.
That makes annotation one of the most resource-intensive parts of building an AI classifier. Human annotators, including community members or trained experts, review social media content and assess it according to a defined framework, considering questions such as whether a piece of content is hateful towards a group, which group is being targeted, and whether it incites violence. Tools such as Label Studio can help organise this process, but the actual judgement still comes from people.
The process becomes even more interesting when multiple annotators look at the same piece of content. People can interpret online hate differently even when they receive the same definitions and training, so researchers do not simply ask one person to provide the final label. There is often strong agreement when content is clearly hateful or clearly non-hateful, but a “grey zone” appears when the meaning is less obvious, particularly when context, sarcasm, coded language, or cultural references are involved.
Instead of treating these disagreements as mistakes to be removed, researchers can use them as information. A model does not necessarily have to produce only a binary “hateful” or “not hateful” result. It can also produce a probability score that reflects how likely a piece of content is to belong to a particular category, allowing researchers to account for the uncertainty that exists in human interpretation.
This means that AI does not simply learn from people agreeing on what counts as hate. It can also learn from where people disagree. Researchers can then set a threshold depending on the purpose of the analysis and what they need the classifier to show to different stakeholders, including communities or law enforcement agencies.
Context Still Matters
These distinctions become even more important when classifiers move across languages and cultural settings. During the Q&A, a participant raised the challenge of applying AI classifiers across Southeast Asia, where language, culture, and the ways extremist narratives are expressed can vary significantly between contexts. In response, Matteo noted that newer generations of large language models have become considerably better at handling different languages and variations in language, offering greater flexibility when working across different linguistic contexts.
For P/CVE research in Southeast Asia, however, language is only one part of context. Slang, cultural references, local events, community dynamics, and the ways narratives develop online can all influence how content should be understood, making local knowledge and careful interpretation important alongside the technology itself.
What Comes After Classification?
The conversation therefore does not end when an AI system flags a piece of content. Researchers still need to consider whether the classification is accurate, what context may be missing, and what the result should actually be used for, particularly when false positives, bias, privacy, and sensitive information are involved.
A tool developed to support research should not automatically become a mechanism for making decisions about individuals simply because it can be used that way. Its purpose, safeguards, access, and limitations need to remain clear, especially in P/CVE contexts where automated classifications may carry consequences beyond the original research question.
Ultimately, the challenge is not simply to build classifiers that can identify more extremist or hateful content. It is to develop and use them with clear definitions, diverse training data, contextual understanding, and human judgement, because before AI can detect a narrative, humans first have to decide what they are looking for.
The conversation continues in SEAN-CSO Webinar 4, “AI Supporting Case Management and Practitioners’ Decisions,” on 6 October 2026, where the discussion will move from detecting extremist narratives to exploring how AI-generated insights can support practitioners when they are faced with real cases and decisions. To continue exploring how AI can support P/CVE practice.
Register now!
https://seancso.org/registerseancso2026
SEAN-CSO provides a platform for civil society organisations in Southeast Asia to network and share resources. Through this network, members can support each other, share knowledge, and coordinate efforts to address the region’s social challenges. In doing so, SEAN-CSO contributes to strengthening the voice of civil society at both regional and international levels.