First Article of Five – The Impact of Polarization and Digital Cacophony on AI-Generated Text – An Introduction
Artificial intelligence (AI) is expected to acquire many skills traditionally seen as the prerogative of humans. From routine tasks to applications across diverse fields, as fraud detection, advancements in autonomous vehicles, and medical diagnosis to, albeit artificially, empathizing with conversation partners, AI is set to permeate an extensive array of activities. Among these, its capacity to persuade users has emerged as a significant and rapidly expanding domain.
Indeed, the intersection of persuasion and AI has long been a reality. For example, CICERO, an AI developed by Meta, has demonstrated human-level performance in the intricate strategy game Diplomacy. It has shown itself capable of “negotiating, persuading, and collaborating with people” by understanding others’ goals and communicating strategically.
As this capability advances, the EU Council, within its AI Act, warns that AI-based manipulation techniques may be used to persuade individuals into unwanted behaviors or to deceptively guide their decisions. Thus, compromising their autonomy, decision-making processes, and freedom of choice.
While the persuasive potential of AI in political or commercial contexts warrants careful regulation, concerns arise also about its interaction with themes associated with harmful behavior, societal polarization, and toxic contents, until the potential erosion of democratic values.
A case in point is Jaswant Singh Chail, who plotted to assassinate Queen Elizabeth II, citing revenge for the 1919 Jallianwala Bagh massacre, while envisioning a mission inspired by Star Wars movies to overthrow empires.Chail exchanged over 5,000 messages with “Sarai,” an AI chatbot he created through the Replika app, which encouraged him to act on his plans. In this case, the AI chatbot “Sarai” acted as a reinforcement mechanism, amplifying his delusions and strengthening his fixation on carrying out a violent plan; rather than challenging his pathological thoughts, the chatbot’s design to agree with users created a feedback loop, that validated and encouraged his harmful intentions. Chail’s case demonstrates the potential danger of AI systems to inadvertently bolster unhealthy or extremist ideologies, by echoing users’ thoughts without critical oversight.
Besides the users’ psychiatric issues, which can amplify the risks related to bot-induced persuasion, the scalability of AI-generated content and its ability to influence users also intersect with a broader societal concern: the escalating polarization and cacophony of digital discourse. Public debate has increasingly shifted from top-down media or institutional communication to bottom-up emotionally charged and divisive narratives. This digital environment, marked by fragmented discussions and algorithm-driven echo chambers, has influenced not only public discourse but has also extended to the ideological frameworks of extremism. Indeed, extremist ideologies now thrive in this polarized landscape, blending misinformation, emotional appeals, and individualized interpretations to erode the boundaries between mainstream and radical narratives. These paths, often self-directed and multifaceted, defy easy categorization and prediction due to the absence of common, unifying factors.
In this respect, FBI Director Christopher Wray referred to this phenomenon as “salad bar ideologies,” where individuals amalgamate disparate beliefs, “a little of this, a little of that,” with the a predisposition to physical or verbal violence, rather than coherent ideas, as a common denominator. This trend is reflective of broader societal incoherence, as Gartenstein-Ross of Valens Global notes: “As we, as a people, are becoming more incoherent, so are extremists.”
As previously mentioned, this fragmentation in extremist ideologies is mirrored in the dynamics of public and political discourse in Western democracies, where emotional appeals have overtaken fact-based dialogue. Increasingly, language is used not to describe reality but to evoke feelings, diminishing the value of truth to prioritize the imperative of “keeping the conversation going,” emboldening anti-democratic and hate-driven narratives, which have gained traction in mainstream discourse. For instance, QAnon theories have inspired political campaigns and mass protests, far-right populism in Germany has reached unprecedented levels of radicalism and popularity, and the Israel-Hamas conflict has unleashed waves of antisemitic and anti-Muslim speech online, often degenerating into physical violence.
The underlying mechanism of this context is the entanglement of emotions with information: a polarized narrative space, where misinformation and emotional manipulation thrive and the symbolic value of truth is eroded, blending falsehoods with facts, reshaping societal perception and decision-making. Without corrective measures, these dynamics could present long-term risks to human rights and democratic values; indeed, even without directly threatening national security, they undermine the foundation of informed public discourse, steering societal perceptions and policies in unpredictable directions.
With these premises, if trained on datasets prioritizing emotion over empirical rigor, AI could exacerbate these trends, configuring the risk of amplifying toxic or biased narratives, by inadvertently persuading users to accept misinformation, or ideologies, that are antithetical to democratic principles.
On the other hand, AI could also offer the right opportunity to analyze the roots of emotional manipulation and adapt public discourse to the realities of contemporary communication.
By addressing the underlying causes of polarization and prioritizing pluralistic dialogue, society could turn the tide on digital cacophony, focusing on constructive engagement as the cornerstones of political and public debate.
How AI Absorbs and Reworks Human Language
To frame the context from which the previous issues emerge, it is useful to examine the links between two macro-areas: the functioning of language-based generative models, particularly large language models (LLMs), and, on the other hand, the reasons why the line between hate speech and legitimate discourse in Western societies has blurred. The emphasis on LLMs, rather than, for example, on AI-generated images and videos, illicit human manipulation of AI models, and technical errors, allows to focus specifically on AI’s capacity to understand the context of the conversation and persuade through language. This, in turn, raises the question of whether it is the social pervasiveness of human biased opinions that contaminates the data foundations of LLMs’ linguistic capabilities or vice versa.
LLMs can be succinctly described as the linguistic systems powering AI tools such as ChatGPT, relying on an algorithmic framework capable of recognizing, summarizing, translating, predicting, and generating texts and other content, based on learned knowledge derived from their dataset (i.e. a structured, organized, and archived collection of data used for analysis or processing). Currently, it is estimated that there are over 25,000 LLMs, each evolving from statistical models, previously employed in natural language processing (NLP), to more advanced, neural network-based models.
These neural networks develop language models based on vast text corpora, learning autonomously the relationships between various words by analyzing enormous quantities of data, without constant human intervention.
Specifically, a neural network allows a computer to process data by mimicking the human brain’s functions, employing a form of machine learning known as deep learning. This learning model operates through interconnected nodes in a layered structure that simulates neurons. Unlike traditional machine-learning models, which store copies of information for training, deep learning allows the model to independently refine its understanding through interaction, subtly adjusting certain data elements in response to new learning.
Neural networks represented a significant advancement, enabling language models to evolve from simple rule-based systems to more sophisticated models. Unlike earlier approaches that relied on statistical methods, which struggled with longer sentences and complex language structures, neural networks enhanced the models’ potential to generate text and, to some extent, comprehend it. However, this technical advancement still imposed to process text sequentially, limiting the understanding of complex language structures.
Indeed, neural network architectures were revolutionized by Transformer language processing, which is allowing AI to develop the ability to handle all parts of a sentence simultaneously, speeding up the processing time while enabling a simulated deeper understanding of context, regardless of how far apart words are in a sentence.
Transformers leverage a mechanism called “attention mechanism,”[i] which allows the model to weigh the importance of each word in a sentence in relation to every other word. The core of this architecture is the interplay between the encoder and decoder. The encoder processes the input sequence, transforming it into a numerical representation that encapsulates the semantic and syntactic meaning. This encoded information is then fed to the decoder which, in turn, generates the output sequence token by token. A token can be a word, a part of a word, or a punctuation mark.[ii] Finally, by considering both the output from the encoder and its own previously generated tokens, the decoder produces output that is both coherent and contextually appropriate.
Thus, unlike older models that might lose track of earlier words in a long sentence, this mechanism of reformulation and keeping track of previous tokens allows Transformers to maintain a comprehensive understanding of the entire context, while also enabling the model to process words it has never seen before, by decomposing them into known subwords.[iii]
To further understand the implications of these mechanisms, it is important to consider not only the remarkable technical advances these architectures have achieved but also their profound effect on human-computer interactions. In fact, the increasing capability of LLMs to mimic human-like conversation makes them not just tools but also entities, which engage with users in a way that evokes human responses, raising questions about trust, emotional engagement, and the evolving nature of communication in the digital age.
[i] “At each step the model is auto-regressive, consuming the previously generated symbols as additional input when generating the next.” Vaswani A. Et al, Attention Is All You Need, 31st Conference on Neural Information Processing Systems (NIPS 2017) in Long Beach – CA, Curran Associates Inc.57, Red Hook, NY, USA, December 4 – 9, 2017, https://arxiv.org/abs/1706.03762
The transformer uses an encoder-decoder architecture: the encoder extracts features from an input sentence, and the decoder uses the features to produce an output sentence. Thus, the architecture is structure in order to allow the encoder to map an input sequence which is then fed into a decoder. The decoder receives the output of the encoder together with the decoder output at the previous time step to generate an output sequence.
KiKaBen, Transformer’s Encoder-Decoder, December 21, 2021, https://kikaben.com/transformers-encoder-decoder/ ; Cristina S., The Transformer Model, Machine Learning Mastery, January 6, 2023, https://machinelearningmastery.com/the-transformer-model/
[ii] “Subword tokenization algorithms rely on the principle that frequently used words should not be split into smaller subwords, but rare words should be decomposed into meaningful subwords. For instance “annoyingly” might be considered a rare word and could be decomposed into “annoying” and “ly”. Both “annoying” and “ly” as stand-alone subwords would appear more frequently while at the same time the meaning of “annoyingly” is kept by the composite meaning of “annoying” and “ly”. This is especially useful in agglutinative languages such as Turkish, where you can form (almost) arbitrarily long complex words by stringing together subwords. Subword tokenization allows the model to have a reasonable vocabulary size while being able to learn meaningful context-independent representations. In addition, subword tokenization enables the model to process words it has never seen before, by decomposing them into known subwords.” Hugging Face, Summary of the tokenizers, https://huggingface.co/docs/transformers/tokenizer_summary
[iii] “The true power of Transformers lies in the synergy between the encoder and decoder. While the encoder provides a deep understanding of the input sentence, the decoder leverages this information to produce an accurate and relevant output. This interaction is mediated through the encoder-decoder attention mechanism, allowing the decoder to query the encoder’s output at each step of the generation process. This collaborative mechanism ensures that the output not only makes sense linguistically but is also a faithful representation or transformation of the input. It’s this encoder-decoder synergy that enables Transformers to excel in a wide range of language processing tasks, from machine translation to content generation.” Truefoundry, Transformer Architecture in Large Language Models, Marc 22, 2024, https://www.truefoundry.com/blog/transformer-architecture