Limits of Large Language Models in Debating Humans
Authors:
James Flamino,
Mohammed Shahid Modi,
Boleslaw K. Szymanski,
Brendan Cross,
Colton Mikolajczyk
Abstract:
Large Language Models (LLMs) have shown remarkable promise in their ability to interact proficiently with humans. Subsequently, their potential use as artificial confederates and surrogates in sociological experiments involving conversation is an exciting prospect. But how viable is this idea? This paper endeavors to test the limits of current-day LLMs with a pre-registered study integrating real…
▽ More
Large Language Models (LLMs) have shown remarkable promise in their ability to interact proficiently with humans. Subsequently, their potential use as artificial confederates and surrogates in sociological experiments involving conversation is an exciting prospect. But how viable is this idea? This paper endeavors to test the limits of current-day LLMs with a pre-registered study integrating real people with LLM agents acting as people. The study focuses on debate-based opinion consensus formation in three environments: humans only, agents and humans, and agents only. Our goal is to understand how LLM agents influence humans, and how capable they are in debating like humans. We find that LLMs can blend in and facilitate human productivity but are less convincing in debate, with their behavior ultimately deviating from human's. We elucidate these primary failings and anticipate that LLMs must evolve further before being viable debaters.
△ Less
Submitted 5 February, 2024;
originally announced February 2024.
Dynamics of Ideological Biases of Social Media Users
Authors:
Mohammed Shahid Modi,
James Flamino,
Boleslaw K. Szymanski
Abstract:
Humanity for centuries has perfected skills of interpersonal interactions and evolved patterns that enable people to detect lies and deceiving behavior of others in face-to-face settings. Unprecedented growth of people's access to mobile phones and social media raises an important question: How does this new technology influence people's interactions and support the use of traditional patterns? In…
▽ More
Humanity for centuries has perfected skills of interpersonal interactions and evolved patterns that enable people to detect lies and deceiving behavior of others in face-to-face settings. Unprecedented growth of people's access to mobile phones and social media raises an important question: How does this new technology influence people's interactions and support the use of traditional patterns? In this article, we answer this question for homophily-driven patterns in social media. In our previous studies, we found that, on a university campus, changes in student opinions were driven by the desire to hold popular opinions. Here, we demonstrate that the evolution of online platform-wide opinion groups is driven by the same desire. We focus on two social media: Twitter and Parler, on which we tracked the political biases of their users. On Parler, an initially stable group of Right-biased users evolved into a permanent Right-leaning echo chamber dominating weaker, transient groups of members with opposing political biases. In contrast, on Twitter, the initial presence of two large opposing bias groups led to the evolution of a bimodal bias distribution, with a high degree of polarization. We capture the movement of users from the initial to final bias groups during the tracking period. We also show that user choices are influenced by side-effects of homophily. Users entering the platform attempt to find a sufficiently large group whose members hold political biases within the range sufficiently close to their own. If successful, they stabilize their biases and become permanent members of the group. Otherwise, they leave the platform. We believe that the dynamics of users' behavior uncovered in this article create a foundation for technical solutions supporting social groups on social media and socially aware networks.
△ Less
Submitted 22 July, 2024; v1 submitted 27 September, 2023;
originally announced September 2023.
