CLIP Talk: Rupak Sarkar & Philip Resnik (CS)

PairScale: Analyzing Attitude Change with Pairwise Comparisons

Abstract: We introduce a text-based method for measuring attitudes in communities regarding issues of interest, going beyond the pro/con/neutral of conventional stance detection to extract both implicit and explicit attitudes and place them on a continuous scale. The method exploits LLMs both to extract attitudes and to perform pairwise comparisons that yield unidimensional attitude scores via the classic Bradley-Terry model. We validate the LLM-based steps using human judgments, and illustrate the utility of the method for social science by examining the evolution of attitudes on two high-profile issues in U.S. politics in two political communities on Reddit over the period spanning Donald Trump's presidential campaign to the 2022 mid-term elections. WARNING: Discussion of potentially offensive political content.