Toward Detecting Modern Social Media Agents: Dataset, Experimental Analysis, and an Agent-Based Solution
Abstract
AbstractWith the advancement of large language models (LLMs) and agent technologies, social bots have shed traditional stereotypes and evolved into autonomous, highly intelligent social media agents. This paper studies whether current social bot detection research is sufficient to identify such agents and discusses potential detection directions. We construct a social media agent dataset from authentic agent accounts on Weibo and Moltbook to evaluate prior detection approaches. Experimental results across various detection algorithms reveal that existing methods suffer from out-of-distribution (OOD) failures and exhibit a cognitive blind spot regarding agent-specific features. Through multidimensional feature comparisons, we find that such agents counterintuitively do not represent a transitional form between humans and traditional bots, but instead follow a distinct pattern that explains the observed generalization failures and feature limitations. Finally, motivated by the excellent performance of LLMs, we propose Agent-Detect-Agent, an online detection agent with tool calling. Experiments demonstrate that this agent further enhances the detection capability of base LLMs and achieves consistently good performance across different account types.