Skip to content
conftrace_
2026 ICML ICML 2026

Position: From Crowdsourcing to Crowd-LLM-Sourcing and LLM-Sourcing

Crowdsourcing has been widely adopted for large-scale data collection and problem solving, yet its outcomes are often noisy and inconsistent, making quality control and aggregation central concerns. Meanwhile, Large Language Models (LLMs) have shown strong capabilities in generation, annotation, evaluation, and reasoning. These developments can be framed as an emerging paradigm at the intersection of crowdsourcing and LLMs, encompassing two directions: (1) Crowd-LLM-Sourcing, where humans and LLMs jointly participate in workflows, and (2) LLM-Sourcing Inspired by Crowdsourcing, where crowdsourcing principles guide LLM-driven generation, annotation, evaluation, and inference. Many existing studies on LLMs overlook decades of prior work in crowdsourcing, even though the two domains are grounded in closely related principles on some topics. The central position of this paper is that, in scenarios where an LLM can be regarded as an LLM worker, LLM research should draw upon the rich body of crowdsourcing literature. At the same time, LLM workers differ from human workers in their correlated errors and context-dependent capabilities. This paper therefore highlights not only the relevance of crowdsourcing, but also the need to adapt its mechanisms for collective intelligence with model-based agents.

More from ICML 2026

All ICML 2026 papers →