Multi-Agent RL with Invisible Collaborators: Marginal Advantage Estimation for Indirect Cooperation

Cooperative Multi-Agent Reinforcement Learning ({MARL}) has primarily focused on direct cooperation among agents. However, many real-world systems exhibit structurally asymmetric cooperation, where agents are physically constrained and cannot directly coordinate with those on whom they depend. In such settings, accurately estimating individual contributions under high uncertainty is challenging. We propose Marginal Advantage Estimation (MAE), a representation-level contribution estimation method for cooperative {MARL} under this structural isolation. MAE employs a synchronised feature-masking mechanism to evaluate marginal contributions without action-level counterfactual perturbations, thereby reducing variance and providing more informative learning signals. We provide a theoretical analysis of its bias–variance properties and demonstrate consistent performance improvements across 18 tasks in Partitioned MPE and Basilisk benchmarks over strong cooperative {MARL} baselines.