conftrace_

2019 NeurIPS NeurIPS 2019

Multiclass Performance Metric Elicitation

Abstract

Metric Elicitation is a principled framework for selecting the performance metric that best reflects implicit user preferences. However, available strategies have so far been limited to binary classification. In this paper, we propose novel strategies for eliciting multiclass classification performance metrics using only relative preference feedback. We also show that the strategies are robust to both finite sample and feedback noise.

Authors

Gaurush Hiranandani, Shant Boodaghians, Ruta Mehta, Oluwasanmi O Koyejo

Topics

Artificial Intelligence > Core AI > Causal Inference Machine Learning > Core Methods > Classification Machine Learning > Core Methods > Metric Learning Machine Learning > Learning Types > Evaluation

Keywords

preference learning multi-class classification performance metric preference feedback metric elicitation

More from NeurIPS 2019

All NeurIPS 2019 papers →

Gradient Dynamics of Shallow Univariate ReLU Networks

Chirality Nets for Human Pose Regression

TAB-VCR: Tags and Attributes based VCR Baselines

Assessing Social and Intersectional Biases in Contextualized Word Representations

Likelihood-Free Overcomplete ICA and Applications In Causal Discovery

MaCow: Masked Convolutional Generative Flow