Position: Vision Encoders should be Image Size Agnostic and Task Driven
Abstract
This position paper argues that the next generation of vision encoders should be image size agnostic and task driven. The source of our inspiration is a behavioral trait of biological vision – efficiency. We focus on a couple of ways in which vision in nature is efficient, but modern vision encoders not. Humans and animals deal with vast quantities of visual data, and need to be smart where they focus their limited energy. It is our belief that vision encoders should be dynamic and the computational complexity should depend on the task at hand rather than the size of the image. To realize this, we introduce SOVA (Sequential Observation with Visual Attention) – a proof-of-concept solution for image classification. Despite classification not being representative for what we are trying to achieve, it shows that our approach is feasible and promising. Code is available at: https://github.com/insait-institute/sova.