Reparametrization invariance in Bayesian approximations

Bayesian analysis of deep neural networks has largely been unsuccessful, often resulting in significant under- or over-fitting. We argue that this is due to inappropriate treatment of the overparametrization that trivially follows from increasing network depth. We give a geometric characterization of overparametrization and show that respecting this geometric structure results in significant improvements to Bayesian approximations. We further discuss the numerical aspects.