Mind over metrics: How can we tell if two brains (or AI models) are alike?

[cm0002@mander.xyz]

Indeed, the computational literature is now hopelessly replete with competing methods that quantify some form of similarity in neural population codes. One cluster of methods frames the problem through the lens of geometry, asking whether two systems arrange their responses in the same shape. This includes the framework of representational similarity analysis (RSA), as well as linear centered kernel alignment (CKA), which has become the de facto standard in the machine-learning research community. Others favor prediction, gauging similarity by how well the activity of one system can be used to predict that of the other. This perspective is prevalent in initiatives, such as Brain-Score, that use regularized linear regression performance as a metric of similarity. Some approaches, such as Procrustes shape distance, combine elements of both geometric similarity and prediction.

The summary above is highly incomplete—a recent review of the literature documented well over 30 methods in use. This proliferation of approaches gives us a deep well to draw from, but it also represents a serious concern. Many neuroscience practitioners—even those with computational and mathematical backgrounds—simply do not have the time to sift through this complex literature and understand its nuances. A skeptic may even feel that we are overcomplicating the problem.

https://www.thetransmitter.org/neural-dynamics/mind-over-metrics-how-can-we-tell-if-two-brains-or-ai-models-are-alike/