There is a lot of interesting research into predictive coding as an alternative means to solve the credit assignment problem that might be a more plausible model of what happens in the brain.
I really liked this paper that showed using a predictive coding learning rule leads to the exact same gradients as backprop in arbitrary networks:
Oh wow the theoretical implications in neuroscience exite me here - is this a potential model of Fristons Markov Blanket concept
“ Probably the most ambitious and all-encompassing version of the ‘Bayesian turn’ in cognitive science is
the free energy principle (FEP). The FEP is a mathematical framework, developed by Karl Friston and
colleagues (Friston, Kilner, and Harrison 2006; Friston et al. 2010; Friston 2010; Friston et al. 2017a;
Friston 2019), which specifies an objective function that any self-organizing system needs to minimize in
order to ensure adaptive exchanges with its environment. One major appeal of the FEP is that it aims for
(and seems to deliver) an unprecedented integration of the life sciences (including psychology,
neuroscience, and theoretical biology). The difference between the FEP and earlier inferential theories
(e.g., Gregory 1980, Grossberg 1980, Rao and Ballard 1999, Lee and Mumford 2003) is that not only
perceptual processes, but also other cognitive functions such as learning, attention, and action planning
can be subsumed under one single principle: the minimization of free energy through the process of active
inference (Friston 2010; Friston et al. 2017). ”
It might be beneficial while not being optimal on its own.
The obvious example is if it has different behaviour around local minima, it could be an altenate pathway out.
I have often wondered if doing training with radically different aproaches for the first few iterarions would avoid any method specific artifacts before the weights had time to denoise.
I really liked this paper that showed using a predictive coding learning rule leads to the exact same gradients as backprop in arbitrary networks:
Predictive Coding Approximates Backprop Along Arbitrary Computation Graphs https://direct.mit.edu/neco/article/34/6/1329/110646/Predict...
“ Probably the most ambitious and all-encompassing version of the ‘Bayesian turn’ in cognitive science is the free energy principle (FEP). The FEP is a mathematical framework, developed by Karl Friston and colleagues (Friston, Kilner, and Harrison 2006; Friston et al. 2010; Friston 2010; Friston et al. 2017a; Friston 2019), which specifies an objective function that any self-organizing system needs to minimize in order to ensure adaptive exchanges with its environment. One major appeal of the FEP is that it aims for (and seems to deliver) an unprecedented integration of the life sciences (including psychology, neuroscience, and theoretical biology). The difference between the FEP and earlier inferential theories (e.g., Gregory 1980, Grossberg 1980, Rao and Ballard 1999, Lee and Mumford 2003) is that not only perceptual processes, but also other cognitive functions such as learning, attention, and action planning can be subsumed under one single principle: the minimization of free energy through the process of active inference (Friston 2010; Friston et al. 2017). ”
How does it do on CIFAR-10, or even better, ImageNet?
Interesting research, not sure it's a backprop alternative.
===
EDIT: accuracy on MNIST is not ~90%. It's ~85%.
The obvious example is if it has different behaviour around local minima, it could be an altenate pathway out.
I have often wondered if doing training with radically different aproaches for the first few iterarions would avoid any method specific artifacts before the weights had time to denoise.
Their goal is to understand how distributed systems which cannot do backprop (the brain) can still do learning.
I didn't see ImageNet. TinyImageNet is something else.
[1]: https://arxiv.org/abs/2605.31022