Editing Legendre polynomials (section)

=== In recurrent neural networks ===

A [[recurrent neural network]] that contains a {{math|''d''}}-dimensional memory vector, <math>\mathbf{m} \in \R^d</math>, can be optimized such that its neural activities obey the [[linear time-invariant system]] given by the following [[state-space representation]]:
<math display="block">\theta \dot{\mathbf{m}}(t) = A\mathbf{m}(t) + Bu(t),</math>
<math display="block">\begin{align}
A &= \left[ a \right]_{ij} \in \R^{d \times d} \text{,} \quad
&& a_{ij} = \left(2i + 1\right)
\begin{cases}
  -1 & i < j \\
  (-1)^{i-j+1} & i \ge j
\end{cases},\\
B &= \left[ b \right]_i \in \R^{d \times 1} \text{,} \quad
&& b_i = (2i + 1) (-1)^i .
\end{align}</math>

In this case, the sliding window of <math>u</math> across the past <math>\theta</math> units of time is [[Approximation theory|best approximated]] by a linear combination of the first <math>d</math> shifted Legendre polynomials, weighted together by the elements of <math>\mathbf{m}</math> at time <math>t</math>:
<math display="block">u(t - \theta') \approx \sum_{\ell=0}^{d-1} \widetilde{P}_\ell \left(\frac{\theta'}{\theta} \right) \, m_{\ell}(t) , \quad 0 \le \theta' \le \theta .</math>

When combined with [[deep learning]] methods, these networks can be trained to outperform [[long short-term memory]] units and related architectures, while using fewer computational resources.<ref>{{cite conference |last1=Voelker |first1=Aaron R. |last2=Kajić |first2=Ivana |last3=Eliasmith |first3=Chris |title=Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks |url=http://compneuro.uwaterloo.ca/files/publications/voelker.2019.lmu.pdf |conference=Advances in Neural Information Processing Systems |conference-url=https://neurips.cc |year=2019 }}</ref>