when trees fall...

While playing around with the MNIST dataset and the example code, I tried to visualise the weights of the connections from the weights to the hidden layer. These can be thought of as feature extractors of the input.

$$
\newcommand{\precisionmat}{\mathbf{B}}
\newcommand{\pixel}{\mathbf{p}}
$$
We want to learn feature extractors for the input in terms of Gaussian filters with varying width height and rotations.

  <p>
    Let the Gaussian function be,
  </p>
  
  <p>
    $$ g(\pixel;\precisionmat,\boldsymbol{\mu}) = \exp\left(-{\left\| \precisionmat (\pixel &#8211; \boldsymbol{\mu}) \right\|}^2\right) $$
  </p>
  
  <p>
    Then we define a weight matrix $\mathbf{W}$ between layers such that,
  </p>
  
  <p>
    $$ \underbrace{\mathbf{W}}_{(n,h)} = \underbrace{\mathbf{G}}_{(n,k)} \underbrace{\mathbf{M}}_{(k,h)}$$
  </p>
  
  <p>
    where $\mathbf{M}$ is a standard transformation matrix freely tuned by gradient descent, and $\mathbf{G}$ is a matrix with each column representing a Gaussian filter:
  </p>
  
  <p>
    $$(\mathbf{G})_{ij} = g(\left[\text{row}(i),\text{col}(i)\right];\precisionmat_j,\boldsymbol{\mu}_j)$$
  </p>
  
  <p>
    where $\text{row}(i)$ and $\text{col}(i)$ give the row and column of input $i$. As a result, the free variables for tuning the matrix $\mathbf{G}$ are then the matrices $\precisionmat$
  </p>
  
  <p>
    In this way, $\mathbf{W}$ represents a layer in which features are captured in the form of linear combinations of Gaussian shaped feature extractors from the input, which is assumed to have some form of two-dimensional topology.
  </p>
  
  <p>
    So if we were to visualise a column from $\mathbf{G}$ in it&#8217;s 2D representation:
  </p>
</div>