when trees fall...

Controller Outputs

  <p>
    $$<br /> \newcommand{\memory}{\mathbf{M}}<br /> \newcommand{\read}{\mathbf{r}}<br /> \newcommand{\erase}{\mathbf{e}}<br /> \newcommand{\add}{\mathbf{a}}<br /> \newcommand{\weight}{\mathbf{w}}<br /> \newcommand{\key}{\mathbf{k}}<br /> \newcommand{\shift}{\mathbf{s}}<br /> \newcommand{\Shift}{\mathbf{S}}<br /> $$<br /> A huge part of my problems stem from not knowing what the architecture of the controller looks like. The paper describes it&#8217;s inputs as coming from the input sequence, and the read head of the memory. It&#8217;s outputs are the write heads, and the output sequence.
  </p>
  
  <p>
    It then goes on to describe the reading and writing mechanism, sometimes referring to &#8220;multiple heads&#8221;. This got me confused as to whether there were separate read, erase, and write heads and corresponding weights to go along with each of these. Another possible way to implement this was to have, for each head, an erase, and an add head, with one set of weight parameters (parameters from which we calculate the current weight for this timestep from the previous). This is the version I finally settled with.
  </p>
  
  <p>
    The other problem was to decide where the head[s] would go. Looking at previous LSTM papers, the write/forget gates are directly dependent on the input data as well, so this led me to think maybe the heads was a single layer network that directly gave a read, writes and erase on seeing input. I eventually settled for the heads being connected after the hidden layer.
  </p>
  
  <p>
    I took a look at <a href="http://arxiv.org/pdf/1308.0850v5.pdf">this paper</a> to figure out how Graves usually models outputs of different domains. This is what I&#8217;ve come up with:
  </p>
  
  <p>
    $$<br /> \hat{y} = \left(\hat{\key},\hat{\beta},\hat{g},\hat{\shift},\hat{\gamma}\right) = \mathbf{h}^\top\mathbf{W} + \mathbf{b}<br /> $$
  </p>
  
  <p>
    $$<br /> \begin{align}<br /> \key &= \hat{\key} &\\<br /> \beta &= \exp\left(\hat{\beta}\right) &\Longrightarrow &\beta > 0 \\<br /> g &= \sigma\left(\hat{g}\right) &\Longrightarrow & g \in (0,1) \\<br /> (\shift)_i &= \frac{\exp((\hat{\shift})_i)}{\sum_j \exp((\hat{\shift})_j)} &\Longrightarrow<br /> & (\shift)_i \in (0,1),\sum_i (\shift)_i = 1 \\<br /> \gamma &= \log\left(e^{\hat{\gamma}} + 1 \right) + 1 &\Longrightarrow & \gamma \geq 1 \\<br /> \end{align}<br /> $$
  </p>
  
  <p>
    I have no particular justification to choose the softplus over the $\exp$. In my opinion, exponentiating seems like it would be much more likely to be unstable numerically as opposed to the softplus.
  </p>
</div>