FizzBuzz in Theano, or, Backpropaganda Through Time.

Interviewer: Hi! Can I get you coffee or anything? Do you need a break?

  <p>
    <strong>Me</strong>: No, but thanks!
  </p>
  
  <p>
    <strong>Interviewer</strong>: Let&#8217;s get started then! Are you familiar with &#8220;fizzbuzz&#8221;?
  </p>
  
  <p>
    <strong>Me</strong>: &#8230; I know about 15 <em>fizzbuzz</em> different ways to do it. See what I did there?
  </p>
  
  <p>
    <strong>Interviewer</strong>: Right, so I need you to print the numbers from 1 to 100, except that if the number is divisible-
  </p>
  
  <p>
    <strong>Me</strong>: I said I know the problem. I&#8217;ll code in python.
  </p>
  
  <p>
    <strong>Interviewer</strong>: That&#8217;s fine.
  </p>
  
  <p>
    <strong>Me</strong>: Some standard imports&#8230;
  </p>
</div>

In [ ]:

<div class="inner_cell">
  <div class="input_area">
    <div class=" highlight hl-ipython2">
      <pre><span></span><span class="kn">import</span> <span class="nn">theano_toolkit</span>

import theano import theano.tensor as T import numpy as np

from theano_toolkit.parameters import Parameters from theano_toolkit import updates

Interviewer: Geez another neural nut…

  <p>
    <strong>Me</strong>: What?
  </p>
  
  <p>
    <strong>Interviewer</strong>: I said that&#8217;s a good start. Go on.
  </p>
  
  <p>
    <strong>Me</strong>: We&#8217;re gonna have to generate labels to train on, so let&#8217;s implement fizzbuzz&#8230;
  </p>
</div>

In [2]:

<div class="inner_cell">
  <div class="input_area">
    <div class=" highlight hl-ipython2">
      <pre><span></span><span class="k">def</span> <span class="nf">fizzbuzz</span><span class="p">(</span><span class="n">N</span><span class="p">):</span>
<span class="c1"># 0 for copy,</span>
<span class="c1"># 1 for fizz,</span>
<span class="c1"># 2 for buzz,</span>
<span class="c1"># 3 for fizzbuzz</span>
<span class="n">output</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="n">N</span><span class="p">,),</span><span class="n">dtype</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">int8</span><span class="p">)</span>
<span class="n">idx</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="n">N</span><span class="p">)</span>
<span class="n">output</span><span class="p">[(</span><span class="n">idx</span><span class="o">%</span><span class="k">3</span>)==0] = 1
<span class="n">output</span><span class="p">[(</span><span class="n">idx</span><span class="o">%</span><span class="k">5</span>)==0] = 2
<span class="n">output</span><span class="p">[(</span><span class="n">idx</span><span class="o">%</span><span class="k">15</span>)==0] = 3
<span class="k">return</span> <span class="n">output</span>

Interviewer: Okay, that’s enough, that function already demonstrates you know the problem, and frankly, sitting through this once is enough.

  <p>
    <strong>Me</strong>: Aha! But I&#8217;m betting the other candidate didn&#8217;t use RNNs did he?
  </p>
  
  <p>
    <strong>Interviewer</strong>: Wha-
  </p>
  
  <p>
    <strong>Me</strong>: Recurrent neural networks! It can solve all problems known to man! Let&#8217;s initialise some model parameters.
  </p>
</div>

In [3]:

<div class="inner_cell">
  <div class="input_area">
    <div class=" highlight hl-ipython2">
      <pre><span></span><span class="n">hidden_size</span> <span class="o">=</span> <span class="mi">5</span>

P = Parameters() P.W_hidden_hidden = 0.1 np.random.randn(hidden_size,hidden_size) P.b_hidden = np.zeros(hidden_size) P.init_hidden = 0.1 np.random.randn(hidden_size) P.W_output = np.zeros((hidden_size,4)) P.b_output = np.zeros(4) parameters = P.values()

Interviewer: Only 5 hidden units?

  <p>
    <strong>Me</strong>: Oh, that&#8217;s a good question I-
  </p>
  
  <p>
    <strong>Interviewer</strong>: You know what? Pretend I never asked.
  </p>
  
  <p>
    <strong>Me</strong>: It&#8217;s okay I&#8217;ll explain later, but <a href="http://research.microsoft.com/apps/video/?id=239083">success is guaranteed</a>. Let&#8217;s define the model&#8230;
  </p>
</div>

In [4]:

<div class="inner_cell">
  <div class="input_area">
    <div class=" highlight hl-ipython2">
      <pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">iscalar</span><span class="p">(</span><span class="s2">"x"</span><span class="p">)</span>

lbls = T.ivector("lbls") def step(prev_hidden): return T.nnet.sigmoid(T.dot(prev_hidden,P.W_hidden_hidden) + P.b_hidden)

init = T.nnet.sigmoid(P.init_hidden) hiddens,_ = theano.scan( fn=step, outputs_info=[init], n_steps=x ) outputs = T.nnet.softmax(T.dot(hiddens,P.W_output) + P.b_output)

Me: We need to do that concatenation of the first hidden representation or we’d have an off by one error.

  <p>
    <strong>Interviewer</strong>: <em>Yawns</em>
  </p>
  
  <p>
    <strong>Me</strong>: Okay let&#8217;s train this using the standard cross-entropy cost, with an L2 penalty&#8230;
  </p>
</div>

In [5]:

<div class="inner_cell">
  <div class="input_area">
    <div class=" highlight hl-ipython2">
      <pre><span></span><span class="n">loss</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">T</span><span class="o">.</span><span class="n">nnet</span><span class="o">.</span><span class="n">categorical_crossentropy</span><span class="p">(</span><span class="n">outputs</span><span class="p">,</span><span class="n">lbls</span><span class="p">))</span>

cost = loss + np.float32(1e-4) * sum(T.sum(T.sqr(w)) for w in parameters) gradients = T.grad(cost,wrt=parameters)

Me: Great, now we’ll prepare three functions for training, validation, and a third one just for analysis purposes later…

  <p>
    <strong>Interviewer</strong>: <em>Zzzzz&#8230;</em>
  </p>
</div>

In [6]:

<div class="inner_cell">
  <div class="input_area">
    <div class=" highlight hl-ipython2">
      <pre><span></span><span class="n">train_fizzbuzz</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">function</span><span class="p">(</span><span class="n">inputs</span><span class="o">=</span><span class="p">[</span><span class="n">x</span><span class="p">,</span><span class="n">lbls</span><span class="p">],</span><span class="n">outputs</span><span class="o">=</span><span class="n">cost</span><span class="p">,</span>
                            <span class="n">updates</span><span class="o">=</span><span class="n">updates</span><span class="o">.</span><span class="n">adam</span><span class="p">(</span><span class="n">parameters</span><span class="p">,</span><span class="n">gradients</span><span class="p">))</span>

fizzbuzz_accuracy = theano.function(inputs=[x,lbls], outputs=[ T.mean(T.neq(T.argmax(outputs,axis=1),lbls)), loss ]) fizzbuzz_hiddens = theano.function(inputs=[x],outputs=hiddens)

Me: Time to train this thing! So, I’m thinking… 200,000 iterations, with a checkpoint every 10,000 iterations. We’ll validate it on a longer fizzbuzz sequence to make sure it’s generalising right.

  <p>
    <strong>Inteviewer</strong>: <em>Chokes while snoring</em>
  </p>
</div>

In [7]:

<div class="inner_cell">
  <div class="input_area">
    <div class=" highlight hl-ipython2">
      <pre><span></span><span class="n">min_val</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">inf</span>

model = [] for _ in xrange(20): for _ in xrange(10000): length = np.random.randint(4) * 15 + 16 train_fizzbuzz(length,fizzbuzz(length)) acc, val_cost = fizzbuzz_accuracy(301,fizzbuzz(301)) print acc, val_cost, if val_cost < min_val: model = [ w.get_value() for w in parameters ] min_val = val_cost print "Save" else: print

    <div class="output_subarea output_stream output_stdout output_text">
      <pre>0.332225913621 0.764216542244 Save

0.332225913621 0.662550151348 Save 0.325581395349 0.651876211166 Save 0.325581395349 0.648342370987 Save 0.325581395349 0.649337947369 0.318936877076 0.631394803524 Save 0.0 0.0484583750367 Save 0.0 0.0167257916182 Save 0.0 0.0113316513598 Save 0.0 0.0111983753741 Save 0.0 0.0111334351823 Save 0.0 0.0111326910555 Save 0.0 0.0111096147448 Save 0.0 0.0111302910373 0.0 0.0111406939104 0.0 0.0111133335158 0.0 0.0110623119399 Save 0.0 0.0111135747284 0.0 0.0111036580056 0.0 0.0110982181504

Me: And we’re done-

  <p>
    <strong>Interviewer</strong>: <em>Wakes up</em> Wh- Oh. Great! Finally.
  </p>
  
  <p>
    <strong>Me</strong>: -with the training. We have to analyse the model&#8217;s hidden states now..
  </p>
  
  <p>
    <strong>Interviewer</strong>: Uh. That&#8217;s fine. We&#8217;ll get back to you, thanks for coming out to interview today, bye! <em>Sneaks out the door</em>
  </p>
  
  <p>
    <strong>Me</strong>: I&#8217;ll just load the best model based on the &#8220;validation&#8221; we have and test it.
  </p>
</div>

In [8]:

<div class="inner_cell">
  <div class="input_area">
    <div class=" highlight hl-ipython2">
      <pre><span></span><span class="k">for</span> <span class="n">w</span><span class="p">,</span><span class="n">w_save</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">parameters</span><span class="p">,</span><span class="n">model</span><span class="p">):</span> <span class="n">w</span><span class="o">.</span><span class="n">set_value</span><span class="p">(</span><span class="n">w_save</span><span class="p">)</span>

test_length = 46 hidden_states = fizzbuzz_hiddens(46).T

In [11]:

<div class="inner_cell">
  <div class="input_area">
    <div class=" highlight hl-ipython2">
      <pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="kn">as</span> <span class="nn">plt</span>

import matplotlib %matplotlib inline plt.figure(figsize=(15,5)) plt.xticks(np.arange(test_length)) plt.imshow(hidden_states,interpolation='None')

Out[11]:

    <div class="output_text output_subarea output_execute_result">
      <pre>&lt;matplotlib.image.AxesImage at 0x7fa9ca2b7910&gt;</pre>
    </div>
  </div>
  
  <div class="output_area">
    <div class="prompt">
    </div>
    
    <div class="output_png output_subarea ">
      <img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA1wAAAB1CAYAAABaruCnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz

AAALEgAACxIB0t1+/AAAH+FJREFUeJzt3Xl0VGW67/FnJ6RBQUQkkQwFCQQCCQSDaYjpEAYRsBEl GgeQA6hHtNW+6jJgt8IhaUWPiLfVbgTxANpLoA+tDIoDNsokECQyyBwiUyYGhcgspLLvH3Sdmxtr 1977rbdyIX4/a+21CPXWL+/eefLUflNVuwzTNAUAAAAAoF/Y/+8JAAAAAEBDxYILAAAAAEKEBRcA AAAAhAgLLgAAAAAIERZcAAAAABAiLLgAAAAAIEQa6QgxDINrywMAAAD4RTNN06j7f1oWXCIiZkzg 2/NPiuRfZZMx7Gfz+5mCtaZMzAw8LuzVVbY5IrNE5IGAIwzZZ5tiykIxJCfgmKrGI21zXqoW+aPN T6P0J/vjIyLyppjyqAQe2+XJGvugwnyRjPzAY+5xMKH/yhf5d5ucTAc5Zr6IETinrXenbUxVwVRp MfGxgGNGh6c4mJDICtOUPkbgY73TO9s2Z3vBIkmZODTgmPnhTewnZL4vYuQGHGJsuts+ZnqBGI9M DDhmVWq6bc6sgkp5YGJ0wDFZfTfa5oiI5O8XyY8PPMZMcNBDNpsy8XqbHvLOZgczmiYivws4wpBv bFNMWSyG3B5wTFXjwL1KxFkPOeSwh/xFTPm9TQ9JetxBD/k6X6RHfuAxoxxMaEa+yBibnB4Ochz0 kDhvsW3MiYK/SPOJvw845sHwTg4m5KyHFHv/yzZna8Fi6ToxcB3NC29qPyEnPWSDgx4yo0CMMYF7 yPLuGbY57xSUy+iJsQHH9O77tW2Ok/4h0nB7yKmm9j1k0nmR534VeEzZafvj46R/iNBDdPQQJ/1D hB5Sbz1khSmGxc+DlxQCAAAAQIiw4AIAAACAEKm3BVcfm6epnert0ZMjkqYpx9nLRuxkafxJ/FpX UFwfPTndNeWInpwmvbUdIYnXlBPZW08diSTriUnvrSUmrXczLTkiIn1a6Mnp3VpPjoj9SyqdSdKS orOHOHlljSOxffTk3KApR1MPadxb2xHS1kOieuupI2095AY9PeT63jbvRXBIV/8Qabg9pFe4lhh9 /UOEHmIjXkvKRfSQwILtIbYP0cuXL++blJS0OyEhYd/48eNfUP1GfRqr3rNOjsfZexDs6VlwGdJZ S04vrQsuTcfoUltwGXpymvTReLLk4LXTTkT10bTgMvQ0OiO9j5actD56Gp2IvhOmPq119RA9C3dD 0x9tdPaQnrp6yKV2sqSphzTu01NLjoi+HnLdpdZDNP3Mru/TXEuOzgVXQ+0h2ZoWXNr6hwg9xIau /iFCD7ET0gWXaZrGQw899PYHH3xwZ0lJSeKyZcv6r1u37sbgviUAAAAA/DIEXHBt3rz5+pYtWx7r 0qXLtvDwcO+IESPeW7BgwR31NTkAAAAAuJwFXHCVl5fHxsbGlvu+9ng8peXl5YGvrQgAAAAAEBGb z+Gq+4HGXq/X8hW++Sf/77/7/Erfe7YAAAAA4FKzouriJiIi+fmW4wIuuGJjY8vLysrifF+XlZXF xcXFlfkba/ehxgAAAADQUPRpUeuCGvn5UlBQ4HdcwJcUpqamfnvs2LGW3377beqFCxci5syZc9/Q oUMX6Z4sAAAAADREAZ/hCgsLq3n77bcfys3Nff/cuXNNRowY8V5mZuba+pocAAAAAFzOAi64RET6 9ev3ZXFxccf6mAwAAAAANCQaPyoTAAAAAFAbCy4AAAAACBEWXAAAAAAQIiy4AAAAACBEDNM07UfZ hRiGacgsDdM5oCFDRJpM1BLjvcfyc55d+eE9LTES5f1CT5CImMYqLTnG0/+hJccboedYm7MMLTnh h9dryREREeNjPTE3aDrWQ/Qc6wt/1hIjTapW6gkSERFdvyOaesgYTT1kppYYifLqO9am8aWWHOMJ TXV9laYeMkNXD9moJUdERAw9n8ZidNN0rHMutR6i8+LJn2nKoYfYoYcERg+xd6n1kBozQwzDENM0 f1YEPMMFAAAAACHCggsAAAAAQoQFFwAAAACECAsuAAAAAAgR2wXXiBEj3ouKijrStWvXrfUxIQAA AABoKGwXXGPGjJnxySef/LY+JgMAAAAADYntgis7O3tVixYtqupjMgAAAADQkPAeLgAAAAAIkUa6 gkxZXOurJDGkk65oAAAAALikmLJRRC5+SHV+vvUHp2tbcBlyu64oAAAAALikGdJdRLqLiEh+foYU FBT4HcdLCgEAAAAgRGwXXHfccceC7OzsVbt3707yeDyls2fPvr8+JgYAAAAAlzvblxQuWLDgjvqY CAAAAAA0NLykEAAAAABChAUXAAAAAIQICy4AAAAACBEWXAAAAAAQIto+h2ujPBh0Rpq8pWEmIkYr LTHy0bt6cobKVD1BzfroyRGRt07dpCXn4SmJWnIaSbWWnC+0lXSFphwRo/1/aMlZU6Rn38K/maUl xzBHasmZr68NyV3yDy05hpzRkvPuDC0x8oCuHtIkS0+OiLx1rreWnIdfa6g95ICmHI09ZLOmHrLl b1pyDHO4lhx6iD16iD16iD16iJ0ay1t4hgsAAAAAQoQFFwAAAACECAsuAAAAAAgRFlwAAAAAECIs uAAAAAAgRGwXXKWlpZ7+/fsv83g8pYmJiSVTp059rD4mBgAAAACXO9vrIBqGYebn5+dnZWV9dfTo 0cju3btv7Nev35edO3feWR8TBAAAAIDLle0zXHFxcWVZWVlfiYhERkYeTUpK2l1ZWRkd+qkBAAAA wOXN1Sd9FRcXdywuLu7Ys2fP9XVvmy7m//w7XUTSxQh+dgAAAABwCdoupmz/17+35edbjnO84Kqq qmpxzz33/Pfbb7/9UNOmTU/Xvf0RFlgAAAAAfiFSxJCUf/07Nz9fCgoK/I5zdJXCc+fONRk6dOii J5544vWBAwcu1TVJAAAAAGjIbBdcXq83/O67754/aNCgz0aPHv1OPcwJAAAAABoE2wXXypUrey9Z suTWqVOnPubxeEo9Hk/p4sWLb6+PyQEAAADA5cz2PVz9+vX7sqamhg9IBgAAAACXWEgBAAAAQIiw 4AIAAACAEGHBBQAAAAAhYpimaT/KLsQwzDBZGXTOGukTdIaIyFotKSI3afposTeq39CSM7tRXy05 IiLe97toySm4U89BytX0OW6pt1dryZHFZ/TkiIj33qu05Cz4e/C/qyIiZYaev7M8+Y1XS05Y979r yRER2S/DteT8Q0uKSLamHvJe9SQtOW80ulNLjoiI96MkLTkv3qrnIN2uq4fcqamHfEAPsfPkFk09 JJUeYoceYo8eYq+h9hBDUw+pMe8VwzDENM2fFRPPcAEAAABAiLDgAgAAAIAQYcEFAAAAACHCggsA AAAAQoQFFwAAAACEiO2CyzRNo0ePHl8nJCTsi4+P3z927NhX6mNiAAAAAHC5s11wGYZhLlmy5NZ9 +/Yl7N69O2nt2rWZS5cuHVgfkwMAAACAy5mjlxRGRUUdERHxer3hNTU1YYZh6LkQPwAAAAA0YI7f w5WSkrK9VatW36empn47YMCAz0M5KQAAAABoCBwvuLZv355SWVkZXVJSkrh+/fqedW83ZXatbZPe WQIAAADAJcSUHWLK+2LK+5Kfn285ztVVCq+++uofBw4cuHTJkiW31r3NkPtrbWnuZwwAAAAAlwlD ksWQXDEkN7gF19GjRyMPHDjQVkSkqqqqxeLFi29PTk7eoW+qAAAAANAwNbIbUFVV1SI3N/f9H374 4dpGjRpVjxgx4r1hw4bNq4/JAQAAAMDlzHbB1aFDhz1btmzpVh+TAQAAAICGxNV7uAAAAAAAzrHg AgAAAIAQYcEFAAAAACHCggsAAAAAQsQwTTP4EMMwRVYHnROWkRl0hohI9UHba4E4El6h6QOcjXgt MTvNllpyREQ6R1TrCbqwRUuM9+buWnLC/lmjJceQUi05IiLeDm215ITv8WrJMWS7lpwLaXqupRNe qWe/RETCDunJOhweoSUnyrtLS44YzbXEfGvGaskREUnV1EOMC0Vacqpv7qklhx5i75LrIUc19pAy ekgg9BB79BB7DbWHmKUihmGIaZpG3dt4hgsAAAAAQoQFFwAAAACECAsuAAAAAAgRFlwAAAAAECKO Flw1NTVhGRkZhb169Qr+yhgAAAAA8AvhaMH11ltvPdyuXbu9F69GCAAAAABwwnbBdeTIkaj58+ff /fjjj//V32UOAQAAAAD+2S648vLypkyaNOm58PBwfR90AQAAAAC/AAE/IXj58uV9w8LCajIzM9cW FhZmBI6aVevfaf/aAAAAAKAB+mnFxU1E8vOthwVccK1bt+7GZcuW9U9ISNj3008/NT5+/Pg1Q4cO XbRo0aKhPx/9gPpkAQAAAOBy0rjPxU0uLrgKCgr8Dgv4ksJnn332xbKysrh9+/YlLFy4MCc9Pb3I /2ILAAAAAFCX48/hMk3T4CqFAAAAAOCc4wVXRkZG4apVq7JDORkAAAAAaEgcL7iCt0lLinlihZac FT/pebLOlCI9OeZXWnJERL4WTftWs0JPjqZjtOKYridYV2jKETFlnZacFWd01eMKTTkbtOSsOKnx SfHzK7TEmLJSS84aU9fPbL2eHHOtlhwRkQ2XXA/5RksOPcReg+0h/3pTuw70kMB09Q8ReogdXf1D hB5iK8gectktuOSEnka38ryWGBFNiwkRnQsuTUw9x1rXMVp5XEuM6DxZEinUkrLyrJYYEU0nAqKp 0a08pSXmIk0LLl3HaI22cwpdv7E6F1yaaOshG7Wk0EMcJWnKucR6iMYFFz0kMG39Q4QeYktP/xCh h9i6fBZcAAAAAPDLEvCy8G5079404O2Vlb+S6OjAY8KS7L9PxVmRGLtxV3e3DyqpEEmMCTike+sr 7OdT2Uhiom3GGfbr2ooKQ2JiAo9rbDrYLxEJr6yQxtE2++bgJ19RLhITazOo2skxirA/Rh0c7FtV hUgHm/36wT6mslIkOjrwGEMi7INEpKIyTGKibca2cbBvuypEOtns21VO5iMSY7tvTRzkNJKYaJtx nRzsV3WFSGeb/Yq0jxERqfxOJLp94DFhgb+ViDg7RuHh9vsWVl4h4bE2++Zt7GA+jSQm2macYf8L W1ERJjExgcc1cdhDGlVWSJN66iFG9ZX2OZUREhNtM44e8svtIa3tYyr3iER3sB8XFmU/hh4SmJP+ IUIP0dFDHPUPEXpIPfYQK4ap4TXEXL0QAAAAwC+daZpG3f/TsuACAAAAAPwc7+ECAAAAgBBhwQUA AAAAIRLyBdfy5cv7JiUl7U5ISNg3fvz4F1RzRowY8V5UVNSRrl27bg1mPqWlpZ7+/fsv83g8pYmJ iSVTp059TCXHNE2jR48eXyckJOyLj4/fP3bs2FeCmVdNTU1YRkZGYa9evVarZkRGRh71eDylHo+n tHPnzjtVc44ePRp52223fRgdHV3Zvn377zZv3ny9Ss7u3buTfPPxeDylV1555ZkpU6bkqWRNnz79 keTk5B3Jyck7cnJyFp46daqZSs7rr7/+ROfOnXcmJSXtfuWVV8Y6vZ+/+jtx4kTz3/72t5+0a9du b3Z29qrDhw9fp5Izd+7c4cnJyTvCw8O9GzdudHZVA4usP/zhD/8ZHx+/Pz4+fn9ubu77J06caK6S 8+KLLz7brl27vQkJCfv69u27fN++fQkqOT5Tp059LCwsrGbv3r3tVHLy8vKmXHvttT/4aumzzz4b pDqfyZMnj/PlFBQUTLTLscoaPHjwx76cyMjIoykpKdtVcnbu3Nk5MzNzbdeuXbempaVtWrlyZW+V nC1btnTLyMgoTE5O3nHLLbd8WlVV1cIux6ofuq1tqxy3tW2Vo1LXVllua9vuMcNpbVvluK3tQPNx U9tWOW7r2ipHpa6tstzWttXjs9u6tspxW9dWOW7r2ipHpV/bncM4rWurHLd1HWg+buraKsdtXVvl qNS1VZZKzxb5+fmiyrmIvxzVc5G6OSr92l+OSl37y/Fxcx5ilaVyLvI/TNMM2VZTU2O0b9++ZOvW rV2qq6vDe/bsWbh27dobVbJWrlyZvWHDhvQuXbpsDWZOpaWlcatXr84yTVOOHDkSGRcXV7pjx47O KlmHDx+OMk1Tzp071zgzM3PNZ599NlB1Xm+++ebvhg0bNrdXr16rVDNat25dqePndscdd3zw0ksv /cE0TTl16lTTY8eOXaMjt23btvtLSkrau73fsWPHromMjDzy448/NjdNU0aOHPnua6+99oTbnK1b t3Zp167ddydPnmx2/vz5iBtvvHHt9u3bk1Xrb8KECX8aN27cy6ZpyhtvvPH7MWPGvKWSs3379uTi 4uIOWVlZq7/55pvuTvfHX9aCBQtyzpw5c4VpmjJmzJi3nnnmmf9UyfHVtm/f7r333nkqOaZpSmVl Zev+/fv/Mykpadd3333XTiUnLy/vlTlz5gx38/P2l/Phhx8O6dGjx3pfLR04cKCNalbtbcKECX8a P3788yo5OTk5C/72t7/9m2ma8sUXX/S7/vrrN6nkpKenb/joo49u9f3Mxo4dO9kux6ofuq1tqxy3 tW2Vo1LXVlluazvQY4ab2rbKcVvbVjlua9vJY6GTurbKUalrqyyV2vb3+KzSs/3lqPRsfzkqde0v R6VfW2W5rWurHJWe7S9HpWfbnZs57df+clTq2ipLpa5N8+fniyp17S9H9Vykbo5KXfvLUa1rf+fT bmvaKkulrn1bSJ/h2rx58/UtW7Y81qVLl23h4eHeESNGvLdgwYI7VLKys7NXtWjRoirYOcXFxZVl ZWV9JXLxGaGkpKTdlZWVNhew9C8qKuqIiIjX6w2vqakJU71a45EjR6Lmz59/9+OPP/5X08+VTerT oUOHWq9Zs+Y3vr/ANG3a9PQ111wT9Ef9rV69uldUVNSR9u3bf+f2vqZpGqZpGmfOnLnS6/WGnz17 9orY2Nhytzk7d+7s/Otf/3pDs2bNTkVERFzIzs5e9cEHH9zp5L7+6u/DDz+8bfTo0e+IiIwaNerd hQsX5qjkJCcn7+jQocMeF7timZWTk7PwiiuuOCsikpWV9VV5ebndhf395vhqW0Tk/Pnzv3JS21a/ o2PHjn1l0qRJzzn9/bDKcfu74S9n2rRpvxs/fvwLzZs3PyEi0qZNm4PBzMln3rx5w4YPHz5XJScs LKzG94ztyZMnr3JS2/5ydu7c2fnmm2/+p4hI//79lzmpbX/9sKKiIsZtbVv1Vbe1bZWjUtdWWW5r O9BjhpvatjrWIu5q2ypn+vTpj7ipbSePhU7q2mo+KnVtlaVS23Ufn0XUera/x3mVnu0vR6Wu/eWo 9GurLBH3PdvfsRZx37P95bit60D75eO0X/ubj0pdW2Wp1LW/80WVuvaXo1LX/nJU6tpfjkpdW51P u63pQFmq5+khXXCVl5fH1i5Gj8dT6uTA15fi4uKOxcXFHXv27LleNSMlJWV7q1atvk9NTf12wIAB n6tk5OXlTZk0adJz4eHhXtV5iFz8Re7YsWNxly5dts2YMWOMSkZJSUlimzZtDo4aNerdlJSU7Q8+ +ODMM2fO2H/IhY25c+cOv+++++ao3Ldly5bHXnrppT8mJiaWxMbGlldXVzfKzc19321O165dt65b t+7Go0ePRp46darZ559/PqCsrCxOZU4i/299N2/e/MSFCxciLly44OzDd0LMNE3j3XffHTVkyJCP VDMmTJjwfOvWrQ+9+eabj06ePHmcSsayZcv6X3nllWd69Ojxteo8fJ555pmX27Vrt3f06NHvOH3p RV179uzpUFhYmJGWlrapV69eq4uKitKDndf69et7Nm3a9LTqy3gnT5487tVXX326TZs2Bx955JHp r7322pMqOd26ddvi+4PWwoULc9z22uLi4o579uzp0LNnz/XB1LaOvmqVo1rXdbNUa7t2TjC17cvJ yMgoFFGv7drzKS4u7qha2/6OtUpd196vYOu69pxUa7v24/PAgQOXqta1jsf5QDlu69pfjmpN181S reu6x1pEra7r5qjWtdWxdlvXdecTTF3XzVKpa3/niyp1reu8M1COm7q2ynFb1/5yVGvaak6q/Tqk C666K0mv1xseyu/nRlVVVYt77rnnv99+++2HmjZtelo1Z/v27SmVlZXRJSUlievXr+/p9v7Lly/v GxYWVpOZmbk22Ge3ioqK0ouLizt+/PHHg1999dWn16xZ8xu3GdXV1Y02btzY/dFHH31z27ZtXcLD w72qJ9u1MxcuXJhz7733/l3l/qdPn246e/bs+3ft2tWpvLw8NiIi4sK0adN+5zanU6dOuyZOnFgw aNCgzwYPHvxxenp6UVhYWI3KnPzxPROnKy8Y48ePfyEmJqbi7rvvnq+a8fzzz084dOhQ60cfffRN lffenT9//lcTJkx4/sUXX3zW93+qx+epp57684EDB9ru2rWrU7NmzU6pvmeyurq60fHjx6/ZtGlT 2qRJk54L5vj4BPPHBBGRmTNnPpiXlzfl4MGDbWbMmDFGNWvWrFkPvPPOO6PT09OLfvrpp8aNGjWq dnpfXz+cMWPGmGbNmp2qe7vT2tbVV61yVOraX5ZKbdfOiYiIuKBa23Xno1rbtXOaNWt2SrW2rY61 27qumxNMXdfdN9Xarv34XFhYmFH3dqd1HezjvF2O27r2l6Par2tnrVq1Klu1ruvOSbWu6+ao1rXV sXZb13Vzgqnrullu69rp+aJdXes677TLcVrXgXLc1LW/HNXzEKs5BXUuovI6RKfbpk2brk9PT9/g +/r111//X05fo+pv27NnT2Kw7+EyTVPOnj3bpHfv3itmz549Wte+vvzyy+OcvCa47jZp0qRnY2Nj y+Lj4/dFR0dXNGnS5Oztt9++KNj55OXlvaLyPqc9e/YkxsbGlvm+/vTTTwcFO58lS5YM7t+//z9V 7//555/fPHjw4CW+r+fOnTts+PDhc4I9Ro8//vhf/vznPz/p5tjUrr9u3bpt3rZtW4ppmlJVVXV1 ZGTkEZUc3+b2ddNWWX/9618fGzBgwNILFy40CibHtx06dOi6uLi4Urc5e/fuTYiMjDwSHx+/Lz4+ fl9ERMR5j8dzcM+ePYnBzGfr1q1dnL5uvm5OVlbW6mXLlt3k+/q66647VFVVdbXqMaqurg6PiYkp Ly0tjVM91rGxsWXff//9tb6vGzdufM7Jzy7QMSoqKrohNTV1i5P5+OuHKrUdqK+6qW2rHJW6tuv1 Tmu7bo5qbdvNx2lt+8tRqW2r+bita385qnVtd4zc1LZv8z0+q/bsujkqdW2Vo1LXVvNxU9P+su67 7773VHt2oDm56dl1c4Lp2XXno9Kv6+ao1rXdMXJS1/7OF2+77bbFbuva7rzTaV0HynFT107Og53U tb+cIUOGfKhS007m5LauXRWc283r9Ya1a9fuuy1btqSeP38+okePHuvXrFmTqZqnY8FVXV0dPmTI kA99F4VQ3Y4cORK5f//+tqZpyvHjx1tkZmaumTt37rBgMgsLC3tmZWWtVrnv8ePHW/jeYHj48OGo lJSUbV9++WVflaxu3bpt/vbbb7uapilPPfXU/1ZZSNbehg8fPmfWrFn3q95/586dnaKjoyuOHj3a qqamxhg1atQ7L7zwwnMqWXv37k0wTVM2btyYFhMTU167cbqtvwkTJvwpLy/vFdM05bXXXnvioYce mqGS49uysrJWFxUV3eBmf+pmzZs3794ePXqsP3XqVNNgcjZs2JBeU1NjmKYpkydPHjtgwIClweyb aZrSqVOnnU7frFo3Z9euXUmmebGnjBs37uWRI0e+q5IzZcqUp31vLt64cWNa27Zt96seI9O8+MeA 7OzslcEc6xtvvHGt703YX375Zd/ExMQ9KjkHDx70eL3esNOnT185aNCgT6dNm/aIXYZVP3Rb23Z9 1WltW+Wo1LVVltvadvKY4aS2rXLc1rZVjtvaDrRfburaKkelrq2y3Na21eOz27q2e5x3WtdWOW7r 2iqnqKjoBrf92sk5jJO6tsrZvXt3Rzd1bZXjtq4D7ZebuvaXM2fOnOEqdW01J5We7dtqny+qnovU zXFb11Y5quchdXNUz0Os9stpTQfKUj0XMc0QL7hM8+JVXDp06FDs8XgO/vGPf3xRNScnJ2dBdHR0 RURExPm4uLhS1RP4L774op9hGDVxcXGlvm3RokW3u80pLi7ukJqauiU2Nrasbdu2+5977rkXgj1W 69aty1C9SuGOHTs6JyUl7YqNjS1LSEjY+/LLL49TnUdhYWHP1NTULR07dtydm5v7j5MnTzZTzTp9 +vSV11577fe+KwypblOmTHm6ffv2JR06dCi+66675qv8IpumKTfddNOy1q1bV3bq1Gnn0qVLBwRT fz/++GPzW2655ROPx3MwKytrdWVlZWu3OTNnznxg3rx598bFxZU2adLk7HXXXXdo0KBBn6rMaebM mQ+0adPmQKtWrY76atvJM4H+ckaOHPludHR0RVxcXOktt9zyie/Bwu0xqn2700bnbz533XXXfN98 cnJyFtS+epGb+Zw9e7bJXXfdNb99+/YlaWlpG7/66qvfqP78TdOU+++/f9b06dMfDqaOvvnmm+7p 6ekbOnbsuDstLW2jkz9K+TtG06dPfzg2NrYsKirqcF5e3iu+B6pAm1U/dFvb/nIWLlw41G1tW+Wo 1LVVltvadvKY4aS2rebjtrat5uO2tgPtl5u6tspRqWurY+S2tq0en93WtVXO3Llzh7mpa6sct3Vt laPSr52cwzipa6sct3VtleO2rgPtl5u6tspRqWurrGnTpj3itmf7ttrniyrnIv5y3Na1VY5Kv/aX o1LX/nLc1nSgrNzc3H+4PRfxbYZpKl1YDwAAAABgI+QffAwAAAAAv1QsuAAAAAAgRFhwAQAAAECI sOACAAAAgBBhwQUAAAAAIcKCCwAAAABC5P8A9hhpzltMq8YAAAAASUVORK5CYII= " />

Me: Let me just clean that up..

In [13]:

<div class="inner_cell">
  <div class="input_area">
    <div class=" highlight hl-ipython2">
      <pre><span></span><span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">15</span><span class="p">,</span><span class="mi">5</span><span class="p">))</span>

plt.xticks(np.arange(test_length)) plt.imshow(hidden_states>0.5,interpolation='None')

Out[13]:

    <div class="output_text output_subarea output_execute_result">
      <pre>&lt;matplotlib.image.AxesImage at 0x7fa9ca18fdd0&gt;</pre>
    </div>
  </div>
  
  <div class="output_area">
    <div class="prompt">
    </div>
    
    <div class="output_png output_subarea ">
      <img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA1wAAAB1CAYAAABaruCnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz

AAALEgAACxIB0t1+/AAAG1BJREFUeJzt3X10VOWBx/FnErIipIhIImEyMAkhgQSC0DSENEShKS+l KKFBBbOA7YFaZdf1NGCrIAFFlxe32BVBOIj2COzSmiDFFyyCgEAokXcIeZEEkhBILER5LSR59g+d bjrOnbn3mTvjZPh+zrnnEDLz47n3/vLMfZiXWKSUAgAAAABgvpDvegAAAAAAEKxYcAEAAACAj7Dg AgAAAAAfYcEFAAAAAD7CggsAAAAAfIQFFwAAAAD4SDszQiwWC58tDwAAAOCWJqW0OP+dKQsuIYSY 6+H7nwgh7jPh39GTM8/jaPQm6eE5Z66YZ0KKfnqyAu0YBVqOnnOmLyl4j3Ug9tqsnEA7Z4F4rAPt GAVaDnOI5xx/9tqsHL1ZgXbOAvFYB9oxCrQcM89ZsB7rQOt1vpTCYvnWWksIwUsKAQAAAMBnWHAB AAAAgI/4bcFlD7CcQBuROSlmZ5mVFKw5gZgUWDnmpJibZVZOoI3InBSzs8xKCtacQEwKrBxzUvj5 8F+Kucc68EYVWDnmpJidFlg55qR4n+NxwbV9+/ZhCQkJpTExMZWzZ89+QfUfsqve0Uc5gTYic1LM zjIrKVhzAjEpsHLMSTE3y6ycQBuROSlmZ5mVFKw5gZgUWDnmpPDz4b8UFgH+zDEnxey0wMoxJ8XH Cy4ppWXatGmr3nnnnZ9VVFTEbd26NWvv3r1DvPw3AQAAAOCW4HbBdejQoXu6dOlyoV+/fsdCQ0Ob c3Nz3y4oKBjvr8EBAAAAQFvmdsFVW1trtVqttY6vbTZbdW1trdX3wwIAAACAts/t7+Fy/oXGzc3N oVq3/aTVn+3C7NeVAgAAAEDgqPpmE0KI/Px8zdu5fYbLarXW1tTURDu+rqmpiY6Ojq5xddv7Wm12 3cMEAAAAgLbHLv5//aO84EpOTj5y4cKFLkeOHEm+efNm2Nq1ax8ZN27cRvOGCQAAAADBy+1LCkNC QlpWrVo1LScn50/Xr19vn5ub+3Z6evoefw0OAAAAANoytwsuIYQYPnz4trKysnh/DAYAAAAAgonH X3wMAAAAAFDDggsAAAAAfIQFFwAAAAD4CAsuAAAAAPARi5TS8608hVgsUoi5JgwnsMwV877rIfyT eUF4jB3MOtZSWEzJmS+eMyUnENFr/+FY+w9ziP/Qa//hWPsPc4j/BGuvpcwXFotFSCm/VQKe4QIA AAAAH2HBBQAAAAA+woILAAAAAHyEBRcAAAAA+IjHBVdubu7bkZGR9f379z/qjwEBAAAAQLDwuOCa Pn36yvfff/8n/hgMAAAAAAQTjwuuzMzMnZ07d270x2AAAAAAIJjwHi4AAAAA8JF25kV90urP9m82 AAAAAAhGVd9sQuTn52veysRnuO5rtdnNiwUAAACAgGMXjvWPnxZcAAAAAIDWPC64xo8fX5CZmbmz tLQ0wWazVa9Zs+ZRfwwMAAAAANo6j+/hKigoGO+PgQAAAABAsOElhQAAAADgIyy4AAAAAMBHWHAB AAAAgI+w4AIAAAAAHzHtFx8/J+Z7nTFfPGfCSALPPDH3ux7Ct5hxvoQw75yZlWPWfgUis/bNrD5a hDQlx8xzxhziP8whbQ9ziGfMIf7DHNL2BOscMlfMMyVHiHzN7/AMFwAAAAD4CAsuAAAAAPARFlwA AAAA4CMsuAAAAADAR1hwAQAAAICPeFxwVVdX27KysrbabLbquLi4imXLlj3hj4EBAAAAQFvn8WPh LRaLzM/Pz8/IyPi0oaEhYtCgQQeGDx++rW/fviX+GCAAAAAAtFUen+GKjo6uycjI+FQIISIiIhoS EhJK6+rqonw/NAAAAABo2wz94uOysrL4srKy+MGDB+9z/t6OVr98rKcQwi4s3o8OAAAAAAJQ1Teb EELk5+dr3k73h2Y0NjZ2fuihh/531apV0zp27HjF+fv3Css/NhZbAAAAAIKZXQhx3zeb1wuu69ev tx83btzGJ5988pWRI0du8X54AAAAABD8PC64mpubQx988MENo0aN+nDq1Klv+mFMAAAAABAUPC64 duzYce/mzZt/umzZsidsNlu1zWarfvfddx/wx+AAAAAAoC3z+KEZw4cP39bS0sIvSAYAAAAAg1hI AQAAAICPsOACAAAAAB9hwQUAAAAAPmKRUnq+lacQi0VaxHNe5zwn5nudEYjmibnf9RC+Za6Y910P 4Z9Ik35323wTemg2jrV7FuH9HOTAHOI/9Np/ONbuMYd4xhziWaD12kwca/+QMl9YLBYhpfzWDvIM FwAAAAD4CAsuAAAAAPARFlwAAAAA4CMsuAAAAADAR1hwAQAAAICPeFxwSSktqampf42Jiam02+1V M2fOXOyPgQEAAABAW+dxwWWxWOTmzZt/WllZGVNaWpqwZ8+e9C1btoz0x+AAAAAAoC3T9ZLCyMjI eiGEaG5uDm1paQmxWCzm/dILAAAAAAhSut/DlZSUdLxr165fJCcnHxkxYsRHvhwUAAAAAAQD3Quu 48ePJ9XV1UVVVFTE7du3b7Dz96XY0WqrMnWQAAAAABBYqoQQnwghPhH5+fmatzL0KYV33HHHlyNH jtyyefPmnzp/zyLubbXZDQ0VAAAAANoWuxDiPiHEfd4tuBoaGiJOnz7dUwghGhsbO7/77rsPJCYm njBnkAAAAAAQvNp5ukFjY2PnnJycP/3tb3+7q127dk25ublvT5w4cb0/BgcAAAAAbZnHBVfv3r3L Dx8+PMAfgwEAAACAYGLoPVwAAAAAAP1YcAEAAACAj7DgAgAAAAAfYcEFAAAAAD7i8UMz9JLC4nXG fPGcCSMR4jkx35SceWKuKTlmMWu/hDBv3yxCmpJj5r4Fq2A9Z2b93JuZxRziWbD2MZgF6zljDvEf 5pBbW7CeMzPnEC08wwUAAAAAPsKCCwAAAAB8hAUXAAAAAPgICy4AAAAA8BFdC66WlpaQtLS0oqFD h+7y9YAAAAAAIFjoWnC9/vrrv4yNjT1lsVjM+VgRAAAAALgFeFxw1dfXR27YsOHBGTNmvCql9P6z 3wEAAADgFuFxwZWXl7dkwYIFz4aGhjb7Y0AAAAAAECzcLri2b98+LCQkpCU9PX2P52e3Pmm1VZky OAAAAAAITFXCsf7Jz8/XvJXbBdfevXuHbN26NSsmJqZy/PjxBcXFxSnjxo3b6PrW97Xa7IaHCwAA AABth1041j/KC65nnnnmxZqamujKysqYwsLC7JSUlOKNGzeOM3OYAAAAABCsdP8eLimlhU8pBAAA AAD9dC+40tLSinbu3Jnpy8EAAAAAQDDRveDyXpUpKdKkHHNSzEwyK0eIKmHWE5FVpqQE7zkzL8uc FPOSzDtnZj4pXmVKSvD20awc5hB/JjGHuBdoXRQimPtoTg7zvj+TzMoJvKRgm0Pa3IJLiNOmpFSZ kmJmklk5Zh0hIThn/ssyJ8XMJHPOmXldFCLQ9q3KlBQzk8zKYQ7xZxJziD9ShDD3nAVrH83JYd73 Z5JZOYGYFFxziB8XXAAAAABwa2lnVtCgQVFuv19X9z0RFeX+NhYdT/udrQsX3T3kRIlBHnO+V1fn cTyDhPvvC6Fvv/TQk6Nnv4QQIrzurIiK6u72Nmbt2618zvRmBdq++fOcmdVFIQJv3wKtj8whbe+c 6c0KtH1ri3OI3nN2K/fRrC7qOWdCtM0+tsVzJkTg7VswzyFaLFJ6/9pGPr0QAAAAwK1OSmlx/jtT FlwAAAAAgG/jPVwAAAAA4CMsuAAAAADAR3y+4Nq+ffuwhISE0piYmMrZs2e/oJqTm5v7dmRkZH3/ /v2PejOe6upqW1ZW1labzVYdFxdXsWzZsidUcqSUltTU1L/GxMRU2u32qpkzZy72ZlwtLS0haWlp RUOHDt2lmhEREdFgs9mqbTZbdd++fUtUcxoaGiLuv//+TVFRUXW9evX6/NChQ/eo5JSWliY4xmOz 2ao7dOhwdcmSJXkqWStWrHgsMTHxRGJi4ons7OzCy5cvh6vkvPLKK0/27du3JCEhoXTx4sUz9d7P Vf+++uqrTj/5yU/ej42NPZWZmbnz/Pnzd6vkrFu3blJiYuKJ0NDQ5gMHDuj7VAONrN/85jf/abfb q+x2e1VOTs6fvvrqq04qOS+++OIzsbGxp2JiYiqHDRu2vbKyMkYlx2HZsmVPhISEtJw6dSpWJScv L2/JXXfd9TdHlz788MNRquNZtGjRLEfOvHnz5nrK0coaM2bMe46ciIiIhqSkpOMqOSUlJX3T09P3 9O/f/+jAgQMP7tix416VnMOHDw9IS0srSkxMPDF69OgPGhsbO3vK0ZoPjXZbK8dot7VyVHqtlWW0 254eM/R2WyvHaLfdjcdIt7VyjPZaK0el11pZRrut9fhstNdaOUZ7rZVjtNdaOSrztadrGL291sox 2mt34zHSa60co73WylHptVaWypwtxLevF1WuRVzlqF6LOOeozNeuclR67SrHwch1iFaWyrXIP0gp fba1tLRYevXqVXH06NF+TU1NoYMHDy7as2fPEJWsHTt2ZO7fvz+lX79+R70ZU3V1dfSuXbsypJSi vr4+Ijo6uvrEiRN9VbLOnz8fKaUU169fvy09PX33hx9+OFJ1XK+99tqvJk6cuG7o0KE7VTO6detW Z8Z5Gz9+/DsvvfTSb6SU4vLlyx0vXLhwpxm5PXv2rKqoqOhl9H4XLly4MyIiov7LL7/sJKUUkydP fmvp0qVPGs05evRov9jY2M8vXboUfuPGjbAhQ4bsOX78eKJq/+bMmTN/1qxZC6WU4ve///2/TZ8+ /XWVnOPHjyeWlZX1zsjI2PXZZ58N0rs/rrIKCgqyr169eruUUkyfPv31p59++j9Vchzdduzbww8/ vF4lR0op6urqumVlZf0lISHh5Oeffx6rkpOXl7d47dq1k4ycb1c5mzZtGpuamrrP0aXTp0/3UM1q vc2ZM2f+7Nmzn1fJyc7OLvjDH/7wr1JK8fHHHw+/5557DqrkpKSk7P/zn//8U8c5mzlz5iJPOVrz odFua+UY7bZWjkqvtbKMdtvdY4aRbmvlGO22Vo7Rbut5LNTTa60clV5rZal029Xjs8qc7SpHZc52 laPSa1c5KvO1VpbRXmvlqMzZrnJU5mxP12Z652tXOSq91spS6bWU375eVOm1qxzVaxHnHJVeu8pR 7bWr62mjndbKUum1Y/PpM1yHDh26p0uXLhf69et3LDQ0tDk3N/ftgoKC8SpZmZmZOzt37tzo7Zii o6NrMjIyPhXi62eEEhISSuvq6pQ+5zEyMrJeCCGam5tDW1paQlQ/rbG+vj5yw4YND86YMeNV6eKT Tfzp3Llz3Xbv3v1Dx//AdOzY8cqdd9550dvcXbt2DY2MjKzv1avX50bvK6W0SCktV69e7dDc3Bx6 7dq1261Wa63RnJKSkr4/+MEP9oeHh18OCwu7mZmZufOdd975mZ77uurfpk2b7p86deqbQggxZcqU twoLC7NVchITE0/07t273MCuaGZlZ2cX3n777deEECIjI+PT2tpaq0qOo9tCCHHjxo1/0dNtrZ/R mTNnLl6wYMGzen8+tHKM/my4ylm+fPmvZs+e/UKnTp2+EkKIHj16nPFmTA7r16+fOGnSpHUqOSEh IS2OZ2wvXbr0PT3ddpVTUlLS98c//vFfhBAiKytrq55uu5oPz549291ot7XmVaPd1spR6bVWltFu u3vMMNJtrWMthLFua+WsWLHiMSPd1vNYqKfXWuNR6bVWlkq3nR+fhVCbs109zqvM2a5yVHrtKkdl vtbKEsL4nO3qWAthfM52lWO01+72y0HvfO1qPCq91spS6bWr60WVXrvKUem1qxyVXrvKUem11vW0 0U67y1K9Tvfpgqu2ttbauow2m61az4H3l7KysviysrL4wYMH71PNSEpKOt61a9cvkpOTj4wYMeIj lYy8vLwlCxYseDY0NLRZdRxCfP2DHB8fX9avX79jK1eunK6SUVFREdejR48zU6ZMeSspKen4L37x i9VXr17t4M24hPj6qepHHnlkrcp9u3TpcuGll176bVxcXIXVaq1tampql5OT8yejOf379z+6d+/e IQ0NDRGXL18O/+ijj0bU1NREq4xJiH/ud6dOnb66efNm2M2bN8NU88wkpbS89dZbU8aOHftn1Yw5 c+Y8361bt3Ovvfba44sWLZqlkrF169asDh06XE1NTf2r6jgcnn766YWxsbGnpk6d+qbel144Ky8v 711UVJQ2cODAg0OHDt1VXFyc4u249u3bN7hjx45XVF/Gu2jRolkvv/zyr3v06HHmscceW7F06dL/ UMkZMGDAYcd/aBUWFmYbnWvLysriy8vLew8ePHifN902Y17VylHttXOWardb53jTbUdOWlpakRDq 3W49nrKysnjVbrs61iq9br1f3va69ZhUu9368XnkyJFbVHttxuO8uxyjvXaVo9pp5yzVXjsfayHU eu2co9prrWNttNfO4/Gm185ZKr12db2o0muzrjvd5RjptVaO0V67ylHttNaYVOdrny64nFeSzc3N ob7894xobGzs/NBDD/3vqlWrpnXs2PGKas7x48eT6urqoioqKuL27ds32Oj9t2/fPiwkJKQlPT19 j7fPbhUXF6eUlZXFv/fee2NefvnlX+/evfuHRjOampraHThwYNDjjz/+2rFjx/qFhoY2q15st84s LCzMfvjhh/9H5f5XrlzpuGbNmkdPnjzZp7a21hoWFnZz+fLlvzKa06dPn5Nz586dN2rUqA/HjBnz XkpKSnFISEiLyphccTwTZ1aeN2bPnv1C9+7dzz744IMbVDOef/75OefOnev2+OOPv6by3rsbN278 y5w5c55/8cUXn3H8nerxeeqpp353+vTpnidPnuwTHh5+WfU9k01NTe0uXrx458GDBwcuWLDgWW+O j4M3/5kghBCrV6/+RV5e3pIzZ870WLly5XTVrDfeeOPnb7755tSUlJTiv//977e1a9euSe99HfPh ypUrp4eHh192/r7ebps1r2rlqPTaVZZKt1vnhIWF3VTttvN4VLvdOic8PPyyare1jrXRXjvneNNr 531T7Xbrx+eioqI05+/r7bW3j/Oecoz22lWO6nzdOmvnzp2Zqr12HpNqr51zVHutdayN9to5x5te O2cZ7bXe60VPvTbrutNTjt5eu8sx0mtXOarXIVpj8upaROV1iHq3gwcP3pOSkrLf8fUrr7zy73pf o+pqKy8vj/P2PVxSSnHt2rX299577ydr1qyZata+Lly4cJae1wQ7bwsWLHjGarXW2O32yqioqLPt 27e/9sADD2z0djx5eXmLVd7nVF5eHme1WmscX3/wwQejvB3P5s2bx2RlZf1F9f4fffTRj8eMGbPZ 8fW6desmTpo0aa23x2jGjBn//bvf/e4/jByb1v0bMGDAoWPHjiVJKUVjY+MdERER9So5js3o66a1 sl599dUnRowYseXmzZvtvMlxbOfOnbs7Ojq62mjOqVOnYiIiIurtdnul3W6vDAsLu2Gz2c6Ul5fH eTOeo0eP9tP7unnnnIyMjF1bt279kePru++++1xjY+MdqseoqakptHv37rXV1dXRqsfaarXWfPHF F3c5vr7tttuu6zl37o5RcXHx95OTkw/rGY+r+VCl2+7mVSPd1spR6bWnuV5vt51zVLvtaTx6u+0q R6XbWuMx2mtXOaq99nSMjHTbsTken1XnbOcclV5r5aj0Wms8RjrtKuuRRx55W3XOdjcmI3O2c443 c7bzeFTma+cc1V57OkZ6eu3qevH+++9/12ivPV136u21uxwjvdZzHayn165yxo4du0ml03rGZLTX hgpndGtubg6JjY39/PDhw8k3btwIS01N3bd79+501TwzFlxNTU2hY8eO3eT4UAjVrb6+PqKqqqqn lFJcvHixc3p6+u5169ZN9CazqKhocEZGxi6V+168eLGz4w2G58+fj0xKSjq2bdu2YSpZAwYMOHTk yJH+Ukrx1FNP/ZfKQrL1NmnSpLVvvPHGo6r3Lykp6RMVFXW2oaGha0tLi2XKlClvvvDCC8+qZJ06 dSpGSikOHDgwsHv37rWtJ06j/ZszZ878vLy8xVJKsXTp0ienTZu2UiXHsWVkZOwqLi7+vpH9cc5a v379w6mpqfsuX77c0Zuc/fv3p7S0tFiklGLRokUzR4wYscWbfZNSij59+pTofbOqc87JkycTpPx6 Tpk1a9bCyZMnv6WSs2TJkl873lx84MCBgT179qxSPUZSfv2fAZmZmTu8OdZDhgzZ43gT9rZt24bF xcWVq+ScOXPG1tzcHHLlypUOo0aN+mD58uWPecrQmg+NdtvTvKq321o5Kr3WyjLabT2PGXq6rZVj tNtaOUa77W6/jPRaK0el11pZRrut9fhstNeeHuf19lorx2ivtXKKi4u/b3S+1nMNo6fXWjmlpaXx RnqtlWO01+72y0ivXeWsXbt2kkqvtcakMmc7ttbXi6rXIs45RnutlaN6HeKco3odorVfejvtLkv1 WkRKHy+4pPz6U1x69+5dZrPZzvz2t799UTUnOzu7ICoq6mxYWNiN6OjoatUL+I8//ni4xWJpiY6O rnZsGzdufMBoTllZWe/k5OTDVqu1pmfPnlXPPvvsC94eq71796apfkrhiRMn+iYkJJy0Wq01MTEx pxYuXDhLdRxFRUWDk5OTD8fHx5fm5OT88dKlS+GqWVeuXOlw1113feH4hCHVbcmSJb/u1atXRe/e vcsmTJiwQeUHWUopfvSjH23t1q1bXZ8+fUq2bNkywpv+ffnll51Gjx79vs1mO5ORkbGrrq6um9Gc 1atX/3z9+vUPR0dHV7dv3/7a3XfffW7UqFEfqIxp9erVP+/Ro8fprl27Nji6reeZQFc5kydPfisq KupsdHR09ejRo993PFgYPUatv693onM1ngkTJmxwjCc7O7ug9acXGRnPtWvX2k+YMGFDr169KgYO HHjg008//aHq+ZdSikcfffSNFStW/NKbHn322WeDUlJS9sfHx5cOHDjwgJ7/lHJ1jFasWPFLq9Va ExkZeT4vL2+x44HK3aY1HxrttqucwsLCcUa7rZWj0mutLKPd1vOYoafbWuMx2m2t8Rjttrv9MtJr rRyVXmsdI6Pd1np8NtprrZx169ZNNNJrrRyjvdbKUZmv9VzD6Om1Vo7RXmvlGO21u/0y0mutHJVe a2UtX778MaNztmNrfb2oci3iKsdor7VyVOZrVzkqvXaVY7TT7rJycnL+aPRaxLFZpFT6YD0AAAAA gAc+/8XHAAAAAHCrYsEFAAAAAD7CggsAAAAAfIQFFwAAAAD4CAsuAAAAAPARFlwAAAAA4CP/B8Gk PtlxEUKuAAAAAElFTkSuQmCC " />

Me: So this diagram shows the hidden state at every iteration, it’s a bit how our own neurons, and this is how the brain works. Check this out. There’s a- Wait, where’s everyone?

I was thinking about Joel Grus’ parody about learning fizz buzz using neural networks, and I felt like it’d be an fun way to demonstrate learning some kind of state machine using RNNs.

  <p>
    His approach was using a binary encoding of the input and making a prediction about fizzing or buzzing. That actually seems like a harder problem to me, just thinking about it in terms of <a href="http://homepage.cs.uiowa.edu/~jones/bcd/mod.shtml">boolean operations that need to be performed</a>. Moreover, it doesn&#8217;t generalise to numbers bigger than the one pre-defined. Another way of approaching this without that limitation would then be to count up from 1 and output as we go. Deterministic finite-state automatons (DFAs) do that very nicely, and it&#8217;s easy to see how a DFA could be encoded in an RNN. And even though the hidden states of an RNN are not finite, in practice, it is possible to learn a working DFA with an RNN. There&#8217;s been literature (that I haven&#8217;t read) on the <a href="http://binds.cs.umass.edu/papers/1996_Siegelmann_JCompInt.pdf">relationship</a> <a href="https://clgiles.ist.psu.edu/papers/JACM-1996-stable-encoding-rnn-automata.pdf">between</a> <a href="http://axc.ulb.be/uploads/2015/11/89-nc.pdf">them</a> that date back to it&#8217;s <a href="http://cns-classes.bu.edu/cn550/Readings/mcculloch-pitts-43.pdf">inception</a>.
  </p>
  
  <p>
    I figured in order for this to happen, conceptually, there&#8217;d be 2 state machines running in &#8220;parallel&#8221;: A 3-state DFA maintaining divisibility by 3, and a 5-state DFA maintaining divisibility by 5. 3-states requires 2 bits to encode, 5-states requires 3, so together, 5 hidden units. We might even be able to bring that down to 4 hidden units, seeing as the <a href="https://www.quora.com/Deterministic-finite-automata-intersection-and-union-How-do-I-use-product-construction">product of these two DFAs</a> would have 15 states, which would require 4 bits to encode.
  </p>
  
  <p>
    If you look at that last diagram, there&#8217;s a repeating pattern from one &#8220;fizzbuzz&#8221; to another. The cleaned version shows a similar state between (6, 12), (21, 27), and (36, 42), but looking at the uncleaned version, you&#8217;ll see that the activation isn&#8217;t so strong in one of the units. There are other patterns just looking at individual rows of the hidden activations too. I was hoping there would be an obvious pattern where a unit would be activated every 3 iterations (there is: see row 2) and another activated every 5 (row 0 sort of does this, but not as consistently as I&#8217;d like), and both of these would be activated together every 15.
  </p>
  
  <p>
    It would be great if the hidden layer size for real-life problems could be worked out in a similar way. Hopefully one day we can!
  </p>
</div>

@misc{tan2016-05-30,
  title        = {FizzBuzz in Theano, or, Backpropaganda Through Time.},
  author       = {Tan, Shawn},
  howpublished = {\url{https://blog.wtf.sg/2016/05/30/fizzbuzz-in-theano-or-backpropaganda-through-time/}},
  year         = {2016}
}