Can Chinese Rooms Think?

There’s a tendency as a machine learning or CS researcher to get into a philosophical debate about whether machines will ever be able to think like humans. This argument goes so far back that the people that started the field have had to grapple with it. It’s also fun to think about, especially with sci-fi […]

Computing Log Normal for Isotropic Gaussians

Details of the Hierarchical VAE (Part 2)

So, as a recap, here’s a (badly drawn) diagram of the architecture:

Details of the Hierarchical VAE (Part 1)

The motivation to use a hierarchical architecture for this task was two-fold: Learning a vanilla encoder-decoder type of architecture for the task would be the basic deep learning go-to model for such a task. However, the noise modeled if we perform maximum likelihood is only at the pixel level. This seems inappropriate as it implies […]

Samples from the Hierarchical VAE

Each of the following plots are samples of the conditional VAE that I’m using for the inpainting task. As expected with results from a VAE, they’re blurry. However, the fun thing about having a hierarchy of latent variables is I can freeze all the layers except for one, and vary that just to see the […]

Hierarchical Variational Autoencoders

$$\newcommand{\expected}[2]{\mathbb{E}_{#1}\left[ #2 \right]} \newcommand{\prob}[3]{{#1}_{#2} \left( #3 \right)} \newcommand{\condprob}[4]{{#1}_{#2} \left( #3 \middle| #4 \right)} \newcommand{\Dkl}[2]{D_{\mathrm{KL}}\left( #1 \| #2 \right)} \newcommand{\muvec}{\boldsymbol \mu} \newcommand{\sigmavec}{\boldsymbol \sigma} \newcommand{\uttid}{s} \newcommand{\lspeakervec}{\vec{w}} \newcommand{\lframevec}{\vec{z}} \newcommand{\lframevect}{\lframevec_t} \newcommand{\inframevec}{\vec{x}} \newcommand{\inframevect}{\inframevec_t} \newcommand{\inframeset}{\inframevec_1,\hdots,\inframevec_T} \newcommand{\lframeset}{\lframevec_1,\hdots,\lframevec_T} \newcommand{\model}[2]{\condprob{#1}{#2}{\lspeakervec,\lframeset}{\inframeset}} \newcommand{\joint}{\prob{p}{}{\lspeakervec,\lframeset,\inframeset}} \newcommand{\normalparams}[2]{\mathcal{N}(#1,#2)} \newcommand{\normal}{\normalparams{\mathbf{0}}{\mathbf{I}}} \newcommand{\hidden}[1]{\vec{h}^{(#1)}} \newcommand{\pool}{\max} \newcommand{\hpooled}{\hidden{\pool}} \newcommand{\Weight}[1]{\mathbf{W}^{(#1)}} \newcommand{\Bias}[1]{\vec{b}^{(#1)}}$$ I’ve decided to approach the inpainting problem given for our class project IFT6266 using a hierarchical […]

FizzBuzz in Theano, or, Backpropaganda Through Time.

Deciding When To Feedforward (or WTF gates)

Another paper of mine, titled “Towards Implicit Complexity Control using Variable-Depth DNNs for ASR Systems” got accepted to the International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2016 in Shanghai, which happened not too long ago. The idea behind this one was the intuition that in a classification task, some instances should be simpler […]

Constraining Hidden Layers for Interpretability (eventually, hopefully…)

I haven’t written much this past year, so I guess as a parting post for 2015, I’d talk a little bit about the poster I presented at ASRU 2015. The bulk of the stuff’s in the paper, plus I’m still kind of unsure about the legality about putting stuff that’s in the paper on this […]

Learning to Transduce with Unbounded Memory – The Neural Stack

DeepMind has in the past week released a paper proposing yet another approach to having a memory structure within a neural network. This time, they implement a stack, queue and a deque “data structure” within their models. While this idea is not necessarily new, it incorporates some of the broad ideas seen in the Neural […]