2014 Michalski, V., Memisevic, R., Konda, K.
Modeling Deep Temporal Dependencies with Recurrent “Grammar Cells”
Neural Information Processing Systems (NIPS 2014)
[pdf][supplementary][bibtex]
Abstract
We propose modeling time series by representing the transformations that take a frame at time t to a frame at time t+1. To this end we show how a bi-linear model of transformations, such as a gated autoencoder, can be turned into a recurrent network, by training it to predict future frames from the current one and the inferred transformation using backprop-through-time. We also show how stacking multiple layers of gating units in a recurrent pyramid makes it possible to represent the ”syntax” of complicated time series, and that it can outperform standard recurrent neural networks in terms of prediction accuracy on a variety of tasks.
Supplementary Material
Bottom-layer PGP filter pairs
Filters pairs (left/right input receptive fields) of PGP models trained on the accelerated transformation data sets introduced in our paper, the bouncing balls data set [1] and NORBvideos [2]:
Accelerated Rotations |
Accelerated Shifts |
Bouncing Balls |
NORBvideos |
 |
 |
 |
 |
Generated Sequences
Bouncing Balls
Some sequences generated by a three-layer PGP on the bouncing balls data set (generated with the script released with
(the first 4 frames are seeded, the remaining frames are generated by the model):
Some shorter predicted sequences (right) together with ground truth (left) from preliminary experiments with a 2-layer PGP (3 seed frames not shown):
Chirps
Additional chirp predictions not shown in the paper, because of space restrictions. After seeing 5 windows of 10 frames (frames 1-50) each the models predicted the remaining sequence. The comparison models are the Conditional Restricted Boltzmann Machine (CRBM) [3] trained with contrastive divergence and a vanilla RNN trained with backpropagation through time.
References
[1] I. Sutskever, G. E. Hinton, and G. W. Taylor. The recurrent temporal restricted boltzmann machine. In Advances in Neural Information Processing Systems 21, pages 1601–1608, 2008.
[2] R. Memisevic and G. Exarchakis. Learning invariant features by harnessing the aperture problem. In Proceedings of the 30th International Conference on Machine Learning, 2013.
[3] G. W. Taylor, G. E. Hinton, and S. T. Roweis. Modeling human motion using binary latent variables. In Advances in Neural Information Processing Systems 20, pages 1345–1352, 2007.