Gaussian discrete restricted Boltzmann machine : theory and its applications : a thesis presented in partial fulfilment of the requirements for the degree of Master of Engineering in Electronics and Computer Engineering at Massey University, Albany, New Zealand
Loading...
Date
2015
DOI
Open Access Location
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Massey University
Rights
The Author
Abstract
Restricted Boltzmann Machine (RBM) is a two-layer neural network, popular for its
efficient training methodology in many applications involving data recall, classification,
and recognition. Traditionally RBM is designed with binary neurons in both layers.
RBMs with Gaussian (continuous-valued) neurons in visible layer have been introduced
for ease of integration with real data. However, the hidden layer still consists of binary
neurons. Recently, theoretical studies in discrete RBM with discrete visible and hidden
nodes have shown that increasing the number of hidden states improves reconstruction
error. Motivated by this finding, the research in this thesis aims to develop an RBM
with a Gaussian visible layer and a discrete multi-state hidden layer, called the Gaussian
Discrete RBM (GDRBM). The equations governing this new model have been worked
out and a contrastive divergence training algorithm has been developed based on these
equations. Performance results using the MNIST and CBCL benchmark datasets show
that the performance of a GDRBM with 4-state hidden neurons is approximately the
same as that of other Gaussian RBMs with binary hidden neurons when the size of the
hidden layer is doubled. This GDRBM has also been used to form one layer of a deep
autoencoder. This is the first time an autoencoder has been designed with a multi-state
discrete layer. Initial experimental results show that a GDRBM-based deep autoencoder
is able to reconstruct the inputs reasonably well. However the pretraining is not very
effective and the amount of initial reconstruction error need to be reduced to make it
perform at the same level of a traditional deep autoencoder. Further research will be
needed to understand how GDRBM could be used in a deep autoencoder.
Description
Keywords
Restricted Boltzmann machine (RBM), Neural networks (computer), Machine learning, Gaussian discrete RBM