Gaussian discrete restricted Boltzmann machine : theory and its applications : a thesis presented in partial fulfilment of the requirements for the degree of Master of Engineering in Electronics and Computer Engineering at Massey University, Albany, New Zealand

Thumbnail Image
Open Access Location
Journal Title
Journal ISSN
Volume Title
Massey University
The Author
Restricted Boltzmann Machine (RBM) is a two-layer neural network, popular for its efficient training methodology in many applications involving data recall, classification, and recognition. Traditionally RBM is designed with binary neurons in both layers. RBMs with Gaussian (continuous-valued) neurons in visible layer have been introduced for ease of integration with real data. However, the hidden layer still consists of binary neurons. Recently, theoretical studies in discrete RBM with discrete visible and hidden nodes have shown that increasing the number of hidden states improves reconstruction error. Motivated by this finding, the research in this thesis aims to develop an RBM with a Gaussian visible layer and a discrete multi-state hidden layer, called the Gaussian Discrete RBM (GDRBM). The equations governing this new model have been worked out and a contrastive divergence training algorithm has been developed based on these equations. Performance results using the MNIST and CBCL benchmark datasets show that the performance of a GDRBM with 4-state hidden neurons is approximately the same as that of other Gaussian RBMs with binary hidden neurons when the size of the hidden layer is doubled. This GDRBM has also been used to form one layer of a deep autoencoder. This is the first time an autoencoder has been designed with a multi-state discrete layer. Initial experimental results show that a GDRBM-based deep autoencoder is able to reconstruct the inputs reasonably well. However the pretraining is not very effective and the amount of initial reconstruction error need to be reduced to make it perform at the same level of a traditional deep autoencoder. Further research will be needed to understand how GDRBM could be used in a deep autoencoder.
Restricted Boltzmann machine (RBM), Neural networks (computer), Machine learning, Gaussian discrete RBM