Augmenting Differentiable Neural Computer with Read Network and Key-Value Memory

Abstract

A Differential Neural Computer (DNC) is a type of neural network architecture that can make use of external memory. A DNC model can represent complex data sequences and reason about them. However, training a complex DNC with large memory matrices was slow, hence this study focused on improving a DNC model to learn faster and perform bAbi question-answering tasks more accurately. The attempted improvements were to use key-value pairs for memory locations instead of arrays and to obtain a read vector from the memory matrix with a neural network. Evaluation of the improved DNC models on the bAbi dataset showed that their compute time was 13% shorter and their error rate was 6.6% lower. To conclude, the attempted improvements were successful, and the training speed of the key-value memory model also became shorter.