Local Sigmoid Method: Non-Iterative Deterministic Learning Algorithm for Automatic Model Construction of Neural Network

Abstract

A non-iterative learning algorithm for artificial neural networks is an alternative to optimize the neural network parameters with extremely fast convergence time. Extreme learning machine (ELM) is one of the fastest learning algorithms based on a non-iterative method for a single hidden layer feedforward neural network (SLFN) model. ELM uses a randomization technique that requires a large number of hidden nodes to achieve the high accuracy. This leads to a large and complex model, which is slow at the inference time. Previously, we reported analytical incremental learning (AIL) algorithm, which is a compact model and a non-iterative deterministic learning algorithm, to be used as an alternative. However, AIL cannot grow its set of hidden nodes, due to the node saturation problem. Here, we describe a local sigmoid method (LSM) that is also a sufficiently compact model and a non-iterative deterministic learning algorithm to overcome both the ELM randomization and AIL node saturation problems. The LSM algorithm is based on "divide and conquer" method that divides the dataset into several subsets which are easier to optimize separately. Each subset can be associated with a local segment represented as a hidden node that preserves local information of the subset. This technique helps us to understand the function of each hidden node of the network built. Moreover, we can use such a technique to explain the function of hidden nodes learned by backpropagation, the iterative algorithm. Based on our experimental results, LSM is more accurate than other non-iterative learning algorithms and one of the most compact models.