Notes on the generalized backpropagation algorithm for contextual neural networks with conditional aggregation functions

Abstract

In this paper, we show that a contextual neural network with artificial neurons performing a conditional aggregation of signals can be trained by the generalized backpropagation algorithm. To allow this algorithm to be used for training contextual neural networks, we derive appropriate generalized delta rules. Our approach is constructed on the basis of introduced generalized representation of the aggregation function in an ordered groups space and division of its attention function into binary scan-path and contribution functions. The advantage of the proposed representation is that it clarifies the description of the aggregation process by using Stark’s scan-path theory and allows us to achieve results independent from the actual form of the attention functions used during aggregation. As such, the proposed solution is valid for the whole presented family of conditional aggregation functions and is a considerable extension of the previously reported results. In particular, the obtained results are valid for the introduced exemplary attention functions which illustrate performed calculations. Moreover, the presented solution can be further extended by considering real valued, non-binary contribution functions inside ordered aggregation functions. Especially promising are its possible applications in large deep neural networks and energy-limited systems.