Why Layering in Recurrent Neural Networks? A DeepESN Survey

The extension of Recurrent Neural Networks (RNNs) in the direction of deep learning is a topic that is gaining more and more attention in the neural networks community. The study of deep RNNs opened a number of intriguing research questions on the actual role played by layering in the architectural design of RNNs. Recently, the introduction of the Deep Echo State Network (DeepESN) model allowed to start addressing such open issues in literature, contributing to shed light on the intrinsic properties of state dynamics developed by hierarchical compositions of recurrent layers. This contribution intends to present a unified view over the major advancements in the study of DeepESNs, enabling to directly point out the natural advantages of a layered construction of recurrent networks for temporal data processing.