Rectified Linear Unit
Better performance compared to the conventional S curve, tanh, was shown in the following paper
Glorot, X., Bordes, A. and Bengio, Y., 2011, June. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics (pp. 315-323).
http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf
This page is auto-translated from /nishio/ReLU using DeepL. If you looks something interesting but the auto-translated English is not good enough to understand it, feel free to let me know at @nishio_en. I’m very happy to spread my thought to non-Japanese readers.