Transformer Performance: Hopfield Theory & Cross-Entropy Loss Data

Table 1: Table of selected related works for Hopfield network, enumerating their domain, energy function, and memory capacity. For all the works above, n represents the dimension of the input vector. W is the outer product of the patterns. M is the matrix of patterns. r is the order of polynomial F(·), d is the number of patterns, and c is a positive constant.

Table 2: Large transformer-based language models and their reported cross-entropy loss.

Authors:

(1) Xueyan Niu, Theory Laboratory, Central Research Institute, 2012 Laboratories, Huawei Technologies Co., Ltd.;

(2) Bo Bai baibo ([email protected]);

(3) Lei Deng ([email protected]);

(4) Wei Han ([email protected]).

This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license.

← Previous

New Regularization-Free Energy Function for Transformer Analysis

Up Next →

LogSumExp Function Properties: Lemmas for Energy Functions