The Conformer Encoder May Reverse
the Time Dimension

Abstract

We sometimes observe monotonically decreasing cross-attention weights in our Conformer-based global attention-based encoder-decoder (AED) models. Further investigation shows that the Conformer encoder internally reverses the sequence in the time dimension. We analyze the initial behavior of the decoder cross-attention mechanism and find that it encourages the Conformer encoder self-attention to build a connection between the initial frames and all other informative frames. Furthermore, we show that, at some point in training, the self-attention module of the Conformer starts dominating the output over the preceding feed-forward module, which then only allows the reversed information to pass through. We propose several methods and ideas of how this flipping can be avoided. Additionally, we investigate a novel method to obtain label-frame-position alignments by using the gradients of the label log probabilities w.r.t. the encoder input frames.

I Introduction & related work

Figure 1: Cross-attention weights of a model with reversed encoder vs. standard encoder.

The Conformer encoder [1] is a popular choice for the encoder of automatic speech recognition (ASR) systems. It combines self-attention and convolutional layers in order to model both long-range as well as local dependencies in the input sequence and has been shown to be superior [1, 2] to the Transformer encoder [3]. The attention-based encoder-decoder (AED) framework uses a decoder which autoregressively produces an output sequence while attending to the whole encoder output at every step [4, 5].

We observe that the Conformer encoder of an AED model sometimes reverses the time dimension of the input sequence, as the cross-attention weights show (Figure 1), i.e. the cross-attention is flipped in the time dimension. In this work, we investigate different aspects of this phenomenon such as why it occurs and how it can be avoided.

Furthermore, we propose a novel way of obtaining label-frame-position alignments (for each output label, the start/end time frames in the input) by leveraging the gradients of the label log probabilities w.r.t. the encoder input frames. We use methods related to saliency maps and other gradient-based attribution methods such as the ones studied in [6, 7]. Moreover, [8, 9] shows how gradients could be used to better interpret the behavior of a model.

II AED Model

Our baseline is the standard AED model adapted for ASR [4, 5]. The encoder consists of a convolutional frontend, which downsamples the sequence $x_{1}^{T^{\prime}}$ of 10ms frames into a sequence ${h_{0}}_{1}^{T}$ of 60ms frames, followed by a stack of $N=12$ Conformer blocks, which further process the downsampled sequence resulting in the encoder output $h_{1}^{T}$:

$${h_{0}}_{1}^{T} =\operatorname{ConvFrontend}(x_{1}^{T^{\prime}}),$$

$${h_{i}}_{1}^{T} =\operatorname{ConformerBlock}_{i}({h_{i-1}}_{1}^{T}),\quad h=h_{N},i=1,\dots,N.$$

The probability of the output label sequence $a_{1}^{S}$ given the encoder output $h_{1}^{T}$ is defined as

$$p(a_{1}^{S}\mid h_{1}^{T})=\prod_{s=1}^{S}p(a_{s}\mid a_{1}^{s-1},h_{1}^{T}).$$

At each step $s$, the decoder autoregressively predicts the next label $a_{s}$ by attending over the whole encoder output $h_{1}^{T}$. We define $a_{S}=\texttt{EOS}{}$ as the end-of-sequence token, which implicitly models the probability of the sequence length. We use an LSTM [10] decoder with single-headed MLP cross-attention [11] following our earlier setup [12]. We use BPE subword label units [13] (1k and 10k vocab. sizes).

We train our model using the standard label-wise cross-entropy (CE) criterion $L=-\log p(\overline{a}_{1}^{S}\mid h_{1}^{T})$ using the target transcriptions $\overline{a}_{1}^{S}$. In the experiments, where we observe the flipping of cross-attention weights, there is no further loss, but we also tested to add CTC [14] as an auxiliary loss [15].

We use label-synchronous beam search for recognition.

III Experimental Setup

Our experimental setup follows the global AED baseline from [16]. We perform all our experiments on the LibriSpeech 960h [17] corpus using the RETURNN framework [18] based on PyTorch [19]. Our experimental pipeline is managed by Sisyphus [20]. All models are trained for 100 epochs using the AdamW optimizer [21] and we apply on-the-fly speed perturbation and SpecAugment [22]. As hardware, we either use a single Nvidia A10 GPU or 4x Nvidia 1080 GTX GPUs in parallel with parameter syncing every 100 batches [23, 24]. All the code to reproduce our experiments is published.

TABLE I: Comparing the baseline in this paper – Conformer AED with BPE 1k without CTC aux. loss – vs related models. Results on LibriSpeech, without external language model.

AED Model	Label Units	CTC aux. loss	WER [%]
			dev		test
			clean	other	clean	other
E-branchformer-Trafo [25]	BPE 5k	Yes	2.0	4.6	2.1	4.6
Conformer-LSTM	BPE 1k	No	2.8	6.7	3.0	6.8
	BPE 10k		2.6	6.1	2.8	6.0
		Yes	2.4	5.4	2.6	5.8

Table I shows the word error rate (WERs) of our baseline model compared to related models. BPE 1k without CTC aux. loss is the base configuration of all experiments in the paper.

IV Analysis

IV-A Initial Development of Cross-Attention Weights

Figure 2: Development of cross-attention weights into the flipping behavior over the initial training epochs.

Figure 3: Showing $G_{0}$, i.e. the logarithm of the $L_{2}$ norm of the gradients of the target label log probabilities w.r.t. first Conformer block inputs very early in training (after 2 epochs).

In all of our experiments, we observe that the decoder cross-attention initially only attends to the first few frames of the encoder output (Figure 2). We hypothesize that the choice of these frames is not due to their usefulness for predicting labels but rather because they have distinct features which makes it simple to attend to them. The convolutional zero padding on the sequence boundaries can provide those distinct features, and/or the initial silence. The first frame is probably easier to recognize than the last frame, as the last frame can have different padding in it due to the batching, and also the sequences on LibriSpeech have slightly more silence at the beginning (260 ms vs. 230 ms on average).

We conducted an experiment where we force the model to only attend to the center frame by setting the attention weight of this frame to one and all other weights to zero. The initial CE losses of these models are almost identical (6.30 vs 6.29 after 1 epoch), proving that the choice of frame for the initial cross-attention weights is not important for the prediction of labels in the beginning of training.

In order to see how the model utilizes the frames of the individual encoder layers, we can look at the gradients of the target label $\overline{a}_{s}$ log probabilities w.r.t. the corresponding encoder layer input or output $h_{i}$ or even the input to the convolutional frontend $h_{-1}=x$. This gives us a $D$-dimensional vector per target label position $s$ and frame $t$, which we reduce to a scalar by taking the logarithm of its $L_{2}$ norm:

$${G_{i}}_{s,t}=\log\left\|\nabla_{{h_{i}}_{t}}\log p(\overline{a}_{s}\mid\overline{a}_{1}^{s-1},h_{1}^{T})\right\|_{2}.$$ (1)

Very early in training (after 2 epochs), we see from $G_{0}$ (Figure 3), that the gradients are initially more focused on individual label frames and especially less focused on the silence frames. Furthermore, we can see that, in this early stage in training, for some output labels, the gradients over all input frames are stronger, meaning the encoder output is more important for those labels than for others. It is specifically more pronounced for the labels ”You” and EOS.

IV-B How is Time Reversal possible?

Figure 4: Self-attention energies averaged over the 8 heads of the 10th Conformer block for initial epochs. After this, all further layers are flipped.

Figure 5: Gradients $G_{9}$ and $G_{10}$ w.r.t. the output of blocks 9 and 10 after 12 epochs. For $G_{9}$, we have the crossing of information from the residual and the self-attention. In $G_{10}$, only the flipped information is left.

The flipping can only occur in the self-attention (Figure 4). But what about the residual connection? In every Conformer block, there is a final layernorm layer, so there is no direct residual path from the input to the output of the encoder. There is one Conformer block where the time reversal occurs. In the flipping block, the self-attention module has by far the largest activations in magnitude (mean $L_{2}$ norm 60 for some example sequence), so that the residual connection (norm 23) from the input of this block and from the first feed-forward module do not have much effect anymore. The remaining convolution module and second feed-forward module also do not add much (norm 13 and 15 respectively). The final layernorm then removes all the original frame-wise information and only the reverse order is kept.

It can be seen that the flipping occurs in Conformer block 10 and not in the other blocks (Figure 5). The gradients $G_{9}$ (Figure 5) show that there is still some information remaining from the inner residual connection in block 9.

We do not expect that such flipping is easy to perform by an encoder where there is a residual connection from input to output like in the standard Transformer [3].

IV-C Reasons for Time Reversal

1) The cross-attention initially only attends to the first few frames of the encoder output for all decoder frames (Figure 2, epoch 2) as discussed before (Section IV-A).

2) The decoder acts initially like a language model and can learn independently from the encoder. To slightly improve the prediction perplexity, having some information of the encoder is useful, and that is where it uses the fixed cross-attention to the first frame. The first encoder frame attends globally to more informative frames of lower layers (Figure 4, epoch 2 to 6.4) as this is most useful at this stage to collect global information from all labels. Attending globally to multiple frames is also more informative than just first label or another single label for predicting the whole sequence.

3) To further improve the sequence prediction perplexity, the cross-attention learns to focus on another frame, which happens to be some random frame towards the end (Figure 2, epoch 8). The next most easily usable information is to know about the first label in the sequence. Thus this encoder frame uses self-attention to attend to the frames of the first label (Figure 4, epoch 6.8 to 8), where the first label is originally located in lower layers.

4) The next labels follow, one after the other. For the cross-attention, it is easier to choose some position right next to the previous position. So this will lead to the flipping (Figure 2 and Figure 4, epoch 8 to 10).

The specific vocabulary and also the average sequence length will all influence those training dynamics. Specifically, when we skip sequences longer than 75 labels during training, we do not observe the flipping, both for BPE1k and BPE10k. For BPE1k, the output sequences are naturally longer.

The training dynamics are also stochastic and depend on the random initialization. When running the training multiple times with different random seeds, we observe that the flipping happens in 12 out of 13 experiments (for BPE1k). In 2 out of the 12 flipping cases, in addition to the flipping, we also observed some shuffling of segments.

IV-D Measures to Avoid Time Reversal

1)  Use CTC auxiliary loss [15]

CTC allows only monotonic alignments between input frames and output labels, so the encoder output can not be flipped. We never observed such flipping in any experiment with CTC auxiliary loss. This is actually very commonly used (by default in ESPnet [26] and many RETURNN setups [12, 16]), which is why the flipping was maybe not observed before.

2)  Disabling self-attention in the beginning

The flipping occurs in the self-attention. We can fix the self-attention weights to be the identity matrix for the first epoch and then only later use learned attention weights. This effectively means that we only use the linear transformation for the values in the self-attention module. This experiment shows faster convergence (after 12 epochs: baseline reaches 0.83 CE loss, baseline with CTC aux loss reaches 0.56 CE, disabling self-attention reaches 0.45 CE), and also no flipping occurs.

3)  Hard attention on center frame

We argued before that the initial focused cross-attention to the first frame might lead to the flipping. We did an experiment where we forced the cross-attention weights to the center of the encoder output sequence for the first full epoch. In this experiment, no flipping did occur.

V Alignments from Model Input Gradients

Figure 6: Gradients $G_{0}$ w.r.t. Conformer input after 12 epochs. Alignment path is visible.

As can be seen in Figure 6, the gradients of the log label probabilities w.r.t. the Conformer input $G_{0}$ show an alignment between output labels and input frames. We were asking the question: Can we use those gradients to estimate an alignment, i.e. the best alignment path (boundaries/positions of each label and word)? This has been done before using the attention weights [27] but that can be problematic due to multiple cross-attentions (e.g. Transformer with multiple layers and heads) or because the encoder shifts or transforms the input, as our work shows here. When the model has reasonable performance, the gradients w.r.t. the model input cannot have those artifacts like flipping or shifting.

We use either $G_{0}$ (Equation 1) to get an alignment with 60ms frame shift or $G_{-1}$ (before the conv. frontend) with 10ms frame shift. We allow any number of $\epsilon$ (blank) labels between any of the real labels, we allow the real label to be repeated multiple times over time, and we exclude the final EOS label. This is very similar to the CTC label topology except that we do not enforce an $\epsilon$ between two equal labels. We search for an allowed state sequences $r_{1}^{T}:a_{1}^{S}$ for state indices $r_{t}\in\{1,\dots,2\cdot S-1\}$ corresponding to states $Y=(\epsilon,1,\epsilon,2,\dots,S-1,\epsilon)$ which maximizes

$$\operatorname{GradScore}(r_{1}^{T})=\sum_{t=1}^{T}\operatorname{GradScore}(r_{t})$$ (2)

$$\operatorname{GradScore}(r_{t})=\begin{cases}\log\operatorname{softmax}_{\overline{t}}(G_{i})_{Y_{r_{t}},t},&Y_{r_{t}}\neq\epsilon,\\ \gamma_{\epsilon},&Y_{r_{t}}=\epsilon\end{cases}$$ (3)

for some fixed blank score $\gamma_{\epsilon}$ which is a hyperparameter. The best $r_{1}^{T}$ can be found via dynamic programming. We obtain the final alignment label sequence $y_{1}^{T}$ with $y_{t}=\begin{cases}a_{Y_{r_{t}}},&Y_{r_{t}}\neq\epsilon,\\ \epsilon,&Y_{r_{t}}=\epsilon\end{cases}$.

We measure the time-stamp-error (TSE) [28, 29, 30] of word boundaries, i.e. the mean absolute distance (in milliseconds) of word start and end positions against a reference GMM alignment, irrespective of the silence. Additionally, we also compute the TSE w.r.t. the word center positions, which might be a better metric for peaky alignments like CTC [31]. We summarize our findings in Table II. Our method still performs worse in terms of TSE compared to a well tuned phoneme-level CTC model from earlier work [30], however we improve over all our BPE-based CTC alignments by far. We also see that our method still works even when the encoder flips the sequence.

TABLE II: Alignment quality in terms of time-stamp-error (TSE) for word left/right boundaries and center positions against reference GMM alignment on a random 10h subset of the training data of LibriSpeech. ^∗: On the whole training data, and the computation is different: more consistent alignment due to same feature extraction as the GMM, while our models here use a different feature extraction.

Best Path Scores	Model	Train Phase	Flip	Frame Shift [ms]	Label Units	TSE [ms]
						Left/ Right	Center
Probs.	CTC [30]	full	no	40	Phonemes	38∗	-
Probs.	CTC	full	no	60	BPE1k	83	61
					BPE10k	312	306
Grads.	AED	early	no	10	BPE1k	56	43
				60		61	50
			yes	10		72	53
				60		75	61

VI Conclusions

In this work, we have shown that the Conformer encoder of an attention-based encoder-decoder model is able and, under certain conditions (smaller BPE 1k vocab. and no CTC aux. loss), seems to prefer to reverse the time dimension of the input sequence. We studied which functionality of the Conformer makes this behavior possible and how the initial cross-attention weights of the decoder push the model to do so. Furthermore, we proposed several methods to avoid this flipping such as disabling the self-attention in the beginning, which also improves convergence.

We also showed that the gradients of the label log probabilities w.r.t. the encoder input frames can be used to obtain alignments, even early in training, even when the encoder reverses the sequence, and its time-stamp-errors are even better than a normal CTC forced alignment.