Analysis on the number of layers in the transformer-based model for neural machine translation

Dongxing Li; Zuying Luo

doi:10.1117/12.2661982

28 December 2022 Analysis on the number of layers in the transformer-based model for neural machine translation

Dongxing Li, Zuying Luo

Author Affiliations +

Proceedings Volume 12506, Third International Conference on Computer Science and Communication Technology (ICCSCT 2022); 1250625 (2022) https://doi.org/10.1117/12.2661982
Event: International Conference on Computer Science and Communication Technology (ICCSCT 2022), 2022, Beijing, China

Abstract

Recently, the transformer-based models have been widely used in sequence-to-sequence (seq2seq) tasks, especially neural machine translation (NMT). In the original transformer, the layer number in encoder is equal to the layer number in decoder. However, the structure is more complex and task is more difficult in decoder than those in encoder, so the layer number should not be same. In order to verify how many layer number in encoder and decoder is properly valued, we improve transformer as our model and conduct four experiments on four translation tasks of IWSLT2017. The experimental results show that the layer number in decoder should be larger than that in encoder, which can bring better translation performance.

Citation Download Citation

Dongxing Li and Zuying Luo "Analysis on the number of layers in the transformer-based model for neural machine translation", Proc. SPIE 12506, Third International Conference on Computer Science and Communication Technology (ICCSCT 2022), 1250625 (28 December 2022); https://doi.org/10.1117/12.2661982

Access the abstract

PROCEEDINGS
7 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Computer programming

Transformers

Analytical research

Data modeling

Head

Lithium

Matrices

Show All Keywords

RELATED CONTENT

The study of L2 mispronunciation detection based on Mandarin landmarks
Proceedings of SPIE (March 02 2022)

Experimental analysis of position embedding in graph transformer networks
Proceedings of SPIE (May 12 2022)

Cross corpus named entity recognition on Chinese electronic medical records...
Proceedings of SPIE (November 30 2022)

BERT-based regression model for micro-edit humor classification task
Proceedings of SPIE (October 15 2021)

Research on English translation technology based on machine learning
Proceedings of SPIE (October 19 2022)

Chinese named entity recognition incorporating syntactic information
Proceedings of SPIE (April 28 2023)

Subscribe to Digital Library

Receive Erratum Email Alert

Show All Keywords

Keywords/Phrases

Search In:

Publication Years