TY - GEN
T1 - A Study of Pre-trained Language Models in Natural Language Processing
AU - Duan, Jiajia
AU - Zhao, Hui
AU - Zhou, Qian
AU - Qiu, Meikang
AU - Liu, Meiqin
N1 - Publisher Copyright:
© 2020 IEEE.
PY - 2020/11
Y1 - 2020/11
N2 - Pre-trained Language Model (PLM) is a very popular topic in natural language processing (NLP). It is the rapid development of pre-trained language models (PLMs) that has led to the achievements of natural language today. In this article, we give a review of important PLMs. First, we generally introduce the development history and achievements of PLMs. Second, we present several extraordinary PLMs, including BERT, the variants of BERT, Multimodal PLMs, PLMs combined with Knowledge Graph and PLMs applied to natural language generation. In the end, we summarize and look into the future of PLMs. We expect this article will provide a practical guide for learners to understanding, using and developing PLMs with the abundant literature existing for various NLP tasks.
AB - Pre-trained Language Model (PLM) is a very popular topic in natural language processing (NLP). It is the rapid development of pre-trained language models (PLMs) that has led to the achievements of natural language today. In this article, we give a review of important PLMs. First, we generally introduce the development history and achievements of PLMs. Second, we present several extraordinary PLMs, including BERT, the variants of BERT, Multimodal PLMs, PLMs combined with Knowledge Graph and PLMs applied to natural language generation. In the end, we summarize and look into the future of PLMs. We expect this article will provide a practical guide for learners to understanding, using and developing PLMs with the abundant literature existing for various NLP tasks.
KW - BERT
KW - Cross-modal
KW - Embedding
KW - KG
KW - Natural Language Generation
KW - Pre-trained
UR - https://www.scopus.com/pages/publications/85098480175
U2 - 10.1109/SmartCloud49737.2020.00030
DO - 10.1109/SmartCloud49737.2020.00030
M3 - 会议稿件
AN - SCOPUS:85098480175
T3 - Proceedings - 2020 IEEE International Conference on Smart Cloud, SmartCloud 2020
SP - 116
EP - 121
BT - Proceedings - 2020 IEEE International Conference on Smart Cloud, SmartCloud 2020
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 5th IEEE International Conference on Smart Cloud, SmartCloud 2020
Y2 - 6 November 2020 through 8 November 2020
ER -