Multimodal Representation Learning for Blastocyst Assessment

Youcheng Wang; Zhe Zheng; Na Ni; Guoqing Tong; Nuo Cheng; Kai Li; Ping Yin; Yuanyuan Chen; Yingna Wu; Guangping Xie

doi:10.1109/ISBI53787.2023.10230468

Multimodal Representation Learning for Blastocyst Assessment

Youcheng Wang
, Zhe Zheng
, Na Ni
, Guoqing Tong
, Nuo Cheng
, Kai Li
, Ping Yin
, Yuanyuan Chen
, Yingna Wu
, Guangping Xie

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

Blastocyst selection based on morphology grading is crucial in in vitro fertilization (IVF) treatment. Several research studies based on convolutional neural networks (CNNs) have been reported to select the most viable blastocyst automatically. In this paper, we propose a multimodal representation learning framework in which the text description is firstly streamed as a complementary supervision signal to enrich the visual information. Moreover, we redefine the blastocyst assessment problem to an image-text retrieval task to solve the data imbalance. The experimental results show that the performance metrics, e.g., accuracy, outperform the unimodal classification (+1.5%) and image retrieval counterparts (+1.2%), which demonstrates our proposed model's effectiveness.

Original language	English
Title of host publication	2023 IEEE International Symposium on Biomedical Imaging, ISBI 2023
Publisher	IEEE Computer Society
ISBN (Electronic)	9781665473583
DOIs	https://doi.org/10.1109/ISBI53787.2023.10230468
State	Published - 2023
Externally published	Yes
Event	20th IEEE International Symposium on Biomedical Imaging, ISBI 2023 - Cartagena, Colombia Duration: 18 Apr 2023 → 21 Apr 2023

Publication series

Name	Proceedings - International Symposium on Biomedical Imaging
Volume	2023-April
ISSN (Print)	1945-7928
ISSN (Electronic)	1945-8452

Conference

Conference	20th IEEE International Symposium on Biomedical Imaging, ISBI 2023
Country/Territory	Colombia
City	Cartagena
Period	18/04/23 → 21/04/23

Keywords

Blastocyst Assessment
Image-text Retrieval
Multimodal Representation Learning
Visual Transformer

Access to Document

10.1109/ISBI53787.2023.10230468

Cite this

Wang, Y., Zheng, Z., Ni, N., Tong, G., Cheng, N., Li, K., Yin, P., Chen, Y., Wu, Y., & Xie, G. (2023). Multimodal Representation Learning for Blastocyst Assessment. In 2023 IEEE International Symposium on Biomedical Imaging, ISBI 2023 (Proceedings - International Symposium on Biomedical Imaging; Vol. 2023-April). IEEE Computer Society. https://doi.org/10.1109/ISBI53787.2023.10230468

@inproceedings{f26a71be146342069689561a7b20e723,

title = "Multimodal Representation Learning for Blastocyst Assessment",

abstract = "Blastocyst selection based on morphology grading is crucial in in vitro fertilization (IVF) treatment. Several research studies based on convolutional neural networks (CNNs) have been reported to select the most viable blastocyst automatically. In this paper, we propose a multimodal representation learning framework in which the text description is firstly streamed as a complementary supervision signal to enrich the visual information. Moreover, we redefine the blastocyst assessment problem to an image-text retrieval task to solve the data imbalance. The experimental results show that the performance metrics, e.g., accuracy, outperform the unimodal classification (+1.5\%) and image retrieval counterparts (+1.2\%), which demonstrates our proposed model's effectiveness.",

keywords = "Blastocyst Assessment, Image-text Retrieval, Multimodal Representation Learning, Visual Transformer",

author = "Youcheng Wang and Zhe Zheng and Na Ni and Guoqing Tong and Nuo Cheng and Kai Li and Ping Yin and Yuanyuan Chen and Yingna Wu and Guangping Xie",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 20th IEEE International Symposium on Biomedical Imaging, ISBI 2023 ; Conference date: 18-04-2023 Through 21-04-2023",

year = "2023",

doi = "10.1109/ISBI53787.2023.10230468",

language = "英语",

series = "Proceedings - International Symposium on Biomedical Imaging",

publisher = "IEEE Computer Society",

booktitle = "2023 IEEE International Symposium on Biomedical Imaging, ISBI 2023",

}

Wang, Y, Zheng, Z, Ni, N, Tong, G, Cheng, N, Li, K, Yin, P, Chen, Y, Wu, Y & Xie, G 2023, Multimodal Representation Learning for Blastocyst Assessment. in 2023 IEEE International Symposium on Biomedical Imaging, ISBI 2023. Proceedings - International Symposium on Biomedical Imaging, vol. 2023-April, IEEE Computer Society, 20th IEEE International Symposium on Biomedical Imaging, ISBI 2023, Cartagena, Colombia, 18/04/23. https://doi.org/10.1109/ISBI53787.2023.10230468

Multimodal Representation Learning for Blastocyst Assessment. / Wang, Youcheng; Zheng, Zhe; Ni, Na et al.
2023 IEEE International Symposium on Biomedical Imaging, ISBI 2023. IEEE Computer Society, 2023. (Proceedings - International Symposium on Biomedical Imaging; Vol. 2023-April).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Multimodal Representation Learning for Blastocyst Assessment

AU - Wang, Youcheng

AU - Zheng, Zhe

AU - Ni, Na

AU - Tong, Guoqing

AU - Cheng, Nuo

AU - Li, Kai

AU - Yin, Ping

AU - Chen, Yuanyuan

AU - Wu, Yingna

AU - Xie, Guangping

PY - 2023

Y1 - 2023

N2 - Blastocyst selection based on morphology grading is crucial in in vitro fertilization (IVF) treatment. Several research studies based on convolutional neural networks (CNNs) have been reported to select the most viable blastocyst automatically. In this paper, we propose a multimodal representation learning framework in which the text description is firstly streamed as a complementary supervision signal to enrich the visual information. Moreover, we redefine the blastocyst assessment problem to an image-text retrieval task to solve the data imbalance. The experimental results show that the performance metrics, e.g., accuracy, outperform the unimodal classification (+1.5%) and image retrieval counterparts (+1.2%), which demonstrates our proposed model's effectiveness.

AB - Blastocyst selection based on morphology grading is crucial in in vitro fertilization (IVF) treatment. Several research studies based on convolutional neural networks (CNNs) have been reported to select the most viable blastocyst automatically. In this paper, we propose a multimodal representation learning framework in which the text description is firstly streamed as a complementary supervision signal to enrich the visual information. Moreover, we redefine the blastocyst assessment problem to an image-text retrieval task to solve the data imbalance. The experimental results show that the performance metrics, e.g., accuracy, outperform the unimodal classification (+1.5%) and image retrieval counterparts (+1.2%), which demonstrates our proposed model's effectiveness.

KW - Blastocyst Assessment

KW - Image-text Retrieval

KW - Multimodal Representation Learning

KW - Visual Transformer

UR - https://www.scopus.com/pages/publications/85172142305

U2 - 10.1109/ISBI53787.2023.10230468

DO - 10.1109/ISBI53787.2023.10230468

M3 - 会议稿件

AN - SCOPUS:85172142305

T3 - Proceedings - International Symposium on Biomedical Imaging

BT - 2023 IEEE International Symposium on Biomedical Imaging, ISBI 2023

PB - IEEE Computer Society

T2 - 20th IEEE International Symposium on Biomedical Imaging, ISBI 2023

Y2 - 18 April 2023 through 21 April 2023

ER -

Multimodal Representation Learning for Blastocyst Assessment

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this