RMIT student uses machine learning to decode doctors’ handwriting

RMIT student uses machine learning to decode doctors’ handwriting

An RMIT University student has helped solve a big challenge standing in the way of digitising massive Vietnamese medical records: deciphering doctors’ notoriously difficult-to-read handwriting.

Working closely with the Hospital of Tropical Disease (HTD) and Oxford University Clinical Research Unit (OUCRU), Bachelor of Software Engineering student Phung Minh Tuan successfully developed an end-to-end pipeline to recognise text on scanned Vietnamese medical records, potentially accelerating the digitisation of medical health records that the Government initiated in 2019. 

news-1-rmit-student-uses-machine-learning-to-decode-doctors-handwritingBachelor of Software Engineering student Phung Minh Tuan successfully developed an end-to-end pipeline to perform text recognition on scanned Vietnamese medical records.

Overcoming the complexities of Vietnamese handwriting

“There have been great advances in handwritten text recognition, however, most existing methods have been developed for English language and there is little-to-no methods dedicated for Vietnamese language,” shared Tuan, now in his final year with the School of Science & Technology at RMIT.

“Vietnamese handwritten text recognition is fundamentally more challenging than that of English due to the presence of more character classes, complex vocals and tonal symbols.”

It took Tuan more than three months of trial and error to figure out the most effective way to turn an image of a hard-copy medical document into an electronic version. 

news-2-rmit-student-uses-machine-learning-to-decode-doctors-handwritingAn illustration of perspective warping operation

“We tackled those challenges by enhancing various tasks in the text recognition pipeline,” he said.

“We applied a denoising process, performed text segmentation at a word level, and applied the Bigram language model to improve the probabilities of the possible correction given the neighbouring words.

“Most importantly, we combined and implemented a deep learning architecture which consists of a ResNet for feature extraction, a BiLSTM network for text sequence modelling, [both artificial neural networks], and CTC for final transcription task.

“At that point, the final output sequence aligned to a lexicon to further boost the accuracy.”

news-3-rmit-student-uses-machine-learning-to-decode-doctors-handwritingPhung Minh Tuan (pictured left) and RMIT School of Science & Technology lecturer and his supervisor Dr Dinh Ngoc Minh (pictured right).

A door has been opened to a better treatment process

RMIT School of Science & Technology lecturer and Tuan’s supervisor Dr Dinh Ngoc Minh emphasised the promising pipeline results, which can ultimately facilitate the digital transformation of medical centres and hospitals in Vietnam, to improve their readiness for adopting modern electronic health record management systems.

“The proposed work can accelerate the digitisation process of medical health record systems,” Dr Minh said.

“With the help of the machine in processing all the records, health facilities could gradually shift to an electronic system without any sudden change in protocol.

“Such systems would also allow remote medical centres or field health workers with limited computer access to continue with paper systems which could then be digitised easily.”

Dr Minh said that the patient records could be shared simply between departments, which could help reduce unnecessary tests and optimise treatment, to eventually improve health care quality.

“And most importantly, Tuan’s work can generate a digital medical-note dataset for a variety of potential medical machine-learning solutions,” Dr Minh said.

“In fact, our collaborators HTD and OUCRU are planning to use the generated data to develop expert diagnostic systems, improve the treatment process and minimalise errors in healthcare practices.

“Tuan has already proven himself as a professional software developer, and a quick-learner who has expanded his machine learning knowledge significantly during the six months of working on this project.”

Through this work, Tuan has been offered an internship at the OUCRU, and his work has been presented at the ACIS2020, an A-ranked international conference, the AHT conference, as well as the University’s recent student showcase.

Story: Hoang Ha

08 December 2020

Share

  • Research
  • Student projects
  • Engineering
  • Community
Engage

Related news

ai-training-ethinic-minority-students-1200x800.jpg

MOET and RMIT Vietnam successfully conclude AI skills development program for over 5,400 ethnic minority students
Icon / Small / CalendarCreated with Sketch. 10 Nov 2026

Vietnam’s Ministry of Education and Training (MOET) and RMIT University Vietnam have successfully wrapped up the AI skills development program for ethnic minority students on 7 November 2026.

nine people standing behind a table smiling

RMIT Sustainability Week 2026: building capabilities, driving impact
Icon / Small / CalendarCreated with Sketch. 10 Nov 2026

RMIT Sustainability Week 2026 creates a space for learning, connection and hands-on experiences where individuals can develop their sustainability capabilities in ways that align with their strengths and aspirations.

Two people standing in front of a Scholarship Award Ceremony 2026 backdrop, with one holding a framed certificate and award.

RMIT Vietnam announces VND183 billion scholarships fund for future students and researchers
Icon / Small / CalendarCreated with Sketch. 06 Nov 2026

The 2027 Scholarship program comprises 106 undergraduate and Master’s scholarships worth 66 billion VND, alongside 50 fully funded PhD scholarships worth approximately 117 billion VND to support the next generation of researchers and innovators.

two newlyweds

Later marriage: a new stage of adulthood?
Icon / Small / CalendarCreated with Sketch. 04 Nov 2026

People across many countries are marrying later than ever. RMIT University academic Dr Gordon Ingram argues that this reflects a shift in how young people experience the transition to adulthood.

Load more +