Benchmarking Automated Parts Of Speech Tagging Across 52 Languages For Multilingual Clinical Language Analysis

Loretta Gasparini, Daniel Shaw, Thayabaran Kathiresan, Devon Borysiewicz, Geoffrey W. Stuart, Cross-linguistic POS Consortium, Adam P. Vogel (Presenter)

Twenty-five language varieties have promising POS tagging capabilities. This warrants replication using longer samples. We propose next steps for the 11 languages with unclear results and the 16 with poor performance.

INTRODUCTION

Part-of-speech (POS) tagging using NLP models could aid analysis of impaired language for diagnosis and monitoring. But the accuracy of available POS taggers has been reported with varied methods and limited transparency. This study aimed to develop a standard methodology across languages for evaluating POS taggers and report initial results.

METHODS

Linguists tagged POS across 52 language varieties (see Figure 1). Cohen’s kappa for IRR was calculated if two linguists were available. Tokenization and POS tags were compared with NLP models (MCC and 95% CIs).

RESULTS

25 languages had strong performance (>80% IRR and MCC). 11 yielded unclear results and 16 had relatively poor performance.

CONCLUSIONS

Languages with strong results warrant replication using longer samples. Improved models or ground truth are needed for certain languages. Our methods and data are transparently reported to facilitate next steps.