Yi Ting Huang

Abstract

Purpose: Advances in automatic speech recognition (ASR) and natural language processing have the potential to meaningfully impact language assessment. Despite the increasing availability of these technologies, little is known about their performance in routine assessment tasks such as language sample analysis (LSA), which is commonly used in the identification of developmental language disorder (DLD). This study examined the performance of ASR in transcribing child language samples, as well as the concurrent validity and diagnostic accuracy of metrics derived from an automated LSA system.

Method: We used an existing corpus of narrative samples from 66 children with DLD and 285 with typical language (TL), which included audio recordings and transcripts. We conducted automated LSA using the proprietary software LSA+, which integrates ASR and natural language processing. We evaluated transcription accuracy by comparing ASR transcripts to those included in the corpus. We then examined associations between language ability metrics from ASR and manual transcripts, as well as the diagnostic accuracy of each, by using LSA+ as a common means of analysis for all transcripts.

Results: ASR produced less accurate transcripts for children with DLD than for those with TL and for younger children. Despite differences in transcription accuracy, automated LSA measures exhibited positive correlations with their manual counterparts, ranging from r = .79 to .98. Manual LSA measures in isolation exhibited better diagnostic accuracy than that of their automated counterparts. However, composite models with multiple automated metrics yielded the highest accuracy when including mean length of utterance, number of different words, and total number of words, in addition to ASR-derived mean length of speech segment and pauses per word.

Discussion: Full automation of LSA demonstrates the potential to improve the efficiency and robustness of language assessment, particularly when utilizing composite measures. Despite its advantages, automated LSA may result in inaccurate estimates of language ability if used without consideration of its limitations. Continued examination of ASR and related technologies is needed to better understand their efficacy across languages, ages, and types of communication disorders.

Language acquisition research has a long tradition of including individuals with disabilities as research subjects. Numerous early works had the goal of using what was “missing” in their development to inform theories of how the system “ought” to function. More recent decades have seen a shift toward understanding individuals with disabilities as their own functional systems that are worth describing and also toward ensuring that this research benefits individuals from these communities. This shift has, to some extent, created a disconnect between work on “typical” development and on clinical populations, which have largely progressed independently of each other. The former is concentrated more in departments of Linguistics and (Cognitive) Psychology, while the latter is concentrated more in departments of Communication Sciences and Disorders (and its other associated names). Their shared past and overlapping constructs are masked by differences in goals—to understand basic acquisition processes vs. to support functional communication in those who may not fit within normative societal standards.