Visual speech recognition is an important research area in human-computer interaction, encompassing the complementary tasks of lip reading and sign language recognition. Lip reading, often referred to as visual speech recognition (VSR), involves decoding spoken language from visible articulatory movements of the lips, teeth, and tongue. Sign language recognition (SLR), in contrast, interprets the semantically rich combination of manual gestures, non-manual markers, and spatial grammar that characterize natural sign languages [1]. Although these tasks appear methodologically distinct, they share core computational properties that motivate a unified treatment.

The alignment between lip reading and SLR arises from deep structural similarities in how visual language is conveyed. Both modalities operate at comparable frame rates, typically 25–30 frames per second, and require models capable of handling variable-length sequences that may span hundreds of frames [2]. Both depend on fine-grained spatial control of biological articulators, whether the orofacial musculature or the hands and arms. Furthermore, both exhibit strong contextual dependence, as isolated visual patterns are often ambiguous without their temporal and linguistic surroundings, leading to viseme-level confusion in lip reading and coarticulation effects in SLR [3]. These similarities suggest that representations learned from one modality could benefit the other.

In many real-world contexts, both modalities co-occur. Proficient signers often mouth spoken words while signing, providing complementary linguistic cues [4]. Existing systems that process these channels independently disregard this inherent multimodal redundancy, leaving major gains untapped.

Current visual speech recognition techniques face several limitations that hinder practical deployment. Most studies treat lip reading and sign language recognition as separate tasks, resulting in distinct architectures, training pipelines, and evaluation protocols that limit cross-task knowledge transfer and increase computational overhead [5]. Many approaches rely on recurrent models such as LSTMs [6] to capture temporal dependencies, but these models struggle with the long-range dependencies present in continuous sign language sequences and restrict parallelization during training [7]. In addition, conventional CNN-based spatial representations often fail to adequately capture the structural relationships of hand configurations, while multimodal fusion strategies typically rely on simple operations such as concatenation or averaging, which cannot effectively model the dynamic, context-dependent interactions between lip and sign cues.

This work addresses these limitations through the following contributions: We introduce UniModal-LSR, a unified architecture that simultaneously handles lip reading and sign language recognition, enabling bidirectional knowledge transfer and shared representations across modalities. The unified architecture also reduces redundancy by 25.9% and improves generalization. To capture temporal dynamics, we design a Hierarchical Temporal-Spatial Encoder (HTSE) that combines 3D convolutions for local spatiotemporal feature extraction with multi-head self-attention for modeling long-range dependencies, enabling efficient processing of continuous visual speech sequences. We further propose a Cross-Modal Attention Fusion (CMAF) module with bidirectional cross-attention and adaptive gating to dynamically integrate lip and sign cues. A Contrastive Semantic Alignment (CSA) objective aligns modality-specific embeddings within a shared space to enhance semantic consistency and reduce modality-specific noise. Additionally, a Spatial-Temporal Graph Convolutional Network (ST-GCN) models hand skeletal structures to capture joint topology and better distinguish visually similar signs. Extensive experiments on multiple benchmarks demonstrate state-of-the-art performance, and comprehensive ablation studies quantify the contribution of each component; Table 1 further clarifies the novelty of the proposed framework.

To capture temporal patterns at multiple scales, we design a Hierarchical Temporal-Spatial Encoder (HTSE) that fuses three-dimensional convolutions for local spatiotemporal feature extraction with multi-head self-attention for modeling long-range temporal dependencies. Arranging these modules hierarchically facilitates the processing of long continuous visual-speech sequences.

The remainder of the paper is organized as follows. Section 2 situates our approach within the existing literature. Section 3 details the proposed framework. Section 4 presents an architectural and representational analysis. Section 5 describes the experimental setup. Section 6 reports empirical results. Section 7 discusses implications and limitations. Section 8 concludes the paper.

This section describes the UniModal-LSR architecture in detail. An overview is provided in Fig. 1.

This section presents an analysis of the architectural properties and representational capacity of the proposed framework. Rather than restating general theoretical results, we focus on aspects directly relevant to the CMAF and HTSE designs that justify our specific architectural choices.

Table 3 enumerates the benchmark datasets used in our evaluation. Figs. 2 and 3 show example frames from the lip reading and sign language datasets, respectively. LRS2-BBC [14], LRS3-TED [31], and GRID [32] provide sentence- and word-level lip reading benchmarks collected from BBC broadcasts, TED talks, and controlled laboratory environments. For sign language recognition and translation, PHOENIX-2014 [33] and its translation counterpart PHOENIX-2014T [1] serve as standard continuous German Sign Language benchmarks, while CSL [34] and WLASL [35] provide isolated sign datasets for Chinese and American Sign Language, respectively. Finally, How2Sign [36] enables multimodal and cross-modal experiments as it contains synchronized lip and sign annotations for the same utterances. Performance for lip reading and continuous sign language recognition is measured using Word Error Rate (WER).

Face detection uses RetinaFace (confidence 0.9), with lip ROIs extracted from facial landmarks 48−67 and resized to 88×88 pixels via affine normalization. Hand pose estimation uses MediaPipe Hands (confidence 0.7), and the fewer than 2% of failed detections are filled by linear interpolation to maintain temporal continuity. Data augmentation is applied asymmetrically: horizontal flipping is used only for sign language due to handedness constraints, while both modalities use temporal jittering (±2) frames, color jittering (±0.2), and random erasing (p=0.1). Decoding employs beam search (width 10, length normalization 0.6) without an external language model; tokenization uses character-level encoding for lip reading (28 tokens) and BPE with 1000 merges for sign language.

For reproducibility, all experiments use random seed 42, and results averaged over three runs show a standard deviation below 0.3% WER. Official dataset splits are used for all benchmarks, and the code along with pretrained models will be released upon publication to support replication and further research.

Table 4 reports Word Error Rates on lip reading benchmarks, including recent baselines from 2022–2023 for comprehensive comparison. All results are reported without external language models to ensure fair comparison. The unified model achieves the lowest WER across all evaluated datasets, with relative improvements of 5.7% on LRS2 and 5.1% on LRS3 compared to the strongest prior work.

Table 5 shows results for continuous and isolated sign language recognition. The unified model achieves state-of-the-art performance on PHOENIX-14 with 18.3% WER, representing a 13.7% relative improvement over the previous best result.

Table 6 presents BLEU scores on PHOENIX-2014T. Our model achieves 24.89% BLEU-4, representing an improvement of 2.25 points over the previous state of the art.

Table 7 analyzes each component’s contribution through cumulative addition, starting from a single-modality baseline and progressively adding each proposed component.

Table 8 provides detailed analysis of CMAF design choices, examining the impact of attention directionality and gating mechanisms.

The results confirm several design principles. Bidirectional attention outperforms unidirectional variants because both modalities contain complementary information that benefits the other. Sign-to-lip transfer provides slightly larger gains than lip-to-sign transfer, likely because sign language’s richer spatial vocabulary provides additional discriminative cues for disambiguating visually similar lip movements. Adaptive element-wise gating achieves the best performance by allowing dimension-specific modality weighting, enabling the model to selectively combine different aspects of each modality’s representation.

Table 9 shows sensitivity to the CSA loss weight λCSA, and Fig. 4 visualizes the cross-modal alignment effect using t-SNE projections of the learned embeddings.

The visualizations demonstrate that without CSA loss, lip and sign embeddings occupy distinct regions of the representation space, limiting cross-modal transfer. With the CSA loss at λCSA=0.1, semantically matched pairs from both modalities cluster together, enabling more effective information sharing. The optimal weight balances alignment strength against task-specific discriminability; larger weights improve alignment scores but begin to degrade recognition performance as the representations become overly constrained.

Table 10 evaluates performance when one modality is degraded or missing during inference, addressing concerns about robustness to partial input availability.

The model exhibits graceful degradation when one modality is impaired. Performance drops range from 3% to 14% depending on the degradation type and target task, demonstrating that cross-modal training provides implicit robustness. As expected, lip reading performance depends more heavily on the lip stream, while sign recognition depends more on the sign stream. Partial degradations such as frame dropout or occlusion have smaller effects than complete modality removal, indicating that the model can leverage whatever partial information remains available. Table 11 compares different pre-training strategies, confirming that joint multimodal pre-training provides the strongest performance gains.

Table 12 evaluates generalization by training on one dataset and testing on another without fine-tuning.

The unified model shows improved cross-domain generalization, particularly for lip reading where the 8–9% WER reduction suggests that cross-modal learning provides regularization benefits that transfer across dataset boundaries. The improvement on sign language transfer is smaller but still positive, indicating that the learned representations capture generalizable visual speech features rather than dataset-specific patterns.

Table 13 demonstrates performance under reduced training data availability, showing relative improvements of up to 19% when training with only 10% of the data.

Table 14 compares computational requirements between separate single-task models and the unified architecture. Latency measurements are end-to-end, including all preprocessing steps, conducted on a single NVIDIA A100 GPU with batch size 1.

We address practical deployment scenarios beyond the full multimodal inference setting. When only one modality is available, such as lip-only or sign-only video, the model can operate with a single branch while still benefiting from joint training. Table 15 shows that single-branch variants extracted from the unified model outperform separately trained single-task models, demonstrating that the cross-modal training provides transferable improvements even when inference uses only one modality. Table 16 presents performance of reduced-capacity variants suitable for edge deployment with limited computational resources.

Table 17 evaluates performance under reduced temporal sampling, which is relevant for bandwidth-constrained applications where transmitting full frame-rate video is impractical.

Fig. 5 decomposes errors by category. The unified model reduces errors across all categories, with notable improvements in viseme confusion (28% reduction) and boundary detection (33% reduction), which are areas where cross-modal information provides the greatest benefit.

Our work builds on the foundation established by multimodal architectures for speech-related tasks. The dual-tower framework proposed by Ge et al. [26] demonstrates the effectiveness of unified multimodal modeling for audio-text fusion in air traffic control communications, achieving improved recognition accuracy through coordinated processing of acoustic and textual information. While their approach improves multimodal speech recognition, it addresses a fundamentally different modality combination. Our CMAF module addresses the unique challenges of visual-visual fusion, where both modalities share temporal structure but differ in spatial semantics. The bidirectional cross-attention design enables more flexible information exchange than parallel tower processing, which is important when the usefulness of lip and sign cues varies across contexts. Furthermore, the adaptive gating mechanism provides context-dependent weighting that is especially suited to the dynamic relationship between visual speech modalities.

Despite encouraging results, several limitations remain. The benchmark coverage may not fully capture real-world variability in illumination, camera pose, and occlusion, and evaluation on more diverse conditions would strengthen practical applicability. The model is language-specific and must be retrained for each target spoken or signed language, highlighting the need for multilingual or language-agnostic extensions. Training is computationally expensive, requiring eight A100 GPUs during 72 h; although lightweight variants reduce inference cost, they do not address training efficiency. The framework also assumes both lip and sign modalities are available during inference, and while performance degrades gracefully when one modality is missing, explicit strategies such as modality dropout could improve robustness. Finally, the CSA loss relies on semantically aligned lip-sign pairs, and constructing such pairs across datasets may introduce noise, suggesting that future work could explore self-supervised alignment methods that enable training on unpaired data.

We introduced UniModal-LSR, a unified multimodal architecture for joint lip reading and sign language recognition. Through hierarchical temporal-spatial encoding, graph-enhanced hand modeling, cross-modal attention fusion with adaptive gating, and contrastive semantic alignment, the model achieves state-of-the-art performance while reducing parameters by 25.9% relative to separate task-specific models. Detailed ablations confirm the contribution of each component, and analysis demonstrates robustness to modality degradation and improved cross-domain generalization.

Future work will explore integration of additional modalities such as audio and full-body pose to capture a more complete picture of visual communication. Language-agnostic representation learning could enable a single model to serve multiple linguistic communities. Self-supervised pre-training methods may reduce labeled data requirements, making the technology more accessible for under-resourced languages. Integration with dialogue systems, such as the full-duplex speech dialogue schemes based on large language models, could enable more natural and responsive human-computer interaction.