The audio component ai(t) is processed by an automatic speech recognition model fASR(⋅), implemented using Whisper. The model generates a set of Ni(a) speech segments:(2)𝒮i(a)={(sij(a),τijstart,τijend)}j=1Ni(a)where sij(a) denotes the transcribed text and (τijstart,τijend) are the corresponding temporal boundaries. Equivalently, each segment is obtained via the mapping:(3)fASR:ai(t)↦(sij(a),τijstart,τijend)providing temporally localized linguistic evidence aligned with the spoken narrative of the video.

The visual stream Ii(t) is sampled to obtain a set of representative keyframes:(4)𝒦i={Ii(tik)}k=1Ni(v)where the sampling instants {tik} are determined either by scene-change detection or by uniform temporal intervals Δt. Each keyframe is converted into a natural-language description using a vision–language captioning model fVLM(⋅), instantiated here as Gemini. Video transcripts are generated using Whisper, while visual captions are produced using Gemini. The resulting set of visual sentences is:(5)𝒮i(v)={(sik(v),tik)}k=1Ni(v)where sik(v)=fVLM(Ii(tik)) encodes the semantic content of frame Ii(tik) at time tik.

In the next step, the two modalities are temporally synchronized into a unified sequence ordered by time:(6)𝒮i=sortt(𝒮i(a)∪𝒮i(v))={(sim,τimstart,τimend)}m=1Liwhere sorting ensures chronological coherence. This unified representation 𝒮i is then used for subsequent preprocessing, semantic chunking, and embedding steps.

Each segment sim is normalized using the function s~im=fnorm(sim), which removes non-linguistic tokens (e.g., [music], [applause]), repeated punctuation, and capitalization inconsistencies. Segments with ‖s~im‖< εlen (shorter than a minimum token threshold) are discarded. Moreover, duplicate or near-duplicate sentences are eliminated using a cosine-similarity filter.

Long ASR sentences are split into smaller parts using punctuation and grammar rules. These parts are then combined again when they are temporally close and syntactically relevant.

If ℬ(s~im)={s~im(1),…,s~im(r)} denote the boundary refinement operator producing r refined sub-sentences, the updated sequence after boundary refinement is (8)𝒮i′=⋃m=1Liℬ(s~im)

In Eq. (8), 𝒮i′ combines both audio and visual textual items. Next, a temporal alignment function falign is used to ensure consistent ordering and gap filling. The function is defined as:(9)𝒮i′′=falign(𝒮i′)={(sip,tipstart,tipend,ηip)}p=1Piwhere ηip∈{audio,visual} indicates the source modality.

In this step, audio transcripts and visual captions are fused at the text level by aligning modality-specific segments based on timestamps and combining them into a single sequence while preserving temporal order. We define a temporal overlap threshold θt to resolve any potential conflicts between modalities. The temporal overlap between two segments is defined as:(10)overlap(sip,siq)=max(0,min(tipend,tiqend)−max(tipstart,tiqstart))

If the overlap between an audio and visual segment exceeds θt, the segments are considered temporally redundant. In such cases, the visual segment is either suppressed or merged into the corresponding audio segment, with priority given to the audio transcript.

Alternative fusion strategies, such as early feature-level fusion or late fusion of independently processed modalities, were considered. However, early fusion increases model complexity and requires cross-modal embedding alignment, while late fusion may lead to fragmented or inconsistent context across modalities. In contrast, the proposed temporally aligned text-level fusion provides a simple, interpretable, and computationally efficient mechanism that preserves both semantic coherence and temporal structure.

Consecutive sentences in 𝒮i′′ that are semantically related and temporally contiguous are merged into higher-level units, termed semantic chunks. Let ϕsem(⋅) denote a semantic affinity function between two adjacent sentences:(11)ϕsem(sip,si(p+1))=hip⊤hi(p+1)‖hip‖‖hi(p+1)‖where hip are contextual embeddings from a pretrained sentence encoder.

We introduce a semantic similarity threshold θsem to control chunk formation, and a temporal gap threshold δgap to enforce temporal continuity. The threshold δgap defines the maximum allowable time difference between two consecutive sentences for them to be considered temporally adjacent. Two sentences are merged if ϕsem(sip,si(p+1))>θsemandti(p+1)start−tipend<δgap,ensuring both semantic coherence and temporal continuity.

The merged chunk cin is defined as (12)cin=⨁p=pspesip,tinstart=tipsstart,tinend=tipeendwhere ⨁ denotes string concatenation preserving sentence order. The resulting set of chunks 𝒞i forms the temporally grounded, semantically consolidated representation of video vi.

After pre-processing, each video vi is represented by 𝒞i={(cin,tinstart,tinend)}n=1Ni, where each cin corresponds to a normalized, semantically coherent textual unit which is associated with a temporal interval within the video. This structured representation serves as input to the embedding and indexing stage, as discussed in Section 4.3.

The embedding model femb(⋅) is instantiated using a pre-trained Bidirectional Generalized Embedding (BGE) encoder, denoted as BGEbase. For each textual input cin composed of Lin tokens, the encoder produces token-level hidden states Hin=[hin(1),…,hin(Lin)], which are mean-pooled to obtain the chunk-level embedding:(14)zin=1Lin∑ℓ=1Linhin(ℓ)

The embeddings are then ℓ2-normalized:(15)z^in=zin‖zin‖2

The embedding dimension d is fixed (d=768 for BGEbase).

All normalized embeddings from the dataset are aggregated into a single matrix: Z=[z^11,z^12,…,z^in,…]⊤∈RM×d, where M=∑i=1NNi is the total number of chunks across all videos. Each embedding z^in is associated with metadata μin={video_id=i,chunk_id=n,tinstart,tinend}, linking the vector to its source and temporal span.

For efficient approximate nearest-neighbor (ANN) search, we employ the FAISS (Facebook AI Similarity Search) library to construct an index ℐ:(16)ℐ=FAISS.build(Z,metric=cosine)where FAISS organizes the embeddings using an inverted-file index with product quantization (IVF-PQ) for sub-linear retrieval. This yields a mapping: findex:z^in↦ℐ, allowing rapid lookup of semantically similar vectors.

Given a query embedding z^q=femb(q) derived from a textual question q, semantic similarity between the query and a candidate chunk z^in is computed using cosine similarity:(17)sim(z^q,z^in)=z^q⊤z^in

We define a retrieval similarity threshold θr to characterize relevance, and only those chunks with similarity greater than θr are selected as semantically relevant candidates. For large-scale retrieval, FAISS efficiently estimates the top-k nearest neighbors:ℛq(k)=TopK(z^in)[sim(z^q,z^in)],where k denotes the number of retrieved candidates.

The resulting set is a ranked list of chunk identifiers and corresponding similarity scores:(18)ℛq(k)={(i,n,simin)∣simin=z^q⊤z^in}top-k

Each retrieved embedding in ℛq(k) retains its temporal and source metadata μin. This structure allows downstream modules to apply temporal constraints during the temporal-aware retrieval phase. The representation and indexing phase produces a searchable, semantically rich vector space, given by:(19)𝒟index={(z^in,μin)}i,n=1Ni

This vector space 𝒟index serves as the knowledge base for efficient, timestamp-aware video question answering.

Each user question q is first encoded into a dense query vector, given by:(20)z^q=femb(q),‖z^q‖2=1using the same embedding function femb(⋅) defined in Section 4.3. An initial candidate set is obtained from the FAISS index by retrieving the top-K semantically nearest neighbors:(21)ℛq(K)=TopK(z^in)[sim(z^q,z^in)],sim(z^q,z^in)=z^q⊤z^in

The resulting retrieval set ℛq(K)={(ik,nk,simiknk)}k=1K contains both the similarity scores and corresponding temporal metadata μiknk.

To exploit temporal structure and maintain coherence, a timestamp-based filtering function ftemp(⋅) is applied to ℛq(K). The parameter Δt represents a temporal tolerance window that expands the query-specific interval to account for minor misalignment between annotated timestamps and retrieved segments. (22)ℛqtemp={(ik,nk)∈ℛq(K)|ik=j, [tiknkstart,tiknkend]∩[tqstart−Δt,tqend+Δt]≠∅}

The system then applies local temporal smoothing to group adjacent chunks belonging to the same video:(23)ℛqsmooth=merge(ℛq(K);δt)where merge(⋅) is used to aggregate consecutive results whose time gaps are below δt. This temporal-aware filtering ensures that retrieved evidence is not only semantically relevant but also temporally contiguous and contextually stable.

We refine the top-M candidates (M<K) of the retrieval using a cross-encoder scoring model fce(⋅,⋅). This cross-encoder scoring model evaluates the pairwise relevance between the query q and each candidate chunk cin:(24)rin=fce(q,cin)where rin is a scalar relevance score in [0,1]. The final ranked set is:(25)ℛq∗=sortrin({(i,n,rin)∣(i,n)∈ℛqtemp})

This set is ordered by descending rin. fce is implemented using a transformer-based cross-encoder fine-tuned on QA relevance pairs. This re-ranking step yields high-precision evidence selection, improving over embedding-only retrieval.

From the top-L elements of ℛq∗, a compact context window ℰq is constructed:(26)ℰq={(c~ℓ,t~ℓstart,t~ℓend)}ℓ=1L,(c~ℓ,t~ℓstart,t~ℓend)∈ℛq∗

The corresponding text snippets are concatenated into a temporally ordered context:(27)Cq=⨁ℓ=1L[[t~ℓstart,t~ℓend]:c~ℓ]

An instruction-tuned LLM fLLM(⋅,⋅) then generates a grounded natural-language answer:(28)aq=fLLM(q,Cq)subject to the grounding constraint that all information used in aq must originate from Cq.

This constraint instructs the LLM to act primarily as a reasoning engine by restricting answer generation to the retrieved video-derived context.

Fig. 2 summarizes the structured input formulation used in the proposed framework.

For Open-ended questions, we used standard text-based and semantic evaluation metrics. Token-level precision, recall, and F1 are computed as:(29)Precision=|Ty∩Ty^||Ty^|,Recall=|Ty∩Ty^||Ty| (30)F1=2PRP+Rwhere Ty and Ty^ denote the sets of tokens in the ground-truth and generated answers, respectively.

The next used metric is BLEU score which measures n-gram overlap between y and y^ and is defined as:(31)BLEU=BP⋅exp(∑n=14wnlog⁡pn),wn=14where BP is the brevity penalty and pn denotes modified n-gram precision.

The metric METEOR emphasizes recall and incorporates a penalty for fragmented matches:(32)METEOR=Fα(1−Ppen),Fα=10PRR+9Pwhere P and R denote precision and recall, respectively. The penalty term Ppen accounts for fragmented alignments and is defined as:(33)Ppen=γ(chm)βwhere m is the number of matched unigrams, ch is the number of contiguous chunks in the alignment, and γ and β are tunable parameters (γ=0.5, β=3).

Next, ROUGE-1 is used, which measures unigram recall:(34)ROUGE-1=|Ty∩Ty^||Ty|where Ty and Ty^ denote the reference and generated unigram sets, respectively.

To account for paraphrasing and semantic equivalence beyond surface-form overlap, we compute cosine similarity between sentence embeddings:(35)Sim=ey⋅ey^‖ey‖2‖ey^‖2where embeddings ey and ey^ are generated using all-MiniLM-L6-v2.

Multiple-choice questions (MCQs) are treated as a classification task, since each question contains four answer options with exactly one correct choice. Accordingly, we report objective classification metrics. The accuracy measures the proportion of correctly selected options as:(36)Accuracy=NcorrectNtotal

For MCQs, precision, recall, and F1 are computed over the predicted and ground-truth option labels, providing a balanced view of classification performance, particularly under varying difficulty levels.

In summary, text-generation and semantic metrics (F1, BLEU, METEOR, ROUGE, and semantic similarity) are used exclusively for open-ended questions. In contrast, classification metrics (Accuracy, Precision, Recall, and F1) are used exclusively for multiple-choice questions.

Table 4 shows the results of open-ended QA for LLaVA-1.6 and the proposed framework. It can be seen that the performance of both systems decreases as the difficulty level of the questions increases. However, the proposed framework consistently outperforms LLaVA-1.6 on all metrics and difficulty levels. The proposed Temporal-Aware RAG improves F1 from 15.30% to 23.52% and ROUGE-1 from 21.70% to 29.76%, indicating more faithful overlap with the reference answers and better coverage of the key facts required by the questions.

LLaVA-1.6 exhibits moderate semantic similarity and a relatively low F1 and ROUGE-1. This suggests that LLaVA-1.6 can often infer the question’s general topic and produce broadly plausible answers. However, it frequently fails to respond to the specific lecture segment where the correct information is stated. This behavior is consistent with the absence of an explicit evidence-selection mechanism. The model implicitly searches over long transcripts and sparsely sampled visual context, which increases the likelihood of generating generic, underspecified, or partially correct answers.

In contrast, the proposed framework demonstrates higher similarity and significantly improved lexical and coverage-based scores. These results suggest that temporally filtered retrieval and re-ranking not only identify conceptually relevant evidence but also enable the generator to produce answers that more closely align with the reference content.

Table 5 reports MCQ performance. Since each MCQ contains four answer options with exactly one correct choice, we treat MCQs as a classification task and report Accuracy (ACC), Precision, Recall, and F1-score.

For MCQs, the proposed Temporal-Aware RAG framework achieves consistently higher performance. The accuracy, precision, recall, and F1 values were higher for our method than for LLaVA-1.6. The performance of LLaVA-1.6 decreases steadily from “Simple” to “Very Hard” questions. This shows that it struggles when questions require careful distinction between similar options or when information must be combined from different parts of the lecture. This happens because LLaVA-1.6 uses only a few keyframes and a full, unstructured transcript. As a result, it is easily affected by irrelevant information and by details that appear far apart in time.

On the other hand, the proposed framework shows a gradual decline as difficulty rises, but it still performs better than LLaVA-1.6. These results show that explicit retrieval, temporal filtering, and re-ranking make systems more robust, even when questions require detailed reasoning or distributed evidence.

Overall, the results across both open-ended and MCQ settings demonstrate that explicit, timestamp-aware evidence selection is essential for lecture-style VideoQA. While LLaVA-1.6 provides a strong general-purpose multimodal baseline, the proposed Temporal-Aware RAG framework yields more accurate and reliable answers by combining multimodal cues with structured retrieval, temporal constraints, and re-ranking.

We next compare the proposed Temporal-Aware RAG framework against standard retrieval-based baselines that progressively introduce multimodal evidence while explicitly excluding temporal reasoning. This controlled comparison isolates the contribution of temporal grounding beyond retrieval and multimodality alone.

Text-only RAG (no temporal awareness): A FAISS index is constructed over semantically chunked Whisper ASR transcript segments, and the top-k chunks are retrieved using embedding similarity. The generator produces answers using only the retrieved transcript evidence, without visual captions or timestamp constraints.

Multimodal RAG (no timestamps): ASR transcript chunks and Gemini-generated visual caption chunks are jointly indexed in a single retrieval space. Top-k chunks are retrieved by similarity without enforcing any timestamp constraints, allowing retrieved evidence to be semantically relevant but temporally misaligned.

Table 6 reports results for open-ended question answering. The results show that retrieval based on ASR transcripts alone is not robust as question difficulty increases. While “Simple” questions achieve 24.37% F1, performance drops sharply for “Hard” (9.01%) and “Very Hard” (3.89%) questions, with a corresponding decline in semantic similarity. This indicates that similarity-based retrieval over transcripts struggles to retrieve precise, question-specific evidence for complex queries.

Incorporating visual captions substantially improves semantic alignment across all difficulty levels, increasing overall similarity from 48.24% to 61.00%. This highlights the complementary value of slide content and on-screen information. However, without timestamp constraints, multimodal RAG remains vulnerable to temporal drift and can retrieve conceptually related but contextually incorrect segments.

To assess the contribution of each individual component in the proposed framework, an ablation study was conducted on open-ended questions. The ablation study evaluates four model variants, each created by removing one key component: visual captions, temporal filtering, the cross-encoder re-ranker, or normalization.

Table 7 presents the ablation results. It can be seen that removing the temporal filter from the framework had the greatest impact on performance, as the overall F1 score drops to 11.40 and the semantic similarity to 36.67. This performance decline is especially more evident for “Very Hard” questions. The decline in performance is due to the retriever becoming less effective at localizing relevant segments without temporal constraints.

The ablation study after removing the cross-encoder re-ranker shows that the performance in the simple category remains relatively stable (F1 = 31.15). However, for the “Hard” and “Very Hard” questions, a relatively sharp decline is observed. This indicates that, without re-ranking, the framework struggles with hard questions due to its inability to prioritize the most relevant evidence.

When visual captions were removed, a moderate decline in performance was observed as the overall F1 score decreased to 16.40, and semantic similarity fell to 55.80. Again, the decline is more evident in the more difficult categories. This indicates that visual captions are an important clue to answer complex questions.

Finally, the ablation study after removing the normalization component also resulted in a moderate performance decrease. The “Very Hard” category shows a more substantial decrease in semantic similarity (to 28.48). These results suggest that normalization supports consistency in more challenging scenarios.