As shown in Fig. 2a, we begin by collecting crisis events from authoritative sources and then gather documents.

Crisis Event Collection

The first stage of corpus construction is to identify a large and diverse set of crisis events that have received sufficient media coverage and generated widespread social media discussion. We gather crisis events (2018–2025) from two sources: the EM-DAT international disaster database [17] for standardized records, and Wikipedia’s curated disaster lists for events with high public interest. After deduplication against 95 seed events from prior datasets, we obtain 368 new events.

EM-DAT International Disaster Database: Maintained by the Center for Research on the Epidemiology of Disasters (CRED), providing standardized records with verified impact metrics.

Document Collection

For each event, we collect multi-source documents from two parallel streams.

News Articles: To ensure both quality and relevance, we employ a two-phase collection strategy. We begin by crawling articles from the vetted “References” section of an event’s official Wikipedia page (e.g., the page for the “2025 Table Mountain fire”⁴4

https://en.wikipedia.org/wiki/2025_Table_Mountain_fire (accessed on 01 September 2025).

To fulfill the multifaceted requirements of the OPOR-Gen task, we perform three distinct annotation tasks (as shown in Fig. 2b): (1) Reference Annotation, to extract key factual attributes about each event; (2) Social Media Annotation, to classify the author type of each post; and (3) Timeline Annotation, to pinpoint the start and end dates of each key phase in the public opinion lifecycle. The first two tasks are automated via our protocol-guided LLM framework, while the third, more complex task is handled by human experts to ensure the highest quality.

LLM-Based Annotation Framework

To address the prohibitive expense and time of manual annotation, we develop a protocol-guided LLM framework with three key components: (1) clear label definitions, (2) detailed annotation criteria, and (3) diverse few-shot examples.

We evaluate a set of frontier LLMs on the Social Media Annotation task. We use the pre-labeled data from the CrisisLexT26 dataset [12] as the ground truth for this evaluation. Each model is prompted to classify the author type of tweets from the dataset. As shown in Table 1, gpt-4o-mini demonstrates the optimal balance between annotation quality and cost, so we select it as the primary model for our protocol-guided annotation tasks.

Reference and Social Media Annotation

In the Reference Annotation task, our framework distills key factual attributes (e.g., location, time, cause) from Wikipedia pages to produce Zi, the structured metadata used for factual evaluation. The subsequent Social Media Annotation task classifies tweet authors into Netizens or Authoritative Institutions, providing a crucial prerequisite for Event Focus section.

Human Timeline Annotation

Given the intricate nature of identifying public opinion phases, the timeline annotation is performed entirely by human experts to ensure high quality and reliability. Six in-house experts, all Master’s or PhD students familiar with public opinion lifecycles, conduct the annotation using a dedicated tool⁶6

Our dedicated annotation tool significantly streamlines the process by addressing two primary challenges: (1) providing a unified interface where annotators can view all documents and visualizations simultaneously, and (2) automatically enforcing standardized JSON output to ensure data consistency and eliminate manual formatting errors. Internal tests show this tool reduces annotation time per event by over 50%.

Volume and Length Distribution

As shown in Table 2, the OPOR-Bench provides comprehensive multi-source coverage for each crisis event. The substantial token count per event (averaging 32K+) highlights the significant information distillation challenge inherent in the OPOR-Gen task.

Geographic Distribution

Table 3 shows our dataset spans 108 countries across six continents, ensuring geographic diversity and mitigating potential regional bias.

Type and Category Distribution

Fig. 3 shows our dataset’s balanced coverage of natural disasters (44.9%) and human-caused crises (55.1%). Table 4 reveals distinct media coverage patterns across event types—“Political” events generate more news coverage while “Wildfires” trigger higher social media activity—underscoring the necessity of multi-source integration.

The OPOR-Eval Agent manages the evaluation process by analyzing report com ponents, selecting appropriate tools, and producing the comprehensive score vector. Following a reasoning-acting protocol, the agent explicitly externalizes reasoning before each action (see Appendix B for implementation details). This process is formalized as: (3)Si=Agent(Ri,Yi,Zi)=Si,fact⊕Si,opin⊕Si,solwhere ⊕ denotes vector concatenation and the components are produced by each specialized tool.

The Fact-Checker Tool (Tfact) verifies the factual accuracy of the report’s Title, Summary, and the Date Accuracy of the Event Timeline by comparing them against the reference data (Zi). This tool evaluates the model’s Factual Consistency—its ability to stay faithful to the provided source material. (4)Tfact(Ri,title,Ri,summary,Ri,timeline,Zi)=Si,fact=(si,1,si,2,…,si,7)

The Opinion-Miner Tool (Topin) evaluates the Sub-Events of the Event Timeline and the entire Event Focus section by comparing them against the source social media posts (Yi). It measures the model’s Multi-source Synthesis—extracting key insights from large volumes of noisy, unstructured text. (5)Topin(Ri,timeline,Ri,focus,Yi)=Si,opin=(si,8,si,9,si,10,si,11)

The Solution Counselor Tool (Tsol) leverages the LLM’s internal knowledge and reasoning to evaluate the report’s Event Suggestions based on criteria such as their feasibility, relevance, and innovation. This directly tests the model’s Practical Reasoning—generating novel, actionable solutions from parametric knowledge. (6)Tsol(Ri,suggestions)=Si,sol=(si,12,si,13,si,14,si,15)

We evaluate five frontier LLMs with 128 K+ context windows: GPT-4o, DeepSeek-R1, DeepSeek-V3, Gemini 2.5 Pro, and Llama-3.3-70B. For each model, we employ two distinct generation strategies (Fig. 5): modular generation creating sections independently then assembling them [19], and end-to-end generation producing complete reports in a single pass. See Appendix C for prompt templates.

We use temperature 0.7 for generation and 0.3 for evaluation tasks. All other hyperparameters follow model defaults. Complete configuration details are provided in our dataset repository.

We implement OPOR-Eval with GPT-4o and DeepSeek-V3 to evaluate the generated reports across 15 dimensions following identical protocols. Additionally, we conduct human evaluation on a subset of these reports to validate the overall effectiveness.

To validate OPOR-Eval, we conduct a two-phase human evaluation with three experts who also design the scoring criteria. Using a dedicated annotation tool ensuring blind evaluation, experts independently score each report. The protocol includes calibration and formal evaluation phases:

The calibration phase utilizes a preliminary set of 50 reports (5 events by our five baseline models under both generation strategies) to ensure all three experts share a consistent understanding of the criteria. In this phase, the experts independently rate the reports, and their agreement is measured using the Intraclass Correlation Coefficient (ICC).

In this section, we analyze the results of our human evaluation to answer two key questions: (1) How reliable are our human experts? and (2) How strong is the agreement between our human experts and the OPOR-Eval agents?

Answer 1: Our human experts demonstrate high inter-rater reliability.

Table 5 shows high inter-rater reliability among human experts. Based on the classification where Intraclass Correlation Coefficient (ICC) values are categorized as poor (<0.50), moderate (0.50–0.75), good (0.75–0.90), and excellent (>0.90), most dimensions achieve good to excellent agreement with ICC > 0.75. The relatively lower agreement on Event Suggestions (ICC = 0.64, moderate) reflects the inherent subjectivity in evaluating recommendation quality.

Answer 2: The OPOR-Eval framework achieves strong human-agent alignment with GPT-4o.

Correlation is generally categorized as weak (<0.50), moderate (0.50–0.70), or strong (>0.70). For MAE, lower values indicate better alignment. Our human-agent agreement analysis (Table 6) reveals that the GPT-4o achieves a strong overall alignment (ρ=0.69, MAE = 0.53). In contrast, DeepSeek-V3 shows moderate performance, with notably poor correlation to subjective Opinion Mining tasks (ρ=0.13). This confirms that while OPOR-Eval is effective with a strong model, human oversight remains valuable for these subjective dimensions.

Analysis of GPT-4o evaluation results (Table 7) reveals a notable weakness across all models in the Event Timeline dimension. This weakness is rooted in a dramatic failure on the Date Accuracy sub-dimension (average: 1.25). This failure is consistent even for the top-performing model (Gemini 2.5 Pro: 1.28) and stems from the task’s demand for complex temporal reasoning—identifying inflection points in data trends rather than simply extracting dates from documents.

A consistent pattern across all models (Fig. 6) suggests this relationship: performance is significantly higher for Human-caused Disasters than for Natural Disasters. Further examination at the sub-category level (Fig. 7) reveals a clearer distinction: models excel on events with well-defined information structures, like “Industrial” and “Traffic” accidents, but struggle with events characterized by diffuse information, such as “Wildfires” and “Floods”. We hypothesize that this disparity stems from the fact that human-caused disasters typically feature clear causal chains and structured data (e.g., official investigation reports) that are easily processed by LLMs. In contrast, natural disasters generate fragmented information from diverse sources with ambiguous temporal boundaries, posing a fundamental challenge to fact extraction and timeline segmentation.

Our evaluation framework evaluates three distinct capabilities: Factual Consistency (via Fact-Checker Tool), Multi-source Synthesis (via Opinion-Miner Tool), and Practical Reasoning (via Solution Counselor Tool). As shown in Table 8, model performance rankings remain remarkably stable across these three capabilities. Pearson correlations between all category pairs exceed 0.84 (p < 0.001), indicating high consistency in model performance across dimensions. This high inter-category correlation provides statistical evidence that current LLMs demonstrate coherent performance across factual verification, information integration, and strategic reasoning, rather than excelling in isolated dimensions.

Our analysis reveals that report quality does not simply increase with source document volume, highlighting a critical limitation in the information overload and inefficient multi-document synthesis of current LLMs. Since Title and Summary generation primarily relies on news articles while Timeline and Focus depend on social media data, we analyze their correlations separately.

News Articles: Table 9 shows a negative correlation between article count and the scores for both the Title and Summary, suggesting information overload. Fig. 8a identifies the optimal range: 10–20 articles.

Social Media: Table 9 shows opposing effects: Timeline accuracy benefits from more tweets (r = +0.21, p < 0.001), while Event Focus suffers dramatically (r = −0.44, p < 0.001). Fig. 8b reveals a critical threshold approximately 700 tweets where this trade-off becomes pronounced.

DeepSeek-V3 consistently assigns higher scores than GPT-4o for identical reports (average: 4.02 vs. 3.62, p < 0.001). Fig. 9a visualizes this bias: GPT-4o shows lower median scores with wider distribution, indicating more stricter criteria and better discrimination, whereas DeepSeek-V3’s scores cluster higher with less variance. This systematic difference proves that absolute scores from different LLM evaluators are not directly comparable, highlighting the need for score normalization or calibration in practical applications.

To investigate self-evaluation bias, we compare the scores an agent assigns to its own reports (“Self-Evaluation”) vs. those from other models (“Cross-Evaluation”). Our results Fig. 9b show that while self-evaluation biases are statistically significant, their practical impact is negligible: DeepSeek-V3 shows a minor self-preference (+0.03 average score, p < 0.001), while GPT-4o exhibits slight self-criticism (−0.02 average score, p < 0.05). These differences represent less than 1% of the rating scale. This key finding confirms a high degree of objectivity in these LLM evaluators, enhancing the credibility and practical viability of LLM-based evaluation systems.