Welcome to SciClaimEval, a pilot task on the verification of scientific claims against tables and figures from scientific articles.
The task is organized as part of NTCIR-19 and aims to evaluate systems that can reliably check the truthfulness of scientific statements using multi-modal evidence.
Scientific claim verification involves determining whether claims made in research papers are supported or refuted by accompanying evidence, such as experimental results, tables, and figures. With the rapid rise of generative AI and large language models (LLMs), the volume of scientific submissions has increased substantially, creating a growing demand for tools that can assist reviewers in assessing the validity and consistency of paper claims.
The SciClaimEval pilot task focuses on cross-modal scientific claim verification, aiming to assess whether textual claims in scientific papers are adequately supported by evidence from diverse modalities, namely tables and figures. We introduce a new benchmark dataset, constructed by extracting claims and their corresponding evidence from scientific articles across multiple domains, including biomedicine, machine learning, and natural language processing. The figure below illustrates an example from the benchmark dataset. The task involves determining whether a claim is supported or refuted given a piece of evidence (a table or a figure) and optional contextual information (preceding text in the original paragraph).
The task dataset will be published in three rounds. First, we publish a development dataset (dev set) in order to let everyone explore parts of the data on January 31. Second, the formal test dataset will be released only for task participants in March. Participants are required to submit their results on this formal run dataset! Following the NTCIR conference, we make all data subsequently publicly available by the end of 2026.
This task includes two subtasks. Participants can submit solutions to either or both subtasks.
Dataset
The data consists of JSON files of all claims alongside figures (PNG), tables (PNG), and the full paper texts (JSON).
Each claim contains a unique path to an evidence file (evi_path for subtask 1 or evidence_id_1/evidence_id_2 for subtask 2) and evi_type that indicates the type of evidence (either table or figure).
Additionally, we provide contextual information, including the caption (caption), immediate context of the claim (context), and a path to the full paper content (paper_path). The use_context field indicates whether additional context is necessary to potentially disambiguate the claim. Specifically, use_context contains either no (no additional context required), yes (requires the context field for disambiguation), or other sources (requires the full paper for disambiguation).
Optional: For tables, we provide access to the original source format of the table (LaTeX or HTML). Participants are allowed to additionally submit results for subtask 1 on these formats besides the mandatory PNG formats. evi_path_original can be used as a unique path to the original variant.
Subtask 1: Claim Label Prediction Task
In this subtask, you predict if a given claim (text) is either Supported or Refuted by the given evidence (tables and figures in PNG format).
Prediction Format: All participants of subtask 1 are required to submit a results file in this format. The claim_id matches the claim_id in the data. The pred_label (prediction label) contains either Supported or Refuted. If your solution produces a confidence score for the prediction, you can add the optional score field to the submission. Any other fields will be ignored. A submission on other formats (see optional description above) will be distinguished in the submission form and not the result format.
Test Format: Example of a test entry for subtask 1.
[{"paper_id":"2403.19137","claim_id":"val_tab_0001","claim":"Table 1 shows that our probabilistic inference module consistently outperforms its deterministic counterpart in terms of Avg and Last accuracy.","caption":"Table 1 : Performance comparison of different methods averaged over three runs. Best scores are in bold . Second best scores are in blue . The results for L2P, DualPrompt, and PROOF are taken from [ 92 ] . See App. Table 14 for std. dev. scores.","evi_type":"table","evi_path":"tables_png/dev/val_tab_0001.png","evi_path_original":"tables/dev/val_tab_0001.tex","context":"To understand our probabilistic inference modules further, we examine their performance against the deterministic variant of ours (Ours w/o VI).","domain":"ml","use_context":"yes","paper_path":"papers/dev/ml_2403.19137.json","license_name":"CC BY 4.0","license_url":"http://creativecommons.org/licenses/by/4.0/"}]
Subtask 2: Claim Evidence Prediction Task
In this subtask, you predict which of the two given pieces of evidence (tables and figures) supports the claim (text).
Prediction Format: All participants of subtask 2 are required to submit a results file in this format. The sample_id matches the sample_id in the data. The pred_label (prediction label) contains either evidence_id_1 or evidence_id_2 depending on which evidence supports the claim. As for subtask 1, participants can provide an additional score flag to indicate the model’s confidence.
Test Format: Example of a test entry for subtask 2.
[{"sample_id":"val_0071","evidence_id_1":"figures/dev/val_fig_0113.png","evidence_id_2":"figures/dev/val_fig_0114.png","claim":"As shown in Figure 4(b) , increasing the value of \\alpha can prevent the model from outputting more sensitive information, but it may also lead to the loss of necessary information.","context":"For unlearning, we found that adjusting the value of \\alpha can serve as a balance between forgetting and retaining .","caption":"(a) Impact on instruction tuning; (b) Impact on unlearning; Impact of strength coefficient \\alpha on performance","domain":"ml","evi_type":"figure","paper_id":"2410.17599","use_context":"other sources","paper_path":"papers/dev/ml_2410.17599.json","license_name":"CC BY 4.0","license_url":"http://creativecommons.org/licenses/by/4.0/"}]
All submissions will be evaluated on precision, recall, macro F1, and accuracy. In order to minimize the risk of model bias on subtask 1, the primary evaluation metric here is accuracy on claim pairs (a claim pair are two entries in the dataset with the same claim but opposing evidence labels). This stricter metric only counts correct results if both entries of a pair were correctly predicted (i.e., the supported claim and refuted claim of the same claim text were correctly identified).
Results
The following tables show the results of all submissions on the evaluation set. In each table, only the best submission is shown directly. To see all run submissions from the same team, click on the specific row. Approaches and notes descriptions were given by the teams. All tables show the team names and baselines.
Subtask 1 was sorted by the primary metric ‘pair accuracy’ while subtask 2 was sorted by the primary metric ‘accuracy’.
Subtask 1
Team Name
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy ▾
SciTrue▸
98.0
98.0
98.0
98.0
98.2
Approach
Notes
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy
Claude Fable ...
Claude Fable 5 (PNG-only)
98.0
98.0
98.0
98.0
98.2
routed ensemb...
routed ensemble + GPT-5.5 + Claude Fable 5 + GPT-5.6 Sol + pair prior (PNG-only, leak-free)
pipline run purely on table image with random seed 22
93.0
93.1
93.0
93.0
94.0
Black Socks▸
94.1
93.7
93.8
93.8
90.0
Approach
Notes
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy
Gemma4 31B Th...
Gemma4 31B Think High + Pairwise Inference
94.1
93.7
93.8
93.8
90.0
Gemma4 Think ...
Gemma4 Think High + CoT
86.7
86.6
86.6
86.6
74.5
Pro²Crast▸
87.2
87.2
87.2
87.2
86.8
Approach
Notes
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy
v7_rev2
87.2
87.2
87.2
87.2
86.8
v7_rev2
87.2
87.2
87.2
87.2
86.8
v7.5
87.5
87.5
87.5
87.5
86.6
v6_rev2
85.2
85.2
85.2
85.2
85.2
v7.5
supported
86.2
86.1
86.2
86.2
84.3
v5_zuy_cot_re...
v5_zuy_cot_revertlowconf_5ctxtwindow
79.0
78.9
78.9
79.0
61.1
base
73.8
73.7
73.7
73.8
52.8
University of Southampton
85.8
85.9
85.8
85.9
85.9
Denso IT Laboratory▸
74.1
74.1
74.0
74.1
75.9
Approach
Notes
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy
evi=image, co...
evi=image, context=no
74.1
74.1
74.0
74.1
75.9
baseline + pa...
baseline + pairwise estimation
68.2
68.3
68.2
68.2
69.7
baseline
59.8
58.0
55.3
57.0
17.6
TakasuLab
86.7
86.4
86.5
86.5
73.8
SEBIS TUM
72.7
72.8
72.7
72.7
72.9
VerifyAI
85.3
85.4
85.3
85.3
72.2
ProvideQ▸
84.8
84.9
84.9
84.9
71.1
Approach
Notes
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy
Table:HTML pa...
Table:HTML parsed from PNG with chandra+target CoT+qwen3.6:35B, Figure: PNG+two round prompting+stage one：qwen3-vl:32b,stage2:qwen3.6-plus
84.8
84.9
84.9
84.9
71.1
Table:HTML pa...
Table:HTML parsed from PNG with chandra+target CoT+qwen3.6:35B, Figure: PNG+two round natural prompting+qwen3-vl:32b
82.6
82.6
82.5
82.6
66.9
Table:HTML pa...
Table:HTML parsed from PNG with chandra+target CoT improve version+qwen3-max, Figure: PNG+two round natural prompting+qwen3-vl:32b
82.2
82.2
82.1
82.1
65.5
able:HTML par...
able:HTML parsed from PNG with chandra+target CoT improve version+qwen3.6:35B, Figure: PNG+two round natural prompting+qwen3-vl:32b
81.3
81.3
81.1
81.1
64.1
HTML that par...
HTML that parsed by chandraOCR+qwen3.6:35B, two round figure +qwen3vl:8B
80.5
80.5
80.5
80.6
62.0
Table:HTML pa...
Table:HTML parsed from PNG with chandra+target CoT+qwen3.6:35B, Figure: PNG+two round prompting+qwen3-vl:32b
80.5
80.5
80.5
80.6
62.0
Table:Qwen3:´...
Table:Qwen3:´8B+HTML parsed from OCR, two round prompting for figure
71.9
72.0
71.9
72.0
47.2
SIB Text Mining▸
81.3
81.3
81.1
81.1
64.1
Approach
Notes
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy
SIBTM_T1_R01
Custom prompt...
Custom prompt Qwen3-VL-32B-Instruct:
we started from the baseline prompt, and then iteratively refined it looking at the mismatches.
81.3
81.3
81.1
81.1
64.1
Qwen3-VL-32B-...
Qwen3-VL-32B-Instruct + fine-tuning
QLoRA fine-tu...
QLoRA fine-tuning of Qwen3 on automatically-perturbed claim-evidence pairs derived from SIBiLS/PMC OA tables and figures
80.0
79.9
79.7
79.7
62.0
Baseline Qwen...
Baseline Qwen3-VL-32B-Instruct
Baseline eval...
Baseline evaluation of the claim label prediction task (Task 1) on the test split, without fine-tuning, providing a reference point for future runs
78.3
78.2
77.9
77.9
58.3
Baselines▸
80.4
79.5
79.8
79.4
61.1
Approach
Notes
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy
o4-mini
80.4
79.5
79.8
79.4
61.1
Qwen3-VL-30B-...
Qwen3-VL-30B-A3B
72.5
70.7
71.5
70.6
46.5
Qwen3-VL-8B
75.8
65.8
70.3
65.7
44.4
Qwen3-VL-4B
72.8
67.1
69.8
67.1
43.1
InternVL3_5-3...
InternVL3_5-38B
71.8
68.8
68.5
69.5
42.4
InternVL3_5-1...
InternVL3_5-14B
69.5
66.2
65.4
67.1
36.3
InternVL3_5-8...
InternVL3_5-8B
67.0
64.0
62.9
64.9
35.0
Llama-3.2-11B...
Llama-3.2-11B-Vision
56.7
51.8
48.4
53.2
14.6
SciNex▸
77.5
77.5
77.4
77.4
57.4
Approach
Notes
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy
subtask1-one-...
subtask1-one-stage-png
-
77.5
77.5
77.4
77.4
57.4
subtask1-two-...
subtask1-two-stage-png
-
75.5
75.4
75.4
75.5
56.0
CAISALab
62.3
59.8
58.2
60.8
26.9
Team Name
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy ▾
SciTrue▸
98.1
98.1
98.1
98.1
98.4
Approach
Notes
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy
Single-model ...
Single-model verification with Claude Fable 5. Each evidence item (claim + caption + context + a structured markdown rendering of the official table data, for tables + the evidence image, read directly from disk via the Claude Code CLI's Read tool) is scored by Fable 5, which returns a calibrated support score in [0,1]. We then apply a legal pair prior: claims are grouped by identical claim text (a field visible at test time) to recover the Supported/Refuted contrastive pairs, and within each pair the higher-scoring evidence is labeled Supported, the lower Refuted; ties are broken by the continuous score, never by item ordering. Unpaired (singleton) claims are labeled by Fable 5's absolute judgment.
98.1
98.1
98.1
98.1
98.4
SciTrue-T1-En...
SciTrue-T1-Ensemble+GPT5.5+PairPrior
97.3
97.4
97.3
97.3
97.7
Training-free...
Training-free ensemble with evidence-type routing and a legal pair prior. Four models — Claude Opus 4.8, Gemma-4-31B, GPT-5.5, and Claude Fable 5 — score every item (claim + caption + context + table markdown/LaTeX source for tables + evidence image); GLM-4.6V is added to the mix for figure-type evidence, where models are weaker on average. Scores are averaged within the routed model set, then the same legal pair prior described above (recover pairs by claim text, rank by fused score, leak-free tie-breaking) converts the fused scores into final Supported/Refuted labels.
96.9
96.9
96.9
96.9
97.2
Description: ...
Description: A training-free system built from five multimodal models with full test coverage (Claude Fable 5, GPT-5.5, Gemma-4-31B, Claude Opus 4.8, GLM-4.6V-Flash). Each model returns a JSON verdict with a support score in [0,1] under one fixed fact-checker prompt (no chain-of-thought, no examples). We exploit the fact that a claim's evidence type (evi_type) is visible at test time and apply evidence-type routing, fusing a different, per-type-optimal subset of models by signed-sum of support scores: Table claims → Fable 5 + GPT-5.5 + Gemma-4-31B + Opus 4.8 Figure claims → GPT-5.5 + Opus 4.8 (figures reward only the strongest few; pooling weaker models dilutes them) We then apply the legal pair prior: pairs are recovered from identical claim text (a visible field — never the hidden claim_id_pair), and within each recovered pair the higher-scoring member is labeled Supported, the other Refuted. This turns two absolute judgments into one relative ranking. The method is leak-free: ties are broken by the continuous fused score, never by row position; the ~45 unpaired supported-only claims are labeled Supported. On the development set (352 pairs) this routed pair prior reaches 96.9 leak-free pair-accuracy, above the fixed-routing baseline (96.2). All operations use only test-visible fields, so the gains are expected to transfer.
image input + stg1 data extraction + stg2 optional code execution + gemma4 31b think high
79.9
79.9
79.9
79.9
62.7
Pro²Crast▸
83.6
83.7
83.6
83.7
83.6
Approach
Notes
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy
v6
83.6
83.7
83.6
83.7
83.6
v7
83.0
83.0
83.0
83.0
82.6
TakasuLab▸
85.0
84.9
84.9
85.0
71.3
Approach
Notes
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy
Gemma4 Pipeli...
Gemma4 Pipeline
Commit ID: 3e...
Commit ID: 3ece2e4
85.0
84.9
84.9
85.0
71.3
Gemma4 Pipeli...
Gemma4 Pipeline
Commit ID: 3e...
Commit ID: 3ece2e4
85.1
84.8
84.9
85.0
70.6
Qwen3-VL-8B-I...
Qwen3-VL-8B-Instruct Pipeline
Commit ID: 3e...
Commit ID: 3ece2e4
71.2
70.6
70.8
70.8
44.9
SciNex▸
78.3
78.1
77.8
77.8
59.7
Approach
Notes
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy
subtask1-one-...
subtask1-one-stage
78.3
78.1
77.8
77.8
59.7
subtask1-two-...
subtask1-two-stages
75.9
75.5
75.5
75.7
56.0
subtask1-one-...
subtask1-one-stage-thinking
73.0
73.0
73.0
73.1
53.2
CAISALab▸
65.7
64.7
64.4
65.2
32.6
Approach
Notes
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy
baseline_phi4...
baseline_phi4_qwenvl
65.7
64.7
64.4
65.2
32.6
baseline_phi4...
baseline_phi4_14B_qwen_vl
65.7
64.7
64.4
65.2
32.6
Team Name
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy ▾
SciTrue▸
98.1
98.1
98.1
98.1
98.4
Approach
Notes
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy
Single-model ...
Single-model verification with Claude Fable 5. Each evidence item (claim + caption + context + a structured markdown rendering of the official table data, for tables + the evidence image, read directly from disk via the Claude Code CLI's Read tool) is scored by Fable 5, which returns a calibrated support score in [0,1]. We then apply a legal pair prior: claims are grouped by identical claim text (a field visible at test time) to recover the Supported/Refuted contrastive pairs, and within each pair the higher-scoring evidence is labeled Supported, the lower Refuted; ties are broken by the continuous score, never by item ordering. Unpaired (singleton) claims are labeled by Fable 5's absolute judgment.
98.1
98.1
98.1
98.1
98.4
SciTrue-T1-En...
SciTrue-T1-Ensemble+GPT5.5+PairPrior
97.3
97.4
97.3
97.3
97.7
Training-free...
Training-free ensemble with evidence-type routing and a legal pair prior. Four models — Claude Opus 4.8, Gemma-4-31B, GPT-5.5, and Claude Fable 5 — score every item (claim + caption + context + table markdown/LaTeX source for tables + evidence image); GLM-4.6V is added to the mix for figure-type evidence, where models are weaker on average. Scores are averaged within the routed model set, then the same legal pair prior described above (recover pairs by claim text, rank by fused score, leak-free tie-breaking) converts the fused scores into final Supported/Refuted labels.
96.9
96.9
96.9
96.9
97.2
Description: ...
Description: A training-free system built from five multimodal models with full test coverage (Claude Fable 5, GPT-5.5, Gemma-4-31B, Claude Opus 4.8, GLM-4.6V-Flash). Each model returns a JSON verdict with a support score in [0,1] under one fixed fact-checker prompt (no chain-of-thought, no examples). We exploit the fact that a claim's evidence type (evi_type) is visible at test time and apply evidence-type routing, fusing a different, per-type-optimal subset of models by signed-sum of support scores: Table claims → Fable 5 + GPT-5.5 + Gemma-4-31B + Opus 4.8 Figure claims → GPT-5.5 + Opus 4.8 (figures reward only the strongest few; pooling weaker models dilutes them) We then apply the legal pair prior: pairs are recovered from identical claim text (a visible field — never the hidden claim_id_pair), and within each recovered pair the higher-scoring member is labeled Supported, the other Refuted. This turns two absolute judgments into one relative ranking. The method is leak-free: ties are broken by the continuous fused score, never by row position; the ~45 unpaired supported-only claims are labeled Supported. On the development set (352 pairs) this routed pair prior reaches 96.9 leak-free pair-accuracy, above the fixed-routing baseline (96.2). All operations use only test-visible fields, so the gains are expected to transfer.
96.0
96.0
96.0
96.0
95.6
task1_routed_...
task1_routed_gptfable_SUBMIT
96.0
96.0
96.0
96.0
95.4
Bonn-Juelich Informatics▸
96.7
96.7
96.7
96.7
97.7
Approach
Notes
Precision
Recall
Macro-F1
Accuracy
Pair Accuracy
random seed 2...
random seed 22 + riskscore threshold 5 + HITL
96.7
96.7
96.7
96.7
97.7
random seed 2...
random seed 22 + riskscore threshold 5.5 + HITL fixed
95.3
95.4
95.4
95.4
96.3
random seed 2...
random seed 22 + risk score threshold 5.5 + HITL
95.1
95.2
95.1
95.1
96.1
random seed 2...
random seed 22+risk threshold 0.6 + HITL
94.7
94.7
94.7
94.7
95.6
Hybrid task 1...
Hybrid task 1 with task 2 predictions
92.9
93.0
92.9
92.9
93.8
top 3 majorit...
top 3 majority voting
93.1
93.2
93.2
93.2
93.8
Best performi...
Best performing pipeline with random seed 23
92.9
93.0
92.9
92.9
93.5
Best performi...
Best performing pipeline with random seed 25
92.9
93.0
92.9
92.9
93.5
Parser fix
92.7
92.8
92.7
92.7
93.5
Ensemble resu...
Ensemble results with random seed 23-24-26
92.9
93.0
92.9
92.9
93.5
Ensemble resu...
Ensemble results with random seed 22-24-26
92.7
92.7
92.7
92.7
93.3
Task 2 no cor...
Task 2 no correction process
92.5
92.5
92.5
92.5
93.3
Best performi...
Best performing pipeline with random seed 26
92.6
92.6
92.6
92.6
93.3
This submissi...
This submission uses the updated Task 2 conflict tiebreaker with explicit original-vs-swapped diagnosis
92.2
92.3
92.3
92.3
93.1
Best performi...
Best performing pipeline with random seed 24
92.2
92.3
92.3
92.3
92.8
Best performi...
Best performing pipeline with random seed 27
91.9
92.0
91.9
91.9
92.6
SciClaim-Reas...
SciClaim-Reasoner, a modular claim verifi- cation framework built on top of Qwen3.6-35B-A3B
Direct pairwise evidence selection with a single model. Claude Fable 5 is shown the claim (with caption and context) and both candidate evidence images side by side, and selects the one that supports the claim, returning a confidence score for each option. No ensembling is used, since Subtask 2 is already a relative-comparison task.
98.4
SciTrue-T2-GP...
SciTrue-T2-GPT5.5
97.7
Score-weighte...
Score-weighted combination of two strong single models for pairwise evidence selection. GPT-5.5 and Claude Fable 5 each independently choose which of two evidence images supports the claim, with Fable 5 additionally providing continuous per-image confidence scores. The two models' signals are summed (GPT-5.5's discrete choice contributes ±1, Fable 5 contributes its score difference), and the evidence item with the higher combined score is selected — so the two models agree on the large majority of items, and Fable 5's confidence arbitrates the small number of disagreements.
97.7
Bonn-Juelich Informatics▸
98.2
Approach
Notes
Accuracy
purly table i...
purly table image seed 22 + threshold 5 + HITL
98.2
purly table i...
purly table image seed 22 + threshold 5 + HITL
97.9
purly table i...
purly table image seed 22 + threshold 5.5 + HITL
96.5
purly table i...
purly table image seed 22 + threshold 6 + HITL
96.1
pipline run p...
pipline run purely on table image with random seed 22
Apply Task 1 to each image (SIBTM_T1_R01). If no definitive verdict can be reached—because both are refuted or both are confirmed—call the model with both images. In this case, use one prompt for images and a separate prompt for tables.
90.5
Qwen3-VL-32B-...
Qwen3-VL-32B-Instruct + fine-tuning + prompting
Step-by-step ...
Step-by-step reasoning prompt to support the model used for SIBTM_T2_R03
80.8
Qwen3-VL deci...
Qwen3-VL decider + chart/table advisor
ChartGemma an...
ChartGemma and PaddleOCR act as advisors, extracting figure interpretations and table structure respectively, which Qwen uses to make the final decision
80.1
Qwen3-VL-32B-...
Qwen3-VL-32B-Instruct + fine-tuning
QLoRA fine-tu...
QLoRA fine-tuning of Qwen3 on automatically-perturbed claim–evidence pairs derived from SIBiLS/PMC OA tables and figures.
72.2
Baseline Qwen...
Baseline Qwen3-VL-32B-Instruct
Baseline eval...
Baseline evaluation of the evidence selection task (Task 2) on the test split, without fine-tuning, providing a reference point for future runs
65.3
SciNex▸
88.2
Approach
Notes
Accuracy
subtask2-two-...
subtask2-two-stage-png
-
88.2
subtask2-one-...
subtask2-one-stage-png
-
83.6
Baselines▸
81.9
Approach
Notes
Accuracy
o4-mini
81.9
Qwen3-VL-8B
56.2
Qwen3-VL-30B-...
Qwen3-VL-30B-A3B
53.5
InternVL3_5-1...
InternVL3_5-14B
52.5
InternVL3_5-3...
InternVL3_5-38B
52.1
InternVL3_5-8...
InternVL3_5-8B
51.6
InternVL3_5-1...
InternVL3_5-1B
46.1
Qwen3-VL-4B
41.2
Llama-3.2-11B...
Llama-3.2-11B-Vision
35.9
LMU & TUM NLP▸
81.5
Approach
Notes
Accuracy
2step_op_8tab...
2step_op_8tabpng_json_extraction_fallback_30fig
81.5
2step_op_8tab...
2step_op_8tabpng_indep_desc_30fig
78.7
2step_op_refr...
2step_op_reframed_antibias_8tab30fig
70.4
Pro²Crast▸
80.8
Approach
Notes
Accuracy
v3.5
label 2
80.8
v8
80.3
v7
80.1
v3_rev2
80.1
v3.5
default label...
default label 1
78.9
v3.5
label 1 impro...
label 1 improved
77.1
v9
fixed
76.2
v6
75.9
v3
improved
75.9
v5
75.7
v3.5
label 2 impro...
label 2 improved
75.2
v2_rev2
74.5
v4
default label...
default label 1
68.5
v4
label 2
66.9
v1-png
60.9
v9
50.2
Denso IT Laboratory
61.3
SEBIS TUM
60.6
Team Name
Accuracy ▾
TakasuLab
93.5
SciNex
89.3
Black Socks
84.5
LMU & TUM NLP▸
83.6
Approach
Notes
Accuracy
2step_op_8tab...
2step_op_8tabjson_json_extraction_fallback_30fig
83.6
2step_op_8tab...
2step_op_8tabjson_indep_desc_30fig
80.8
2step_op_8tab...
2step_op_8tabjson_reframed_antibias_30fig
78.0
2step_op_8tab...
2step_op_8tabjson_reframed_antibias_30fig_others
76.8
VerifyAI▸
66.7
Approach
Notes
Accuracy
extractor_eva...
extractor_eval_test
66.7
EXTRACTOR-EVA...
EXTRACTOR-EVALUATOR method
0.0
Extractor2Eva...
Extractor2Evaluator_RENAMED
0.0
Pro²Crast▸
66.7
Approach
Notes
Accuracy
v2
66.7
v3
63.4
Denso IT Laboratory
65.5
Team Name
Accuracy ▾
Bonn-Juelich Informatics▸
97.7
Approach
Notes
Accuracy
random seed 2...
random seed 22 + riskscore threshold 5 + HITL
97.7
random seed +...
random seed + riskscore threshold 5.5 + HITL fixed
96.3
random seed 2...
random seed 22 + risk score threshold 5.5 + HITL
96.1
random seed 2...
random seed 22+risk threshold 0.6 + HITL
95.6
Hybrid task 1...
Hybrid task 1 with task 2 predictions
93.8
top 3 majorit...
top 3 majority voting
93.8
Best performi...
Best performing pipeline with random seed 23
93.5
Best performi...
Best performing pipeline with random seed 25
93.5
parser fix
93.5
Ensemble resu...
Ensemble results with random seed 23-24-26
93.5
Ensemble resu...
Ensemble results with random seed 22-24-26
93.3
Task 2 no cor...
Task 2 no correction process
93.3
Best performi...
Best performing pipeline with random seed 26
93.3
SciClaim-Reas...
SciClaim-Reasoner, a modular claim verifi- cation framework built on top of Qwen3.6-35B-A3B
93.1
This submissi...
This submission uses the updated Task 2 conflict tiebreaker with explicit original-vs-swapped diagnosis
93.1
Best performi...
Best performing pipeline with random seed 24
92.8
Best performi...
Best performing pipeline with random seed 27
92.6
News
[2026-08-03] All run results are now available. Congratulations and thanks to all participants. Paper submissions instructions have been added.
[2026-07-21] Submissions are now closed. Thanks to all participants. Remember the paper submission.
[2026-06-17] EvalBot is now live. It automatically evaluates your submission and informs you via e-mail about your results within 30min.
All deadlines are 11.59 pm UTC -12h (Anywhere on Earth (AoE)).
Registration for Participation
The registration is now closed.To participate in the SciClaimEval task, participants must (1) register via the 19th NTCIR online registration system and (2) submit a signed memorandum.
All participants are required submit a paper explaining their approaches by 1st September.
Further information are available on research.nii.ac.jp/ntcir/ntcir-19/papers. Please use the template provided. Also note the checklist before submission:
Your paper must include the Task Name SciClaimEval
Your paper must include your Team Name (i.e., your Group Name as shown in the result tables above)
You need to reference the SciClaimEval Task Overview Paper (a bibtex will be provided shortly after)
Please direct any questions or corrections regarding the task to: sciclaimeval (at) gmail.com
Cite Us
To cite this work, please use the following BibTeX.
We will update the URL and page numbers once the official LREC 2026 proceedings are released.
@inproceedings{ho-etal-2026-sciclaimeval,title={SciClaimEval: Cross-modal Claim Verification in Scientific Papers},author={Ho, Xanh and Wu, Yun-Ang and Kumar, Sunisth and Xia, Tian Cheng and Boudin, Florian and Greiner-Petter, Andre and Aizawa, Akiko},booktitle={Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026)},month={May},year={2026},pages={11060--11071},address={Palma, Mallorca, Spain},publisher={European Language Resources Association (ELRA)},editor={Piperidis, Stelios and Bel, Núria and van den Heuvel, Henk and Ide, Nancy and Krek, Simon and Toral, Antonio},doi={10.63317/4ap9rg2gnwmf}}
Organizers
Akiko Aizawa (National Institute of Informatics, Japan)
SciClaimEval
