As mentioned in our paper, all samples in the dataset were created by humans. However, human
annotators can occasionally make mistakes due to workload, oversight, or other factors. To
date, we are aware of only a few (two at the moment) problematic samples in the dataset.
To ensure a fair comparison among all participating teams, we have decided to keep the dataset
unchanged at this time. If you encounter any errors, please don't hesitate to inform us.
If you encounter such ambiguous claims, please refer to the use_context field in
our dataset. This field can have three values: no (no additional context
required), yes (requires the context field for disambiguation), or
other sources (requires the full paper for disambiguation). Based on our
experience creating and reviewing the dataset, we have found that most ambiguous claims can be
clarified using the context information (either the provided short paragraph or the full
paper).
No, there is no limitation on how many submissions you can make.
No, we do not provide a leaderboard before the run submission deadline (July 19th). However, on our website
(Evaluations & Baselines), you can compare your
results against the baseline scores. When you upload a new submission, our evaluation bot (EvalBot) will send out an e-mail
to your full team containing the results (or an error message if something went wrong). You are encouraged to upload more
runs to further improve your scores but you cannot see other participant's scores before the deadline.
We publish a full leaderboard after the run submission deadline.
Yes, after you submit your file, our evaluation bot (EvalBot) sends your team an e-mail containing your results.
If you haven't received an e-mail after 30min, please send an e-mail to the organizers (sciclaimeval (at) gmail.com) as there was likely an error during the evaluation.
We deployed an automated evaluation bot (EvalBot) that automatically verifies and evaluates your result, but it may take up to 30min for you to receive the e-mail. The e-mail will have the subject 'SciClaimEval Submission Update' and
come from sciclaimeval@gmail.com. If you haven't received an e-mail after 30min, please make sure you entered the correct Group ID assigned to your team. If you do not have a group ID yet, you need to register for the task first to obtain one.
If you are sure you entered the correct Group ID, please send an e-mail to the organizers (sciclaimeval (at) gmail.com) as there was likely an error during the evaluation.
The best run on the primary metric (task 1: pair accuracy, task 2: accuracy) decides your team rank versus other team's best submission regardless of the submission date/order.
However, we will still show and analyse all other submissions if they differ sufficiently (for example, different architecture/model differs sufficiently; hyperparameter tuning does not).
Hence, we encourage all teams to explain each submission during the upload in the Google form and explore different ideas. The final leaderboard will contain all submissions eventually.
Information regarding the paper template and formal requirements (including the page limit and
formatting guidelines) is available on the NTCIR-19 submission instructions page. Please refer
to the
NTCIR-19 Paper Submission Instructions
for details.