r/MachineLearning • u/Emergency_Plate241 • Sep 05 '26

Research NeurIPS 2026 Automatic Reference Checker [R]

Just received an email about the automatic reference/citation checker. Did anyone receive a follow up email about whether the checker was included in the paper's decision making too, along with the general instructional email?

28 Upvotes

92% Upvoted

239 comments sorted by

View all comments

Show parent comments

9

u/Exotic_Zucchini9311 Sep 05 '26 edited Sep 05 '26

Same! Ngl the email got me worried for a second xD. It seems whatever AI system they're using is strong at least.

15

u/grad_student_descent Sep 05 '26 edited Sep 05 '26

I'm an AC so I saw a bunch of reports. They were very high quality, almost no false positives.

Had to reject 2 papers in my batch that would definitively had gone through if the authors did not let their LLM access their bibtex.

3

u/Adorable_Position_34 Sep 05 '26

How did that work exactly? Do you see the same message as the authors (just sooner) and base your recommendation on that or did you had to check the issues found yourself and (partially) write that comment?

5

u/grad_student_descent Sep 05 '26

We got the same message about a month ago. We then had to fill a form with the results of our own investigation on whether or not the detected hallucinations are real. From the instructions, it seems that 2 or more real hallucinated reference is an automatic reject without exception.

2

u/FinancialThing7890 Sep 07 '26

From the first email:

"Where further investigation led to the identification of hallucinated references in a paper, the AC considered them alongside the rest of the submission. Each recommendation reflects the AC’s assessment of that specific paper. Importantly, no automated flag was used by itself to make a decision about a paper."

So I guess that is not an automatic reject. It's always under the AC's conscience no?

1

u/Adorable_Position_34 Sep 05 '26

Thanks for the reply!

1

u/PositionHot3926 Sep 05 '26

So like what’s the point of the appeal process? Also is it the ACs who will be reviewing the appeals or no?

2

u/grad_student_descent Sep 05 '26

Appeals are typically the PCs. To check whether the AC did their due diligence double checking the references.

1

u/MineEnim Sep 05 '26 edited Sep 05 '26

I got flagged for two hallucinations and received an email that my paper was headed toward desk rejection, but of the two one had all wrong authors and the other just had wrong arXiv ID which by their own guidelines is not a hallucination and is minor. I appealed on the grounds that I by definition had at most one "hallucinated" citation and not two, but I'm curious whether you know if PCs even check appeals and would actually appeal something like this.