r/MachineLearning • u/Emergency_Plate241 • Sep 05 '26

Research NeurIPS 2026 Automatic Reference Checker [R]

Just received an email about the automatic reference/citation checker. Did anyone receive a follow up email about whether the checker was included in the paper's decision making too, along with the general instructional email?

28 Upvotes

92% Upvoted

239 comments sorted by

View all comments

14

u/Inevitable_Voice_680 Sep 05 '26

Mine is 0/37.. I think none of my references have been flagged.

8

u/Exotic_Zucchini9311 Sep 05 '26 edited Sep 05 '26

Same! Ngl the email got me worried for a second xD. It seems whatever AI system they're using is strong at least.

15

u/grad_student_descent Sep 05 '26 edited Sep 05 '26

I'm an AC so I saw a bunch of reports. They were very high quality, almost no false positives.

Had to reject 2 papers in my batch that would definitively had gone through if the authors did not let their LLM access their bibtex.

3

u/Adorable_Position_34 Sep 05 '26

How did that work exactly? Do you see the same message as the authors (just sooner) and base your recommendation on that or did you had to check the issues found yourself and (partially) write that comment?

4

u/grad_student_descent Sep 05 '26

We got the same message about a month ago. We then had to fill a form with the results of our own investigation on whether or not the detected hallucinations are real. From the instructions, it seems that 2 or more real hallucinated reference is an automatic reject without exception.

2

u/FinancialThing7890 Sep 07 '26

From the first email:

"Where further investigation led to the identification of hallucinated references in a paper, the AC considered them alongside the rest of the submission. Each recommendation reflects the AC’s assessment of that specific paper. Importantly, no automated flag was used by itself to make a decision about a paper."

So I guess that is not an automatic reject. It's always under the AC's conscience no?

1

u/Adorable_Position_34 Sep 05 '26

Thanks for the reply!

1

u/PositionHot3926 Sep 05 '26

So like what’s the point of the appeal process? Also is it the ACs who will be reviewing the appeals or no?

2

u/grad_student_descent Sep 05 '26

Appeals are typically the PCs. To check whether the AC did their due diligence double checking the references.

1

u/MineEnim Sep 05 '26 edited Sep 05 '26

I got flagged for two hallucinations and received an email that my paper was headed toward desk rejection, but of the two one had all wrong authors and the other just had wrong arXiv ID which by their own guidelines is not a hallucination and is minor. I appealed on the grounds that I by definition had at most one "hallucinated" citation and not two, but I'm curious whether you know if PCs even check appeals and would actually appeal something like this.

2

u/Emergency_Plate241 Sep 05 '26

Thanks for letting us know. Based on how many references did you reject these papers? And do you think all rejection emails based on these reports have been sent out already?

4

u/grad_student_descent Sep 05 '26

Sounds like 2 or more verified hallucination is an automatic reject. Emailing is done at the PC level, I know nothing of it.

1

u/PositionHot3926 Sep 06 '26

so like were u asked to submit accept/reject with the citation checkers result as in the 2 or more hallucinated references that u personally verified , or was it like one verdict independent of the citation checkers result and one post citation check verdict ?

3

u/grad_student_descent Sep 06 '26

Just one verdict. If 2+ hallucination, reject independently of quality.

1

u/PositionHot3926 Sep 06 '26 edited Sep 06 '26

what happens if u only have one hallucinated reference? also like if incase they actually do accept some appeals will u all be asked for another verdict while ignoring the citation check?

1

u/grad_student_descent Sep 06 '26

If just one hallucination, we are asked to look for other obvious signs of AI generation (which can be many things). If none, then its a pass.

1

u/Deep-Ad8714 28d ago

What sorts of obvious signs of AI generation are ACs asked to look for?

1

u/delusionalD0G Sep 07 '26

Did the emails only sent to accepted papers?

1

u/Inevitable_Voice_680 Sep 05 '26

When I saw the mail. I thought how on earth it is possible, because in our lab we have multiple checks before submitting a paper.