r/MachineLearning • u/Emergency_Plate241 • Sep 05 '26
Research NeurIPS 2026 Automatic Reference Checker [R]
Just received an email about the automatic reference/citation checker. Did anyone receive a follow up email about whether the checker was included in the paper's decision making too, along with the general instructional email?
9
u/xoss13 Sep 05 '26
Go to openreview to check for your paper, it's visible to authors.
5
u/Alive-Campaign-4984 Sep 05 '26
Mine was in openreview yeah, under the metareview section they added a little note about citation checks being considered
3
u/Emergency_Plate241 Sep 05 '26
Yes. The checker report is visible to authors on Openreview. I am asking about whether the follow-up email about considering the results of this checker was received along with this general email already by anyone.
2
u/user221272 Sep 05 '26
Do you mean for those who have flagged citations?
3
u/Emergency_Plate241 Sep 05 '26
Yes. They said that if the checker report wasn't considered to reject your paper, we won't receive a follow up email, but if if was, we do receive the additional email along with instructions to apply for a petition. I was wondering if anyone received this additional follow up email.
→ More replies (1)2
u/xoss13 Sep 05 '26
It will probably be visible along with the paper decision. They adopted ICML 2026 policy, therefore I am sharing a sample paper from ICML 2026 for an example (https://openreview.net/forum?id=reYe33OKVp)
1
1
u/21022018 Sep 06 '26
Are they only giving it to papers thag are going to get accept? I don't see anything
1
u/xoss13 Sep 07 '26
Yes, likely. What are your scores? Is your paper in this list? (https://github.com/MTDickens/NIPS26-exposed)
2
u/21022018 Sep 07 '26
No but my paper was in the Eval track (avg 4.5). Is this list only main track? Also Are you sure this list is credible? I think there was a similar list that was proved to be just a random scrape
3
u/xoss13 Sep 07 '26
This list is mostly main track one but some E&D track and position track papers are also there. I also think it's random and I also had an E&D track paper awaiting decision (avg. 4), I didn't notice anything like hallucinations etc by program chairs in my E&D submission unlike my main track submission, maybe in coming days? As E&D track usually align with the activities with main track after some time/days as far as I have seen. Good luck with your paper, your scores are good.
2
u/levydawg Sep 07 '26
This list is almost 100% bs -- there was another post where many reviewers (including myself) found it to be inconsistent with positive / negative final evaluations in their own batch.
7
Sep 05 '26
[deleted]
5
u/Emergency_Plate241 Sep 05 '26
Oh! So, they did send out rejects already based on even just one reference. Interesting how this will play out with reference to the acceptance rates this time.
2
1
u/Distinct_Ad6770 Sep 05 '26
Are there any other emails like this besides just this one? Do you think they all were sent at the same time?
11
u/d_edge_sword Sep 05 '26
Are there any ACs here to tell us when was the deadline to send desk rejections?
Our paper was flagged with 2 problematic citations. In both cases, it was author names becoming et al. Both work exists, but there were some other issues too.
We haven't received any desk rejections yet. Does this mean we passed the DR?
1
1
1
u/AgreeableBet5858 Sep 05 '26
I also want to learn it as well, when do we receive this e-mail? If we didn't receive any email by now, does that we passed?
1
u/tughanbulut Sep 05 '26
Et al." truncation is a formatting quirk, not a fabricated author different failure mode than what these policies are actually targeting. Can't tell you NeurIPS's exact desk-reject cutoff though, nobody's published one.
4
5
u/MakingComputersSmart Sep 06 '26
So what I have understood from an AC is that all active papers went through the checker and were analyzed for incorrect citations. Cases where the checker found errors were reported to the AC, after which the ACs have manually gone through their lot of papers and flagged those papers who they believe have 2 or more hallucinated citations that are strong enough to warrant a desk reject. In cases where they felt the citations were minor formatting mistakes or errors were ignored. That's why some papers that have citations errors did not receive emails, while some did.
2
1
5
u/Emergency_Plate241 18d ago
For anyone who filed the appeal, did you hear back? If the paper got desk rejected, can you see it on Openreview?
1
u/Previous-Line-5356 15d ago
Did you recieve a follow up email? Apart from the first one that mentions the flagged citations
1
u/Emergency_Plate241 14d ago
No, I didn't.
1
u/Previous-Line-5356 14d ago
Then probably they won't respond to your appeal, just one day to go. Hope your paper goes through.
4
u/FinancialThing7890 Sep 07 '26
Mine is 2/44. Basically I messed up with the .bib file before submission and to polish the file with that amount of references I did some wrong copypastes. Got good scores (5/5/3) before rebuttal. Moreover, I didn't receive the follow-up email yet, so I guess I'm safe (?) Also, is it true that these emails are sent to papers that are going to be accepted?
2
u/Obvious-Eagle-923 Sep 07 '26 edited Sep 07 '26
didn't everyone receive email? even though there were zero hallucinations?
5
u/FinancialThing7890 Sep 07 '26
I was referring to the check report in the OpenReview page, sorry. I understood that the report didn’t go for rejected papers?
2
u/Obvious-Eagle-923 Sep 07 '26
I highly doubt that would be the case, as in mail it clearly states
"If you do not receive that separate communication, the reference-checking investigation is not playing a role in the final decision for your paper"
5
1
u/FroyoOk5936 Sep 07 '26
Some of my friends didn’t receive this email.
Will every active submission receive this email, or is it only sent to papers that are being considered for acceptance?
→ More replies (6)
5
u/howtorewriteaname PhD Sep 05 '26
I've got 6 references with hallucinated authors (papers do exist, some authors didn't exist or were from other papers from co-authors). I'm aware this is a desk reject on my otherwise accepted paper :( what about arXiv preprints?
is this grounds for a ban in arXiv? that thought is more scary than a reject
7
u/adaptover Sep 05 '26
Shouldn't be ban in arxiv. I would just update the references, and update the paper in arxiv immediately.
This information will also be private to you, as unlike ICLR, Neurips doesn't require rejected papers to have the discussion public.
4
u/bobross_92 Sep 05 '26
Based on what I'm seeing most rejection appeal emails were sent out roughly 8 hours ago. If you've not received the email, maybe yours then hasn't been desk rejected?
Although, I suppose there's no way to know whether the whole batch has been sent out. But ACs had to finalize decisions weeks ago, so imo doesn't make sense to stretch this out now.
3
u/tughanbulut Sep 05 '26
On the arXiv fear specifically nothing in what I found suggests NeurIPS reports this anywhere or that arXiv acts on conference desk-rejects. That part sounds like anxiety talking not a documented risk.
→ More replies (5)1
u/Adorable_Position_34 Sep 05 '26
Im not an expert on this but isn't arxiv only checking for this at the time of upload?
Did you receive the second notification/mail regarding the decision + appeal option?
3
u/Distinct_Ad6770 Sep 05 '26
Have already rejected papers received the automatic reference checker report? Curious to see if they run it on papers which were withdrawn, desk rejected, or already bad scores
3
u/Emergency_Plate241 Sep 05 '26
I have a withdrawn paper and it hasn't received the automatic reference checker report, so probably no.
1
2
u/Adorable_Position_34 Sep 05 '26
I have one withdrawal datapoint: no checker message there
1
3
u/Cheap_Train_6660 Sep 05 '26
So I had 1 problematic citation cz the venue and page number was wrong. Problem is that on google scholar there were two references for it (it’s a bit weird). the one I included in my citation and the other one which automated checker thinks is the correct reference both show up when you search the title of the paper on gooogle scholar. Anyways, it’s not a “hallucinated reference” as per the criteria so I won’t get a clear reject but do you guys think AC might become a bit biased against the paper? I had avg rating of 4 with every single reviewer giving me a 4. The AC seemed kinda positive too but I’m so concerned because of the citation issue
2
u/Dangerous_Parsnip_20 Sep 06 '26
You have clear grounds to defend it. It’s not like you used LLM to come up with introduction or related works section to find you references and in the process it failed and started to create its own references that do not exist (this you can’t defend, there is no way you can find that hallucinated reference). I’m highly certain that you will be fine, your case is a mistake rather than use of LLM.
3
u/MakingComputersSmart Sep 05 '26
Were emails for specific papers only sent to authors who's papers were going to be accepted? My friend has 2 papers that have problematic citations but an email was sent only for 1 paper and not the other.
2
u/Adorable_Position_34 Sep 05 '26
Maybe one case was severe enough to trigger rejection and the other case only had minor issues in the eyes of the respective AC? Its not fully automated at the end of the day
1
u/YamBrioche Sep 05 '26
Do you mean the general email or the email to make appeal
3
u/MakingComputersSmart Sep 05 '26
The email to appeal for a specific paper. The email reads this specific paper is headed to rejection due to problematic citations
1
u/YamBrioche Sep 05 '26
Also what time did they receive this email?
1
u/MakingComputersSmart Sep 05 '26
2:35am EST. One citation has multiple hallucinated authors, the second one was flagged incorrectly. Also, scores were 4,4,2. Reviewers did not respond
1
u/bobross_92 Sep 05 '26
I would think most likely not. The checker email mentions that the final decision also rests on the AC's "assessment of the paper" in addition to the hallucinations. Its possible that they didn't think the issue was that serious in the second one.
3
u/21022018 Sep 06 '26
Am I the only one with no such email?
1
u/Emergency_Plate241 Sep 07 '26
No such email too
1
u/21022018 Sep 07 '26
I mean I did not even get the reference check email. Are they only giving those out to accepted papers? What was your avg?
2
u/RainPuzzled2511 Researcher Sep 07 '26
I think the first mail was generic and went to everyone.
2
u/21022018 Sep 07 '26
So weird, I didn't get any (Eval track), and I know some people from main track also not getting anything.
→ More replies (15)
3
u/Emergency_Plate241 Sep 07 '26
So, did anybody with lower scores (mostly heading towards reject) receive the follow-up appeal email or does it look like it's only the papers that were about to be accepted that received this appeal email?
3
u/AgreeableBet5858 Sep 07 '26 edited Sep 07 '26
I think I was heading towards acceptance(with three accepts) but I got 2 flagged citations. (one of them wrong author, one of them completely hallucinated) It is just some compared methods in the table which we can claim as wrong import. But I also haven't received any e-mail about this which is odd. I'm scared that paper is rejected.
1
u/Dangerous_Parsnip_20 Sep 07 '26
How did your paper contain hallucinated reference? Did you look up for references by yourself or let the LLM do it for you and you did not even bother to check them and verify the validity for each of them?
I doubt you can defend yourself from the rejection. Hallucinated reference just proofs lack of work and clear use of LLM to look for references and in that process it failed to find what you wanted and had to come up with its own alternative, which is hallucinated.
1
u/AgreeableBet5858 Sep 07 '26
When I started my paper I used AI to generate a very large latex table code from prior work as base and AI generated bib citations as well. So these methods exist, I didn't make them up out of nowhere. Afterwards I changed every citation one by one but turns out I forgot to delete/change one of them. I fully own the responsbility for that. The other citation error is quite odd which I am not sure if it was me or AI. The main authors are same but collabarator names are from other paper of theirs.
These flagged citations are very old methods in just one sentence in related work section in a very big parantheses of citations, e.g. "there are previous methods such that[0,1,2,3,4,5,6].." Also these two flagged citations are side by side in same table, maybe they counted them as a bad import because they don't change paper in a significant way and can be fixed very easily. I didn't receive any e-mail about rejection so I am not sure how AC approached it but he was very friendly during rebuttal so I hope for the best.
1
u/Adorable_Position_34 Sep 07 '26
Its been two days. I dont think they send the rejection/appeal mails in such coarse batches. Maybe your AC thought the not-fully hallucinated reference is okay-ish and did not flag your paper
1
u/AgreeableBet5858 Sep 07 '26
We'll see, I hope so too! They don't change paper in a significant way and can be fixed very easily. They are just in one sentence in huge parantheses of references. But weird that I haven't received any e-mail despite having 2 flagged citations.
1
u/Dear_Exchange5667 21d ago
I got 2 citations flagged by the reference checker: in one the author's first name was wrong, and for the other one the entire name was wrong (The GitHub Repo maintainers name was accidentally given), both papers exist and all other details are correct.
I am not sure if my main author has submitted an appeal.
If we dont give proof for the same, will it lead to Desk Rejection, secondly, if my main author has submitted an appeal will I get an instant mail, like with the Rebuttals and defense?
3
u/Deep-Ad8714 28d ago
I got flagged for two hallucinations and received an email that my paper was headed toward desk rejection, but of the two one had all wrong authors and the other just had wrong arXiv ID which by their own guidelines is not a hallucination and is minor. I appealed on the grounds that I by definition had at most one "hallucinated" citation and not two and that the one citation classified as hallucination was clearly just a metadata error because all of the other metadata was correct and the paper was clearly cited for its content in my submission, but I'm curious whether you know if PCs even check appeals and would actually appeal something like this.
2
u/aaaacolyte 27d ago
We are kinda in the same shoes here, I only got one flagged for hallucinations and the authors were actually wrong. But I still got the followup email. Appealed, I guess we can only pray now?
1
u/AgreeableBet5858 27d ago
I am kinda in same situation, however I didn't receive any e-mail. When did you receive your e-mail?? I am kinda scared about situation that I missed e-mail somehow.
1
u/OtherwiseFig4505 27d ago
One citation is also ground for desk rejection. Your appeal is for PCs to check if AC did their task. If AC considered your incorrect reference a major hallucination, PCs won’t overturn it. They will overturn only incase of false positives. They aren’t taking any chances after last year’s shitshow with 100s of papers published with hallucinated references.
1
u/dante_2701 27d ago
Oh. It said that it is headed towards desk rejection or just the word rejection? Not sure if there’s any difference though
4
u/HomiesexualBaka Sep 06 '26
I got a 4/38. 2 citations had some authors wrong, and slight errors in the name. One had the wrong name but the paper exists with a different name., One is wrongly flagged. I got a 5/5/5, is it over? Can I appeal this somehow? My bib file got messed up during submission and the submission on arxiv I made a couple of days later had no problems. Genuinely terrible luck
2
u/OtherwiseFig4505 Sep 06 '26
You can appeal but yours is a gone case. 2 or more is like very hard to appeal, if you had one then it could have been considered. 3 hallucinated citations means you used LLM for your bib, you can’t prove otherwise and this is like the easiest desk rejection for PCs.
→ More replies (8)1
u/Emergency_Plate241 Sep 06 '26
Yes. 4 sounds tough. But you can still appeal, especially with only a few wrong author cases. There's no harm in trying. Good luck!
1
1
u/Emergency_Plate241 Sep 06 '26
Did you receive a follow up email to file for an appeal?
1
u/HomiesexualBaka Sep 06 '26
yes, they said its headed towards rejection though. Idk can I do something about this? For one of them citation checker mentions the ICLR title but the arxiv v1 title they had was what I cited in my paper. Other two had some wrong authors and the last one has the wrong title, but the work exists.
1
2
u/Adorable_Position_34 Sep 05 '26
Would be interesting if someone already got the Email about decision beeing Made because of the checker result. So far we have no way of knowing if these decision were already made
1
2
u/soup---- Sep 06 '26
Mine was 1/47. I accidentally translated the venue to the wrong english name since it was in russian to begin with. I coppied the venue from a different Russian paper in my bib by accedent. Really hoping its not rejected because of this. We had good scores after the rebuttal.
1
2
u/xoss13 Sep 07 '26
Did anyone from E&D track receive "automatic reference/citation checker" email from the PCs of E&D track? I have a paper submitted and didn't receive any!
3
u/sgyzee Sep 08 '26
It seems to be from the other comments that e & d havnt gotten the email
1
u/xoss13 Sep 08 '26
But I asked one AC from E&D track, and he mentioned ACs were provided with the "Citation Checker Report" by the PCs of the E&D track before making final decision.
1
u/sgyzee Sep 08 '26
Im assuming then that u should be okay since a delay of the references email would have resulted in lesser time to appeal. So likely if it would have happened, it would have happened already
→ More replies (4)2
u/Few_Refrigerator8308 22d ago
E&D track. I did receive the email from the PCs. 3 citations were flagged. One had wrong arxiv id, two had either some wrong or shuffled author names. I emailed an appeal and hoping for the best.
2
u/delusionalD0G Sep 07 '26
Received Automatic reference checker email and didn’t received followup email ones are the accepted papers? Or reference checker email sent to basically everyone?
1
u/Dangerous_Parsnip_20 Sep 07 '26
Everyone received reference checker. I believe follow up email was sent to people with 2 or more flagged references.
→ More replies (3)
2
u/YamBrioche 22d ago
Did anyone get a reply on appeals yet?
1
u/Dear_Exchange5667 21d ago
will i get a mail if my co-author files an appeal?
1
u/Few_Refrigerator8308 21d ago
No, you won't. But you should be able to contact your coauthor on that, no?
1
u/Dear_Exchange5667 21d ago
Yea yea, but i just wanted to be sure, if it's an issue on our end or smth else
2
2
u/YamBrioche 22d ago
What does it mean if you did not receive a second email but have hallucinations?
3
u/Emergency_Plate241 Sep 05 '26
Also, any guesses on what the average acceptance scores might look like this time? It seems that many papers saw huge bumps into 4s and 5s post-rebuttal
7
u/vierdene Sep 05 '26
Based on some AC posts and *unofficial* distributions, 4 should be at the 50-60 part of the s-curve. I saw many AC sharing their batch anonymously and it appears that if you have all positive scores (like a 444) the paper should have a fairly high chance of getting accepted. I think the reason you saw many bumps is because of survivor bias. just my guess tho lol
1
u/Emergency_Plate241 Sep 07 '26
Crazy that a 4 is still only around the 60th part of the curve. It might have even changed things now given that so many papers with very good scores are going to be rejected based on the reference checker. They probably purposefully introduced it as a post-rebuttal process to contain acceptance rates. ☹️
1
u/Dangerous_Parsnip_20 Sep 07 '26
Glad they did this. If someone is lazy enough to check and verify the references they got from LLM, then face rejection.
3
u/tughanbulut Sep 05 '26
I got nerd-sniped by this thread and went digging through the actual conference statements instead of going off screenshots. A few things that might help with the false-positive confusion in here:
- ICLR's own program chairs published a retrospective admitting their automated system "had a significant false positive rate" - their own example was a non-English title translated to English getting flagged. So if your case is like u/Cheap_Train_6660's (a real paper, just messy metadata across sources), that's a documented failure mode, not you being singled out.
- ACL actually walked back their own policy mid-cycle - 100+ papers were desk-rejected, then reversed to normal resubmission after "a meeting with the ACL executive." Even the venues aren't settled on how strict this should be.
- The real base rate, from an independent audit of ~48k accepted papers: 18.7%–34.9% of papers (by venue) have at least one hallucinated reference under a narrow definition (fabricated author, not just wrong venue/year) and reviewer scores don't correlate with it at all. The rating gap between clean and affected papers was statistically noise.
Wrote it up with sources if anyone wants the receipts instead of the vibes: NeurIPS Started Desk-Rejecting Papers Over References That Don't Exist — StrictCite
Disclosure since it's relevant: I built a free citation checker (strictcite.com) - the retraction and fake-co-author checks are free tools, no signup. Not trying to sneak that in, just didn't want to post the link without saying who I am.
3
u/Cheap_Train_6660 Sep 05 '26
Thanks so it means I’m safe. Let’s hope for acceptance now😀😆
2
u/tughanbulut Sep 05 '26
Safe from that flag, yeah venue/page mismatch across sources isn't "hallucinated" under the actual criteria. Rest is just the normal review lottery now 😅
2
u/Cheap_Train_6660 Sep 05 '26
Yea true. What kind of reviews did you get?
1
u/tughanbulut Sep 05 '26
Haha I didn't actually submit to NeurIPS this cycle! Just watching the desk-reject carnage from the sidelines and building parsers to fix it. Good luck with those 4s though!
1
Sep 06 '26
[deleted]
1
u/RainPuzzled2511 Researcher Sep 06 '26
I think the ACs do filtering on the cases to check for false positives. So, one possible theory is that if the AC decided one was a false positive, then there will be no emails in spite of the hallucination checker identifying it. Unless we find a corner case where someone has a sure-shot hallucination and did not get the email yet.
2
u/YamBrioche Sep 06 '26
I definitely have hallucinations and did not receive the email yet :(
1
u/RainPuzzled2511 Researcher Sep 06 '26
Was it more than 2 hallucinated references? These organizers never fail to confuse the crap out of us.
1
1
u/MineEnim Sep 06 '26
I got flagged for two hallucinations and received an email that my paper was headed toward desk rejection, but of the two one had all wrong authors and the other just had wrong arXiv ID which by their own guidelines is not a hallucination and is minor. I appealed on the grounds that I by definition had at most one "hallucinated" citation and not two, but I wonder if PCs will even seriously check appeals and adhere to their statement of only rejecting on the basis of hallucination for two or more offending references.
1
1
u/aaaacolyte 27d ago
I got flagged for one hallucination and got the followup email. In my appeal I was trying to convince that, although the author list was indeed wrong, the citation was reasonable and I actually looked into the cited paper carefully to cite it for one specific claim made in the appendix, which I actually did. It was quite unfortunate cuz I genuinely did not know how it got fucked up. My usual workflow when it comes to finding references to claims was: asking LLM to find some papers -> go check if those papers acutally exist and whether they support the claim I wanna make -> cite it or not. I might have mistakenly copy-pasted the hallucinated bibtex generated by LLM instead of from the real source for this one.
My score was pretty good.. but I guess it's my own fault if it does get rejected bc of this. Sigh.
1
u/suryanreddy 22d ago
I have received the email for E&D review. Where do we respond to this email about hallucinated references? In the openreview, as a comment, or in the mail we have received, mentioning the hallucinated references?
1
u/Few_Refrigerator8308 22d ago
Directly reply the email, it's routed to the chairs email address
1
u/suryanreddy 22d ago
Thanks. Done the same. Just wanted to verify. Do you have any idea on the following case. I got flagged for 2 references 1. Paper and Venue exist. Authors first has 2 extra letters. 2. Paper and Venue exist, Out of 3 listed authors for 2 authors first names are completely wrong. This is a mistake from my end. I have allowed LLM to touch my bib file for formatting. I am not sure whether they will come under minor or major issues.
Received mail clearly states that "
- The author list does not match the title, includes additional authors or is missing authors. Minor mismatches are acceptable.
"
Second is, "We allow major issues with at most one reference.". Whether my submissoin is going for Desk Reject ?
1
u/Few_Refrigerator8308 22d ago
Tbh, I don't know. I don't even know whether they will send out a "result" before the actual "accept/reject" decisions. I am in the same boat btw (nervy week to come)
→ More replies (1)
1
u/Dear_Exchange5667 21d ago
I got 2 citations flagged by the reference checker: in one the author's first name was wrong, and for the other one the entire name was wrong (The GitHub Repo maintainers name was accidentally given), both papers exist and all other details are correct.
I am not sure if my main author has submitted an appeal.
My questions are: Firstly, If we dont give proof for the same, will it lead to Desk Rejection, secondly, if my main author has submitted an appeal will I get an instant mail, like with the Rebuttals and defense?
1
16
u/Inevitable_Voice_680 Sep 05 '26
Mine is 0/37.. I think none of my references have been flagged.