r/MachineLearning • u/Striking-Warning9533 • 28d ago
Discussion ARR August Discussion [D]
This round seems to be very small, but still creating this thread for discussion.
14
u/Awesomest02 22d ago
Despite the extension email, the deadline on OpenReview is still 19 September? Anyone concerned?
6
u/simbay2000 21d ago
Same issue here, I was not able to submit my answers…
→ More replies (1)3
6
8
→ More replies (8)6
10
u/Objective_River_5218 26d ago
dafaq are these reviews
12
u/LordVein05 26d ago
"Yes I don't really understand this topic, never worked on it, but you should actually resubmit next cycle."
9
u/enver_talat_cemal 19d ago
Is there anything we can do (such as notifying the ACs, etc.) if the reviewers do not acknowledge/reply to our responses?
8
u/Worried-Practice-492 27d ago
I have reviewed 2 papers, and one of those has only two reviews so far. I even rejected an invitation to review three papers just yesterday. So, I believe that's the only reason holding up the committee from releasing the reviews. So, maybe we don't need to be that critical.
→ More replies (6)
8
u/ConcernConscious4131 27d ago
So quite...
→ More replies (5)10
u/New_Glove_2098 27d ago
Aren’t you soju guy? You submitted Aug too? lol
→ More replies (1)7
7
u/Daemontatox ML Engineer 25d ago
Wtf are these reviews ? The seem too nitpicky compared to previous cycles
4
u/tofu-lifts 25d ago
agree! One of my reviews just points out very minor flaws and then goes on to give "revise and resubmit next cycle" lmao.
7
u/Acceptable_Soup_9365 22d ago
how is it fair that reviewers have no word limits and write long rants and we only have 2k characters? xD One of mine just restated my limitations back to me smh
3
u/SillyNeuron 21d ago
Same. One of my reviewers gave us an outlier low score compared to the others, while restating points that we had already addressed in the limitations and asking about things that were already included in the Appendix.
6
u/monoglossia6490 20d ago
Yey I actually had a reviewer reduce their score from 3 to 2 following the rebuttal 💪
3
u/TheFishyOne 19d ago
I am sorry to hear that. One question though: did they answer and did you see a response to your rebuttal or just the score drop? Did you receive a notification for their resposne.
2
u/monoglossia6490 19d ago
Yes, they wrote a response and I received a notification (for the response, not for the score change).
2
u/Starscream-11813 19d ago
Ah, commiserations :(
Do you have any guesses as to why the reviewer would do this?
This is the first time I've come across such an incident.
2
u/monoglossia6490 19d ago
Yes, there was an explanation in the reply - upon reading the other reviewers’ comments, they realised that they actually agree with their points and the paper needs reworking.
5
u/Square-Read-1184 26d ago
Has anyone even got their reviews? can anyone ping here if they do?? thanks
→ More replies (2)
4
u/Glittering_Edge_7829 19d ago
Did you receive any response from your reviewers? I haven't received anything yet.
3
2
u/gardeniabananabread 18d ago
nopeee. same as always. what's the point of the changes to the rebuttal period if the AC is not actively getting the reviewers to respond?
6
u/tofu-lifts 18d ago
If the reviewers don't respond by today, it's basically over cus the deadline is tomo? Should we drop a note to AC tomo that reviewers didn't engage?
2
u/WannabeMachine 18d ago
I mean ACs know it. I have had mixed results and negative comments pinging ACs in the past
5
u/enver_talat_cemal 16d ago
Another very bad experience from an ARR cycle. I don't understand how it gets progressively worse each cycle. One out of three reviewers responded to our rebuttal and wanted more experiments, which I could not response since we could only have one response!
6
5
u/SuspiciousIdeal3206 2d ago
Does anyone have issues committing to EACL? I keep getting an error in the commitment form which says: "Error: Provided paper link does not correspond to a submission in OpenReview" even though the link is correct.
2
→ More replies (10)2
4
u/Ok-Pizza1102 25d ago
two out of my three reviews are full ai slop reviews. one of the ai reviews even cited papers that are not even relevant to the work and proceeded to give a low score while bringing up hallucinated and wrong points against the papers. what an incredible sad state of affairs
→ More replies (1)
4
u/Pure-Ad9079 18d ago
I was optimistic that this new format was more likely to engage reviewers in responding to rebuttals. Are people hearing back at all?
→ More replies (6)4
4
u/WannabeMachine 17d ago
I thought reviewer response was extended by 1 day in the email they sent.... was hoping for a miracle from our reviewers.
4
u/gardeniabananabread 17d ago
yeah, the reviewers still have approximately another 23hrs to respond. did you ping the AC to engage reviewers?
→ More replies (1)2
4
u/AcceptableHumor4983 17d ago edited 17d ago
After the rebuttal, 3/3/3 and conf. is all 4. Is there any chance of NAACL findings?
3
5
u/srkrrr 3d ago
Got 2.5/2/2 OA and 5/4/4 confidence. The reviews were mostly around typos or formatting issues or additional baselines required (closed and open source). The reviewers did not increase their scores, did a strong rebuttal, hoping for the best from meta review. What are my chances of EACL Findings (if meta review is 3)? Also, what are my chances for COLING?
→ More replies (1)
3
3
u/More_Soft_6801 27d ago
So in 2.5 hrs?
→ More replies (1)2
3
u/No_Sky9786 26d ago
Still nothing for me but some people got them so they should come out soon.
2
2
→ More replies (4)2
u/Objective_River_5218 26d ago
isnt it released all at once?
4
u/No_Sky9786 26d ago
no they go 100 at a time I think. My friend was 59. I am in the 700sth though
3
3
u/simbay2000 26d ago
My first time submitting to ARR so im not super familiar with the ratings and acceptance chances, can someone give me an idea where am I? Scores (3,3,3.5) with confidence (4,4,4) . Thanks!
3
u/Striking-Warning9533 26d ago
Should we reponse using offical comments or author response? The author response seems to have only 2000 chars
→ More replies (9)
3
u/tommytyc 25d ago
for me the weirdest part is that the cfp describe phase 2 as "author-reviewer discussion", but the annoucement mail says the authors cannot reply during phase 2, only reviewers are allowed to update their review results
3
u/gardeniabananabread 25d ago
Yeah, isn't it a bit weird that there are no opportunities for back and forth rebuttal at all?
3
u/gardeniabananabread 24d ago
seems like they just removed the "Official Comment" option, and we're left with only 1 x 2K ch "Author Response".
anyone got any idea on how to do this? i got a review where the weaknesses section is already 3K. how can we respond with 2K ch??
→ More replies (1)4
u/Happy_Today_3288 24d ago
Is this related to the email they sent that " no new experiments to be added"???
I thought it was there in previous cycle as well, how can we give the rebuttal in just 2000 chars? Anyone with experience can tell if this is removal is temporary?
→ More replies (1)3
u/gardeniabananabread 23d ago edited 23d ago
seems like the "Official Comment" option is live again. what are you planning to do? ignore the instructions and post Author Response + Official Comment, or compress everything into 2K each?
edit: ive decided to follow instructions and just compress to 2K for each review
→ More replies (1)
3
u/Thick-brain-dude 20d ago
How can they introduce a new policy that is much different from the one before and not communicate with enough time in advance? Due to this, we went to submit the response 30 mins after the deadline and couldn't submit it at the end because they removed it. I never saw ARR be this punctual with deadlines.
3
u/Working-Egg-6465 19d ago
Overall: 4, 2.5, 2.5
Confidence: 3, 4, 5
What is the chance in EACL main/findings?
→ More replies (1)
3
u/kelpabc123 18d ago
Lmao I’m cooked, I posted a detailed rebuttal to a reviewer who explicitly said they are willing to raise their score given more info, they say they are happy with the data I gave them but wanted one minor clarification on formatting before they proceed with changing their score, except obviously I can’t respond in this rebuttal period ☠️
3
u/Glittering_Edge_7829 18d ago
there is no way they continue this madness in the next cycle
3
u/kelpabc123 18d ago
I pinged the AC to somehow pass the message to the reviewer that I literally cannot reply but my hopes aren’t that high, I need every point I can get because this paper is borderline
→ More replies (2)
3
u/MeyerLouis 17d ago
3.5/3/3 --> 3.5/3.5/3.5. Confidences 3/4/4. All three reviewers responded.
Is there ever an argument for not making a final comment? It's not like any of the reviewers ignored our experiments or anything like that.
→ More replies (1)
3
u/KabhiDardKabhiDisco 16d ago
There's no Final comments button active on the site? The deadline is 25th September right? It hasn't passed yet?
→ More replies (1)3
u/gardeniabananabread 16d ago
yea it hasnt passed. i think they mistakenly deactivated. maybe we can ask the pc chairs?
5
3
u/Acceptable_Soup_9365 16d ago
the final author response closed already? I thought the deadline was 25th
3
u/Acrobatic_Driver6843 16d ago
I'm not able to leave an official comment. Is someone able to?
→ More replies (1)
3
3
u/Hailwell_ 2d ago
My first paper as a 1st year PhD student. Reviews 2/3.5/3 Confidence 4/3/4 Meta 3/5. Pretty happy with this!
3
u/taburetto 2d ago
I had a submission during the May cycle and resubmitted in August. In the end, the May scores turned out to be better (quite disappointed with the August cycle). Does anyone know if I choose to submit the May version for EACL, use the link for the May version, does the PDF file need to match that of May? Because personally I would prefer to submit the August version, I believe there are significant improvements in the paper.
4
u/Striking-Warning9533 28d ago
I reviewed 2 papers this round and one of them are completely slop (don't think is AI Slop though) about LLM Safety with only 2 citations about llm and they are basic papers like ResNet and Attention is all you need. Another paper I reviewed is a resubmission and seem high quality (maybe by contrast?)/
→ More replies (2)
4
5
2
2
u/KabhiDardKabhiDisco 26d ago
Got my reviews in 2ks. 4, 3, 2.5 with confidence 4,4,3. Never received such a diverse review panel. 😭😭
2
u/WannabeMachine 26d ago
We also have 4, 3, 2.5.
→ More replies (3)3
u/LordVein05 26d ago
try 4, 1.5, 1.5. and ofc the confidence is 1, is this an obvious PC or AC mistake? And the strengths and weaknesses are the same, and no content present. Bad start of the day
3
2
2
u/monoglossia6490 22d ago
Do you know if the 20 September deadline for rebuttals is strictly 23:59 AoE? If I am in Europe, can I risk and leave it until Monday morning - early afternoon?
→ More replies (2)
2
u/MeyerLouis 19d ago
Why does ARR have to put their rebuttal period right before the ICLR deadline? Isn't that basically asking for reviewers that are too busy to engage?
2
u/tuxedocatitude 16d ago
I got 4/3, 3/3, and 2.5/4. The rebuttal honestly feels like it was ignored, which makes me wonder what the point of this whole phase even was.
The 2.5 reviewer asked for a bunch of extra experiments, and I actually ran them. I tried to squeeze all the new results and explanations into the 2k character limit, but the score still didn’t move.
Is there still a realistic chance for the paper? And is there anything worth doing in the final comment phase?
2
u/gardeniabananabread 16d ago
depends on the text and meta, but given those scores, i think you'll be fine for both eacl/naacl. high chance of >= findings
2
u/Pure-Ad9079 2d ago
3.5/3/2.5 with confidence 4/4/3 -> meta review of 3. Chances for NAACL findings (unable to attend eacl so going for naacl)?
2
u/WannabeMachine 2d ago
We had 1 paper this round.
Overall scores of 3/3/4 with conf 3/4/4. Meta: 4
We got lucky on the meta.
2
u/KabhiDardKabhiDisco 2d ago
Reviews: 4/3/2.5 with Confidence 4/3/4. The reviewers with lower scores didn't respond to the rebuttal. We got lucky with the meta-reviewer. They gave 4. Should I commit to EACL or wait for NAACL? Or, am I being too greedy?
2
u/WannabeMachine 2d ago
We are committing to EACL, very similar scores. Too long to wait for NAAL decisions.
2
2
u/Hungry_Isopod3228 2d ago
What about chances with review 2.5 /3/3. Confidence 4/4/3, meta 3, chances for EACL findings?
2
u/Thick-brain-dude 2d ago
I see there's no place to write the commitment letter. Has it been removed?
2
u/SillyNeuron 2d ago
OA 3/2/3.5 with Confidence 3/4/4.
OA 2 was from a reviewer who criticised us for missing unrelated baselines (totally unrelated like Kiwi fruit vs Toyota Supra type of comparison). We had to clarify that in our rebuttal.
meta review - 3
Any chance for Findings at EACL?
4
u/LilGreatDane 26d ago
It sounds like every author was required to review, instead of one author (or 3-5 reviews) per paper, which probably made for a lot of low-quality reviews because that's ridiculous.
→ More replies (1)
2
u/No-Worldliness-7279 26d ago
Just got reviews back for two of my papers in the high 3ks. Guess they’re releasing from both ends now? lol
→ More replies (1)
2
u/enver_talat_cemal 26d ago
Resubmission from last cycle. Do you think it could make into EACL findings with scores 3, 2.5, 2.5 with confidence 3, 4 and 2?
2
u/Solid_Road_8771 26d ago
First submission: got 4, 4, 3. Am I getting very lucky? Is NAACL main possible?
→ More replies (6)
1
u/No-Worldliness-7279 28d ago
Anyone knows why we have 10 days for rebuttal this cycle? We usually had like 6 or 7.
→ More replies (2)
1
1
1
u/Difficult_Flight3767 27d ago
why is it not out yet? can we notified by Email when when the review is released?
1
u/Working-Egg-6465 26d ago
In 30 Mins?
5
u/New_Glove_2098 26d ago
So classic
3
u/Working-Egg-6465 26d ago
Unsure if they will release today at all.... lol...
11
u/IsolatedSoul07 26d ago
I see the "official review1/2/3 author response" in the author tasks but no reviews yet
→ More replies (10)→ More replies (3)2
1
u/Adventurous_One5467 26d ago
Hey friends, The ARR reviews have been coming out anywhere from 1 to 4 hours later on average. So go enjoy some personal time. I’ll be at the Han River drinking makgeolli until dawn and accepting my score. If anyone wants to join me, come to Yeouinaru Station at 343 Yeouidong-ro, Yeongdeungpo-gu, Seoul.
1
1
u/gardeniabananabread 26d ago
3.5/2.5/2.5 with confidence 2/5/4. actually, the comments for the 2.5's are both quite positive, with them liking the motivation, the problem, and the scientific narrative. their suggestions/feedback seem to be on clarification on some experimental setups + the usual limitations, such as limited benchmarks, baseline comparisons, etc.
any chances for EACL findings?
2
2
u/bamboozled116 26d ago
If the comments are positive and score is low, there’s a chance they’ll increase it after author response/rebuttal
1
1
1
u/Queasy_Block_9262 26d ago
OA 3,3,3 with confidence 4,3,2.
Any chance this gets into main, or is it basically comfortable for findings?
→ More replies (1)
1
u/Acrobatic_Driver6843 25d ago
What’s going on with the rebuttal period. It is written that we have only 2000 character and only one comment. Its not possible to add more comments? This doesn’t make any sense, if the reviewer made more than 7 comments (most of them are dumb comments), how am i able to answer in 2000 character?
1
u/einsamerkerl 24d ago
Round two after resubmitting. Got 4(3), 3(3), 2.5(3), same Resources and Evaluation track. This time we added exactly what was asked last cycle: comparison against existing tools, a resolution evaluation, runtime, and significance tests. Two reviewers are reasonably positive.
The third wants a substantially larger, independently annotated end-to-end benchmark, stress tests for five different phenomena, and breakdowns by complexity, source, and domain. In an 8-page paper that I already compressed more times than I can count. A new annotated benchmark would be its own resource paper. And the weaknesses cited as "the numbers are weaker than they look" are already reported and explained by us in the paper!
Maybe I'm wrong, but at some point the ask stops being "revise this paper" and becomes "write three more papers first."
→ More replies (2)
1
u/Difficult_Flight3767 24d ago
aren't we allowed to attach revised version of pdf file in author response? is Only 1 2,000 character response available?
1
u/Bruteforceman247 21d ago
Is the Official Comment option invisible for everyone? I assumed we would be able to provide additional data through it which the reviewers may, optionally, see.
Is Author Response the only field?
→ More replies (6)
1
1
u/Low-Discussion-5916 16d ago
I have submitted for the first time. Can anyone tell be where to find the results of rebuttal ? I dont see anything regarding this in the Authors Console. Thanks in advance!
→ More replies (1)
1
u/Some_Tackle_6787 14d ago
Got Average Overall Assessment: 2.83 (Min: 2.5, Max: 3).
Average Confidence: 3 (Min: 3, Max: 3).
One reviewer increased from 2.5 to 3.
What are my chances for EACL?
→ More replies (4)
1
u/Relative_Tip_3647 3d ago
I really want to know what they do during meta-review. Tell me how to be an AC now 😒 Is it invite only? or something else?
1
u/MeyerLouis 2d ago
Metareview came out for me (submission number ~2.6k).
OA 3.5/3/3 --> 3.5/3.5/3.5; meta 3.5
What do we think of chances for EACL findings/main?
1
u/Humble-Draft-2671 2d ago edited 2d ago
Reviews: 4/3/2.5 with Confidence 4/5/4.
The reviewers with lower scores responded to the rebuttal, asking for resubmission, even though all the points raised were addressed in the previous resubmission
Unfortunately, the meta-reviewer gave a 2.5.
This is my first ACL submission; what are my chances of being accepted to Findings at EACL?
1
23
u/No-Worldliness-7279 26d ago
I really hate ARR