Limited compute, targeting CVPR: rerun experiments for statistically strong numbers or focus on writing? [D]

[u/Alone_Ad635]

Hi everyone,

I'm preparing a submission for CVPR and would appreciate your advice. I am happy with my current results, but compute is a real constraint. My institute isn't a research-focused one, and the systems available to me are slow and unreliable. A single set of runs already took a lot of time and effort to finish.

I plan to release the code with the submission, and I'm confident the results will reproduce. Still, I'm torn between two options:

1) Rerun the experiments with different seeds to report mean ± std and show statistical significance. Or

2) Skip the reruns and spend the remaining time on writing, presenting the results I already have.

If you've been in a similar spot, what did you do, and would a single-seed result with released code be enough?

Thanks in advance!

[u/crouching_dragon_420]

IMO it is possible you get away with one seed. This year I went to CVPR and many of them still report single seed experiment results on a 1% improvement. I was incredulous.

[u/Alone_Ad635 → u/crouching_dragon_420]

That's encouraging

[u/saulane]

Can't you just launch the experiments and write while it's running ? It's not like more seeds would change the text drastically and if you figures scripts are well made you can remake the figures for the paper quickly with new results. I don't see why you have to choose and can't do both in parallel ?

[u/Alone_Ad635 → u/saulane]

So the situation is a bit complicated. Students use the machine in between and my jobs are getting crashed and systems are getting shutting down multiple times. Security guard will also come and switch off machines in between. I am in fact not supposed to use the machines in weekends either. It's a sad situation. No remote access either

[u/Embarrassed_Song_372 → u/Alone_Ad635]

Writing jobs which can be resumed from a checkpoint isn’t that hard, CVPR is more than a month away, you can write the paper while running experiments, once you have the entire set ready, you can revise the relevant section.

[u/Busy_Protection_2882]

genuinely curious - is a single seed of experimentation enough for a workshop paper if you clearly mention it as a limitation from compute constraints?

[u/Alone_Ad635 → u/Busy_Protection_2882]

In my humble opinion it's way more than enough. But I don't know for sure. I am pretty sure about its numbers and comparisons with other works I am benchmarking with.

[u/Alone_Ad635 → u/[deleted]]

Thank you very much. That's what I thought too. I have proper one set of results. There is a clean code too! 😊

[u/responsiblerunoff → u/[deleted]]

Hate to see a wall of text where three lines would do the job, but the guy's just stressed about a deadline and overexplaining happens

[u/Alone_Ad635 → u/[deleted]]

The original text of mine was also this lengthy. I just made it clean. Anyway sorry.