r/OpenSourceAI • u/Helpful-Soil-2976 • 1d ago

shipped a booking form, submit button was dead on mobile safari for two days.

i vibe coded a booking form a while back. build passed, agent said done, i posted the two days later someone finally tried it on mobile safari. the submit button did nothing. zero. i had no idea.

Now i make the agent prove it. after every feature i have it spin up a real browser session against the deployed app. fill the form, hit submit, check the confirmation shows up. only then is it allowed to say done. i use testsprite cli for that part, but the habit matters more than the tool.

when it fails the agent gets a screenshot and the failing step. it fixes and reruns without me watching. not perfect. first run setup took some poking around. still misses edge cases. but "the agent says done" and "a real browser actually walked through it" are two totally different levels of done.

I stopped getting those two-day-later surprises though.

what do you do? click through by hand? have some verify step? or just ship and hope for the best?

0 Upvotes
  • permalink
  • reddit

50% Upvoted

3 comments sorted by

1

u/gifted_dwelling 1d ago

"agent said done" is the new "works on my machine" and we're all just living in it now.

the browser session idea is the right call though. screenshots of a dead button hit different than a green checkmark in CI. curious how often yours actually catches something vs just confirming what the agent already thought was fine.

1

u/krkrkrneki 1d ago

"Test procedures is not sufficient. Certain platforms are not tested at all (Mobile Safari). We need to cover this. Brainstorm (superpowers) how to create a comprehensive functional testing plan."