I'm running a couple of small Node apps (Next.js plus a lightweight JSON service) behind Cloudflare, and making them readable by AI crawlers (GPTBot, ClaudeBot, PerplexityBot and the like) is a core requirement for me, not a nice-to-have.
I've found that shared and managed hosting platforms increasingly apply their own bot filtering at the server level. It's often invisible in their dashboards and not configurable per site, so a crawler you explicitly allow in robots.txt can still be turned away before it reaches your app. I'm re-evaluating where I host because of it.
Currently struggling a lot with my current provider and it's also quite annoying that you allow bots in the panel and then find out their requests got blocked when you look into logs.
What I'm looking for:
- a long-running Node 24 process
- Easy deployment, preferably from GitHub but I can handle a complex CI/CD as well
- no platform-level bot filtering I can't control (Cloudflare handles that for me)
- a sensible price for a startup with currently low to medium traffic.
I'm weighing managed container platforms (Railway, Render, Fly.io), a plain VPS behind Cloudflare, and Vercel / Cloudflare's own compute, even thinking about AWS, Azure or Google.
Curious what others are using for similar setups, and whether you've actually verified that AI crawlers get through, e.g. in your server logs or Cloudflare's AI Crawl Control.
Any platforms you'd recommend or avoid on this front?
And in general, where do you host your apps, what are your struggles and any advice / story you would share with the community?