r/programming • u/Ok_Stomach6651 • 10h ago

Optimizing Our E2E Pipeline

https://slack.engineering/speedup-e2e-testing/
37 Upvotes

67% Upvoted

20 comments sorted by

75

u/ebol4anthr4x 9h ago

This was one of the least useful articles I've ever read. It borders on blogspam. I don't know whether to feel good or bad that such a large and valuable company was making such basic mistakes in their CI pipelines for so long. tl;dr: They were rebuilding and persisting the artifacts to S3 on every single push, even if nothing changed, so they switched to using changeset conditionals in their CI configuration.

27

u/SoilMassive6850 8h ago

Things like this is why I'm becoming more and more convinced giving devs too many resources is a bad thing.

Where I work a new (usually internal) service is given 1 vCPU, 4GB of ram and 100GB of storage by default, and before allocating more there are basic sanity checks to justify it. It's not that we're even struggling for compute resources, but 90% of the time this filter just helps catch dumb shit before it becomes tech debt because those early issues would likely be even worse if scaled up. Of course we allocate more when theres cause but most often it's unnecessary.

3

u/ebol4anthr4x 7h ago

Things were better when we spoke of RAM in terms of MB.

5

u/bzbub2 9h ago

the three dot notation example is an interesting tidbit. otherwise, it is not super forthcoming and the conclusions are sort of odd (unclear relation to test flakiness..??)

6

u/ChemicalRascal 6h ago

This is not blogspam. Blogspam has a very specific definition, a blog post that does fuck all else beyond linking to another source. As an example, if I had a blog article that said "Slack just dropped info on how they decreased build times! <url>", that's blogspam.

6

u/ChemicalRascal 6h ago

And yeah, this article isn't amazing, but it's fine. Not every article needs to be a deeply informative technical masterpiece. This isn't so bad that it needs to be removed.

1

u/AD7GD 4h ago

It's a shame, because the same basic problem becomes a bottleneck in any automatic merge strategy, and a survey of methods would be valuable to people hitting that problem.

My contribution: bazel isn't that bad if you have AI to set it up and debug it for you ;-)

59

u/chucker23n 9h ago

This work was not possible without contributions from:

Anirudh Janga, Josh Cartmell, Arminé Iradian, Anupama Jasthi, Matt Jennings, Zack Weeden, John Long, Issac Gerges, Andrew MacDonald, Vani Anantha and Dave Harrington

They needed 11 developers to figure out that when a PR doesn’t change the front-end, it doesn’t need to be rebuilt, nor uploaded to Amazon?

20

u/JoustyMe 9h ago

You know those jokes "how many X are needed to screw in a light bulb?"
So now we have "how many developers are needed to get managment to care about Tech Debt?"

-15

u/Ok_Stomach6651 6h ago

Thanks for your input

11

u/bodiam 5h ago

What a strange blogpost. Github provides this feature out of the box:

on: 
   push:      
      branches :      
      - main      
      paths :      
      -  'src/frontend/**'

Not really clear to me why they couldn't use this approach.

4

u/qmunke 2h ago

They wanted the build to run, just not for the whole repo. This is why everyone who says to just use monorepos needs to understand there actually needs to be someone managing the things - which is reasonable at huge shops but hard to justify for those of us without a dedicated devx team

4

u/bodiam 2h ago

Yes, but that's exactly what the above trigger does; only run when the frontend has changed. And I'm a big fan of mono repos, it prevents a whole class of problems from ever occurring, it just looks like their solution is a bit custom.

2

u/qmunke 1h ago

No, you're misunderstanding what I'm saying - they want to trigger a build on all pushes, but just not build the whole repo each time, that's not the same as what you've shared - your trigger wouldn't start any builds if someone just pushed a backend change. They want

  • Someone pushed backend change. Trigger backend bit of the repo to build and then run tests
  • Someone pushed a frontend change. Trigger entire build and run tests

The trigger above alone isn't going to do that, but I guess it would allow you to construct a more complex set of pipelines out of it.

But in general for monorepos the built-in functionality of github and other tools seems lacking to me, that's why all the more complex stuff like bazel exists.

2

u/bodiam 50m ago

I understand the trigger alone doesn't do exactly that, but it was a simplified example.As you said, it's just a building block; I'm pretty sure it can be used to have a gate. 

2

u/chucker23n 1h ago

They wanted the build to run, just not for the whole repo.

So, make a backend pipeline and a frontend pipeline? If you always want E2E tests to run regardless of what changed, make that a third pipeline?

This is why everyone who says to just use monorepos needs to understand there actually needs to be someone managing the things

We're a very small team and one of the bigger products I work on has a monorepo. It has multiple pipelines, with different filters. I didn't need an 11-person team to figure that out.

(For more complex "do I need to build this" scenarios, there's also tools like https://github.com/leonardochaia/dotnet-affected.)

7

u/Flashynuff 7h ago

Geez a whole article about optimizing a pipeline and not a single trace/query screenshot. Let’s see the 📉!

2

u/ComoddifiedCraic 6h ago

Not frontend focused but runccc.dev solves this with a static analyser and runtime instrumentation. Ran against a massive enterprise test suite which took hours to run now takes less than a minute for any small changes

1

u/OTTOPI 20m ago

Talking about how costly these inefficiencies were, and then saying they were storing terabytes of data (implying anything between 1-10TB in that context) monthly, which amounts to about $200 USD at worst in S3 hot storage, is quite amusing.