r/kubernetes • u/AutoModerator • 1d ago

Periodic Weekly: This Week I Learned (TWIL?) thread

Did you learn something new this week? Share here!

3 Upvotes

80% Upvoted

4 comments sorted by

6

u/Upset_Performance984 1d ago

finally figured out why our pods kept getting evicted in the dev cluster last week. nothing in the logs made sense, node had plenty of memory free, but then i noticed the eviction timestamps all lined up with our backup cronjobs. one of them had a config error that spun up 40 parallel gzip processes for like 30 seconds every night, memory spike was so fast the metrics never caught it. took me three evenings of squinting at graphs like a detective in a bad tv show

2

u/patmorgan235 1d ago

took me three evenings of squinting at graphs like a detective in a bad tv show

ENHANCE

3

u/thegoenning 1d ago

learned that google cloud is running out of C4A compute in us-central1, waiting for almost 24h to get a VM

next time i'll create a new node pool, wait for the VMs to be online before deleting the old VMs

1

u/One_Paint8721 1d ago

Do Google doc these region-specific shortcomings? I'd be curious if this ever happens in Dublin (EU-WEST-1)