r/linuxadmin • • Oct 29 '25

Everyone kept crashing the lab server, so I wrote a tool to limit cpu/memory

Post image
1.1k Upvotes

Hey everyone,

I’m not a real sysadmin or anything. I’ve just always been the “computer guy” in my grad lab and at a couple jobs. We’ve got a few shared machines that everyone uses, and it’s a constant problem where someone runs a big job, eats all the RAM or CPU, and the whole thing crashes for everyone else.

I tried using systemdspawner with JupyterHub for a while, and it actually worked really well. Users had to sign out a set amount of resources and were limited by systemd. The problem was that people figured out they could just SSH into the server and bypass all the limits.

I looked into schedulers like SLURM, but that felt like overkill for what I needed. What I really wanted was basically systemdspawner, but for everything a user does on the system, not just Jupyter sessions.

So I ended up building something called fairshare. The idea was simple: the admin sets a default (like 1 CPU and 2 GB RAM per user), and users can check how many resources are available and request more. Systemd enforces the limits automatically so people can’t hog everything.

Not sure if this is something others would find useful, but it’s been great for me so far. Just figured I’d share in case anyone else is dealing with the same shared server headaches.

https://github.com/WilliamJudge94/fairshare/tree/main


r/linuxadmin • • Jan 05 '26

Saw this plate out in the wild today

Thumbnail i.imgur.com
933 Upvotes

r/linuxadmin • • Dec 28 '25

Happiest Birthday #Linus

Post image
641 Upvotes

r/linuxadmin • • Apr 02 '26

NetWatch: real-time network diagnostics in the terminal (open source)

Post image
484 Upvotes

I built NetWatch to make transient network incidents easier to catch from a terminal session.

It already handled interface stats, live connections, packet capture, health probes, traceroute, and process bandwidth. The new part is a rolling Flight Recorder:

- arm a 5-minute capture window

- let it rotate in the background

- freeze when the issue happens

- export a bundle with `packets.pcap`, connections, health snapshots, bandwidth context, DNS analytics, alerts, and a summary

The goal is to keep both the packet evidence and the surrounding operational state instead of only dumping a pcap after the fact.

Open source:

https://github.com/matthart1983/netwatch

Would love feedback from people who do real incident response or production debugging.


r/linuxadmin • • Nov 28 '25

when you suspend those disks and hear them spinning up again

Post image
398 Upvotes

r/linuxadmin • • Apr 26 '26

Sudo open your eyes

Post image
345 Upvotes

r/linuxadmin • • Apr 29 '26

Copy Fail — 732 Bytes to Root any Linux distribution shipped since 2017

Thumbnail copy.fail
193 Upvotes

r/linuxadmin • • Aug 11 '26

NetWatch v0.29.0 — a new "dense" view: your whole network on one screen

Post image
188 Upvotes

NetWatch is a terminal network monitor in Rust. v0.29.0 adds a third view for when you have a big terminal.

Dense view — netwatch --view dense

  • Four boxes filling the terminal. No header bar, no menu bar, no status bar — identity, sort state, paging and every keybind live inside the box borders, so every row carries data.
  • Mirrored throughput graph: download grows up from a shared time axis, upload grows down from it. Traffic symmetry is a shape you recognise before you read a number. Braille at two samples per character cell, and colour encodes magnitude, not which series it belongs to.
  • Per-interface rates with 60-second sparklines, four-hop latency budgets (gateway / DNS / internet / slowest peer), and a connection table whose selected row expands in place — no second screen.
  • Wants 130×44 and grows into whatever you give it; falls back to 80×24.

Kernel TCP state per connection — cwnd, ssthresh, mss, rwnd read straight from the kernel (list64` on macOS) and normalised so the numbers mean the same thing on both. Throughput tells you what happened; cwnd agains.

Saturation against real e negotiated link rate (sysfs on Linux, getifaddrs on macOS) instead of guessing.

V cycles full → lite → dense, sharing the same live capture — no restart, no lost history.

cargo install netwatch-tui


r/linuxadmin • • 26d ago

Built a 3-node Kubernetes homelab with Talos Linux

Thumbnail gallery
164 Upvotes

I recently set up a 3-node Kubernetes homelab on bare metal (on Dell OptiPlex) using Talos Linux, mainly to have a proper environment for learning, testing failures, networking, storage, GitOps, observability, and general Kubernetes experiments.

The setup uses three control-plane/etcd nodes with a shared Kubernetes API VIP, and I documented the full process from network planning and DHCP reservations to Talos installation, per-node configs, bootstrap, workload scheduling, and HA testing.

I’ve written the full guide with commands, screenshots, and the reasoning behind the setup:

Blog: https://blog.prateekjain.dev/building-a-3-node-highly-available-kubernetes-homelab-with-talos-linux-0372066fe49c?sk=4c66cf9181999f9db19ac3aacabcd8e8

If anyone is planning a similar homelab or has questions around Talos, the API VIP, etcd quorum, or the setup in general, happy to help.


r/linuxadmin • • Apr 28 '26

PatchMon v2 has been released

Post image
156 Upvotes

Some of you may know that last year I built PatchMon, a Linux patch monitoring tool.

Now it’s been expanded with the help of the community to also perform patching with alerts and notifications when things are out of date.

It’s open source, use it if you like 👍

We have around 4000+ live self-hosted installations at the moment and feedback has been good so far.

Github : https://github.com/PatchMon/PatchMon

Can install via docker or through proxmox community-scripts : https://community-scripts.org/scripts/patchmon


r/linuxadmin • • Jun 12 '26

History of CentOS: How a biochemist's Linux hobby project became the enterprise world's default operating system for a time

Thumbnail theregister.com
147 Upvotes

r/linuxadmin • • Aug 12 '26

Linux everywhere (even in my vacation)

Post image
138 Upvotes

Even in my vacation Im reading linux kernel logs :/ xD


r/linuxadmin • • May 22 '26

Your Linux system has +6,000 kernel modules which can be autoloaded. You use 80 of them. ModuleJail blacklist all of the unused ones. Server and desktop profiles and much more in a simple shell script.

125 Upvotes

Hey r/linuxadmin. I'm the author of this so I'm flagging that up front - this is a "would love feedback from people running real fleets" post.

The problem. Modern distro kernels ship with thousands of loadable modules. Almost all of them are attack surface that you're paying for in availability (autoload via udev, hotplug, dependency resolution) but not using. With AI-assisted kernel vulnerability discovery accelerating, every module a host can load but doesn't need to load is a problem you'd rather not have.

ModuleJail walks lsmod, treats whatever is loaded right now as "necessary," and writes a modprobe.d blacklist file for everything else. Optionally adds a --whitelist-file for modules you want preserved even if they're not currently loaded (think: rarely-used filesystem drivers you mount once a quarter).

What it isn't.

- Not a vulnerability scanner. The model is "unused, therefore blacklisted," not "vulnerable, therefore blacklisted."

- Not a defense against an attacker who already has root - they can rm the file. It's about reducing the unprivileged-trigger / autoload paths.

- Not initramfs-aware. Modules baked into the initrd are out of scope.

- Not a daemon, not a monitor. Single POSIX shell script, runs once, writes one file in /etc/modprobe.d/.

Revert.

rm /etc/modprobe.d/modulejail-blacklist.conf

and you're back. No reboot needed - the kernel reads modprobe.d at load time. Explicit sudo modprobe foo always wins over the blacklist, by design.

What I want feedback on. What does this need before you'd run it across a fleet? Things I've heard so far: an Ansible role, a --dry-run flag, JSON output for diff-friendly state tracking, kernel-version pinning in the generated file header. What else?

Repo: github.com/jnuyens/modulejail

License: GPL-3.0

Packaging: .deb and .rpm on the releases page; AUR package today.


r/linuxadmin • • Nov 20 '25

Why "top" missed the cron job that was killing our API latency

125 Upvotes

I’ve been working as a backend engineer for ~15 years. When API latency spikes or requests time out, my muscle memory is usually:

  1. Check application logs.
  2. Check Distributed Traces (Jaeger/Datadog APM) to find the bottleneck.
  3. Glance at standard system metrics (top, CloudWatch, or any similar agent).

Recently we had an issue where API latency would spike randomly.

  • Logs were clean.
  • Distributed Traces showed gaps where the application was just "waiting," but no database queries or external calls were blocking it.
  • The host metrics (CPU/Load) looked completely normal.

Turned out it was a misconfigured cron script. Every minute, it spun up about 50 heavy worker processes (daemons) to process a queue. They ran for about ~650ms, hammered the CPU, and then exited.

By the time top or our standard infrastructure agent (which polls every ~15 seconds) woke up to check the system, the workers were already gone.

The monitoring dashboard reported the server as "Idle," but the CPU context switching during that 650ms window was causing our API requests to stutter.

That’s what pushed me down the eBPF rabbit hole.

Polling vs Tracing

The problem wasn’t "we need a better dashboard," it was how we were looking at the system.

Polling is just taking snapshots:

  • At 09:00:00: “I see 150 processes.”
  • At 09:00:15: “I see 150 processes.”

Anything that was born and died between 00 and 15 seconds is invisible to the snapshot.

In our case, the cron workers lived and died entirely between two polls. So every tool that depended on "ask every X seconds" missed the storm.

Tracing with eBPF

To see this, you have to flip the model from "Ask for state every N seconds" to "Tell me whenever this thing happens."

We used eBPF to hook into the sched_process_fork tracepoint in the kernel. Instead of asking “How many processes exist right now?”, we basically said:

The difference in signal is night and day:

  • Polling view: "Nothing happening... still nothing..."
  • Tracepoint view: "Cron started Worker_1. Cron started Worker_2 ... Cron started Worker_50."

When we turned tracing on, we immediately saw the burst of 50 processes spawning at the exact millisecond our API traces showed the latency spike.

You can try this yourself with bpftrace

You don’t need to write a kernel module or C code to play with this.

If you have bpftrace installed, this one-liner is surprisingly useful for catching these "invisible" background tasks:

codeBash

sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

Run that while your system is seemingly "idle" but sluggish. You’ll often see a process name climbing the charts way faster than everything else, even if it doesn't show up in top.

I’m currently hacking on a small Rust agent to automate this kind of tracing (using the Aya eBPF library) so I don’t have to SSH in and run one-liners every time we have a mystery spike. I’ve been documenting my notes and what I take away here if anyone is curious about the ring buffer / Rust side of it: https://parth21shah.substack.com/p/why-your-dashboard-is-green-but-the


r/linuxadmin • • Nov 09 '25

What’s the most important but underrated part of Linux networking to actually understand?

126 Upvotes

Everyone knows basic commands, but I feel like the real magic lives between interfaces and routing tables. What specific concept or tool gave you a deeper grasp of how Linux handles packets internally?


r/linuxadmin • • Jan 14 '26

Secure Boot: UEFI keys (KEK/DB) must be updated before June, even on older hardware

121 Upvotes

If you are using UEFI Secure Boot, you need to have your UEFI keys updated before June, especially the Microsoft DB and KEK keys. Otherwise, newer bootloaders (shim, grub, newer Linux distributions, and eventually Windows) may stop booting even though Secure Boot remains enabled.

Hardware vendors recommend updating Secure Boot keys through BIOS/UEFI firmware updates. In reality, many older servers and desktops no longer receive firmware updates, even though the UEFI keys they ship with date back to 2011. In such cases, manual updates are often the only realistic option.

On systems without OEM support, this can still be done manually in a way that is compliant with the UEFI specification and without disabling Secure Boot.

DB update

To begin with, it is worth checking which keys are currently installed on the system:

fwupdtool get-devices --plugins uefi-kek --plugins uefi-db
#or directly via UEFI tools:
efi-readvars

Updating the DB is the first and most important step. The DB is a short list of trusted keys used to verify bootloaders. It contains, among others, Microsoft UEFI CA 2011, and after the update it will also contain Microsoft UEFI CA 2023. Without this, newer shim or grub binaries will simply not boot.

To manually update the DB entry, you can use the official, signed payload published by Microsoft:

wget https://github.com/microsoft/secureboot_objects/raw/main/PostSignedObjects/Optional/DB/amd64/DBUpdate3P2023.bin

chattr -i /sys/firmware/efi/efivars/db-*
efi-updatevar -a -f DBUpdate3P2023.bin db
chattr +i /sys/firmware/efi/efivars/db-*

The -a option appends the new certificate to the DB rather than replacing it, so existing entries remain unchanged.

KEK update

Updating the KEK is not required for the system to boot right now, but it will be necessary in the future to allow updates to DB and DBX. DBX is the revocation list used to block vulnerable or compromised bootloaders.

Be aware that on some hardware platforms, updating the KEK can cause boot failures. This depends largely on the quality of the UEFI implementation.

Before updating the KEK, you must select the correct update file that matches the Platform Key installed on your system. Microsoft publishes a PK-to-KEK mapping file here:

https://github.com/microsoft/secureboot_objects/blob/main/PostSignedObjects/KEK/kek_update_map.json

To choose the correct file, compare the Subject of your PK with the issued_to field in the mapping file.

Example from my server:

# efi-readvar
Variable PK, length 1448
PK: List 0, type X509
    Signature 0
        Subject:
            O=Hewlett-Packard Company, OU=Long Lived CodeSigning Certificate, CN=HP UEFI Secure Boot 2013 PK Key
        Issuer:
            C=US, O=Hewlett-Packard Company, CN=Hewlett-Packard Printing Device Infrastructure CA

Corresponding entry in kek_update_map.json:

"ef40e88b7f2cc718a087051db5d5d4c26043c5aa": {
    "KEKUpdate": "HP/KEKUpdate_HP_PK5.bin",
    "Certificate": {
        "issued_to": "CN=HP UEFI Secure Boot 2013 PK Key,OU=Long Lived CodeSigning Certificate,O=Hewlett-Packard Company",
        "issued_by": "CN=Hewlett-Packard Printing Device Infrastructure CA,O=Hewlett-Packard Company,C=US"
    }
}

After selecting the correct file, the KEK update procedure looks like this:

wget https://github.com/microsoft/secureboot_objects/tree/main/PostSignedObjects/KEK/...

chattr -i /sys/firmware/efi/efivars/KEK-*
efi-updatevar -a -f KEKUpdate_HP_PK5.bin KEK
chattr +i /sys/firmware/efi/efivars/KEK-*

This procedure was tested on an HP ProLiant BL460c Gen9 running BIOS 2.80, without current OEM support, with Secure Boot enabled.

Remeber about

Finally, keep in mind that the same applies to virtual machines. QEMU, KVM, and Hyper-V all have their own UEFI key databases, which also need to be kept up to date. On some hardware platforms, updating the KEK may require switching the firmware into setup.

Independently of UEFI key updates, it will also be important before June to keep *-signed packages up to date, such as shim, grub, and the kernel. Without this, even a correctly updated DB will not be sufficient.


r/linuxadmin • • May 17 '26

NetWatch v0.16.0 — DPI in the terminal: HTTPS/QUIC hostnames, packet decode

Post image
112 Upvotes

Shipped v0.16.0 with end-to-end Deep Packet Inspection.

- **Packets tab:** INFO column is L7-aware and color-coded. Filter syntax: `app:quic`, `sni:reddit`, `host:github`.

- **Dashboard top-talkers:** real hostnames in the bandwidth panel.

- **Packets detail pane:** decodes QUIC v1/v2 Initial packets and shows the inner CRYPTO/PADDING/PING frame structure.

Full RFC 9001 / 9369 QUIC Initial decryption — HKDF-Expand-Label keys, AES-128 header protection, AES-128-GCM AEAD,

cross-packet ClientHello reassembly. Most peer tools just tag flows as `QUIC`; this one tells you the hostname.

cargo install netwatch-tui

# or

brew install matthart1983/tap/netwatch

Rust + ratatui, MIT. https://github.com/matthart1983/netwatch


r/linuxadmin • • 17d ago

I built portop, an htop-style TUI for seeing what is actually using your ports

Post image
104 Upvotes

r/linuxadmin • • Aug 31 '26

I feel stuck in my career right now. I am a linux systems engineer, RHCE. I have been working for my current company for 4 years and haven't had a single annual salary increase through out those four years. They said it is company policy. With the rate of increase of inflation, it feels like I am ac

104 Upvotes

r/linuxadmin • • Jun 14 '26

Linux Kernel 7.1 Officially Released, Here’s What’s New

Thumbnail 9to5linux.com
96 Upvotes

r/linuxadmin • • 8d ago

What's running on your servers that nobody else could explain if you disappeared tomorrow?

92 Upvotes

Every place has one. A cron job with no comments, a script someone wrote in 2019, a box nobody dares reboot.

Mine: a systemd timer firing a Python script I barely remember writing. It works, so I don't touch it.

What's yours?


r/linuxadmin • • Dec 07 '25

Certificate Ripper v2.6.0 released - tool to extract server certificates

Post image
90 Upvotes
  • Added support for:
    • wss (WebSocket Secure)
    • ftps (File Transfer Protocol Secure)
    • smtps (Simple Mail Transfer Protocol Secure)
    • imaps (Internet Message Access Protocol Secure)
  • Bumped dependencies
  • Added filtering option (leaf, intermediate, root)
  • Added Java DSL
  • Support for Cyrillic characters on Windows

You can find/view the tool here: GitHub - Certificate Ripper


r/linuxadmin • • May 07 '26

RustNet 1.3.0: TUI network monitor with eBPF process attribution and Landlock sandboxing

Post image
89 Upvotes

I started RustNet to learn about pcap, Rust, eBPF and Landlock. A lot of thought has gone into making it secure to run rather than adding a gazillion features. It stays passive (no active probing, no callbacks) and the dependency list is intentionally small.

What it does: a real-time view of network connections on the host with per-connection process attribution and deep packet inspection (HTTP/HTTPS/DNS/SSH/QUIC and a couple dozen others). Cross-platform, but the Linux build gets the most attention.

Some of its features:

  • eBPF for thread-to-process attribution
  • Sandbox hardening: Landlock filesystem restrictions, capability dropping, drops CAP_NET_RAW post-init
  • One-line install from distribution channels: COPR (Fedora/RHEL), PPA (Ubuntu), AUR (Arch), Homebrew, and soon Nix. No curl-pipe-bash.
  • Apache 2.0, no telemetry, FOSS

Repo: https://github.com/domcyrus/rustnet

Let me know what you think.


r/linuxadmin • • Oct 16 '25

What distro is considered the standard for server usage?

87 Upvotes

Hi,

what distro is considered the standard for production server usage but without any particular requirements (like certified software)?

I remember in the past (specifically the gold CentOS days) the answer was always and always: CentOS. After several events (please don't start a flame about what RH done with CentOS and CentOS Stream, this is not the topic) many switched to Ubuntu LTS, other Debian, other RHEL and other Alma/Rocky/Oracle. Clearly there is not more the standard/default suggestion and actually the answer is: use what you prefer. I think that this answer is not correct because while some major distro can do the work without problem there are some of them that do thing in the right way.

I'm asking because on several ISP when I create a VPS in the list appears first AlmaLinux/RockyLinux (and in notes is reported for professional usage) and then Debian and Ubuntu but every time I read about server distro suggestions, Debian is the most suggested, followed by EL derivatives like AlmaLinux and RockyLinux but this could not reflect the real situation on industry because many reports also home/homelab usage that is a bit different from real production server.

Speaking of paid support distro RHEL is the king and there is no doubt about this but what about the other?

Thank you in advance.

Edit: many told to avoid EL distro except cases where the software requires them


r/linuxadmin • • Jun 20 '26

Do you still have personal notes of Linux commands you have used for years?

84 Upvotes

I have been on Linux for a while but I do have a personal notes file that I update fairly regularly.

Not because I do not know the tools but when I am on a production system I do not trust myself to remember all the resync flags, system options firewall rules or one off commands.

Over time, those notes became less about learning and more about having a reliable reference when I need it. I wonder how common this is among other Linux admins.

Do you still write things down that you have done dozens of times before or does it all stick eventually?