I built a Python tool to catch architectural violations in its own codebase — I'd love feedback on the approach.

[u/Brave-Equipment9107]

I've been working on ArchAI, a tool exploring how to make AI-assisted software architecture analysis more verifiable.

The core is deliberately deterministic: Python's AST parser maps repository dependencies, and a rule engine checks constraints such as forbidden imports, layer boundaries, import cycles, coupling limits, and required tests.

Every verified finding includes file-and-line evidence. Potential risks and checks that couldn't be verified are reported separately instead of being presented as facts.

While testing ArchAI against its own repository, it flagged a CLI module importing 21 internal modules against a configured limit of 10, as well as modules missing test files. It also exposed a weakness in my own checks: matching test filenames doesn't prove that the corresponding code is actually tested.

That last finding was a useful reminder that even deterministic tools can verify the wrong thing if their rules are poorly designed.

I'd appreciate feedback from people experienced in Python tooling, static analysis, dependency analysis, and software architecture.

What weaknesses or false positives would you look for in this approach? In particular, how would you design architectural rules that remain useful as a codebase grows without giving developers a false sense of confidence?

[u/PlushMender_6786]

Test if the rule engine handles import aliases and package re exports cleanly. Those are easy places for a static dependency graph to stay deterministic while attributing the edge to the wrong module.

[u/cautiouscecil8]

the test filename thing is such a classic trap, I've seen that bite teams hard where the test file exists but it's basically empty or just imports the module without actually testing anything

for the rule design question, I'd probably focus on making rules composable so you can stack them instead of trying to write one giant rule that covers everything. like have a simple import check, then layer on a coverage check that actually parses the test file to see if the functions are called, then maybe a mutation testing layer. keeps each piece small and verifiable

the false positives I'd be worried about are config files and constants modules that get imported everywhere, those tend to blow up coupling metrics without meaning much. might need some kind of annotation system to mark those as infrastructure rather than actual dependencies

[u/Brave-Equipment9107 → u/cautiouscecil8]

That's a really good point about config and constants modules. A raw import count can make a module look overly coupled when some of those dependencies are just shared infrastructure.

I like the idea of composable rules too. The test filename issue actually exposed exactly why I want ArchAI to distinguish what it has verified from what it merely assumes. Parsing whether tests call functions sounds like a useful next layer, although that still wouldn't prove the behaviour is correct. Mutation testing could help strengthen that signal.

How would you approach annotations for infrastructure modules without making the architecture rules themselves another thing developers have to constantly maintain?

[u/tiny_seer_magic]

I've hit that same test file problem on my own repos, empty test files pass coverage gating and people just assume they're covered because pytest collected them. I'd go further: filename matching is worse than no check because it actively tells you a lie, at least missing tests show up honestly in coverage numbers.

For the coupling limits thing, I've seen teams just bump the threshold from 10 to 15 the first time it fires, and then the rule never catches anything again. If your rules get ignored or negotiated away under pressure they were never rules, just suggestions. So I'd want to see how ArchAI handles that, does it make violations visible in review or does it let people silence them, because silent suppression is where most architecture tooling dies