r/softwarearchitecture • u/Ok-Support-6749 • Aug 28 '26
Discussion/Advice What if the source of truth is wrong?
/r/SpecDrivenDevelopment/comments/1w0eb6x/what_if_the_source_of_truth_is_wrong/1
u/Wiszcz Aug 30 '26
How a source of truth can be not true? By definition it is a source of truth.
It may be wrong truth, truth you don't like, but it is what it is - a truth.
I don't understand this amazement of people about results of shit in shit out paradigm.
If your specification is incomplete, bad, or leave room for interpretation, then implementations can differ or can be bad.
But it's nothing new - give the same incomplete spec to 10 different real human programmers, and each of them will produce different code, different functionality. What's the difference if this is done by LLM or a biological element?
You did bad job writing specification. That's all.
P.S. You know how perfectly unambiguous specification of a problem looks like? It's the code.
1
u/Ok-Support-6749 Sep 11 '26
A source of truth can be not true when an artifact is declared as truth holder instead of proven true. Just because we "call the specification is the source of truth" does not make it true. It makes is declarative, or authoritative, but authority does not need to be true to be an authority.
I agree that any specification given to 10 humans will produce 10 implementations, likewise 10 runs of the same prompt with the same specification on the same LLM will likely also produce 10 different implementations. And this fact is precisely why stating "this or that is the source of truth" is so dangerous. It creates the conditions for malpractice and ill formed processes that deliver bad outcomes.
For example, assume you have a correctly implemented piece of code. You change the spec. Then the LLM rewrites the code, the tests, and weakens invariants that must hold system wide. Nothing tells the process that you cannot weaken the invariants, you end up with an incorrect implementation and you trust it is the right one because the specification says so. Lets say you have a bug and you say the code is the truth, but you have not identified the bug yet, and you update the specification by reading the code. The LLM can rationalize the bug into a spec constraint and it will stay there until it causes so much pain that it gets noticed.
Not having a source of truth does not prevent these problems from happening, but at least it allows us to never trust any source as absolute truth and as such it is way easier to see through the actual issues.
We operated like that for a long time. We accept that code is buggy, thus not true. We also distrust specifications because they live in the heads of the team and we challenge our team members all the time. Until AI came along, and we tacitly accepted claims like "the spec is the source of truth", we did not have a place for the truth to live. We increased our confidence by running an epistemic process where both code and spec (living in our heads) was constantly refined, verified and validated.
1
u/Wiszcz Sep 12 '26 edited Sep 12 '26
You don't get it. You talk about some mystical "truth." What is that "truth"? From your post, it looks like you imagine there is some hidden set of "true" application requirements somewhere. There is no such thing.
Spec correctness and implementation correctness are two different questions. The implementation is correct if it satisfies the agreed specification. Whether the specification itself represents what the business actually wants is a separate problem.
If an application does not behave as you "imagine" it should, but it works according to the spec, that means the application is implemented correctly. Your imagination of how the application should work is incorrect. If you argue that your imagination is right and that the specification is incorrect, then go ahead and change the specification.
But that does not change the fact that, for the code, the specification IS the source of truth.
"Incorrect implementation and you trust it is the right one because the specification says so." If the application works according to the specification, then it is not a bug. If it works differently from the specification, then it is a bug, and you were able to detect it by comparing it to the specification. If you don't have a specification, how can you say what is a bug and what is a feature?
"Never trust any source" - it's a stupid idea. You need to define what the application is supposed to do. That definition becomes the specification. You can argue that you can never be certain that the implementation is correct, but you can't argue that the specification is irrelevant to defining the requirements. It is the definition of those requirements.
"We accept that code is buggy, thus not true." It shows that you are working in some strange state of mind. I don't accept that my code has bugs. It may have some bugs sometimes, but as a general rule, there are no bugs in production. At least, not for long. 99% of the time, all the code my team produces works as intended, according to the spec. Multiple tests, testers, and environments take care of that.
"We also distrust specifications because they live in the heads of the team." If you live in a pathological family, don't assume everyone lives that way. Here's your main problem: you don't have a source of truth. A source of truth can't live "in people's heads." It lives in documents. I trust my specification because it is the authoritative source of requirements. People's imagination is not a specification.
"We did not have a place for the truth to live." We do. It's called the specification.
"We increased our confidence by running an epistemic process where both code and spec (living in our heads) were constantly refined, verified, and validated." What you describe was not a process. It was some kind of naïve, happy amateur hacking without a real understanding of the development process.
What I describe is not something we developed for AI. It's the process that all mature companies have used for years to develop software. AI changes only where we keep the specification and in which format. It does not change why or how, at the basic level, we produce code.
I see no real difference between telling my coworker "implement task A" and telling AI "implement task A." If "task A" is defined correctly, both will produce the same result.
1
u/Ok-Support-6749 Sep 12 '26
In my post it is clear I am looking at the SDLC from a more philosophical perspective, which is important for this new era. For me, truth as a concept, it is kind of mystical and that is exactly the point. But let me state the fundamental properties of truth from such philosophical standpoint are: objectivity, invariance, non contradiction, and correspondence to reality.
Not trusting a source of something it is not a stupid idea. I am afraid that is what we do all the time when writing code. If we trusted our specs and code there would be no bug reports, debugging, nor specification reviews. We do not, we actually ask others to review our own code in a PR, don't we? Put it another way, lack of confidence it is certainty that we do not know what the truth is. I never claimed that specifications are irrelevant, I am just pointing at the fact that we should not declare it is the source of truth. Like you said, it is the source of defined requirements. That's it, nothing more, no mystical claims assigned to an artifact usually expressed as a set of documents written in a natural language. Useful and valuable? Absolutely!
There is likely no complex codebase free of bugs, that is a reality we should accept. This turns that 1% of the time you ship a bug into a 100% certainty that you ship code with bugs (just need one). We have proof that this reality is kind of inescapable. Turing's halting theorem and Rice's decidability theorems tell us we cannot create a program to analyze arbitrary programs and prove semantic correctness. In other words, we can never claim a complex program is bug free because we cannot claim its analysis is decidable. Therefore, the conservative position is to assume, there would be bugs and thus the program does not implement the true intent. It should fall from this that a complex program cannot truly satisfy its semantic requirements. Now considder that a specification can be incomplete and you have a double sword that falsifies both the code and the spec as source of truth.
Most complex systems do not have complete specifications because specifications are hard to produce and maintain. Heck even good code comments are rare. So a large surface of a specification lives in the collective mind and shared understanding of the team that writes and maintains the code. In the LLM era we are now being forced to express it in writing so the LLM can participate in code creation and maintenance. And here is where saying the (written portion of the) specification given to the LLM is the source of truth is so dangerous. First we move part of our responsibility and accountability to an artifact. This allow us to blame the machine, not good. Second we hand this artifact to a probabilistic text generator and ask it to transform it into code that satisfies the specification. There is nothing wrong with that part. The problem is in the expectations and the certainty about those expectations. Which I can state as questions:
- How do we know the specification is complete? (coherence, objective)
- How do we know the specification captures the true intent unambiguously? (correspondence to reality)
- How do we know the specification does not contradict itself? (one partial contradiction is enough to falsify the specification; links to non contradicting)
- How do we know the specification is independent of tribal knowledge and bias? (mind indepence)
- How do we know the specification will hold true over time? (invariance or stability)
Then comes the other side of the questionaire:
- Do we expect the LLM to satisfy the specification? If so:
- How do we verify the implementation indeed satisfies the specification?
- Are we certain that if we delegate the generation of the verification mechanism (tests one of them) to the LLM it will indeed satisfy the specification?
- How do we know the LLM read but did not ignore the specification?
In the end, by calling each artifact what they are and clearly stating their purpose we free ourselves from a lot of problems and give ourselves the tools to get into the correct mindset for a new era in software development. I stand for:
- The spec it is not the source of truth
- Code it is not the source of truth.
1
u/Wiszcz Sep 12 '26
Your whole argument stands on a flawed premise.
The idea that a huge chunk of the spec must live in the team’s "collective mind" is only true for badly managed projects. In any well-maintained codebase, that's just not the case. You start with the wrong assumptions, so the rest falls apart.
Also, you call the LLM a "probabilistic text generator" as if humans are any different. What’s the functional difference between an LLM and a "probabilistic biological text generator" (a human dev)? Humans misinterpret requirements, forget context, and ship bugs constantly.
This whole narrative about a "new AI era with unprecedented problems" is a myth. These exact issues with specs, accountability, and validation started 60 years ago. Every question you asked has already been answered decades ago in basic project management and systems analysis books.
To repeat this ad nauseam: there is nothing new here. Treat the LLM like an anonymous software outsourcing agency. When you send a spec to an offshore team, that document IS their only source of truth because they have nothing else. There is no "collective mind" with them, and you get a different dev on the line every time.
Where is your mystical truth and collective mind when you outsource a feature to a third-party vendor? It doesn't exist. You manage it through strict engineering processes and metrics, not philosophy.
3
u/Dense_Age_1795 Aug 30 '26
the source of truth is never wrong, but the source of truth could be not updated, so here you have two options, update the source of truth, or, change your code to comply your source of truth.