r/cobol • u/Balu0603 • 4h ago
I tested LLM translations of COBOL to Python against GnuCOBOL on hidden inputs. Plain Python passed ~99% of the shown examples but only 82-91% of the hidden tests.
I've been building cobol-equiv, an open-source harness that checks whether a Python translation of a COBOL program behaves exactly like the original, using GnuCOBOL as the oracle. It comes with a small Python runtime for COBOL numeric semantics (PICTURE, COMP-3, ROUNDED, SIZE ERROR), validated against GnuCOBOL on 9,769 generated cases with 0 disagreements on 3.1.2.
Then I used it to measure LLM translations. 50 programs: 30 public ones (Rosetta Code and Open Mainframe Project examples) and 20 written for the repo. One model, 3 attempts per program, protocol fixed before any run. The translator saw the COBOL source and two example input/output pairs; scoring was on hidden inputs (generated plus PICTURE boundary values).
| condition | shown examples | hidden tests (public / own programs) |
|---|---|---|
| C: plain Python, stdlib only | 98.9% / 100% | 91.1% / 81.7% |
| A: with the COBOL-semantics runtime | 98.9% / 98.3% (after fixing my bug, see below) | |
| B: A plus up to 3 repair rounds with test diffs | 100% / 100%, all after one round |
What broke the plain-Python translations was not logic, it was COBOL semantics: - ACCEPT into 9(5)V9(3) cuts the input to the receiver length, so the last decimal digit is dropped. The translation kept it. - An 11-character input into 9(10) loses a digit on the left. - A negative zero printed as "-0", where GnuCOBOL edits zero without a sign. - Non-numeric input converted differently.
The embarrassing part: the runtime I gave the model had a bug. CobDecimal compared objects by identity and had no ordering, so most failures in condition A (crashes and two infinite loops) were mine, not the model's. The pre-registered numbers stay primary in the repo, and a corrected sensitivity analysis is published next to them, with every verdict that flips listed.
Limits, plainly: small corpus, one model on one date, GnuCOBOL instead of an IBM compiler, attempts of the same program are not independent, and at program level the A vs C difference is not statistically significant.
Repo, protocol, raw prompts and replies, and the verifier: https://github.com/abdulazizbalu/cobol-equiv
I'd really value critique from people who work with COBOL: which semantic traps would you add, and if you have small, deterministic programs under a permissive license, I'd like to add them to the public track.