Five implementations, one set of trades, and the bug our tests could not see

We price one fixed set of trades through five implementations: our own library (CPd, the engine inside cpStudio) natively in F# and again through its generated Python wrapper, QuantLib natively in C++ and again through its Python bindings, and OpenGamma Strata in Java. Every number is also checked against a closed form derived from first principles, with no pricing library involved. The comparison runs in our CI on every commit, at a relative tolerance of 1e-11.

Building it found two genuine bugs in our own code. Neither was subtle in its effect — one mispriced a leg by a factor of 324 — and both sailed through a test suite we considered thorough. This post is about the method, and about why our existing tests were structurally unable to see either bug.

The bug

Our cross-currency swap exchanges a USD fixed leg against a ZAR floating leg. The ZAR leg's notional is linked to the FX rate: one face value denominates the USD leg, and the ZAR leg exchanges face value times spot. At 18 ZAR per USD, a 1,000,000 USD leg should exchange 18,000,000 ZAR.

The payoff built its FX pair inverted. The ZAR leg exchanged roughly 55,546 ZAR — face value divided by spot, off by a factor of spot², or 324.

The interesting question is why nothing caught it. Two reasons, and both generalise.

First, the existing test solved for the par fixed rate and then repriced the swap at that rate, asserting NPV zero. A par-solve-and-reprice test is insensitive to notional scale by construction: scale the mispriced leg by any factor and the solved rate absorbs it. The test was correct and passing. It just could not see this class of error.

Second, and worse: the obvious validation case is degenerate. If the floating leg projects and discounts on the same curve, a floating leg plus its two notional exchanges is worth exactly zero — for any notional. The notional could be wrong by any factor and a cross-currency test built on one curve per currency would still pass. Our validation spec therefore discounts the ZAR legs on a collateral curve 50bp above the projection curve. That spread is load-bearing, not decorative. Without it, this bug passes; with it, QuantLib and Strata both disagreed with us, they agreed with each other, and the disagreement was real.

That is the case for cross-implementation validation stated plainly. Two independent libraries priced the same trade as ours and got a different answer, and they were right.

The method

The spec is deliberately synthetic and calendar-free: no holiday calendar, no business-day adjustment, Act/365F everywhere, quarterly periods generated by exact three-month steps, a flat continuously-compounded 6% curve, and for the cross-currency cases a flat 3% USD curve, spot 18, and the 6.5% collateral curve above. The point is that schedule generation cannot differ between libraries, so any disagreement is pricing mathematics, not conventions.

Each metric is checked twice, because the two checks catch different failures:

1. Against a closed form computed from first principles. This catches all the libraries being wrong the same way, and catches us deriving a reference circularly from the implementation under test.
2. Against each external library's committed value. This catches errors where a closed form does not exist, or where one would be too easy to derive from our own code.

QuantLib and Strata are not build dependencies. Their values live in a committed CSV with a provenance header (library versions, compiler, interpreter, timestamp), regenerated offline by a script. The comparison test fails for any case that lacks a row from every library that case claims — so a metric cannot be added without every library pricing it, and cross-library coverage cannot silently shrink.

The results

Twenty metrics across fourteen instruments: deposits, FRAs, OIS, a fixed-coupon bond, an FRN, vanilla and tenor-basis swaps, caps and floors, European swaptions, FX outrights and swaps, cross-currency basis swaps, cross-currency swaps with and without mark-to-market notional resets, and a 15-pillar curve bootstrap. Values below are ZAR except where the metric is a rate or discount factor; the deviation column is the worst relative disagreement between our value and any other implementation's.

† Strata only: QuantLib 1.43 has no priceable mark-to-market cross-currency swap, only bootstrap helpers, so that row is a two-library comparison and says so in the case's own declaration rather than by a quietly missing CSV row.

The worst disagreement anywhere is the 15-year bootstrapped discount factor at ~7e-13 — three solvers converging in three different interpolation spaces (we and QuantLib in log-linear discount factors, Strata in linear zero rates). Next is the OIS at 2.7e-13, which compounds roughly 1,100 daily accruals and accumulates rounding the term instruments never see. Everything else sits at 1e-13 or below. These are summation-order effects, not method differences. The 1e-11 gate clears the measured floor by about 15×; 1e-12 would leave under 2× and fail on noise.

On this scope there is no correctness basis for preferring any of the three libraries. That is the honest headline, and we are content with it.

The second axis, and the second bug

Comparing libraries answers whether the mathematics is right. It says nothing about whether a binding faithfully carries it — and we ship a generated Python wrapper whose correctness nothing checked. So each library is priced through its native interface and, where one exists, its Python binding. The two QuantLib generators are line-for-line ports of each other against the same pinned 1.43, so a divergence there isolates SWIG. Same-library pairs agree to the last bit or two (worst ~2e-15), so the gate is set by cross-library disagreement, not cross-binding.

Building our own Python row found three defects in the wrapper's code generator. The dangerous one: reading a union case's payload as obj.<Case> resolves, under pythonnet, to the nested case class rather than the value whenever the two share a name. Where it threw an exception it was merely annoying. Where it did not, it was worse: convention overrides silently never matched, and the run produced plausible numbers — the cross-currency basis swap came out 5% wrong with nothing to indicate it. All three defects are fixed in the generator; the validation generator runs against the fixed wrapper with no workarounds.

Two bugs, then, and a pattern: both produced plausible numbers, both passed the tests we had, and both fell to an independent implementation pricing the same trade.

What about CDS?

Credit default swaps sit outside this comparison, deliberately. Their reference implementation is the ISDA Standard Model, and that model publishes its own test grids — 31,044 rows across six currencies. We reproduce those grids to within $0.01 on $10 million notional (written up separately). Agreeing with QuantLib's and Strata's readings of the ISDA model would be a weaker check than reproducing the model's own published output, so the CDS row points at the grids instead.

What this does not prove

The spec is synthetic on purpose, and that is also its limitation. Real trades carry holiday calendars, spot lags, day-count conventions and business-day rules, and in our experience unexamined convention defaults — not pricing mathematics — are where cross-library differences actually come from in production. This suite proves the mathematics agrees when conventions are pinned. It does not prove two libraries will agree on your trade with each library's defaults left in place. They mostly will not, and the differences will be schedule dates, not discount factors.

Reproducing this

Versions are pinned: QuantLib 1.43 (tag v1.43, commit 6b57206e), both the PyPI wheel on CPython 3.13 and a source build with Apple clang 21; Strata 2.12.73 (tag v2.12.73, commit 1b40934d) on Temurin JDK 21. The spec, the closed forms, the committed reference CSV and the per-library generators are part of CPd's repository, and the comparison runs in the ordinary test gate on every commit — not as a one-off exercise that decays.

The suite exists and fails the build so that is still true next year.

CP Dynamics builds cpStudio, a cross-asset pricing and risk platform.