Originally created by: grynn-in
partner_data_area_id) on every ledger and trial-balance row in the canonical format. Capping elimination at the smaller side survives only as a test: total elimination per account never exceeds that account's consolidated balance.Intercompany Account). IC Elimination Rule stays only for special cases; unrealized_profit keeps working.Canonical partner column (kept minimal)
partner_data_area_id is added to stg_gl_entries and stg_trial_balance, and to every adapter.bronze_general_journal_account_entries (coalesce(…, '')) → silver_gl_entries (both branches of the positional UNION) → gold_trial_balance_by_partner (one row per partner) → gold_consolidated_trial_balance (translated and ownership-applied). gold_trial_balance keeps its account grain (review B1).epm_raw.trial_balance_submissions in bronze_trial_balance_submissions, which also adds the partner to the synthetic recid hash.The engine
gold_ic_reconciliation (rewritten) holds one row per pair: (entity, partner, account) against (partner, entity, counterpart).epm_staging.intercompany_accounts count, and only when the partner itself consolidates line by line into the group in that period.balance_a, balance_b, matched_amount (the smaller side when the sides offset), difference, the group's ic_difference_account and tolerance, and match_status (matched, within_tolerance or over_tolerance).gold_ic_eliminations (rewritten) keeps the old two-legged grain, so gold_fully_consolidated_tb is unchanged.matched rows eliminate the matched amount.difference rows move each side's residual to the group's difference account; with none configured, only the matched amount is eliminated.unrealized_profit rows are unchanged.elimination_kind and the pair key (entity_a, account_a, entity_b, account_b).gold_ic_unmatched: flagged-account balances with no partner, per group, period, entity and account.ic_account_map(): account → counterpart; a pair is symmetric.Tests
assert_ic_elimination_within_balance, per (group, account, period):assert_ic_elimination_nets_zero is rewritten so it can actually fail. It checks each entry, the FCTB ic_elimination layer per group and period, and each pair: the pair nets to 0 once the difference is booked, or to exactly its difference when the group has no difference account.assert_ic_account_in_one_pair, plus accepted_values on elimination_kind and match_status.Staging README: partner_data_area_id is added to the adapter-contract table, because the canonical models select it by name, plus one column rule: emit NULL until the source's partner is mapped, and never guess it.
Schema
clickhouse/init-db.sql gets DDL identical to konsol's: the raw partner column, the group's ic_difference_account and ic_difference_tolerance, and the new epm_staging.intercompany_accounts.gold_consolidated_trial_balance gains a pre-hook ALTER … ADD COLUMN IF NOT EXISTS partner_data_area_id, and the column is last. dbt-clickhouse's append inserts positionally and ignores on_schema_change on that path, so a mid-row column would shift every later column silently. Measured: it failed with NUMBER_OF_COLUMNS_DOESNT_MATCH before this fix.bronze_general_journal_account_entries uses on_schema_change='append_new_columns', which works on its delete+insert path.Either order works.
intercompany_accounts on every migrate and on submit.bronze_trial_balance_submissions reads the raw partner column only if it exists, and gold_ic_reconciliation reads the group's difference settings only if they exist.epm_staging.intercompany_accounts is created by this project's on-run-start (review B4), by konsol's migrate, and by init-db.sql. All three use identical DDL, so this PR also deploys first on an existing volume.Rollback note: main's gold_consolidated_trial_balance fails (26 vs 25 columns) on a table this PR has built. To go back, drop partner_data_area_id from it and from bronze_general_journal_account_entries, or full-refresh them. I did exactly that on the shared stack after testing. Also drop the orphan tables gold_trial_balance_by_partner and gold_ic_unmatched.
Any future column added to gold_consolidated_trial_balance must use the same pattern: add it in the pre-hook (ALTER … ADD COLUMN IF NOT EXISTS) and make it the last output column. dbt-clickhouse's append inserts positionally and ignores on_schema_change, so any other placement shifts every later column silently.
All runs were on konsolidat.local with grynn-in/konsol#173's code hot-copied into the containers. Each project was built from a copy inside the backend container, outside the bind mount, with --project-dir explicit. The container sets DBT_PROJECT_DIR to the deploy checkout, so without the flag dbt builds main from any working directory; I found that out when my first "branch" run built main.
dbt run --select +gold_ic_eliminations+ gold_ic_unmatched --exclude gold_spread_budget+: 36 OK created, PASS=37, ERROR=0.dbt test on the IC tests: 14/14 PASS. The set: assert_ic_elimination_within_balance, assert_ic_elimination_nets_zero, assert_ic_account_in_one_pair, assert_equity_method_no_ic_elim, assert_unrealized_profit_elimination, assert_ic_reconciliation_matched, assert_waterfall_reconciles, assert_fctb_entity_layer_ties, 2 × accepted_values and 3 × not_null.assert_ic_elimination_within_balance FAILs with 4 rows; restored, it PASSes. (nets_zero can't see a duplicated entry, which still nets to zero; that is why the within-balance test exists.)silver_gl_entries, bronze_trial_balance_submissions and the bronze GL.assert_cf_categories_equal_net_change 84, assert_cta_zero_for_same_currency 24, assert_silver_gl_debit_credit_balance 7 (#155), assert_trial_balance_balances WARN 84, and relationships gold_bs_movement → cash_flow_categories 364.dbt parse: exit 0.1eac5f2, sign-convention follow-ups). The only conflicts were in the D365 adapter header comments; each keeps [#158]'s contract pointer and adds the partner note. After the rebase, dbt parse and dbt compile of every touched model exit 0. The runs above were on the pre-rebase commit, whose SQL is identical outside comments.Setup
2100, tolerance 5.4030 (IC revenue) ↔ 5030 (IC expense).| entity | account | partner | balance | |
|---|---|---|---|---|
| ZZA | 4030 | ZZB | −1000 | pair 1 |
| ZZB | 5030 | ZZA | 1000 | pair 1 (matches) |
| ZZA | 5030 | ZZB | 298 | pair 2 |
| ZZB | 4030 | ZZA | −300 | pair 2 (off by 2) |
| ZZC | 5030 | — | 400 | no partner |
| debit entity → credit entity | eliminated |
|---|---|
| ZZA → ZZB | 298 |
| ZZB → ZZA | 1000 |
| ZZC → ZZA | 400 |
| ZZC → ZZB | 300 |
Every ordered entity pair is eliminated, and ZZC's partnerless expense is eliminated twice.
| account | eliminated | balance | exceeds | after elimination |
|---|---|---|---|---|
| 4030 | 1998 | 1300 | yes | +698 (revenue flipped to a debit) |
| 5030 | −1998 | 1698 | yes | −300 |
gold_ic_reconciliation:
| side a | side b | bal a | bal b | matched | diff | status |
|---|---|---|---|---|---|---|
| ZZA/4030 | ZZB/5030 | −1000 | 1000 | 1000 | 0 | matched |
| ZZA/5030 | ZZB/4030 | 298 | −300 | 298 | −2 | within_tolerance (5) |
gold_ic_eliminations: matched 1000 (Dr 5030 ZZB / Cr 4030 ZZA), matched 298 (Dr 5030 ZZA / Cr 4030 ZZB), and a difference of 2 (Dr 2100 / Cr 4030 ZZB).
gold_ic_unmatched: ZZC 5030 400, reason no partner.
| account | eliminated | balance | exceeds | after elimination |
|---|---|---|---|---|
| 4030 | 1300 | 1300 | no | 0 |
| 5030 | −1298 | 1698 | no | 400 (ZZC, unmatched, stays) |
| 2100 | (destination) | −2 (the pair's difference) |
Cleanup:
gold_ic_unmatched table.reconcile_all() and cancelled the build approvals my documents caused.164275c)gold_trial_balance is back at main's account grain. The new gold_trial_balance_by_partner carries the partner and is read only by gold_consolidated_trial_balance, and through it by the IC models.gold_trial_balance (YoY, YTD, P&L views, cash flow, equity method, ownership) see main's grain again.gold_consolidated_trial_balance (FX revaluation, acquisition, disposal, NCI and the tests) all aggregate per entity or account before joining. The one that didn't is fixed under B2.gold_fully_consolidated_tb sums over the partner, so gold_consolidated_ytd runs one total per account.on-run-start creates epm_staging.intercompany_accounts.side_without_pair).assert_partner_grain_not_fanned_out. Its expected figures come from silver and from gold_consolidated_trial_balance summed to the account grain, not from gold_trial_balance. It checks:gold_trial_balance grain;Three builds on the same live data, each from a copy with --project-dir explicit: main, the pre-fix branch 949ea8e, and this fix.
Data: as above, plus two partners on one account. ZZA 4030 carries −1000 with ZZB and −50 with ZZC; ZZC 5030 carries +400 with no partner and +50 with ZZA. FY2098 P1 is added for year-on-year, and FY2099 P2 for YTD.
Two partners on one account (ZZA 4030, FY2099):
| main | pre-fix 949ea8e |
this fix | true | |
|---|---|---|---|---|
gold_trial_balance rows, P1 |
1 | 2 | 1 | 1 |
| YoY rows / current / prior, P1 | 1 / −1050 / −840 | 4 / −2100 / −1680 | 1 / −1050 / −840 | −1050 / −840 |
| YTD per row, P1 · P2 | [−1050] · [−1060] | [−50, −1050] · [−1060] | [−1050] · [−1060] | −1050 · −1060 |
| consolidated YTD per row (entity layer), P1 · P2 | [−1050] · [−1060] | [−1000, −1050] · [−1060] | [−1050] · [−1060] | −1050 · −1060 |
assert_partner_grain_not_fanned_out |
PASS | FAIL 14 | PASS |
Elimination (FY2099 P1):
Tests
OK created, 0 errors.164275c: see the checks.Cleanup: the ZZ data (FY2098–2099), the test user and the approvals it caused are removed. The added columns are dropped, main's models are rebuilt, and the orphan tables are dropped. No ZZ rows remain in silver or gold.
f2d5180)These supersede the matching and elimination rules described above.
gold_ic_reconciliation compares balance_a/balance_b at 100% (translated_amount), never the ownership-weighted group_amount. group_balance_* and share_* show what the group view holds of each side.gold_ic_eliminations eliminates each side at that share. A matched entry covers the share both sides hold, and an nci entry moves the rest of the larger-share side to the NCI line, pseudo-account NCI (var('ic_nci_account') maps it to a chart account), attributed to the minority's entity. It never goes to the difference account.gold_nci_movement_schedule and nci_amount are untouched. Eliminating an intragroup balance doesn't change the minority's share of the subsidiary, so their share of the balance stays in the NCI figures, and the NCI line in the group view is its counterpart. Both stay balanced: the fully consolidated TB's ic_elimination layer nets to zero, and assert_nci_movement_reconciles passes.assert_ic_difference_not_from_ownership.difference_cause:none below 0.005;fx when the functional currencies differ (a trial balance carries no transaction currency or booking rate);booking in one currency when the local amounts don't net;fx in one currency when they do (translation alone).booking counts against the tolerance (within_tolerance/over_tolerance). fx is fx_difference and never trips it.assert_ic_difference_cause.gold_balance_sheet carries it. Eliminations are computed to date and each period posts the change, so a timing difference that reverses next period leaves the difference account then.assert_ic_pair_basis.translated_amount is Nullable, which made such a difference NULL; the previous head had the same flaw.Data: ZZ data, FY2099. ZZA is 100% USD, ZZB 80% USD, ZZC 100% USD and ZZD 100% EUR. Group ZZGRP has difference account 2100 and tolerance 5.
The previous head is 164275c.
| pair (period) | previous head | decisions |
|---|---|---|
| 12: ZZA/1100 +1000 (100%) vs ZZB/2010 −1000 (80%), P1 | 1000 vs −800 → diff 200, over_tolerance | at 100%: 1000 vs −1000, matched, diff 0 |
| 13: ZZA/4030 −300 vs ZZB/5030 +312, both USD, P1 | −300 vs 249.6 → diff −50.4 | diff 12, booking, over_tolerance |
| 13: ZZA/4030 −1000 (USD) vs ZZD/5030 +925 EUR, P1 | diff −30.69, over_tolerance | diff −30.69, fx, fx_difference |
| 14: ZZA/1100 +500 in P1 vs ZZC/2010 −500 in P2 | P1 over_tolerance, P2 over_tolerance again | P1 booking, over_tolerance (open at period end); P2 matched |
| 14: ZZA/4030 −100 vs ZZB/5030 +100, P2 | diff −20 (group amounts) | matched on the movement |
Group view after elimination: 1100, 2010, 4030 and 5030 are all 0 in each period.
booking label. To date that is −21.09.Mutation checks:
| mutation | result |
|---|---|
| match on group_amount | assert_ic_difference_not_from_ownership FAIL 3, assert_ic_pair_basis FAIL 3 |
| every pair on the movement basis | assert_ic_pair_basis FAIL 3 |
| a cross-currency difference labelled booking | assert_ic_difference_cause FAIL 2 |
955a2cf, 632cc7c)assert_partner_grain_not_fanned_out keeps only checks whose models the consolidation scope rebuilds. The ytd and prior_year checks became single-model grain tests, assert_ytd_trial_balance_grain and assert_prior_year_comparison_grain.dbt ls --select +tag:domain:consolidation on a scratch copy (--project-dir explicit): previous head, 44 models and 154 tests; this head, 44 models and 159 tests.gold_prior_year_comparison or gold_ytd_trial_balance. This head picks it with all its models built. The two grain tests are not picked. All the new IC tests' models are built.assert_bs_only_bs_accounts, assert_cf_categories_equal_net_change, assert_consolidated_cf_reconciles, assert_pnl_only_pnl_accounts and assert_ytd_p12_equals_annual.tests/integration/fixtures/ic_two_partners.sql has ZZF 4030 with two partners, in FY2096 and FY2097. test_two_partners_on_one_account_keep_the_account_grain loads it, builds, and asserts the grain.Revert proof, with the fixture alone on the live stack:
| FY2097 P1 | B1 reverted | this head |
|---|---|---|
gold_trial_balance rows |
2 | 1 |
| YTD rows / amount | 2 / −250 | 1 / −150 |
| prior-year rows / amount | 4 / −300 | 1 / −150 |
| the three grain tests | FAIL 2 each | PASS |
F3: layer 6's acquisition rows are summed to the account grain in gold_fully_consolidated_tb.
assert_ic_difference_account_in_chart.gold_trial_balance_by_partner and gold_ic_unmatched are in konsol's managed domain region (konsol#173 registers them), and in EXPECTED_GOLD_TABLES.depends_on hint, fixed in 632cc7c, after which it passes;cec7c3e)nci_amount). Nothing eliminated B's minority share of an intragroup balance, so A 100% / B 80% on 1000/−1000 left −200 on the payable, and the NCI line's +200 printed nowhere.gold_ic_eliminations now also holds the NCI view's entries (elimination_view = 'nci'): the minority's share of each side's elimination and difference, at (1 − share), against the NCI line.gold_fully_consolidated_tb) books only elimination_view = 'group' and is unchanged.assert_ic_full_view_nets_zero. assert_ic_elimination_nets_zero still checks the group view.pair_event 'joined').left row with values 0, which reverses every elimination.assert_ic_pair_left_is_reversed.tests/integration/fixtures/ic_decisions.sql is generated from what konsol writes: FY2095, an 80%-owned side, a balance-sheet pair over P1/P2, a EUR entity, a mid-year acquisition (ZZE, from P3) and a disposal (ZZH, after P2).ic_decisions_expectations.py holds the exact 10 reconciliation rows, 21 entries, and the 100% view. test_ic_decisions_12_to_14_on_data loads, builds and checks.check() reports no mismatches at the rate the warehouse holds (EUR→USD 1.0479), and all 20 IC tests plus 2 schema tests pass.n_a and n_b are computed, lagged and posted separately, each on its own account.adjustment_type = 'ic_elimination_nci' in the fully consolidated TB, and carry the entity on each leg. The waterfall counts them as IC eliminations, and so does the report.assert_consolidated_cf_reconciles still passes. The warn test also flags var('ic_nci_account') mapped to an account outside the chart (WARN 1 when mapped to 9998).assert_ic_difference_cause recomputes the local sides from gold_consolidated_trial_balance (ic_expected_pair_values).632cc7c vs cec7c3e632cc7c |
cec7c3e |
|
|---|---|---|
| 100% view P1: 2010 / 5030 / NCI line | −200 / +62.4 / +140 | 0 / 0 / 0 |
| ZZH disposed after P2 | its P1 elimination never reversed | left row in P3 reverses +400 / −400 |
| ZZE acquired in P3, settled in P4 | P4 "matched", +200 posted back onto ZZA's paid receivable | P3 joined compares the balances at joining; ZZA's receivable ends at 0 |
The group view per account is unchanged in shape: every IC account is 0 while the pair is in the group.
Mutation checks (container copy only):
| mutation | result |
|---|---|
| no NCI-view entries | assert_ic_full_view_nets_zero FAIL 3; the group-view nets-zero test still passes |
| no partner ever treated as having left | assert_ic_pair_left_is_reversed FAIL 1 |
| the old per-slice partner filter | assert_ic_pair_basis FAIL 2, assert_ic_difference_not_from_ownership FAIL 2 |
Each passes once restored.
Report tests (run in the container through a pytest stand-in, since pytest isn't installed there): 8/8.
gold_consolidated_trial_balance never carries them. An intercompany balance that existed at acquisition shows as a booking difference from the joining period (ZZE: 200 in 2100) until the entity's joining-period trial balance carries its opening balances.gold_disposal_adjustments posts only the gain/loss). After the left reversal, its IC balance shows on its own account (ZZH: −400 on 2010 against ZZA's now-external +400).95614f1), rebased on [#176] (2325e0e)These supersede the second round's membership rule ("a side with no data carries its last position").
left row, whatever the reason: a disposal, a sub-group sale while the entity's own ownership stays open, or a move to equity.ownership_resolution_ctes, which gold_entity_ownership now also uses (same output).assert_ic_pair_left_is_reversed is rewritten from its own resolution of ownership_periods and the ancestry. While a balance-sheet pair is not live, its eliminations to date are 0, per view, account and entity.assert_cf_net_income_equals_group_pnl: the net-income line equals the group view's P&L, all layers, for the P&L accounts the mapping doesn't give a line of their own.assert_ic_nci_line_nets_zero (the NCI line over both views, per group and period) and assert_ic_nci_leg_entity (the group view's NCI leg is attributed to the other, partly owned side; the NCI view's to its own side). The fixture gains 70% against 80% on a balance-sheet pair and a P&L pair (the n_b branch), and a balance-sheet booking difference on the 80% side.assert_equity_method_no_ic_elim exempts the left reversal in an entity's first equity period. Those entries only undo what was eliminated while it was fully consolidated; the left test checks they are complete.ic_decisions.sql, FY2095, generated from konsol-written rows): adds ZZ7 at 70%, ZZQ (100% full, then 40% equity from P3), sub-group ZZSUB (sold after P2) holding ZZS (its own period open), ZZA quiet in P3, and a historical equity rate for every node (assert_equity_rate_coverage). The expectations are 19 reconciliation rows, 41 keyed entries and the 100% view.cec7c3e vs 95614f1| case | cec7c3e |
95614f1 |
|---|---|---|
| ZZQ moved to equity after P2 | 150 still eliminated at P4 | P3 left, reversed to 0 |
| ZZSUB (holding ZZS) sold after P2 | 300 still eliminated at P4 | P3 left, reversed to 0 |
| ZZA quiet in P3; ZZB books −50 | no P3 rows at all; ZZH's left and ZZE's joined slip to P4; the −50 is invisible |
P3: 1000 vs −1050, booking −50, over_tolerance; 40 to 2100 in the group view, 10 in the NCI view; P4 matched at 1050 |
| cash flow P1 (ZZ7/ZZB P&L pair's nci entry on 5030) | net income 20 vs group P&L 0 | 0 vs 0 |
| NCI line, both views, each period | 0 | 0 |
100% view at P4: 1100 850 and 2010 −850 (ZZA's now external receivables from ZZH, ZZQ and ZZS, and their own history: the disposal gap), 4030, 5030 and the NCI line 0, and 2100 181.31. The fixture check against live reports no mismatches.
Mutation checks (container copy, live data):
| mutation | caught by |
|---|---|
n_b's NCI leg on the wrong entity |
assert_ic_nci_leg_entity FAIL 2; fixture, 4 mismatches |
| NCI-view legs sent to the difference account | assert_ic_nci_line_nets_zero FAIL 3, assert_ic_nci_leg_entity FAIL 8; fixture, 14 |
no left row |
assert_ic_pair_left_is_reversed FAIL 12; fixture, 8 |
| membership from data (M2 reverted) | fixture, 33 (the P3 quiet row turns into a false left). No dbt test sees it: the fixture is the check, and it runs in the integration suite, not CI |
| the cash flow drops every nci entry (L1 reverted) | assert_cf_net_income_equals_group_pnl FAIL 1 |
no left exemption in assert_equity_method_no_ic_elim |
FAIL 1 |
cec7c3e's models on the same data |
assert_ic_pair_left_is_reversed FAIL 10, assert_cf_net_income_equals_group_pnl FAIL 1 |
Each passes once restored, and the fixture check is back to 0.
Full dbt test on 95614f1: every IC, cash-flow and grain test passes. The failures involve no ZZ rows and no model this PR touches: assert_cf_categories_equal_net_change 84, assert_cta_zero_for_same_currency 24, assert_d365_gl_vouchers_balance 927, test_canonical_gl_journals_balance 927, assert_silver_gl_debit_credit_balance 7, assert_spread_has_12_periods 4, assert_staging_not_stale 1, relationships gold_bs_movement → cash_flow_categories 364, assert_equity_rate_coverage 7, and WARN assert_trial_balance_balances 84. Report tests 8/8 (pytest stand-in in the container).
init-db.sql (both tables kept) and gold_consolidated_trial_balance.sql: [#176]'s rate guard stays the first pre_hook, the partner column's ALTER is second, the DELETE third. _gold__models.yml, deploy.sh and the report script merged cleanly.2325e0e: since [#176] translates only at a governed rate, the fixture carries eight FY2095 EUR→USD rows (Average and Closing, P1–P4, at 1.0479, the rate the expectations were checked at), written by hand rather than approved in konsol's Group Exchange Rate. konsol publishes that table by replacing it whole (EXCHANGE TABLES), so a rate approval during an integration run would drop them and the guard would refuse the build.dbt parse (as CI runs it) exit 0, the changed models compile, the gold-domain check passes, report tests 8/8.2325e0e): against a scratch copy of the governed rates (zz175fx.group_exchange_rates: live's 120 rows plus the 8 FY2095 ZZ rows), with the source repointed in a container copy only; live epm_staging.group_exchange_rates was never written. Build 38 PASS, 0 errors. IC, cash-flow, grain, governed-rate and ownership tests: 35 PASS, 2 WARN (assert_equity_rate_coverage 7 non-ZZ rows; assert_governed_rate_sane 128, because usd_log10 is NaN on the stack). Fixture check: 0 mismatches, ZZD at the governed 1.0479. The A/B output is identical to 95614f1's.2325e0e: 39 passed, 7 failed, 5 skipped, 3 errors.dbt build stopped on assert_d365_gl_vouchers_balance (927 rows of live demo D365 data) and skipped everything downstream. Run the suite's way with dbt run instead of build, both pass: the two-partner grain assertions and the 3 grain tests; the IC fixture's 25 tests (1 WARN, assert_equity_rate_coverage) and 0 expectation mismatches.test_dbt_build_succeeds and test_dbt_build_no_errors_in_output fail on the same D365 test (plus assert_staging_not_stale).test_clickhouse_staging inserts and three test_end_to_end setups get HTTP 500 from ClickHouse on INSERTs into staging tables this PR doesn't touch.test_frappe_sync (its ClickHouse settings don't reach the server from the container).Closes [#148]
🤖 Generated with Claude Code
Tickets: #148
Tickets: #158
Tickets: #170
Tickets: #176
Tickets: #182
Ticket changed by: grynn-in