Respectful Offers: results

Pre-registered simulatorSimulated outcomes from a simulator, not live membersMembers: real, anonymised KKBox subscribersrun real-v2 · commit 2faad55e0a77

Headline, central variant Simulated

Retained per 1,000 members after 3 months: bandit 866.5, rule-following baseline 869.5, with discount spend of 152,247 and 455,274 NTD. Pre-registered claim, extra members kept per 1,000 NTD of discount, bandit minus rule-following baseline: +5.57 (95% interval +5.06 to +6.10; Holm-adjusted p = 0.0001), so the bandit keeps more members per discount dollar. Other variants are in the comparisons table.

Results

Every policy, conservative variant Simulated

Point estimates with 95% block-bootstrap intervals in brackets.
PolicyRetained / 1,000Incremental / 1,000Retained / 1,000 at risk90-day value / member (NTD)Discount cost (NTD)Cost per extra member keptOffers / member-monthNo-offer shareExplorationViolations
No offer858.7 [857.2, 860.2]0.0 [0.0, 0.0]360.7 [355.1, 366.4]345.13 [344.56, 345.70]0n/a0.000 [0.000, 0.000]100.0%n/a0 ✓
Status quo (not respectful)865.1 [863.7, 866.6]6.5 [6.1, 6.8]408.5 [402.7, 414.4]345.32 [344.76, 345.89]447,313346 [331, 363]0.044 [0.043, 0.044]95.6%n/a32,160 ✗
Rule-following baseline863.4 [861.9, 864.8]4.7 [4.3, 5.0]395.3 [389.5, 401.2]345.10 [344.53, 345.66]298,113318 [299, 340]0.081 [0.080, 0.082]91.9%n/a0 ✓
Bandit861.2 [859.7, 862.7]2.5 [2.3, 2.8]378.3 [372.5, 384.3]345.41 [344.85, 345.98]104,364205 [186, 228]0.112 [0.111, 0.113]88.8%2.8%0 ✓
Segment oracle (ceiling)862.1 [860.6, 863.6]3.4 [3.2, 3.7]373.7 [368.0, 379.4]346.31 [345.75, 346.87]37,88955 [50, 61]0.110 [0.109, 0.111]89.0%n/a0 ✓

Every policy, central variant Simulated

Point estimates with 95% block-bootstrap intervals in brackets.
PolicyRetained / 1,000Incremental / 1,000Retained / 1,000 at risk90-day value / member (NTD)Discount cost (NTD)Cost per extra member keptOffers / member-monthNo-offer shareExplorationViolations
No offer858.7 [857.2, 860.2]0.0 [0.0, 0.0]360.7 [355.1, 366.4]345.13 [344.56, 345.70]0n/a0.000 [0.000, 0.000]100.0%n/a0 ✓
Status quo (not respectful)870.3 [868.9, 871.8]11.7 [11.2, 12.1]447.0 [441.1, 453.1]346.01 [345.45, 346.58]673,257289 [280, 298]0.044 [0.043, 0.045]95.6%n/a31,490 ✗
Rule-following baseline869.5 [868.0, 870.9]10.8 [10.3, 11.2]440.5 [434.4, 446.6]346.38 [345.81, 346.94]455,274211 [204, 219]0.084 [0.083, 0.086]91.6%n/a0 ✓
Bandit866.5 [865.0, 868.0]7.8 [7.4, 8.2]406.9 [400.9, 412.9]347.31 [346.74, 347.87]152,24797 [92, 103]0.153 [0.152, 0.154]84.7%1.9%0 ✓
Segment oracle (ceiling)867.6 [866.1, 869.0]8.9 [8.5, 9.3]395.3 [389.5, 401.1]348.40 [347.83, 348.96]53,70330 [28, 33]0.151 [0.150, 0.153]84.9%n/a0 ✓

Every policy, optimistic variant Simulated

Point estimates with 95% block-bootstrap intervals in brackets.
PolicyRetained / 1,000Incremental / 1,000Retained / 1,000 at risk90-day value / member (NTD)Discount cost (NTD)Cost per extra member keptOffers / member-monthNo-offer shareExplorationViolations
No offer858.7 [857.2, 860.2]0.0 [0.0, 0.0]360.7 [355.1, 366.4]345.13 [344.56, 345.70]0n/a0.000 [0.000, 0.000]100.0%n/a0 ✓
Status quo (not respectful)875.0 [873.6, 876.4]16.3 [15.8, 16.9]481.6 [475.5, 488.0]346.53 [345.98, 347.09]881,720270 [264, 276]0.044 [0.043, 0.045]95.6%n/a30,617 ✗
Rule-following baseline876.3 [874.9, 877.7]17.6 [17.0, 18.2]491.0 [484.9, 497.1]348.00 [347.45, 348.55]602,958171 [167, 176]0.088 [0.087, 0.089]91.2%n/a0 ✓
Bandit873.1 [871.7, 874.5]14.4 [13.9, 15.0]443.6 [437.6, 449.7]349.62 [349.08, 350.18]190,11666 [63, 69]0.181 [0.180, 0.182]81.9%1.6%0 ✓
Segment oracle (ceiling)874.9 [873.5, 876.3]16.2 [15.7, 16.8]430.1 [424.1, 436.0]351.01 [350.46, 351.57]70,54422 [20, 23]0.244 [0.242, 0.245]75.6%n/a0 ✓

Retained against spend, conservative variant Simulated

Each point is one budget level; further up and to the left is better.

Retained against spend, conservativeOne line per policy across the budget sweep.8558608658708750246Discount spend per member over the horizon (NTD)Retained per 1,000Status quoRule-followingBandit
The numbers behind this chart
PolicyBudget / 1,000 / monthSpend / memberRetained / 1,000Incremental / 1,000
Status quo (not respectful)00.000858.70.0
Rule-following baseline00.000859.40.7
Bandit00.000859.40.8
Status quo (not respectful)1,0000.761861.42.8
Rule-following baseline1,0000.505861.52.8
Bandit1,0000.290860.92.2
Status quo (not respectful)3,0002.237865.16.5
Rule-following baseline3,0001.491863.44.7
Bandit3,0000.522861.22.5
Status quo (not respectful)6,0004.317868.59.8
Rule-following baseline6,0002.056864.25.5
Bandit6,0000.577861.22.5
Status quo (not respectful)12,0005.668870.311.6
Rule-following baseline12,0002.147864.35.6
Bandit12,0000.577861.22.5

Retained against spend, central variant Simulated

Each point is one budget level; further up and to the left is better.

Retained against spend, centralOne line per policy across the budget sweep.8508608708808900246810Discount spend per member over the horizon (NTD)Retained per 1,000Status quoRule-followingBandit
The numbers behind this chart
PolicyBudget / 1,000 / monthSpend / memberRetained / 1,000Incremental / 1,000
Status quo (not respectful)00.000858.70.0
Rule-following baseline00.000862.03.3
Bandit00.000863.54.9
Status quo (not respectful)1,0001.117863.34.6
Rule-following baseline1,0000.769865.66.9
Bandit1,0000.409865.77.0
Status quo (not respectful)3,0003.366870.311.7
Rule-following baseline3,0002.276869.510.8
Bandit3,0000.761866.57.8
Status quo (not respectful)6,0006.626877.418.8
Rule-following baseline6,0003.521871.612.9
Bandit6,0000.851866.67.9
Status quo (not respectful)12,0008.967881.522.8
Rule-following baseline12,0003.767871.913.2
Bandit12,0000.851866.67.9

Retained against spend, optimistic variant Simulated

Each point is one budget level; further up and to the left is better.

Retained against spend, optimisticOne line per policy across the budget sweep.85086087088089090002.557.51012.5Discount spend per member over the horizon (NTD)Retained per 1,000Status quoRule-followingBandit
The numbers behind this chart
PolicyBudget / 1,000 / monthSpend / memberRetained / 1,000Incremental / 1,000
Status quo (not respectful)00.000858.70.0
Rule-following baseline00.000866.37.6
Bandit00.000869.410.7
Status quo (not respectful)1,0001.460865.16.4
Rule-following baseline1,0001.015871.212.5
Bandit1,0000.531872.413.7
Status quo (not respectful)3,0004.409875.016.3
Rule-following baseline3,0003.015876.317.6
Bandit3,0000.951873.114.4
Status quo (not respectful)6,0008.692885.426.7
Rule-following baseline6,0005.098879.821.1
Bandit6,0000.951873.114.4
Status quo (not respectful)12,00012.229892.233.5
Rule-following baseline12,0005.582880.521.8
Bandit12,0000.951873.114.4

The three variants side by side Simulated

A scenario analysis, not an interval: the variants are never pooled.

Incremental retention by variantDots with 95% intervals; the vertical line is zero.05101520Status quo (not respectful) · conservativeStatus quo (not respectful) · centralStatus quo (not respectful) · optimisticRule-following baseline · conservativeRule-following baseline · centralRule-following baseline · optimisticBandit · conservativeBandit · centralBandit · optimisticSegment oracle (ceiling) · conservativeSegment oracle (ceiling) · centralSegment oracle (ceiling) · optimisticIncremental retained per 1,000 against no offer
The numbers behind this chart
Policy · variantIncremental / 1,000
Status quo (not respectful) · conservative6.5 [6.1, 6.8]
Status quo (not respectful) · central11.7 [11.2, 12.1]
Status quo (not respectful) · optimistic16.3 [15.8, 16.9]
Rule-following baseline · conservative4.7 [4.3, 5.0]
Rule-following baseline · central10.8 [10.3, 11.2]
Rule-following baseline · optimistic17.6 [17.0, 18.2]
Bandit · conservative2.5 [2.3, 2.8]
Bandit · central7.8 [7.4, 8.2]
Bandit · optimistic14.4 [13.9, 15.0]
Segment oracle (ceiling) · conservative3.4 [3.2, 3.7]
Segment oracle (ceiling) · central8.9 [8.5, 9.3]
Segment oracle (ceiling) · optimistic16.2 [15.7, 16.8]

Paired comparisons Simulated

Differences on shared bootstrap resamples; Holm correction at 5% per metric and variant. A tie means the interval includes zero or Holm did not reject.

VariantMetricDifferenceEstimatepVerdict
conservativeincremental retained per 1000Bandit − Rule-following baseline-2.13 [-2.48, -1.78]0.0001separated
conservativeincremental retained per 1000Bandit − Status quo (not respectful)-3.92 [-4.29, -3.54]0.0001separated
conservativeincremental retained per 1000Rule-following baseline − Status quo (not respectful)-1.78 [-2.12, -1.45]0.0001separated
conservativeltv proxy per memberBandit − Rule-following baseline0.31 [0.23, 0.40]0.0001separated
conservativeltv proxy per memberBandit − Status quo (not respectful)0.09 [-0.03, 0.21]0.1428tie
conservativeltv proxy per memberRule-following baseline − Status quo (not respectful)-0.23 [-0.33, -0.13]0.0001separated
conservativeincremental retained per 1000 ntdBandit − Rule-following baseline1.74 [1.28, 2.22]0.0001separated
conservativeincremental retained per 1000 ntdBandit − Status quo (not respectful)1.99 [1.52, 2.47]0.0001separated
conservativeincremental retained per 1000 ntdRule-following baseline − Status quo (not respectful)0.25 [0.08, 0.43]0.0036separated
centralincremental retained per 1000Bandit − Rule-following baseline-2.94 [-3.36, -2.51]0.0001separated
centralincremental retained per 1000Bandit − Status quo (not respectful)-3.81 [-4.34, -3.30]0.0001separated
centralincremental retained per 1000Rule-following baseline − Status quo (not respectful)-0.88 [-1.35, -0.43]0.0001separated
centralltv proxy per memberBandit − Rule-following baseline0.93 [0.83, 1.03]0.0001separated
centralltv proxy per memberBandit − Status quo (not respectful)1.30 [1.13, 1.47]0.0001separated
centralltv proxy per memberRule-following baseline − Status quo (not respectful)0.37 [0.23, 0.51]0.0001separated
centralincremental retained per 1000 ntdBandit − Rule-following baseline5.57 [5.06, 6.10]0.0001separated
centralincremental retained per 1000 ntdBandit − Status quo (not respectful)6.84 [6.29, 7.42]0.0001separated
centralincremental retained per 1000 ntdRule-following baseline − Status quo (not respectful)1.27 [1.12, 1.42]0.0001separated
optimisticincremental retained per 1000Bandit − Rule-following baseline-3.16 [-3.65, -2.68]0.0001separated
optimisticincremental retained per 1000Bandit − Status quo (not respectful)-1.90 [-2.55, -1.24]0.0001separated
optimisticincremental retained per 1000Rule-following baseline − Status quo (not respectful)1.26 [0.72, 1.81]0.0001separated
optimisticltv proxy per memberBandit − Rule-following baseline1.62 [1.50, 1.75]0.0001separated
optimisticltv proxy per memberBandit − Status quo (not respectful)3.09 [2.89, 3.30]0.0001separated
optimisticltv proxy per memberRule-following baseline − Status quo (not respectful)1.47 [1.30, 1.63]0.0001separated
optimisticincremental retained per 1000 ntdBandit − Rule-following baseline9.36 [8.73, 10.02]0.0001separated
optimisticincremental retained per 1000 ntdBandit − Status quo (not respectful)11.49 [10.81, 12.20]0.0001separated
optimisticincremental retained per 1000 ntdRule-following baseline − Status quo (not respectful)2.13 [1.99, 2.28]0.0001separated

Bandit learning curve Simulated

Value of acting greedily on the posterior after each training epoch.

Bandit learning curveOne line per variant.120.5121121.5122122.50246810Training periodsGreedy period-1 value per member (NTD)CentralConservativeOptimistic
The numbers behind this chart
VariantTraining periodsValue
central0120.852
central3121.508
central6121.544
central9121.558
conservative0120.852
conservative3120.936
conservative6121.000
conservative9121.000
optimistic0120.852
optimistic3122.163
optimistic6122.203
optimistic9122.206

Off-policy evaluation, validated Simulated

Off-policy estimates against the exact oracle, conservative variant Simulated

Estimated from the randomised logging pilot at period 1 and scored against the simulator's exact value, the black tick on each row. This validates the estimators; the rollouts are the headline.

Off-policy estimates, conservativeEstimates with 95% intervals; the black tick on each row is the exact oracle value.120.5121121.5122122.5123No offer · IPSNo offer · SNIPSNo offer · Doubly robustRule-following baseline · IPSRule-following baseline · SNIPSRule-following baseline · Doubly robustSegment oracle (ceiling) · IPSSegment oracle (ceiling) · SNIPSSegment oracle (ceiling) · Doubly robustBandit · IPSBandit · SNIPSBandit · Doubly robustPeriod-1 net revenue per member (NTD)
The numbers behind this chart
TargetEstimatorEstimateExact oracleCoveredEffective sample
No offerIPS121.55 [120.62, 122.49]120.97yes ✓52,228
No offerSNIPS120.97 [120.63, 121.30]120.97yes ✓52,228
No offerDoubly robust121.00 [120.73, 121.25]120.97yes ✓52,228
Rule-following baselineIPS121.73 [120.81, 122.65]121.13yes ✓52,081
Rule-following baselineSNIPS121.38 [121.05, 121.70]121.13yes ✓52,081
Rule-following baselineDoubly robust121.22 [120.96, 121.48]121.13yes ✓52,081
Segment oracle (ceiling)IPS121.51 [120.54, 122.45]121.37yes ✓52,008
Segment oracle (ceiling)SNIPS121.39 [121.07, 121.72]121.37yes ✓52,008
Segment oracle (ceiling)Doubly robust121.44 [121.19, 121.69]121.37yes ✓52,008
BanditIPS121.39 [120.51, 122.27]121.09yes ✓54,642
BanditSNIPS121.14 [120.82, 121.46]121.09yes ✓54,642
BanditDoubly robust121.05 [120.79, 121.30]121.09yes ✓54,642

Off-policy estimates against the exact oracle, central variant Simulated

Estimated from the randomised logging pilot at period 1 and scored against the simulator's exact value, the black tick on each row. This validates the estimators; the rollouts are the headline.

Off-policy estimates, centralEstimates with 95% intervals; the black tick on each row is the exact oracle value.120121122123124No offer · IPSNo offer · SNIPSNo offer · Doubly robustRule-following baseline · IPSRule-following baseline · SNIPSRule-following baseline · Doubly robustSegment oracle (ceiling) · IPSSegment oracle (ceiling) · SNIPSSegment oracle (ceiling) · Doubly robustBandit · IPSBandit · SNIPSBandit · Doubly robustPeriod-1 net revenue per member (NTD)
The numbers behind this chart
TargetEstimatorEstimateExact oracleCoveredEffective sample
No offerIPS121.17 [120.24, 122.11]120.97yes ✓52,067
No offerSNIPS120.94 [120.62, 121.25]120.97yes ✓52,067
No offerDoubly robust120.95 [120.69, 121.21]120.97yes ✓52,067
Rule-following baselineIPS121.77 [120.83, 122.74]121.55yes ✓52,046
Rule-following baselineSNIPS121.57 [121.24, 121.89]121.55yes ✓52,046
Rule-following baselineDoubly robust121.56 [121.31, 121.80]121.55yes ✓52,046
Segment oracle (ceiling)IPS121.94 [121.03, 122.85]121.94yes ✓51,948
Segment oracle (ceiling)SNIPS121.94 [121.63, 122.25]121.94yes ✓51,948
Segment oracle (ceiling)Doubly robust121.97 [121.73, 122.21]121.94yes ✓51,948
BanditIPS122.22 [121.34, 123.11]121.64yes ✓54,056
BanditSNIPS121.79 [121.48, 122.10]121.64yes ✓54,056
BanditDoubly robust121.67 [121.42, 121.91]121.64yes ✓54,056

Off-policy estimates against the exact oracle, optimistic variant Simulated

Estimated from the randomised logging pilot at period 1 and scored against the simulator's exact value, the black tick on each row. This validates the estimators; the rollouts are the headline.

Off-policy estimates, optimisticEstimates with 95% intervals; the black tick on each row is the exact oracle value.119120121122123124No offer · IPSNo offer · SNIPSNo offer · Doubly robustRule-following baseline · IPSRule-following baseline · SNIPSRule-following baseline · Doubly robustSegment oracle (ceiling) · IPSSegment oracle (ceiling) · SNIPSSegment oracle (ceiling) · Doubly robustBandit · IPSBandit · SNIPSBandit · Doubly robustPeriod-1 net revenue per member (NTD)
The numbers behind this chart
TargetEstimatorEstimateExact oracleCoveredEffective sample
No offerIPS120.91 [119.96, 121.89]120.97yes ✓51,897
No offerSNIPS121.05 [120.73, 121.37]120.97yes ✓51,897
No offerDoubly robust121.08 [120.83, 121.34]120.97yes ✓51,897
Rule-following baselineIPS122.24 [121.30, 123.21]122.07yes ✓51,990
Rule-following baselineSNIPS122.23 [121.91, 122.56]122.07yes ✓51,990
Rule-following baselineDoubly robust122.28 [122.02, 122.53]122.07yes ✓51,990
Segment oracle (ceiling)IPS122.50 [121.55, 123.44]122.61yes ✓51,891
Segment oracle (ceiling)SNIPS122.72 [122.42, 123.03]122.61yes ✓51,891
Segment oracle (ceiling)Doubly robust122.74 [122.50, 122.98]122.61yes ✓51,891
BanditIPS122.89 [121.96, 123.86]122.28yes ✓53,926
BanditSNIPS122.52 [122.21, 122.82]122.28yes ✓53,926
BanditDoubly robust122.46 [122.22, 122.70]122.28yes ✓53,926

Respect audit Simulated

An independent auditor re-derives every rule from the logs. R1 never re-offer a declined action; R2 a nudge before any discount for members at risk; R3 never contact the holdout; R7 decisions use allow-listed features only; R8 one decision per member per month.

Every respectful rollout audited clean ✓

Every audited rollout (66)
PolicyVariantRolloutRespectfulR1R2R3R7R8
No offerconservativeevaluationyes00000
Status quo (not respectful)conservativeevaluationno7,36124,799000
Rule-following baselineconservativeevaluationyes00000
Banditconservativeevaluationyes00000
Segment oracle (ceiling)conservativeevaluationyes00000
Holdoutconservativeholdoutyes00000
loggingconservativelogging_pilotyes00000
Status quo (not respectful)conservativesweep:0no00000
Rule-following baselineconservativesweep:0yes00000
Banditconservativesweep:0yes00000
Status quo (not respectful)conservativesweep:1000no2,1408,830000
Rule-following baselineconservativesweep:1000yes00000
Banditconservativesweep:1000yes00000
Status quo (not respectful)conservativesweep:3000no7,36124,799000
Rule-following baselineconservativesweep:3000yes00000
Banditconservativesweep:3000yes00000
Status quo (not respectful)conservativesweep:6000no17,04846,942000
Rule-following baselineconservativesweep:6000yes00000
Banditconservativesweep:6000yes00000
Status quo (not respectful)conservativesweep:12000no26,47260,744000
Rule-following baselineconservativesweep:12000yes00000
Banditconservativesweep:12000yes00000
No offercentralevaluationyes00000
Status quo (not respectful)centralevaluationno6,54024,950000
Rule-following baselinecentralevaluationyes00000
Banditcentralevaluationyes00000
Segment oracle (ceiling)centralevaluationyes00000
Holdoutcentralholdoutyes00000
loggingcentrallogging_pilotyes00000
Status quo (not respectful)centralsweep:0no00000
Rule-following baselinecentralsweep:0yes00000
Banditcentralsweep:0yes00000
Status quo (not respectful)centralsweep:1000no1,9458,904000
Rule-following baselinecentralsweep:1000yes00000
Banditcentralsweep:1000yes00000
Status quo (not respectful)centralsweep:3000no6,54024,950000
Rule-following baselinecentralsweep:3000yes00000
Banditcentralsweep:3000yes00000
Status quo (not respectful)centralsweep:6000no15,15147,999000
Rule-following baselinecentralsweep:6000yes00000
Banditcentralsweep:6000yes00000
Status quo (not respectful)centralsweep:12000no24,30463,924000
Rule-following baselinecentralsweep:12000yes00000
Banditcentralsweep:12000yes00000
No offeroptimisticevaluationyes00000
Status quo (not respectful)optimisticevaluationno5,53825,079000
Rule-following baselineoptimisticevaluationyes00000
Banditoptimisticevaluationyes00000
Segment oracle (ceiling)optimisticevaluationyes00000
Holdoutoptimisticholdoutyes00000
loggingoptimisticlogging_pilotyes00000
Status quo (not respectful)optimisticsweep:0no00000
Rule-following baselineoptimisticsweep:0yes00000
Banditoptimisticsweep:0yes00000
Status quo (not respectful)optimisticsweep:1000no1,7078,965000
Rule-following baselineoptimisticsweep:1000yes00000
Banditoptimisticsweep:1000yes00000
Status quo (not respectful)optimisticsweep:3000no5,53825,079000
Rule-following baselineoptimisticsweep:3000yes00000
Banditoptimisticsweep:3000yes00000
Status quo (not respectful)optimisticsweep:6000no12,90348,176000
Rule-following baselineoptimisticsweep:6000yes00000
Banditoptimisticsweep:6000yes00000
Status quo (not respectful)optimisticsweep:12000no21,34966,684000
Rule-following baselineoptimisticsweep:12000yes00000
Banditoptimisticsweep:12000yes00000
PolicyVariantRespectfulR1R2R3R7R8
No offerconservativeyes00000
Status quo (not respectful)conservativeno (reported, OWNER D-17)7,36124,799000
Rule-following baselineconservativeyes00000
Banditconservativeyes00000
Segment oracle (ceiling)conservativeyes00000
No offercentralyes00000
Status quo (not respectful)centralno (reported, OWNER D-17)6,54024,950000
Rule-following baselinecentralyes00000
Banditcentralyes00000
Segment oracle (ceiling)centralyes00000
No offeroptimisticyes00000
Status quo (not respectful)optimisticno (reported, OWNER D-17)5,53825,079000
Rule-following baselineoptimisticyes00000
Banditoptimisticyes00000
Segment oracle (ceiling)optimisticyes00000

Holdout check Simulated

Holdout retention minus no-offer retention on the evaluation members, per 1,000; the interval should contain zero.

VariantDifference / 1,000Verdict
central1.0 [-1.8, 3.8]consistent ✓
conservative1.0 [-1.8, 3.8]consistent ✓
optimistic1.0 [-1.8, 3.8]consistent ✓

Disparity audit Simulated

Central variant. Age band and gender are audited, never used to decide (R7). Groups under 30 members are not shown, nor is one more group when that would reveal them.

PolicyMeasureAttributeGroupEstimate
Banditoffer rateage band18-240.313 [0.308, 0.318]
Banditoffer rateage band25-340.186 [0.183, 0.189]
Banditoffer rateage band35-440.180 [0.175, 0.185]
Banditoffer rateage band45+0.176 [0.168, 0.183]
Banditoffer rateage band<180.467 [0.450, 0.484]
Banditoffer rateage bandunknown0.108 [0.107, 0.110]
Banditoffer rategenderfemale0.220 [0.217, 0.223]
Banditoffer rategendermale0.224 [0.221, 0.228]
Banditoffer rategenderunknown0.108 [0.107, 0.110]
Banditretained per 1000age band18-24735.9 [730.3, 741.7]
Banditretained per 1000age band25-34854.3 [850.7, 857.8]
Banditretained per 1000age band35-44865.4 [859.5, 871.2]
Banditretained per 1000age band45+876.3 [868.0, 884.8]
Banditretained per 1000age band<18613.8 [592.6, 635.0]
Banditretained per 1000age bandunknown897.3 [895.5, 899.0]
Banditretained per 1000genderfemale823.1 [819.4, 826.8]
Banditretained per 1000gendermale820.7 [817.0, 824.4]
Banditretained per 1000genderunknown896.4 [894.6, 898.2]
Rule-following baselineoffer rateage band18-240.260 [0.255, 0.265]
Rule-following baselineoffer rateage band25-340.099 [0.096, 0.102]
Rule-following baselineoffer rateage band35-440.078 [0.074, 0.082]
Rule-following baselineoffer rateage band45+0.066 [0.061, 0.072]
Rule-following baselineoffer rateage band<180.448 [0.428, 0.467]
Rule-following baselineoffer rateage bandunknown0.047 [0.046, 0.049]
Rule-following baselineoffer rategenderfemale0.137 [0.134, 0.140]
Rule-following baselineoffer rategendermale0.144 [0.141, 0.147]
Rule-following baselineoffer rategenderunknown0.048 [0.047, 0.049]
Rule-following baselineretained per 1000age band18-24746.9 [741.3, 752.5]
Rule-following baselineretained per 1000age band25-34858.8 [855.2, 862.3]
Rule-following baselineretained per 1000age band35-44865.9 [860.1, 871.8]
Rule-following baselineretained per 1000age band45+875.8 [867.5, 884.1]
Rule-following baselineretained per 1000age band<18631.2 [609.2, 652.8]
Rule-following baselineretained per 1000age bandunknown898.5 [896.7, 900.2]
Rule-following baselineretained per 1000genderfemale828.7 [824.9, 832.3]
Rule-following baselineretained per 1000gendermale826.4 [822.7, 830.0]
Rule-following baselineretained per 1000genderunknown897.6 [895.8, 899.4]

Churn model card

MeasureValue
AUC0.841 [0.839, 0.842]
Brier score0.0591 [0.0587, 0.0594]
Expected calibration error0.0262
Two-feature baseline AUC0.722 [0.720, 0.725]
Featurestenure_days, auto_renew, cancels_90d, listening_trend_30d, price_30d, plan_days

Model: scikit-learn HistGradientBoostingClassifier (D-29); protected attributes used: [].

Churn model reliability (real KKBox labels, not simulated)

Calibrated March risk against observed March churn; the diagonal is perfect calibration.

Churn model reliabilityBins of predicted risk against observed churn.00.10.20.30.40.500.10.20.30.40.5Predicted churn riskObserved churn rate
The numbers behind this chart
PredictedObservedMembers
0.0050.00596,844
0.0080.02096,843
0.0110.02496,843
0.0140.03796,843
0.0160.04096,843
0.0210.03996,843
0.0250.04296,843
0.0380.05196,843
0.1130.14196,843
0.3750.48896,843

The simulator, quoted in full

Every simulated number above comes from this file (sha256 6ae089599c41b7418c649a2cd7256f6090d4f071ced268adab9eddd0c0a44326), all three variants, byte for byte.

# Respectful Offers response simulator (Tech Spec §5).
#
# STATUS: confirmed by the project owner on 2026-11-17 (OWNER D-47) and pre-registered for
# real KKBox runs; every real evaluation refuses a file whose sha256 is not in docs/prereg.md
# (D-3). Running on your own data? These are only starting values: set what you believe, then
# `offers prereg --scope real` before any real evaluation.
#
# These values are assumptions, not measurements: KKBox data holds no offers. They are chosen
# to be plausible and are bracketed by three variants, and every report quotes all three.
#   risk_bands      low < 5% churn risk <= medium < 15% <= high; R2's threshold is 15%. Base
#                   churn is about 6%, so "high" means two to three times the base rate
#                   (about 13% of members on the synthetic fixture).
#   segment_mix     riskier and declining-listening cells hold more persuadables and lost
#                   causes; sleeping dogs (offers remind them to leave) are 10-15% everywhere.
#   acceptance      about half of persuadables take a discount; sure things take it too (deadweight).
#   exposure_effect persuadables gain 2-4 points of renewal from being shown an offer; sleeping
#                   dogs lose 4-6 points when shown a commercial offer, even if they decline.
#   acceptance_effect: a persuadable who engages with a content nudge gains 12 points.
#   discount_1m     half price for one month; plan_change: the cheaper plan (99/149 = 0.67).
#   conservative    acceptance x0.7, persuadable effects x0.5, sleeping-dog harm x1.5,
#                   5 points of mix share move from persuadables to sleeping dogs (segments
#                   come from one shared uniform per member, so some members move by more
#                   than one segment between variants; only the shares are as stated).
#   plan_change     offered once, from the full plan; nudges cost nothing (not modelled).
#   optimistic      acceptance x1.25, persuadable effects x1.5, sleeping-dog harm x0.5,
#                   5 points of sleeping dogs become persuadables.
schema_version: '1.0'
period_days: 30
horizon_periods: 3
currency: NTD
risk_bands:
  cuts:
  - 0.05
  - 0.15
trend_bands:
  cuts:
  - -0.25
segments:
- sure_thing
- persuadable
- lost_cause
- sleeping_dog
actions:
  content_nudge:
    cost: 0.0
  discount_1m:
    discount_fraction: 0.5
  plan_change:
    price_ratio: 0.67
allow_unconventional: false
variants:
  conservative:
    segment_mix:
      low.stable:
        sure_thing: 0.75
        persuadable: 0.05
        lost_cause: 0.05
        sleeping_dog: 0.15
      low.declining:
        sure_thing: 0.6
        persuadable: 0.15
        lost_cause: 0.08
        sleeping_dog: 0.17
      medium.stable:
        sure_thing: 0.5
        persuadable: 0.2
        lost_cause: 0.1
        sleeping_dog: 0.2
      medium.declining:
        sure_thing: 0.4
        persuadable: 0.25
        lost_cause: 0.15
        sleeping_dog: 0.2
      high.stable:
        sure_thing: 0.25
        persuadable: 0.3
        lost_cause: 0.25
        sleeping_dog: 0.2
      high.declining:
        sure_thing: 0.15
        persuadable: 0.3
        lost_cause: 0.35
        sleeping_dog: 0.2
    acceptance:
      content_nudge:
        sure_thing: 0.245
        persuadable: 0.315
        lost_cause: 0.105
        sleeping_dog: 0.175
      discount_1m:
        sure_thing: 0.385
        persuadable: 0.35
        lost_cause: 0.07
        sleeping_dog: 0.14
      plan_change:
        sure_thing: 0.14
        persuadable: 0.21
        lost_cause: 0.035
        sleeping_dog: 0.07
    exposure_effect:
      content_nudge:
        sure_thing: 0
        persuadable: 0.015
        lost_cause: 0
        sleeping_dog: -0.015
      discount_1m:
        sure_thing: 0
        persuadable: 0.02
        lost_cause: 0
        sleeping_dog: -0.09
      plan_change:
        sure_thing: 0
        persuadable: 0.01
        lost_cause: 0
        sleeping_dog: -0.06
    acceptance_effect:
      content_nudge:
        sure_thing: 0
        persuadable: 0.06
        lost_cause: 0.02
        sleeping_dog: 0
  central:
    segment_mix:
      low.stable:
        sure_thing: 0.75
        persuadable: 0.1
        lost_cause: 0.05
        sleeping_dog: 0.1
      low.declining:
        sure_thing: 0.6
        persuadable: 0.2
        lost_cause: 0.08
        sleeping_dog: 0.12
      medium.stable:
        sure_thing: 0.5
        persuadable: 0.25
        lost_cause: 0.1
        sleeping_dog: 0.15
      medium.declining:
        sure_thing: 0.4
        persuadable: 0.3
        lost_cause: 0.15
        sleeping_dog: 0.15
      high.stable:
        sure_thing: 0.25
        persuadable: 0.35
        lost_cause: 0.25
        sleeping_dog: 0.15
      high.declining:
        sure_thing: 0.15
        persuadable: 0.35
        lost_cause: 0.35
        sleeping_dog: 0.15
    acceptance:
      content_nudge:
        sure_thing: 0.35
        persuadable: 0.45
        lost_cause: 0.15
        sleeping_dog: 0.25
      discount_1m:
        sure_thing: 0.55
        persuadable: 0.5
        lost_cause: 0.1
        sleeping_dog: 0.2
      plan_change:
        sure_thing: 0.2
        persuadable: 0.3
        lost_cause: 0.05
        sleeping_dog: 0.1
    exposure_effect:
      content_nudge:
        sure_thing: 0
        persuadable: 0.03
        lost_cause: 0
        sleeping_dog: -0.01
      discount_1m:
        sure_thing: 0
        persuadable: 0.04
        lost_cause: 0
        sleeping_dog: -0.06
      plan_change:
        sure_thing: 0
        persuadable: 0.02
        lost_cause: 0
        sleeping_dog: -0.04
    acceptance_effect:
      content_nudge:
        sure_thing: 0
        persuadable: 0.12
        lost_cause: 0.02
        sleeping_dog: 0
  optimistic:
    segment_mix:
      low.stable:
        sure_thing: 0.75
        persuadable: 0.15
        lost_cause: 0.05
        sleeping_dog: 0.05
      low.declining:
        sure_thing: 0.6
        persuadable: 0.25
        lost_cause: 0.08
        sleeping_dog: 0.07
      medium.stable:
        sure_thing: 0.5
        persuadable: 0.3
        lost_cause: 0.1
        sleeping_dog: 0.1
      medium.declining:
        sure_thing: 0.4
        persuadable: 0.35
        lost_cause: 0.15
        sleeping_dog: 0.1
      high.stable:
        sure_thing: 0.25
        persuadable: 0.4
        lost_cause: 0.25
        sleeping_dog: 0.1
      high.declining:
        sure_thing: 0.15
        persuadable: 0.4
        lost_cause: 0.35
        sleeping_dog: 0.1
    acceptance:
      content_nudge:
        sure_thing: 0.4375
        persuadable: 0.5625
        lost_cause: 0.1875
        sleeping_dog: 0.3125
      discount_1m:
        sure_thing: 0.6875
        persuadable: 0.625
        lost_cause: 0.125
        sleeping_dog: 0.25
      plan_change:
        sure_thing: 0.25
        persuadable: 0.375
        lost_cause: 0.0625
        sleeping_dog: 0.125
    exposure_effect:
      content_nudge:
        sure_thing: 0
        persuadable: 0.045
        lost_cause: 0
        sleeping_dog: -0.005
      discount_1m:
        sure_thing: 0
        persuadable: 0.06
        lost_cause: 0
        sleeping_dog: -0.03
      plan_change:
        sure_thing: 0
        persuadable: 0.03
        lost_cause: 0
        sleeping_dog: -0.02
    acceptance_effect:
      content_nudge:
        sure_thing: 0
        persuadable: 0.18
        lost_cause: 0.02
        sleeping_dog: 0