Respectful Offers: results
Pre-registered simulatorSimulated outcomes from a simulator, not live membersMembers: real, anonymised KKBox subscribersrun real-v2 · commit 2faad55e0a77
Headline, central variant Simulated
Retained per 1,000 members after 3 months: bandit 866.5, rule-following baseline 869.5, with discount spend of 152,247 and 455,274 NTD. Pre-registered claim, extra members kept per 1,000 NTD of discount, bandit minus rule-following baseline: +5.57 (95% interval +5.06 to +6.10; Holm-adjusted p = 0.0001), so the bandit keeps more members per discount dollar. Other variants are in the comparisons table.
Results
Every policy, conservative variant Simulated
| Policy | Retained / 1,000 | Incremental / 1,000 | Retained / 1,000 at risk | 90-day value / member (NTD) | Discount cost (NTD) | Cost per extra member kept | Offers / member-month | No-offer share | Exploration | Violations |
|---|---|---|---|---|---|---|---|---|---|---|
| No offer | 858.7 [857.2, 860.2] | 0.0 [0.0, 0.0] | 360.7 [355.1, 366.4] | 345.13 [344.56, 345.70] | 0 | n/a | 0.000 [0.000, 0.000] | 100.0% | n/a | 0 ✓ |
| Status quo (not respectful) | 865.1 [863.7, 866.6] | 6.5 [6.1, 6.8] | 408.5 [402.7, 414.4] | 345.32 [344.76, 345.89] | 447,313 | 346 [331, 363] | 0.044 [0.043, 0.044] | 95.6% | n/a | 32,160 ✗ |
| Rule-following baseline | 863.4 [861.9, 864.8] | 4.7 [4.3, 5.0] | 395.3 [389.5, 401.2] | 345.10 [344.53, 345.66] | 298,113 | 318 [299, 340] | 0.081 [0.080, 0.082] | 91.9% | n/a | 0 ✓ |
| Bandit | 861.2 [859.7, 862.7] | 2.5 [2.3, 2.8] | 378.3 [372.5, 384.3] | 345.41 [344.85, 345.98] | 104,364 | 205 [186, 228] | 0.112 [0.111, 0.113] | 88.8% | 2.8% | 0 ✓ |
| Segment oracle (ceiling) | 862.1 [860.6, 863.6] | 3.4 [3.2, 3.7] | 373.7 [368.0, 379.4] | 346.31 [345.75, 346.87] | 37,889 | 55 [50, 61] | 0.110 [0.109, 0.111] | 89.0% | n/a | 0 ✓ |
Every policy, central variant Simulated
| Policy | Retained / 1,000 | Incremental / 1,000 | Retained / 1,000 at risk | 90-day value / member (NTD) | Discount cost (NTD) | Cost per extra member kept | Offers / member-month | No-offer share | Exploration | Violations |
|---|---|---|---|---|---|---|---|---|---|---|
| No offer | 858.7 [857.2, 860.2] | 0.0 [0.0, 0.0] | 360.7 [355.1, 366.4] | 345.13 [344.56, 345.70] | 0 | n/a | 0.000 [0.000, 0.000] | 100.0% | n/a | 0 ✓ |
| Status quo (not respectful) | 870.3 [868.9, 871.8] | 11.7 [11.2, 12.1] | 447.0 [441.1, 453.1] | 346.01 [345.45, 346.58] | 673,257 | 289 [280, 298] | 0.044 [0.043, 0.045] | 95.6% | n/a | 31,490 ✗ |
| Rule-following baseline | 869.5 [868.0, 870.9] | 10.8 [10.3, 11.2] | 440.5 [434.4, 446.6] | 346.38 [345.81, 346.94] | 455,274 | 211 [204, 219] | 0.084 [0.083, 0.086] | 91.6% | n/a | 0 ✓ |
| Bandit | 866.5 [865.0, 868.0] | 7.8 [7.4, 8.2] | 406.9 [400.9, 412.9] | 347.31 [346.74, 347.87] | 152,247 | 97 [92, 103] | 0.153 [0.152, 0.154] | 84.7% | 1.9% | 0 ✓ |
| Segment oracle (ceiling) | 867.6 [866.1, 869.0] | 8.9 [8.5, 9.3] | 395.3 [389.5, 401.1] | 348.40 [347.83, 348.96] | 53,703 | 30 [28, 33] | 0.151 [0.150, 0.153] | 84.9% | n/a | 0 ✓ |
Every policy, optimistic variant Simulated
| Policy | Retained / 1,000 | Incremental / 1,000 | Retained / 1,000 at risk | 90-day value / member (NTD) | Discount cost (NTD) | Cost per extra member kept | Offers / member-month | No-offer share | Exploration | Violations |
|---|---|---|---|---|---|---|---|---|---|---|
| No offer | 858.7 [857.2, 860.2] | 0.0 [0.0, 0.0] | 360.7 [355.1, 366.4] | 345.13 [344.56, 345.70] | 0 | n/a | 0.000 [0.000, 0.000] | 100.0% | n/a | 0 ✓ |
| Status quo (not respectful) | 875.0 [873.6, 876.4] | 16.3 [15.8, 16.9] | 481.6 [475.5, 488.0] | 346.53 [345.98, 347.09] | 881,720 | 270 [264, 276] | 0.044 [0.043, 0.045] | 95.6% | n/a | 30,617 ✗ |
| Rule-following baseline | 876.3 [874.9, 877.7] | 17.6 [17.0, 18.2] | 491.0 [484.9, 497.1] | 348.00 [347.45, 348.55] | 602,958 | 171 [167, 176] | 0.088 [0.087, 0.089] | 91.2% | n/a | 0 ✓ |
| Bandit | 873.1 [871.7, 874.5] | 14.4 [13.9, 15.0] | 443.6 [437.6, 449.7] | 349.62 [349.08, 350.18] | 190,116 | 66 [63, 69] | 0.181 [0.180, 0.182] | 81.9% | 1.6% | 0 ✓ |
| Segment oracle (ceiling) | 874.9 [873.5, 876.3] | 16.2 [15.7, 16.8] | 430.1 [424.1, 436.0] | 351.01 [350.46, 351.57] | 70,544 | 22 [20, 23] | 0.244 [0.242, 0.245] | 75.6% | n/a | 0 ✓ |
Retained against spend, conservative variant Simulated
Each point is one budget level; further up and to the left is better.
The numbers behind this chart
| Policy | Budget / 1,000 / month | Spend / member | Retained / 1,000 | Incremental / 1,000 |
|---|---|---|---|---|
| Status quo (not respectful) | 0 | 0.000 | 858.7 | 0.0 |
| Rule-following baseline | 0 | 0.000 | 859.4 | 0.7 |
| Bandit | 0 | 0.000 | 859.4 | 0.8 |
| Status quo (not respectful) | 1,000 | 0.761 | 861.4 | 2.8 |
| Rule-following baseline | 1,000 | 0.505 | 861.5 | 2.8 |
| Bandit | 1,000 | 0.290 | 860.9 | 2.2 |
| Status quo (not respectful) | 3,000 | 2.237 | 865.1 | 6.5 |
| Rule-following baseline | 3,000 | 1.491 | 863.4 | 4.7 |
| Bandit | 3,000 | 0.522 | 861.2 | 2.5 |
| Status quo (not respectful) | 6,000 | 4.317 | 868.5 | 9.8 |
| Rule-following baseline | 6,000 | 2.056 | 864.2 | 5.5 |
| Bandit | 6,000 | 0.577 | 861.2 | 2.5 |
| Status quo (not respectful) | 12,000 | 5.668 | 870.3 | 11.6 |
| Rule-following baseline | 12,000 | 2.147 | 864.3 | 5.6 |
| Bandit | 12,000 | 0.577 | 861.2 | 2.5 |
Retained against spend, central variant Simulated
Each point is one budget level; further up and to the left is better.
The numbers behind this chart
| Policy | Budget / 1,000 / month | Spend / member | Retained / 1,000 | Incremental / 1,000 |
|---|---|---|---|---|
| Status quo (not respectful) | 0 | 0.000 | 858.7 | 0.0 |
| Rule-following baseline | 0 | 0.000 | 862.0 | 3.3 |
| Bandit | 0 | 0.000 | 863.5 | 4.9 |
| Status quo (not respectful) | 1,000 | 1.117 | 863.3 | 4.6 |
| Rule-following baseline | 1,000 | 0.769 | 865.6 | 6.9 |
| Bandit | 1,000 | 0.409 | 865.7 | 7.0 |
| Status quo (not respectful) | 3,000 | 3.366 | 870.3 | 11.7 |
| Rule-following baseline | 3,000 | 2.276 | 869.5 | 10.8 |
| Bandit | 3,000 | 0.761 | 866.5 | 7.8 |
| Status quo (not respectful) | 6,000 | 6.626 | 877.4 | 18.8 |
| Rule-following baseline | 6,000 | 3.521 | 871.6 | 12.9 |
| Bandit | 6,000 | 0.851 | 866.6 | 7.9 |
| Status quo (not respectful) | 12,000 | 8.967 | 881.5 | 22.8 |
| Rule-following baseline | 12,000 | 3.767 | 871.9 | 13.2 |
| Bandit | 12,000 | 0.851 | 866.6 | 7.9 |
Retained against spend, optimistic variant Simulated
Each point is one budget level; further up and to the left is better.
The numbers behind this chart
| Policy | Budget / 1,000 / month | Spend / member | Retained / 1,000 | Incremental / 1,000 |
|---|---|---|---|---|
| Status quo (not respectful) | 0 | 0.000 | 858.7 | 0.0 |
| Rule-following baseline | 0 | 0.000 | 866.3 | 7.6 |
| Bandit | 0 | 0.000 | 869.4 | 10.7 |
| Status quo (not respectful) | 1,000 | 1.460 | 865.1 | 6.4 |
| Rule-following baseline | 1,000 | 1.015 | 871.2 | 12.5 |
| Bandit | 1,000 | 0.531 | 872.4 | 13.7 |
| Status quo (not respectful) | 3,000 | 4.409 | 875.0 | 16.3 |
| Rule-following baseline | 3,000 | 3.015 | 876.3 | 17.6 |
| Bandit | 3,000 | 0.951 | 873.1 | 14.4 |
| Status quo (not respectful) | 6,000 | 8.692 | 885.4 | 26.7 |
| Rule-following baseline | 6,000 | 5.098 | 879.8 | 21.1 |
| Bandit | 6,000 | 0.951 | 873.1 | 14.4 |
| Status quo (not respectful) | 12,000 | 12.229 | 892.2 | 33.5 |
| Rule-following baseline | 12,000 | 5.582 | 880.5 | 21.8 |
| Bandit | 12,000 | 0.951 | 873.1 | 14.4 |
The three variants side by side Simulated
A scenario analysis, not an interval: the variants are never pooled.
The numbers behind this chart
| Policy · variant | Incremental / 1,000 |
|---|---|
| Status quo (not respectful) · conservative | 6.5 [6.1, 6.8] |
| Status quo (not respectful) · central | 11.7 [11.2, 12.1] |
| Status quo (not respectful) · optimistic | 16.3 [15.8, 16.9] |
| Rule-following baseline · conservative | 4.7 [4.3, 5.0] |
| Rule-following baseline · central | 10.8 [10.3, 11.2] |
| Rule-following baseline · optimistic | 17.6 [17.0, 18.2] |
| Bandit · conservative | 2.5 [2.3, 2.8] |
| Bandit · central | 7.8 [7.4, 8.2] |
| Bandit · optimistic | 14.4 [13.9, 15.0] |
| Segment oracle (ceiling) · conservative | 3.4 [3.2, 3.7] |
| Segment oracle (ceiling) · central | 8.9 [8.5, 9.3] |
| Segment oracle (ceiling) · optimistic | 16.2 [15.7, 16.8] |
Paired comparisons Simulated
Differences on shared bootstrap resamples; Holm correction at 5% per metric and variant. A tie means the interval includes zero or Holm did not reject.
| Variant | Metric | Difference | Estimate | p | Verdict |
|---|---|---|---|---|---|
| conservative | incremental retained per 1000 | Bandit − Rule-following baseline | -2.13 [-2.48, -1.78] | 0.0001 | separated |
| conservative | incremental retained per 1000 | Bandit − Status quo (not respectful) | -3.92 [-4.29, -3.54] | 0.0001 | separated |
| conservative | incremental retained per 1000 | Rule-following baseline − Status quo (not respectful) | -1.78 [-2.12, -1.45] | 0.0001 | separated |
| conservative | ltv proxy per member | Bandit − Rule-following baseline | 0.31 [0.23, 0.40] | 0.0001 | separated |
| conservative | ltv proxy per member | Bandit − Status quo (not respectful) | 0.09 [-0.03, 0.21] | 0.1428 | tie |
| conservative | ltv proxy per member | Rule-following baseline − Status quo (not respectful) | -0.23 [-0.33, -0.13] | 0.0001 | separated |
| conservative | incremental retained per 1000 ntd | Bandit − Rule-following baseline | 1.74 [1.28, 2.22] | 0.0001 | separated |
| conservative | incremental retained per 1000 ntd | Bandit − Status quo (not respectful) | 1.99 [1.52, 2.47] | 0.0001 | separated |
| conservative | incremental retained per 1000 ntd | Rule-following baseline − Status quo (not respectful) | 0.25 [0.08, 0.43] | 0.0036 | separated |
| central | incremental retained per 1000 | Bandit − Rule-following baseline | -2.94 [-3.36, -2.51] | 0.0001 | separated |
| central | incremental retained per 1000 | Bandit − Status quo (not respectful) | -3.81 [-4.34, -3.30] | 0.0001 | separated |
| central | incremental retained per 1000 | Rule-following baseline − Status quo (not respectful) | -0.88 [-1.35, -0.43] | 0.0001 | separated |
| central | ltv proxy per member | Bandit − Rule-following baseline | 0.93 [0.83, 1.03] | 0.0001 | separated |
| central | ltv proxy per member | Bandit − Status quo (not respectful) | 1.30 [1.13, 1.47] | 0.0001 | separated |
| central | ltv proxy per member | Rule-following baseline − Status quo (not respectful) | 0.37 [0.23, 0.51] | 0.0001 | separated |
| central | incremental retained per 1000 ntd | Bandit − Rule-following baseline | 5.57 [5.06, 6.10] | 0.0001 | separated |
| central | incremental retained per 1000 ntd | Bandit − Status quo (not respectful) | 6.84 [6.29, 7.42] | 0.0001 | separated |
| central | incremental retained per 1000 ntd | Rule-following baseline − Status quo (not respectful) | 1.27 [1.12, 1.42] | 0.0001 | separated |
| optimistic | incremental retained per 1000 | Bandit − Rule-following baseline | -3.16 [-3.65, -2.68] | 0.0001 | separated |
| optimistic | incremental retained per 1000 | Bandit − Status quo (not respectful) | -1.90 [-2.55, -1.24] | 0.0001 | separated |
| optimistic | incremental retained per 1000 | Rule-following baseline − Status quo (not respectful) | 1.26 [0.72, 1.81] | 0.0001 | separated |
| optimistic | ltv proxy per member | Bandit − Rule-following baseline | 1.62 [1.50, 1.75] | 0.0001 | separated |
| optimistic | ltv proxy per member | Bandit − Status quo (not respectful) | 3.09 [2.89, 3.30] | 0.0001 | separated |
| optimistic | ltv proxy per member | Rule-following baseline − Status quo (not respectful) | 1.47 [1.30, 1.63] | 0.0001 | separated |
| optimistic | incremental retained per 1000 ntd | Bandit − Rule-following baseline | 9.36 [8.73, 10.02] | 0.0001 | separated |
| optimistic | incremental retained per 1000 ntd | Bandit − Status quo (not respectful) | 11.49 [10.81, 12.20] | 0.0001 | separated |
| optimistic | incremental retained per 1000 ntd | Rule-following baseline − Status quo (not respectful) | 2.13 [1.99, 2.28] | 0.0001 | separated |
Bandit learning curve Simulated
Value of acting greedily on the posterior after each training epoch.
The numbers behind this chart
| Variant | Training periods | Value |
|---|---|---|
| central | 0 | 120.852 |
| central | 3 | 121.508 |
| central | 6 | 121.544 |
| central | 9 | 121.558 |
| conservative | 0 | 120.852 |
| conservative | 3 | 120.936 |
| conservative | 6 | 121.000 |
| conservative | 9 | 121.000 |
| optimistic | 0 | 120.852 |
| optimistic | 3 | 122.163 |
| optimistic | 6 | 122.203 |
| optimistic | 9 | 122.206 |
Off-policy evaluation, validated Simulated
Off-policy estimates against the exact oracle, conservative variant Simulated
Estimated from the randomised logging pilot at period 1 and scored against the simulator's exact value, the black tick on each row. This validates the estimators; the rollouts are the headline.
The numbers behind this chart
| Target | Estimator | Estimate | Exact oracle | Covered | Effective sample |
|---|---|---|---|---|---|
| No offer | IPS | 121.55 [120.62, 122.49] | 120.97 | yes ✓ | 52,228 |
| No offer | SNIPS | 120.97 [120.63, 121.30] | 120.97 | yes ✓ | 52,228 |
| No offer | Doubly robust | 121.00 [120.73, 121.25] | 120.97 | yes ✓ | 52,228 |
| Rule-following baseline | IPS | 121.73 [120.81, 122.65] | 121.13 | yes ✓ | 52,081 |
| Rule-following baseline | SNIPS | 121.38 [121.05, 121.70] | 121.13 | yes ✓ | 52,081 |
| Rule-following baseline | Doubly robust | 121.22 [120.96, 121.48] | 121.13 | yes ✓ | 52,081 |
| Segment oracle (ceiling) | IPS | 121.51 [120.54, 122.45] | 121.37 | yes ✓ | 52,008 |
| Segment oracle (ceiling) | SNIPS | 121.39 [121.07, 121.72] | 121.37 | yes ✓ | 52,008 |
| Segment oracle (ceiling) | Doubly robust | 121.44 [121.19, 121.69] | 121.37 | yes ✓ | 52,008 |
| Bandit | IPS | 121.39 [120.51, 122.27] | 121.09 | yes ✓ | 54,642 |
| Bandit | SNIPS | 121.14 [120.82, 121.46] | 121.09 | yes ✓ | 54,642 |
| Bandit | Doubly robust | 121.05 [120.79, 121.30] | 121.09 | yes ✓ | 54,642 |
Off-policy estimates against the exact oracle, central variant Simulated
Estimated from the randomised logging pilot at period 1 and scored against the simulator's exact value, the black tick on each row. This validates the estimators; the rollouts are the headline.
The numbers behind this chart
| Target | Estimator | Estimate | Exact oracle | Covered | Effective sample |
|---|---|---|---|---|---|
| No offer | IPS | 121.17 [120.24, 122.11] | 120.97 | yes ✓ | 52,067 |
| No offer | SNIPS | 120.94 [120.62, 121.25] | 120.97 | yes ✓ | 52,067 |
| No offer | Doubly robust | 120.95 [120.69, 121.21] | 120.97 | yes ✓ | 52,067 |
| Rule-following baseline | IPS | 121.77 [120.83, 122.74] | 121.55 | yes ✓ | 52,046 |
| Rule-following baseline | SNIPS | 121.57 [121.24, 121.89] | 121.55 | yes ✓ | 52,046 |
| Rule-following baseline | Doubly robust | 121.56 [121.31, 121.80] | 121.55 | yes ✓ | 52,046 |
| Segment oracle (ceiling) | IPS | 121.94 [121.03, 122.85] | 121.94 | yes ✓ | 51,948 |
| Segment oracle (ceiling) | SNIPS | 121.94 [121.63, 122.25] | 121.94 | yes ✓ | 51,948 |
| Segment oracle (ceiling) | Doubly robust | 121.97 [121.73, 122.21] | 121.94 | yes ✓ | 51,948 |
| Bandit | IPS | 122.22 [121.34, 123.11] | 121.64 | yes ✓ | 54,056 |
| Bandit | SNIPS | 121.79 [121.48, 122.10] | 121.64 | yes ✓ | 54,056 |
| Bandit | Doubly robust | 121.67 [121.42, 121.91] | 121.64 | yes ✓ | 54,056 |
Off-policy estimates against the exact oracle, optimistic variant Simulated
Estimated from the randomised logging pilot at period 1 and scored against the simulator's exact value, the black tick on each row. This validates the estimators; the rollouts are the headline.
The numbers behind this chart
| Target | Estimator | Estimate | Exact oracle | Covered | Effective sample |
|---|---|---|---|---|---|
| No offer | IPS | 120.91 [119.96, 121.89] | 120.97 | yes ✓ | 51,897 |
| No offer | SNIPS | 121.05 [120.73, 121.37] | 120.97 | yes ✓ | 51,897 |
| No offer | Doubly robust | 121.08 [120.83, 121.34] | 120.97 | yes ✓ | 51,897 |
| Rule-following baseline | IPS | 122.24 [121.30, 123.21] | 122.07 | yes ✓ | 51,990 |
| Rule-following baseline | SNIPS | 122.23 [121.91, 122.56] | 122.07 | yes ✓ | 51,990 |
| Rule-following baseline | Doubly robust | 122.28 [122.02, 122.53] | 122.07 | yes ✓ | 51,990 |
| Segment oracle (ceiling) | IPS | 122.50 [121.55, 123.44] | 122.61 | yes ✓ | 51,891 |
| Segment oracle (ceiling) | SNIPS | 122.72 [122.42, 123.03] | 122.61 | yes ✓ | 51,891 |
| Segment oracle (ceiling) | Doubly robust | 122.74 [122.50, 122.98] | 122.61 | yes ✓ | 51,891 |
| Bandit | IPS | 122.89 [121.96, 123.86] | 122.28 | yes ✓ | 53,926 |
| Bandit | SNIPS | 122.52 [122.21, 122.82] | 122.28 | yes ✓ | 53,926 |
| Bandit | Doubly robust | 122.46 [122.22, 122.70] | 122.28 | yes ✓ | 53,926 |
Respect audit Simulated
An independent auditor re-derives every rule from the logs. R1 never re-offer a declined action; R2 a nudge before any discount for members at risk; R3 never contact the holdout; R7 decisions use allow-listed features only; R8 one decision per member per month.
Every respectful rollout audited clean ✓
Every audited rollout (66)
| Policy | Variant | Rollout | Respectful | R1 | R2 | R3 | R7 | R8 |
|---|---|---|---|---|---|---|---|---|
| No offer | conservative | evaluation | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | conservative | evaluation | no | 7,361 | 24,799 | 0 | 0 | 0 |
| Rule-following baseline | conservative | evaluation | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | conservative | evaluation | yes | 0 | 0 | 0 | 0 | 0 |
| Segment oracle (ceiling) | conservative | evaluation | yes | 0 | 0 | 0 | 0 | 0 |
| Holdout | conservative | holdout | yes | 0 | 0 | 0 | 0 | 0 |
| logging | conservative | logging_pilot | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | conservative | sweep:0 | no | 0 | 0 | 0 | 0 | 0 |
| Rule-following baseline | conservative | sweep:0 | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | conservative | sweep:0 | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | conservative | sweep:1000 | no | 2,140 | 8,830 | 0 | 0 | 0 |
| Rule-following baseline | conservative | sweep:1000 | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | conservative | sweep:1000 | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | conservative | sweep:3000 | no | 7,361 | 24,799 | 0 | 0 | 0 |
| Rule-following baseline | conservative | sweep:3000 | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | conservative | sweep:3000 | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | conservative | sweep:6000 | no | 17,048 | 46,942 | 0 | 0 | 0 |
| Rule-following baseline | conservative | sweep:6000 | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | conservative | sweep:6000 | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | conservative | sweep:12000 | no | 26,472 | 60,744 | 0 | 0 | 0 |
| Rule-following baseline | conservative | sweep:12000 | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | conservative | sweep:12000 | yes | 0 | 0 | 0 | 0 | 0 |
| No offer | central | evaluation | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | central | evaluation | no | 6,540 | 24,950 | 0 | 0 | 0 |
| Rule-following baseline | central | evaluation | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | central | evaluation | yes | 0 | 0 | 0 | 0 | 0 |
| Segment oracle (ceiling) | central | evaluation | yes | 0 | 0 | 0 | 0 | 0 |
| Holdout | central | holdout | yes | 0 | 0 | 0 | 0 | 0 |
| logging | central | logging_pilot | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | central | sweep:0 | no | 0 | 0 | 0 | 0 | 0 |
| Rule-following baseline | central | sweep:0 | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | central | sweep:0 | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | central | sweep:1000 | no | 1,945 | 8,904 | 0 | 0 | 0 |
| Rule-following baseline | central | sweep:1000 | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | central | sweep:1000 | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | central | sweep:3000 | no | 6,540 | 24,950 | 0 | 0 | 0 |
| Rule-following baseline | central | sweep:3000 | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | central | sweep:3000 | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | central | sweep:6000 | no | 15,151 | 47,999 | 0 | 0 | 0 |
| Rule-following baseline | central | sweep:6000 | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | central | sweep:6000 | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | central | sweep:12000 | no | 24,304 | 63,924 | 0 | 0 | 0 |
| Rule-following baseline | central | sweep:12000 | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | central | sweep:12000 | yes | 0 | 0 | 0 | 0 | 0 |
| No offer | optimistic | evaluation | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | optimistic | evaluation | no | 5,538 | 25,079 | 0 | 0 | 0 |
| Rule-following baseline | optimistic | evaluation | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | optimistic | evaluation | yes | 0 | 0 | 0 | 0 | 0 |
| Segment oracle (ceiling) | optimistic | evaluation | yes | 0 | 0 | 0 | 0 | 0 |
| Holdout | optimistic | holdout | yes | 0 | 0 | 0 | 0 | 0 |
| logging | optimistic | logging_pilot | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | optimistic | sweep:0 | no | 0 | 0 | 0 | 0 | 0 |
| Rule-following baseline | optimistic | sweep:0 | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | optimistic | sweep:0 | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | optimistic | sweep:1000 | no | 1,707 | 8,965 | 0 | 0 | 0 |
| Rule-following baseline | optimistic | sweep:1000 | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | optimistic | sweep:1000 | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | optimistic | sweep:3000 | no | 5,538 | 25,079 | 0 | 0 | 0 |
| Rule-following baseline | optimistic | sweep:3000 | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | optimistic | sweep:3000 | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | optimistic | sweep:6000 | no | 12,903 | 48,176 | 0 | 0 | 0 |
| Rule-following baseline | optimistic | sweep:6000 | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | optimistic | sweep:6000 | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | optimistic | sweep:12000 | no | 21,349 | 66,684 | 0 | 0 | 0 |
| Rule-following baseline | optimistic | sweep:12000 | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | optimistic | sweep:12000 | yes | 0 | 0 | 0 | 0 | 0 |
| Policy | Variant | Respectful | R1 | R2 | R3 | R7 | R8 |
|---|---|---|---|---|---|---|---|
| No offer | conservative | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | conservative | no (reported, OWNER D-17) | 7,361 | 24,799 | 0 | 0 | 0 |
| Rule-following baseline | conservative | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | conservative | yes | 0 | 0 | 0 | 0 | 0 |
| Segment oracle (ceiling) | conservative | yes | 0 | 0 | 0 | 0 | 0 |
| No offer | central | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | central | no (reported, OWNER D-17) | 6,540 | 24,950 | 0 | 0 | 0 |
| Rule-following baseline | central | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | central | yes | 0 | 0 | 0 | 0 | 0 |
| Segment oracle (ceiling) | central | yes | 0 | 0 | 0 | 0 | 0 |
| No offer | optimistic | yes | 0 | 0 | 0 | 0 | 0 |
| Status quo (not respectful) | optimistic | no (reported, OWNER D-17) | 5,538 | 25,079 | 0 | 0 | 0 |
| Rule-following baseline | optimistic | yes | 0 | 0 | 0 | 0 | 0 |
| Bandit | optimistic | yes | 0 | 0 | 0 | 0 | 0 |
| Segment oracle (ceiling) | optimistic | yes | 0 | 0 | 0 | 0 | 0 |
Holdout check Simulated
Holdout retention minus no-offer retention on the evaluation members, per 1,000; the interval should contain zero.
| Variant | Difference / 1,000 | Verdict |
|---|---|---|
| central | 1.0 [-1.8, 3.8] | consistent ✓ |
| conservative | 1.0 [-1.8, 3.8] | consistent ✓ |
| optimistic | 1.0 [-1.8, 3.8] | consistent ✓ |
Disparity audit Simulated
Central variant. Age band and gender are audited, never used to decide (R7). Groups under 30 members are not shown, nor is one more group when that would reveal them.
| Policy | Measure | Attribute | Group | Estimate |
|---|---|---|---|---|
| Bandit | offer rate | age band | 18-24 | 0.313 [0.308, 0.318] |
| Bandit | offer rate | age band | 25-34 | 0.186 [0.183, 0.189] |
| Bandit | offer rate | age band | 35-44 | 0.180 [0.175, 0.185] |
| Bandit | offer rate | age band | 45+ | 0.176 [0.168, 0.183] |
| Bandit | offer rate | age band | <18 | 0.467 [0.450, 0.484] |
| Bandit | offer rate | age band | unknown | 0.108 [0.107, 0.110] |
| Bandit | offer rate | gender | female | 0.220 [0.217, 0.223] |
| Bandit | offer rate | gender | male | 0.224 [0.221, 0.228] |
| Bandit | offer rate | gender | unknown | 0.108 [0.107, 0.110] |
| Bandit | retained per 1000 | age band | 18-24 | 735.9 [730.3, 741.7] |
| Bandit | retained per 1000 | age band | 25-34 | 854.3 [850.7, 857.8] |
| Bandit | retained per 1000 | age band | 35-44 | 865.4 [859.5, 871.2] |
| Bandit | retained per 1000 | age band | 45+ | 876.3 [868.0, 884.8] |
| Bandit | retained per 1000 | age band | <18 | 613.8 [592.6, 635.0] |
| Bandit | retained per 1000 | age band | unknown | 897.3 [895.5, 899.0] |
| Bandit | retained per 1000 | gender | female | 823.1 [819.4, 826.8] |
| Bandit | retained per 1000 | gender | male | 820.7 [817.0, 824.4] |
| Bandit | retained per 1000 | gender | unknown | 896.4 [894.6, 898.2] |
| Rule-following baseline | offer rate | age band | 18-24 | 0.260 [0.255, 0.265] |
| Rule-following baseline | offer rate | age band | 25-34 | 0.099 [0.096, 0.102] |
| Rule-following baseline | offer rate | age band | 35-44 | 0.078 [0.074, 0.082] |
| Rule-following baseline | offer rate | age band | 45+ | 0.066 [0.061, 0.072] |
| Rule-following baseline | offer rate | age band | <18 | 0.448 [0.428, 0.467] |
| Rule-following baseline | offer rate | age band | unknown | 0.047 [0.046, 0.049] |
| Rule-following baseline | offer rate | gender | female | 0.137 [0.134, 0.140] |
| Rule-following baseline | offer rate | gender | male | 0.144 [0.141, 0.147] |
| Rule-following baseline | offer rate | gender | unknown | 0.048 [0.047, 0.049] |
| Rule-following baseline | retained per 1000 | age band | 18-24 | 746.9 [741.3, 752.5] |
| Rule-following baseline | retained per 1000 | age band | 25-34 | 858.8 [855.2, 862.3] |
| Rule-following baseline | retained per 1000 | age band | 35-44 | 865.9 [860.1, 871.8] |
| Rule-following baseline | retained per 1000 | age band | 45+ | 875.8 [867.5, 884.1] |
| Rule-following baseline | retained per 1000 | age band | <18 | 631.2 [609.2, 652.8] |
| Rule-following baseline | retained per 1000 | age band | unknown | 898.5 [896.7, 900.2] |
| Rule-following baseline | retained per 1000 | gender | female | 828.7 [824.9, 832.3] |
| Rule-following baseline | retained per 1000 | gender | male | 826.4 [822.7, 830.0] |
| Rule-following baseline | retained per 1000 | gender | unknown | 897.6 [895.8, 899.4] |
Churn model card
| Measure | Value |
|---|---|
| AUC | 0.841 [0.839, 0.842] |
| Brier score | 0.0591 [0.0587, 0.0594] |
| Expected calibration error | 0.0262 |
| Two-feature baseline AUC | 0.722 [0.720, 0.725] |
| Features | tenure_days, auto_renew, cancels_90d, listening_trend_30d, price_30d, plan_days |
Model: scikit-learn HistGradientBoostingClassifier (D-29); protected attributes used: [].
Churn model reliability (real KKBox labels, not simulated)
Calibrated March risk against observed March churn; the diagonal is perfect calibration.
The numbers behind this chart
| Predicted | Observed | Members |
|---|---|---|
| 0.005 | 0.005 | 96,844 |
| 0.008 | 0.020 | 96,843 |
| 0.011 | 0.024 | 96,843 |
| 0.014 | 0.037 | 96,843 |
| 0.016 | 0.040 | 96,843 |
| 0.021 | 0.039 | 96,843 |
| 0.025 | 0.042 | 96,843 |
| 0.038 | 0.051 | 96,843 |
| 0.113 | 0.141 | 96,843 |
| 0.375 | 0.488 | 96,843 |
The simulator, quoted in full
Every simulated number above comes from this file (sha256 6ae089599c41b7418c649a2cd7256f6090d4f071ced268adab9eddd0c0a44326), all three variants, byte for byte.
# Respectful Offers response simulator (Tech Spec §5).
#
# STATUS: confirmed by the project owner on 2026-11-17 (OWNER D-47) and pre-registered for
# real KKBox runs; every real evaluation refuses a file whose sha256 is not in docs/prereg.md
# (D-3). Running on your own data? These are only starting values: set what you believe, then
# `offers prereg --scope real` before any real evaluation.
#
# These values are assumptions, not measurements: KKBox data holds no offers. They are chosen
# to be plausible and are bracketed by three variants, and every report quotes all three.
# risk_bands low < 5% churn risk <= medium < 15% <= high; R2's threshold is 15%. Base
# churn is about 6%, so "high" means two to three times the base rate
# (about 13% of members on the synthetic fixture).
# segment_mix riskier and declining-listening cells hold more persuadables and lost
# causes; sleeping dogs (offers remind them to leave) are 10-15% everywhere.
# acceptance about half of persuadables take a discount; sure things take it too (deadweight).
# exposure_effect persuadables gain 2-4 points of renewal from being shown an offer; sleeping
# dogs lose 4-6 points when shown a commercial offer, even if they decline.
# acceptance_effect: a persuadable who engages with a content nudge gains 12 points.
# discount_1m half price for one month; plan_change: the cheaper plan (99/149 = 0.67).
# conservative acceptance x0.7, persuadable effects x0.5, sleeping-dog harm x1.5,
# 5 points of mix share move from persuadables to sleeping dogs (segments
# come from one shared uniform per member, so some members move by more
# than one segment between variants; only the shares are as stated).
# plan_change offered once, from the full plan; nudges cost nothing (not modelled).
# optimistic acceptance x1.25, persuadable effects x1.5, sleeping-dog harm x0.5,
# 5 points of sleeping dogs become persuadables.
schema_version: '1.0'
period_days: 30
horizon_periods: 3
currency: NTD
risk_bands:
cuts:
- 0.05
- 0.15
trend_bands:
cuts:
- -0.25
segments:
- sure_thing
- persuadable
- lost_cause
- sleeping_dog
actions:
content_nudge:
cost: 0.0
discount_1m:
discount_fraction: 0.5
plan_change:
price_ratio: 0.67
allow_unconventional: false
variants:
conservative:
segment_mix:
low.stable:
sure_thing: 0.75
persuadable: 0.05
lost_cause: 0.05
sleeping_dog: 0.15
low.declining:
sure_thing: 0.6
persuadable: 0.15
lost_cause: 0.08
sleeping_dog: 0.17
medium.stable:
sure_thing: 0.5
persuadable: 0.2
lost_cause: 0.1
sleeping_dog: 0.2
medium.declining:
sure_thing: 0.4
persuadable: 0.25
lost_cause: 0.15
sleeping_dog: 0.2
high.stable:
sure_thing: 0.25
persuadable: 0.3
lost_cause: 0.25
sleeping_dog: 0.2
high.declining:
sure_thing: 0.15
persuadable: 0.3
lost_cause: 0.35
sleeping_dog: 0.2
acceptance:
content_nudge:
sure_thing: 0.245
persuadable: 0.315
lost_cause: 0.105
sleeping_dog: 0.175
discount_1m:
sure_thing: 0.385
persuadable: 0.35
lost_cause: 0.07
sleeping_dog: 0.14
plan_change:
sure_thing: 0.14
persuadable: 0.21
lost_cause: 0.035
sleeping_dog: 0.07
exposure_effect:
content_nudge:
sure_thing: 0
persuadable: 0.015
lost_cause: 0
sleeping_dog: -0.015
discount_1m:
sure_thing: 0
persuadable: 0.02
lost_cause: 0
sleeping_dog: -0.09
plan_change:
sure_thing: 0
persuadable: 0.01
lost_cause: 0
sleeping_dog: -0.06
acceptance_effect:
content_nudge:
sure_thing: 0
persuadable: 0.06
lost_cause: 0.02
sleeping_dog: 0
central:
segment_mix:
low.stable:
sure_thing: 0.75
persuadable: 0.1
lost_cause: 0.05
sleeping_dog: 0.1
low.declining:
sure_thing: 0.6
persuadable: 0.2
lost_cause: 0.08
sleeping_dog: 0.12
medium.stable:
sure_thing: 0.5
persuadable: 0.25
lost_cause: 0.1
sleeping_dog: 0.15
medium.declining:
sure_thing: 0.4
persuadable: 0.3
lost_cause: 0.15
sleeping_dog: 0.15
high.stable:
sure_thing: 0.25
persuadable: 0.35
lost_cause: 0.25
sleeping_dog: 0.15
high.declining:
sure_thing: 0.15
persuadable: 0.35
lost_cause: 0.35
sleeping_dog: 0.15
acceptance:
content_nudge:
sure_thing: 0.35
persuadable: 0.45
lost_cause: 0.15
sleeping_dog: 0.25
discount_1m:
sure_thing: 0.55
persuadable: 0.5
lost_cause: 0.1
sleeping_dog: 0.2
plan_change:
sure_thing: 0.2
persuadable: 0.3
lost_cause: 0.05
sleeping_dog: 0.1
exposure_effect:
content_nudge:
sure_thing: 0
persuadable: 0.03
lost_cause: 0
sleeping_dog: -0.01
discount_1m:
sure_thing: 0
persuadable: 0.04
lost_cause: 0
sleeping_dog: -0.06
plan_change:
sure_thing: 0
persuadable: 0.02
lost_cause: 0
sleeping_dog: -0.04
acceptance_effect:
content_nudge:
sure_thing: 0
persuadable: 0.12
lost_cause: 0.02
sleeping_dog: 0
optimistic:
segment_mix:
low.stable:
sure_thing: 0.75
persuadable: 0.15
lost_cause: 0.05
sleeping_dog: 0.05
low.declining:
sure_thing: 0.6
persuadable: 0.25
lost_cause: 0.08
sleeping_dog: 0.07
medium.stable:
sure_thing: 0.5
persuadable: 0.3
lost_cause: 0.1
sleeping_dog: 0.1
medium.declining:
sure_thing: 0.4
persuadable: 0.35
lost_cause: 0.15
sleeping_dog: 0.1
high.stable:
sure_thing: 0.25
persuadable: 0.4
lost_cause: 0.25
sleeping_dog: 0.1
high.declining:
sure_thing: 0.15
persuadable: 0.4
lost_cause: 0.35
sleeping_dog: 0.1
acceptance:
content_nudge:
sure_thing: 0.4375
persuadable: 0.5625
lost_cause: 0.1875
sleeping_dog: 0.3125
discount_1m:
sure_thing: 0.6875
persuadable: 0.625
lost_cause: 0.125
sleeping_dog: 0.25
plan_change:
sure_thing: 0.25
persuadable: 0.375
lost_cause: 0.0625
sleeping_dog: 0.125
exposure_effect:
content_nudge:
sure_thing: 0
persuadable: 0.045
lost_cause: 0
sleeping_dog: -0.005
discount_1m:
sure_thing: 0
persuadable: 0.06
lost_cause: 0
sleeping_dog: -0.03
plan_change:
sure_thing: 0
persuadable: 0.03
lost_cause: 0
sleeping_dog: -0.02
acceptance_effect:
content_nudge:
sure_thing: 0
persuadable: 0.18
lost_cause: 0.02
sleeping_dog: 0