Respectful Offers
Case study · simulated offer responses on real KKBox members · data: WSDM Cup 2018 KKBox churn prediction challenge (Kaggle), used for non-commercial research and education
The problem
Subscription services fight churn with discounts, and the usual rule, discount everyone the model flags as at risk, pays members who would have stayed anyway and asks people again after they said no.
What I built
A retention policy with two respect rules written outside the learning: a declined offer stays declined, and members at risk see content before commerce. A calibrated churn model scores each member, a filter removes every forbidden action, and a Thompson sampling bandit chooses among a content nudge, a cheaper plan and a half-price month, within a budget. An independent auditor recomputes every rule from the decision log.
How it was evaluated
On 200,000 real members of the KKBox music service (churn model AUC 0.84). Their data holds no offers, so responses come from a simulator whose values were frozen before the real run, in three variants. Every policy ran on the same members with common random numbers; a 10% holdout and off-policy estimates checked against the simulator's exact answer guard the measurement.
Result (simulated)
The bandit spent 97 NTD of discount per extra member kept, against 211 for the rule-following baseline and 289 for the status quo, which broke the rules 31,490 times. The pre-registered claim, more members kept per 1,000 NTD than the baseline, held in every variant (conservative +1.74, central +5.57, optimistic +9.36). It keeps fewer extra members in absolute terms (7.8 against 10.8 per 1,000): the trade is cheaper retention, not more of it.
What I would do next
Run a real A/B test with the same holdout: it is the only way to learn how members respond to offers, and the result here says the policy is worth that test, under its stated assumptions.