When service teams get AI: faster and better rated — but retrials unchanged, and top agents get worse
A large-scale field experiment in Alibaba's after-sales chat support: the assistant only drafts opening diagnoses and proposals, agents can adopt, modify or ignore — and who is using it decides the outcome.
Key numbers
- Design: a large-scale field experiment with Alibaba randomised whether human chat agents could access a generative AI assistant that drafts issue diagnoses and solution proposals in the opening stage only; agents could adopt, modify or disregard them, so the study estimates effects of both access and usage.
- Average effects: gen AI improved service speed and subjective quality (customer ratings) but had no significant effect on objective quality, measured by customer retrials.
- Interactions were reshaped: treated agents responded faster and took a more proactive role while customers provided less input — patterns that persisted into later chat stages.
- Heterogeneity is the headline: lower-performing agents benefited most, narrowing performance gaps, while top performers saw declines in both subjective and objective quality, consistent with workflow disruption — more shift-away time, response delays and immediate customer retrials.
Method and sample
arXiv paper “Generative AI in Action: Field Experimental Evidence from Alibaba's Customer Service Operations” (2026-02-08): a large-scale field experiment randomising access to a generative AI assistant, with outcomes measured through customer ratings (subjective quality) and customer retrials (objective quality), analysed by agent skill.
What it means for brands
After-sales is the front line of AI adoption, but deploying AI is not the same as improving service — it changes rhythm and division of labour. For brands the takeaway is that when customers ask after-sales questions in AI (warranty, returns, repairs), the answer depends on the policies and facts AI can read, not on agent scripts.
How SourceSight handles it
SourceSight's prompt library covers after-sales and warranty questions (warranty scope, return policy, repair timelines, on-site service) with mentions and position per engine, while the content factory writes policies, service scope and lead times into structured pages.
Limits
The experiment ran in one platform's after-sales setting, so the sample and operating structure are platform-specific; results do not transfer to every industry, especially regulated or high-ticket services.
References
Run it on your own project
What the same prompts return across 10 engines is only knowable by running them.
Related research
Cross-category recommendations: +36% add-to-cart on item pages, +15% on cart pages
Recommending general merchandise to grocery shoppers — a milk frother with the milk — has long been hard; LLM retrieval plus real-time cart context moved production A/B metrics.
Industries · 2026-10-07Home services: 22% of homeowners already use AI to find a provider, and 80% of businesses don't know how to be found
The local services funnel is moving upstream: before a homeowner opens a website or calls, AI has already synthesised reviews, reputation and service descriptions into one answer.