E.M
Back to selected work
AI deploymentEvaluationsHuman-in-the-loop design

Smily AI Inbox

Changed an automated AI feature to keep humans in the loop

Using AI to reduce cognitive load while keeping consequential communication under meaningful human control.

01

Problem and crux

The first version of the AI inbox was designed to answer common guest questions automatically. Testing raised a more important question: did users want maximum automation, or did they still need control over the final message?

The crux was whether automation would save time without removing the oversight users valued.

02

Assumptions to test

We initially expected automatic replies to create more value. User testing showed otherwise. Property managers wanted AI to prepare the response, but they wanted to review it before anything was sent.

03

What I did

I changed the product from automatic replies to AI-generated drafts. Users could approve the draft, edit it or replace it before sending. I also introduced evaluation criteria for relevance, tone, language, booking context and the amount of correction required.

04

Evidence and result

More than half of active accounts used AI Draft in its first month. Some users reported saving up to two hours per day.

>50%Active accounts used it
1 monthAdoption period
Up to 2hDaily time saved

05

Relevance to AI safety operations

This work gave me practical experience with human oversight, evaluation criteria and product decisions shaped by where users need control. Those habits matter when an AI system can act on consequential information.