This strategy is often called a "concierge" MVP. You deliver the service you claim, but behind the scenes everything is incredibly manual. Once you've proved people like the service, you then go make the process less manual. Zappos and Amazon are both famous for doing this.
p.s. -- I already put this in a chain, but the majority of comments are just claiming this is fraud. Thought it might be worth posting something slightly more visible.
There are many times a service like that might not be fraud. For example, if you never explicitly said it was an AI, and if every detail of the service remains the same, from the customer's perspective.
The privacy implications alone make the difference between a human sitting in on your meeting and an actual AI enough to call this fraud. Giving it a fancy name doesn't change that.
When I was teaching product development I called this the Mechanical Turk strategy. Completely agree that it isn’t automatically fraud, it can also be a cheap prototype that lets you start testing hypotheses ASAP.
p.s. -- I already put this in a chain, but the majority of comments are just claiming this is fraud. Thought it might be worth posting something slightly more visible.