AI recommendations - suggested flow

Question

Let me set some context for my case, the dillema and then get to the actual question.

Context
We've been desiging a feature with a target customer of UX/CRO people. The feature fetches a lot of website data (think all interactions from an ecommerce website) and then based on user behavior it generates recommendations for optimisations like "8% of users leave checkout at step #3 due to input "confirm your password", blah blah".

In plain terms this is a Machine Learning Model (chatgpt etc), that looks over all the data, identifies opportunities and makes data-driven recommendations.

Dillema
We have a dedicated page for showing the recommendation and based on some other use-case requirements we have concluded to the following 2 options

• case 1: The "manual"-mode. The user clicks to generate recommendation and then evaluates what is useful. AI is used to generate the recommendations, we just ask the user to initiate those.
• case 2: The auto-mode. The app continuously generates insights. The user just go to recommendations page, reads what is there and evaluates if something is useful.

In general we don't expect users to just come in their "spare time" to see insights. Usually they would have the goal of researching for recommendations when they get asked from their managers.

Question
Is there scientific research or best practises on how to show AI recommendations like those? My preference is case 1 - the manual mode, since it can provide some level of control on the user over AI which could boost engagement with the feature (although I haven't backed this by research or studies).

Answer 1

The question comes down to: Is there a difference in how the recommendations are generated?

In example #1 the impression is that there is some "thorough" process involved to generate the report on recommendations. Probably based on a pre-defined, well-tested set of rules. Implying some level of "trustworthyness".

Example #2 makes the impression of some "casual/on-the-fly" generated recommendations backed or entirely generated by an A.I. - which might not "get" what is really an appropriate recommendation or situation to be improved. It also implies a lot of "fluctuation" in the results. If this window is opened and each time there is a totally different set of recommendations shown than the last time, this might rise suspicion on the user's side on how reliable this information is.

Either way: If there is AI involved and not some human-made, curated ruleset to analyze the data, you probably want to make it clear to the user with a fair warning that they have to deal with the caveats of the AI generated stuff ("illusions" etc.).

Giving the user more control than just a "Generate" button should be considered if this is really going to be used as a research tool. A user might want to specify if the focus is for example on something like churn rates or views/clicks etc. And might also want to have insight into the raw data to confirm things scientifically.

The other difference I consider is performance: If it is cheap to generate those recommendations on the fly, then you can follow example #2 probably. If it takes some time to process the data and present results, you need to stick to #1 anyway.