The first and the most important thing you need to determine is whether you are actually able to run a valid A/B test - do you have a big enough sample size at your disposal?
Which is not the case when you are developing a new product.
A/B testing requires a lot of participants (sample size) - traffic, users and conversions (like a lot). A rough ballpark is 1k monthly transactions (conversions). It is possible with less but it becomes increasingly challenging and less effective with smaller sample sizes.
If you run an a/b test with an insufficient sample size, you'll be running an underpowered a/b test which means that there's a high probability that your results will be imaginary, for example:
- You'll likely see an inconclusive result (no difference between the new and old design) which does not mean that there really isn't a difference (you'll need a bigger sample size to detect that difference)
- You may even see a 'statistically significant' winner (or looser) but it does not mean that the variant is really a winner (or vice versa)
In simple words, if you ignore this, you are simply not going to be able to conclude if your new idea is better than the existing design. There's no way around it, the math is not going to work (statistics is tricky) - you need large amounts of data to detect 'real' differences between two (or more) variants:
Therefore, if you don't have the sample size yet, your best bet is user testing or other qualitative research (surveys, polls, interviews and etc.)
Second thing - A/B testing is widely misunderstood and highly complex. You are basically performing a science experiment (sort of) and there are many ways to screw it up :)
And the chances are high that it may even not give you the answers you are looking for so here are some great in-depth beginner resources to begin with (to learn when and how to use A/B testing properly):
But the answer to your actual question (how to determine if you should a/b test an idea or a suggestion by a user?) - many a/b testing pros use simple prioritization frameworks which are more or less based on the same effort/impact matrix. More details (and framework comparisons):