Perhaps, Google reCAPTCHA v.3 would be an auxiliary solution for you. As it claimed in the documentation:
reCAPTCHA v3 will never interrupt your users, so you can run it
whenever you like without affecting conversion. reCAPTCHA works best
when it has the most context about interactions with your site, which
comes from seeing both legitimate and abusive behavior. For this
reason, we recommend including reCAPTCHA verification on forms or
actions as well as in the background of pages for analytics.
It returns a value that represents a probability whether you're site is dealing with a bot or a human. It's possible to make use of that, at least to reduce the amount of encountered issues by means of reducing the number of users who have to enter a captcha.
Invisible reCAPTCHA v.2
If it's preferred to use reCAPTCHA v.2, its behavior can be improved as well utilizing it with the invisible parameter set to true.
It appears to be a considered option according to the comments. However, since the details are of it implementation is mainly unknown to the general public and hidden behind Google's risk engine, it looks like it may worth a try.
Custom Captcha Implementations Usage Caveat
It requires a bit of investigation and analysis of existing custom solutions. Many of them are even easier for bots than for humans, thus, provide no security from bots. This isn't limited to mathematical operations, but includes text recognition captchas as well. Such implementations from small vendors should be used with a great caution.