We don't have detail about the scenario for which you ask this question, so I hope this answer is relevant to you. Otherwise, just mark it as irrelevant to ensure that no-one else will benefit from the topics I discuss below, for their own scenario, for example.
(Did you see what I did there?)
You're mixing two goals:
1) Provide user with a list of relevant products
2) Improve the relevance of products shown
And your solutions for trying to achieve both is surely going to result in such a tight coupling of dependency and non-traceability, that you won't be able unravel it later.
Goal number 1 - you are by your own admission providing the users with mostly irrelevant results which isn't doing the user any favours in trying to efficiently find products of interest - which is all they care about. No-one likes to have to do work in order to get rid of the rubbish in order to find what they do want.
Goal number 2 - Essentially you are using your users to crowdsource irrelevancy data.
What is irrelevant to one user may be of interest to another and you will never capture the full spectrum of information that will allow you to provide perfect results for all users. You will end up providing only the products in the list that are deemed of interest by the majority of people, and omit those products that have been marked as irrelevant by the most number of people. This can skew results according to number of previous shows; positions in the list on previous showings; geographical interest; colour interest; male/female relevance; cost relevance etc etc (depending on what your products are and who your users are).
This is all gearing up your algorithm for determining relevance to be irrevocably tied in to data that has no hard evidence based reasoning for the majority of decisions it makes.
Let's compare the situation to looking at email:
Most of my email is spam. I get lots of spam. I have filters but they don't catch 100% of spam. I manually delete lots of spam so I can see the wood for the trees and deal with emails I actually want. There's also email server filters that delete 90% of email before it even gets downloaded - except occasionally it very annoying traps an email I did want to see.
Imagine if every time you deleted an email you had to say why it was irrelevant: It's spam; it's no longer relevant; I saw this earlier; I know what this is but it's not for me right now; etc etc. There could be any number of reasons why an email is irrelevant to me, but just because I deleted it, doesn't mean it wouldn't necessarily have been relevant to me last week, next week, if I weren't in a hurry, or just to someone else.
There is one thing that is the saving grace for email - I don't want to receive more of it - it's so annoying. But I put up with it because I need to receive my email. The ones I do receive and read make up for the ones that are irrelevant. So this can act as a spur to mark an email as spam before I delete. For me this means clicking on the 'toggle as junk' column in Thunderbird, before I delete. But even then, this is rare step compared to just hitting delete.
On the other hand I'm quite prepared to mark email for why it IS relevant - I'll add it to a folder according to its relevance (sender, project, pending action etc).
In short - I'm prepared to do more work where I have a positive feeling towards something, but less work for that to which I feel negative.
So consider these questions:
Are you providing enough value to the user in the few products that are relevant for them to be prepared to put up with the work of dealing with all the irrelevant results?
Is your relevancy algorithm going to be testable: Is there going to be a way that you can validate your algorithm - and verify that result inclusion or exclusion is right or wrong.
Are you considering making the user work too hard on things towards which they feel negative, rather than on things to which they feel positive - such as dislikes rather than likes; deletions rather than wishlists and favourites; 'mark as irrelevant' rather than 'just leave alone'; negative actions rather than positive ratings;
Should you be perhaps be relying more on pre-filters: good search; user filters; sorting mechanisms; well implemented faceted search, and generally better structure and organisation of data to show and control what the user sees, rather than getting them to weed out that which is not of interest first. Should you back this up with post-filters: favourites; wishlists; ratings; save for later; users also liked, etc etc.
Can you make it all a positive experience rather than a negative one - and still achieve your goals.