Some product pages contain less than 5 products and on average between 5-10. Would the filter element disturb the customer experience since he has a good overview already since the product count is so low.
The keyword here is consistency. If some of your product or category pages have a filter, then ALL of your pages need to have it. It will look consistent, and it will be easier to maintain.
However, if your pages have 5-10 items per page, you don't need a filter at all. In general, you start considering filters when you have at least one additional page of results. If you have the more or less standardized layouts of 12, 16, or 20 products per page (grid layout) or 10, 20, or 25 (list layout), anything less or equal to 10 itdms makes filters unnecessary.
Now, you can have a higher-level page, like a homepage or products page where you list ALL of your products. In that case, you could use the filter on these higher-level pages and leave it out on the subpages, which will make everything look consistent.
You never know, I doubt you want to start worrying about adding a filter later, if it's already there, it's fine to just leave it. Having a low item count is fine at the beginning so long as you can get the count up again.
But if it really makes you uncomfortable, you can look into placing all the items on one page separated into their different categories a la Google Playstore