and found that there's too many data points to visualise them and not all of them matter.
Sorry, but I think you are mistaken, the outlier on the chart may very well be the one that steals the show or presents new avenues of thought.
Beyond self editing your data set for convenience , which again you do at your own risk, I would suggest you simply change to a different type of chart, a bubble or scatter might be a better fit.
In the context of the viewer, it's generally OK to sacrifice the Ex. N entries if necessary by grouping them together in order to highlight the main entries
I can only speak for myself ( as n= 1 viewer), but I want to see each one even if they are 1000 n-#s, I still think a bubble/scatter chart is your best bet, and again I'll try justifying my point: If one of your n-1 = 1 points grows over time, so it is now n-1=3 for instance, by grouping it you lose the ability to notice it.
You could of course choose to show 2 charts, the complete data set and an edited summarized chart with the most entries, to be accurate you should note your grouping/threshold scheme.
As for the threshold,I would group them by similar attribute, in this case number of entries.
One last thing, you might think I am pedantic, but in the history of displaying quantitative information (see Tufte) and in my own experience ( I used to prepare financial stock charts,research data for publication) this sort of details are the ones that end up causing mistakes.