I would go ahead with your intuition of not supporting legacy browsers if you feel they only constitute a small part of your target population. However, I would recommend clearly highlighting that the browsing/interaction experience would be greatly enhanced if the user uses one of the modern browsers (mention those browsers clearly) and ensure the users see this information prominently.
The way you display this information is dependent on your application but ensure that people are aware of it and don't tend to skim past it.
From an anecdotal point of view, I work on projects for a client which uses only IE and one of our projects required me to design a internal portal for them. The client had certain requirements which would have needed CSS 3 support and though 80 % of the client uses IE 9, there were reservations about whether the remaining 20 % would get affected by this decision. We decided to go with the CSS3 emphasis and we introduced logic in the code so that any detected IE8 users were immediately informed that though they would be able to use the portal, some functionalities would be unavailable to them. Within one month of rolling out, we found that 60 % of the users who had visited the site using IE8 were now using IE9.