David Mulder and Das Beasto have some really good information in there. In response to one comment, the difference in their numbers is because Das Beasto (via Microsoft) is referring to "difficulties" while David Mulder is referring to specific disabilities. For example, someone with dyslexia, or color blindness, or who is simply very myopic has a visual difficulty but is not blind.
I work with disability advocates on accessibility in physical locations, and one of the things we push all the time is that not everyone that has a disability is what is conventionally thought of as "disabled". A kid that broke a finger playing basketball is at a disadvantage using your website in much the same way as an elderly user with arthritic joint stiffening, or a user with Parkinson's disease causing uncontrollable spasming and cramping in their hand. And overall, the population is aging. As it does so, the percentage of users with disabilities is only going to increase. One of the things my coworker likes to throw out is that everyone is going to be disabled at some point in their lives, whether it's because of injury, illness or aging.
The US Census reported in 2010 [pdf] that it did not find a single community in my state (Wisconsin) that did not have people self-reporting as disabled. We have towns of less than 300 people. They also determined that nearly 20% of Americans have some degree of disability, with roughly half of those classifying as a severe disability.
I would also say that if you're designing your website well, it should not be exceedingly difficult to make it accessible. Modern screenreaders and other assistive technology are getting really quite good, and assuming your site follows a good structure, you mostly just need to provide alternatives to video and images. You can easily implement elements that are only visible to assistive devices, which is helpful to make a skip link so that a visually impaired user doesn't have to listen to all of your navigation links on every page. Ensuring that all of your user interface elements are tab-reachable and have a reasonable tab order is not just essential for accessibility, it's good design. I do not have a disability, but if I have to use your page a lot and I can't tab through it cleanly, I'm going to be keeping an eye out for a better designed substitute, and I think you'll find that attitude common among a lot of young professionals (who tend to have disposable income).
Finally, it's good PR. Slap a label on your website touting its accessibility. A lot of people will be impressed like that. It's like calling your product green. Conversely, if I was at a brick and mortar store and I found out the owner wasn't willing to shell out a couple hundred dollars for a threshold ramp, I would walk out. I have no desire to support that kind of miserliness.
Edit
I would add that, at least in the US, the disability community is unusually well organized compared to most demographic groups. They will stage boycotts and start lawsuits if they feel unduly discriminated against.
Edited to add:
The ACLU produced this report on accessibility in online voter registration webpages that's been making the rounds in our office. Our website was not reviewed as we do not offer online registration, but we are incorporating its findings into our other sites.