People - as Don Norman says in this video at UX Week 2008
See also - I am not a user.
Manufacturers often refer to end-user, but I really don't like that term actually. Personally I've no beef with user in a tech environment, but I do use people as a generic term when talking to...well...people! I use the term you often when speaking to someone, and in some scenarios customer is appropriate.
I'm not sure the situation crops up often enough to have to differentiate "People who use the product" and "people who don't use the product", by having to use different terms?
Having laid the groundwork early on in a discussion as to whether you are talking about you or [other] people who use the product, then you can variously refer to them as you, they, people, them, etc... and not often should you need to chop and change the subject between those that do or don't use the product. I'm sure there's exceptions - but I'm speaking generally.
Note, you should avoid the he/she approach - as discussed in this ux.se question
Update: One of the above links goes to a website called 'I am not a user'. They have a poll where people can vote on whether being called a user is ok - or not.
At the time of the OP's question (May 2011) their poll said 52% of people were happy to be called users!
Nearly 6 years later (Jan 2017) there is a swing the other way and 59% of people do not like the term user.
Whether this website attracts a fair audience or not is of course unknown. There is no way to know if this site has attracted an unfair interest from people who don't like to be called 'user' but the biased subject matter of the site should not be dismissed as irrelevant!