Call it "pound" or "hash".
The name 'hash' sounds quite technical. While every programmer do (should) understand what is 'hash', I'm not sure if I should expect it from the non-programmer.
During my stint as a software developer (before I started to care about the people who actually had to use the stuff I wrote) I never referred to the '#' character as "hash", nor did I ever hear it referred to it as such in school or industry. It was not until Twitter that I heard it referenced as "hash" on a significant scale.
When I speak to software developers today I still never call it "hash" in the context of programming.
My point: "hash" is not technical, at all.
Location and Age Matter
I have always called this symbol the "pound sign", or the "number sign".
This is a symbol that suffers a great deal from both chronological and geographical prejudice. As other answers and comments have pointed out, your geographic location has a great deal to do with how to refer to this character. In my case, my perspective is from the United States.
Chronologically you will likely find differences too. I am just old enough to remember when rotary phones were just slightly more common than touch tone phones. In the United States the '#' character has most frequently been referred to as the "pound key". It is still predominately called this, as I still dial into conference calls (to give just one example) and am prompted each time to "enter your pin, followed by the pound sign".
Younger individuals will most likely more immediately identify the symbol as "hash". Given the proliferation of social media the terms used (i.e., "hash") in those medias are becoming more prevalent.
Looking at Wikipedia's entry on the "Number Sign" begins as follows:
Number sign is a name for the symbol #, which is used for a variety of purposes, including (mainly in the United States) the designation of a number (for example, "#1" stands for "number one"). In recent years, it has been used for "hashtagging" on social media websites.
The term number sign is most commonly used when the symbol is used before a number. In the United States the term pound sign is catching on; the telephone key is called the "pound key".1 Outside of North America the symbol is called hash and the corresponding telephone key is called the "hash key" (and the term "pound sign" usually describes the British currency symbol "£").
Notice that "hash" is called out as being more common outside the United States, and the symbol has only more recently been associated with "hash" in the United States since social media.
The Wikipedia entry continues with several "Origin and usage and naming conventions..." sections for the United States, United Kingdom and Ireland, and other names in English.
There is also a section on how the sign is used in programming. Notice that nowhere in that section does the word "hash" appear.
Who are your users?
If you look at the number of Internet users by country you'll notice that the United States is a primary market. Add up the European countries and you will get close to an equal market share, but are all those European markets your targets? Even if you have no specific target, you will want to capture the markets that frequent your site the most.
What are some other examples?
Right here at StackExchange, the help section on headers refers to it as a "hash".
Dictionary.com has an article on this topic: What should you call the #?.
This seemed applicable:
What do you call it?
Your question is specifically calling out a phone. Using "pound" will be widely recognized in the United States, and may be recognized outside the United States in some instances. However, "hash" is more commonly used outside of North America and the term is easily associated with the symbol by individuals, generally regardless of geography, familiar with social media.
Calling it the "hash key" will very likely capture the widest market.