I was having a discussion with my housemate who is a data analyst by trade, and the conclusion that we came to is that there are two sensible options here, depending on the amount of work you personally want to do (we're assuming here that the collection of gender data is actually useful to you, rather than simply of interest in which case it may be better to simply leave it out).
The first option is to have three or four discrete options:
Other, and possibly
Prefer not to say. In my experience, this is an acceptable option for gathering data while being inclusive - it acknowledges that there are people who don't fit the gender binary, and allows them to select a different option. It also allows people to completely opt out if they really don't want to answer (the standard objection is that it'll negatively impact your data collection, but in practice it probably doesn't make much of a difference). Unless their gender is specifically relevant to the application, there's probably little chance of someone being directly upset by being unable to express their exact gender identity (note that if gender identity is important to your application for some reason, then this is not really a sensible or inclusive option).
The second option is to have a textbox and suck it up - it's a data sanitisation problem. A simple find/replace on your dataset will be able to lump your users into a group of man/male/boy responses, a group of woman/female/girl responses, and a group of assorted other responses. Crucially if you're doing demographic analysis, whatever is left over probably isn't statistically significant at an individual level so in your analysis it is acceptable to put them in an internal
Other category. You can then preserve that minority data for further study should you find you need it.
Gender is the correct label for this field, from a descriptive point of view and from a data collection point of view. You'd be surprised how many people think it's hilarious to answer
Sex: with "Yes please".
If you choose to go with the simple dropdown/radio button approach, then
Other is an appropriate label for the third group. It is easily understandable, and non-exclusive in terms of what it might represent.
Transgender is probably not an appropriate label here unless you include additional ones because it excludes people outside the binary who are not transgender or who do not view the label as appropriate for them (while it may technically be inclusive of a wide variety of gender identities, in common use it tends only to represent people who are transsexual). There is one minor problem with the use of the word "other", however - it is directly exclusionary and can potentially feel like the user is being shoved into a box of leftovers - not an ideal experience! For that reason, a text box is probably preferred if you want to make sure you're being inclusive.
Think Outside The Box mirrors these recommendations and has some other interesting guidelines for form construction.