Its useful for the company to know name/email/company of the person requesting the download but isn't it more of a barrier that is not good for UX and puts people off?
That depends. Which is greater: the company's motivation to get people to read whitepapers in which they are not that interested in? Or the user's motivation to read the company's whitepapers which are highly recommended must-reads?
The difficulty or inconvenience of the barrier must be appropriate to the motivation of the user and the value of what is behind the barrier.
With regard to the UX topic: filling out forms is, generally speaking, crappy user experience, regardless, and you know it.
If the material really is valuable, though, I would not mind. Say some renowned author suddenly makes a free electronic copy of a 25 year old classic textbook available on his website in PDF, and he only requests that you grace the guestbook with your
email@example.com. How could anyone object.
The study compared the "reward" approach (e.g. you get the whitepaper if you provide some contact details) with the "reciprocity" approach (e.g. you get the whitepaper, aren't we generous, so please be so kind as to provide some contact details.
The results were very interesting: people exposed to the reciprocity approach were twice as likely to provide their contact details than people exposed to the reward approach. Key take out: the psychological weight of the reciprocity phenomenon is significant.
This depends on the value of the whitepaper to the customer. If the whitepaper is essentially advertising (information about how amazing the company is), then you should have the lowest possible barrier to retrieving them; the company gains value by having that paper read, and this is likely far larger than the value of the identity of who is reading it.
But if the reverse is true, and the whitepaper contains valuable information that users greatly desire, then they may be willing to jump through hoops to get it. Be aware that companies often overestimate the value to users, because they are not the center of the universe... but they think they are.
You could easily get some metrics on this by a simple A/B test. Have the form show up for half the site visitors, but not for the other half. See how many users get blocked by the form, and see how many users input obviously bogus information just to skip the form. If the percentage of useful responses is worth less than the number of users who abandon the attempt then you will have hard data to back up your recommendation.
It's definitely a barrier, but the goal would be to make that barrier easy to move. There's user experience writ large, and then there's the user experience of particular actions deemed necessary by the site creator, content owner, and so on. For example, I find it a bad user experience to have to pay money to buy things, but knowing I will, I am pleased when the experience of doing so is good. (I won't lie, that example is a little exaggerated.)
As a consumer of content, I know (or I should know) that in order to get something -- trusted content that I need or desire -- I have to give up something. If that means I become a lead in someone's CRM tool to get a whitepaper or a scholarly article or a demo of a game or the list goes on, then so be it. Much like how I know that if I want to read a book, I have to purchase it (exchanging info and money) or borrow it from a library (exchanging info via my library card). Yes, there are options in-between, but those become transactions that are usually less straightforward than the standard experience.
So where does that leave the content owner? With decisions to make that will lead the developer to create a workflow that reduces barriers for the user but still remains transactional. With some A/B testing, it's quite possible to come up with a compromise. Some forms are very simple (email address only), some have all fields but only require one, and some have a form but also allow you to jump directly to the link to download. Each of those options are better than a longer form with many required fields, and most likely came into being through discussions and user tests in which the balance was struck and both sides came out with a tangible benefit.