This question has been asked many times and many answers have been given, in this site and in many others. I've seen the answer change a bit during all those years, but mostly, all of them point to not opening new tabs/windows unless it is an external site or a document that is not intended as a web page, like a .pdf, .doc, .ppt, etc.
Some of the advancements in technology, like tabs on a browser, have given arguments to the proponents of opening links externally, the basic argument is that a tab is not as intrusive as a new window and is not blocking or hiding anything. On the other hand, privacy and control advocates state that it still their browser and no one should decide what to do for them. We can add to that last statement that UX concerned people say that the right option is to give users the option to decide what to do, at least on cases where the decision make sense.
In some places you may find references to articles, blog posts, references to studies or strong opinions, like Nielsen's, etc. But no study, by itself seems to be found.
Studies about the subject:
There is only one study that I have found that mentions something related to this subject, 2006-04-13 - Eelco Herder - Ph.D. Thesis - Forward, Back and Home Again Analyzing User Behavior on the Web.. Specifically in section 6 of that study, we can read that most of users on the studied group used new windows and tabs for different tasks, originating that task on a search engine or a specific page. This by itself, doesn't help the question asked here, but the analysis of the situation, tells us that users prefer to open some links on new windows or tabs when they think it's important to compare information, read carefully the linked page, keep the original page on sight or keep the navigation history intact to be able to go back to a previous point.
The study also mentions that even when users wanted to do that, it also loads the brain with extra work, having to remember how they got to that tab/window, having to keep the navigation history, including the branching, on their minds and having to deal with the little help provided by page titles to do that.
It's an interesting study about certain behaviours, not just this subject. The studied group is small and they consider themselves as advanced users, so, as usual, any result has to be taken with proper judgement and be adapted to each situation's circumstances.
Other very common references, but not studies:
1997-10-15 - When to Open Web-Based Applications in a New Window on Nielsen Norman Group. You may also find this article referenced in many sites but in two ways, the old Nielsen usability site, (useit) or the modern Nielsen Group. It's the same article. This article is the most referenced, it doesn't show any data, but we always assume that all the articles from Nielsen are written considering the experience gathered on his multiple usability studies. By the time this article was written, there where no tabs, so all the references and experiences it's based on imply windows. This article gives what is the most common opinion; open links on the same window unless it's something that is not intended for a web browser.
But it's important to realize that this article seems to be based on a very specific situation that years, people and applications have expanded, Nielsen is talking, basically, about applets, their behaviour was very similar to what you would expect from a Flash application, or embedding a word document on a browser now, but with some differences, like applets where used, sparingly, for very specific and specialized tasks, never for amusement or just design, they did look like the web page they where on and the load on the browser and client system was significative. People usually miss that fact, which is relevant and is easy to realize if you are old enough or if you read the linked reference, Navigating the Applet–Browser Divide; in which you can read an addendum from 2007 reminding us about the specific circumstances of the study, but also, telling us that the study and conclusions are still valid.
2000-11-06 - do not change the current window without informing the user on Techniques for Web Content Accessibility Guidelines 1.0 on the W3C. Not a study, but a basic set of recommendations on how to use related mechanisms.
2005-08-29 - Open New Windows for PDF and other Non-Web Documents on Nielsen Norman Group. This article is also cited many times and is much more related to modern times and circumstances, it states what is the most common idea, use new windows/tabs for external documents. The article is focused on documents and media related to specific applications; like .pdf documents on a .pdf reader, .doc documents on a word visor/editor, etc. I addresses the UX aspect of the situation, stating that a different experience has to be on it's specific environment. Again, we assume the base for this article is extensive study, he cites one study on 42 intranets, which you can buy on his site.
2011-01-01 - Top 10 Mistakes in Web Design on Nielsen Norman Group. This updated list from Jakob Nielsen mentions as item #2 having .pdf, and that implies other similar documents, as elements on the browser, recommending to download them, which is related to opening a new tab/browser since another option, instead of having the same or different window/tab is to offer downloadable material. Also, as #9, is a more direct reference to the subject.
2012-31-01 - Why External Links Should Open in New Tabs on UXMovement. This article doesn't claim to be based on any studies, the author supports the idea of opening external links on a new tab, as you could infer from the title. The comments section, is a mixture of all the know opinions, but the vast majority consider that a wrong procedure, the argument range from computer illiteracy, lack of control and disabilities (or the software used to help people with).
2013-09-05 - G200: Opening new windows and tabs from a link only when necessary on Techniques for WCAG 2.0 from the W3C. Not a study but a recommendation. The original version is older than 2013, but I mention the current edition. All the processes on the W3C are lengthy and bureaucratic, but most of the people involved have years of experience on different areas, so if something has reached the level of recommendation, there must be a good reason for it, we may not agree, but we almost positively can say that some studies are involved.
????-02-01 - Beware of opening links in a new window on webcredible. The author of this blog post, who should be good at usability, communication and UX professional, doesn't understand the concept of dates, so we don't really know where the article was written, but considering that he only speaks about new windows, it should have been around 2008. The post gives good usability reasoning for not using new windows/tabs as well as the usual recommendations for when to do it. The only reference they mention is the 1997 article from Jakob Nielsen that most people use.
[????-??-?? - section 10:12 Indicate Internal vs. External Links on usability.gov]. This is just a recommendation, but in the references section mentions some studies that may have relevant information. If somebody can find them and get permission to publish them or at least part of it, we may have more references.
Some other references or related material. Still not research, just articles and opinions based on the same as above.