As with every design answer, it depends. There may be situations where adding responsiveness may be detrimental to the experience, but for the most part I'd say that responsive design is generally a good thing.
Getting responsive design right is incredibly tricky. Here's a great article on the subject: http://stephanierieger.com/a-plea-for-progressive-enhancement/ And here's a great slide show all about what mobile really means: http://www.slideshare.net/bryanrieger/rethinking-the-mobile-web-by-yiibu/
So depending on your situation, here's what I'd suggest.
If you're building an application from scratch.
Go with mobile first. I'm a big fan of mobile first, even if you're not explicitly making a mobile site. You'll find it incredibly helpful to disengage yourself from the stereotypical large desktop. You'll end up making a more useful design.
If you're redesigning an existing site / application.
Look at your users. Look at your analytics. If they really don't use mobile that much, and it would take a lot of effort to re-implement the design using a responsive layout, then skip it. But chances are that since you're asking this question, they're using mobile. In that case, if you have the budget, I would really suggest going responsive. Paying attention to odd sized devices will only help you down the line. (I can't imagine anyone saying to themselves "I really wish I hadn't considered mobile...")
But, if there's something that would be totally ruined by making it responsive, obviously skip it. For example, if there's a game that really requires a fixed layout, or if you have to show charts / graphs at a certain resolution. In these cases, you obviously shouldn't be doing much with them. (Of course, it's case specific and you may have an awesome creative idea that solves the problem. In which case, go for it.)
I'm curious to see what others think about this and I'd love to hear any previous experiences people have had since we've added responsive design to the toolbox. (Comments?)