It depends a bit on the type of app you are developing, but first let's assume a rather standard productivity based app.
My concern is that we will be offering design patterns to the user which might be unfamiliar to them compared to the ones they are used to on their native OS.
Good concern. Doing so will piss your users off and give your competitors market share (as your users leave to use them instead).
What platform did I take this screenshot from?

The app is Microsoft's Outlook Web Access (OWA) app. It's a good app; it integrates with OWA very well and does what you'd need such an app to do on your phone.
It looks pretty nice too. It does a very good job of following the Windows Design Guidelines.
So, what's the trouble? It's OWA for iPhone! It looks nothing like an iOS app, behaves nothing like an iOS app, and suffers because it does not use built in iOS patterns and elements. The app is just a pain to use on iOS, because a user is suddenly thrust into a completely different set of interaction standards!
Take the very first paragraph from the Apple OSX Yosemite HIG:
People love OS X because it gives them the tools and environment they need to create, manage, and experience the content they care about. A great OS X app integrates seamlessly into this environment, while at the same time providing custom functionality and a unique user experience.
Emphasis added by me. Although this describes a desktop app, it is not untrue elsewhere. User expectation of how an app will work is based on the platform they are using.
To reach back to the mobile space, here is the first paragraph from Apple iOS HIG's Consistency section:
Consistency lets people transfer their knowledge and skills from one part of an app’s UI to another and from one app to another app. A consistent app isn’t a slavish copy of other apps and it isn’t stylistically stagnant; rather, it pays attention to the standards and paradigms people are comfortable with and it provides an internally consistent experience.
Emphasis added by me. Keeping a consistent experience within the target platform improves usability and allows your users to quickly learn your app.
Jakob Nielsen wrote an article on Do Interface Standards Stifle Design Creativity? In it he writes:
Since the dawn of time (1984), we have known that consistency is one of the strongest contributors to usability. The Macintosh was based on a detailed book of Apple Human Interface Guidelines that were followed by almost all applications. One of the main benefits of the Mac (and later Windows) over earlier systems was the resulting consistency that made it possible for users to use software right out of the box.
Again, although the language targets desktop applications the philosophy is true across mobile platforms as well.
This doesn't mean your brand changes from one platform to the next. On the contrary you want your brand on Android to feel familiar to an iOS user, and a user should be able to quickly recognize it as yours. But this does not mean you create your own experience, separate from the both Android and iOS.
But, what about more unique applications. Take Yahoo's Weather app for iOS:

or for Android:

... um ... wait ... was it the other way around? No... no, I got the order right! But they look identical. The app doesn't look like an iOS app, it doesn't look like an Android app. The experience is exactly the same on both devices. Yet, the app doesn't violate either the iOS or Android HIG.
The places where it is important to following iOS and Android user expectations are there. Interactions and navigation might be slightly different, but those actions are expected by the users of the platform. They aren't re-learning interactions just to check the weather.
Generally speaking though, the platform HIG should always be by your side and should not be ignored.