You'll be up against the same problem people had when everything was text mode: the keyboard doesn't provide any tipoffs. If your customer can't remember how to do something, they've to go look it up in documentation because the keyboard only shows them its keycaps, and the amount of information a keycap offers is Not Much.
So if you have a simple u/i, one that can be easily learned, then yes, it will make it quite awesome and wonderful for people who'd rather not shift back and from from one implement to another.
If you're providing 2 modes of control as it sounds as though you are, then at the worst people will find the keyboard mode too hard (the way I find the Unix vi and emacs editors too hard for what I want from a text editor (I use geany under FreeBSD)) and they'll use the mixed mode.
So if you think you can do it, go for it! But as you do it, get some of your prospective users to give you advice and feedback while there's still time to make changes.