some compound components can't be derived from base components. so they seem to be orphans that don't exactly fit anywhere. where do you put them.
If a component can not be derived from existing base components (atoms we say) then it is a base component or you are missing something in your base comoponent!. so it is a good idea to make list of all base components you find in your style guide before you move further.
the more you 'atomize' your ui elements
What do you mean by this, I mean It is not possible to atomize UI elements after some point. so again It is better idea to set up your elemental building blocks of your interface, which will definitely serve as building blocks when you hit complex UI components.
If you still feel that your atomic components are not getting reused then either you need to work on your atomic components again or add this new one to the list of existing atomic components (If it makes sense).
where do you strike the balance between completeness and complexity
One of the biggest advantages atomic design provides is the ability to quickly shift between abstract and concrete.
best way to test this will be is to pick up a new page and use your style guide to build the page. If you think your style guide achieves this with less effort(I mean more your atomic components are getting used) then you are there. I don't think it will ever be complete though :).
good read on atomic design
hope this helps :)