Multi-selecting through dragging, Ctrl-clicking and Shift-clicking are reasonably usable when the user normally only selects one object at a time. They have weak discoverability, but are very efficient for users. They, along with the deselection method you describe, are standards, so a lot of users already know how to do this, so discoverability isn’t as much of an issue as one might think. Besides, multi-selection is usually an expert shortcut. In most apps, anything a user can do with multi-select they can also do with single-select –it’ll just take longer, so discoverability is not considered to be a requirement. Deviating from the standard, on the other hand, will likely confuse many of your users.
Single click to select is a good idea when the user must select multiple items to complete a key task successfully. It’s faster than Ctrl-clicking but not as fast as Shift-clicking or dragging for larger number of proximate objects. On the other hand it has much better discoverability. Putting a checkbox-like control next to each icon effectively communicates that each selection persists through multiple single click. You may need a Clear button to clear all selections.