Your primary consideration should be whether the UI behaves the way the user expects. If it's intuitive, drag and drop away. You can use hallway testing to check if your UI is intuitive. In that situation, dragging and dropping appears to be the most sensible as well as the easiest course of action, so there are few (if any) reasons you shouldn't use drag and drop.
The only other thing you might want to consider is strain caused by too much movement. If the user has to drag something very far (or very often), it's a different experience than using a mouse. With your updated question, it seems like this shouldn't be an issue. One way to counteract this, however, would be to allow users to select groups of people that can then be dragged and dropped.
Finally, when you say there's no drag-and-drop functionality in the iPad's native apps, you're missing two things. First, there probably isn't a need for it (so if there's a better way to implement your functionality, do that instead). Second, iOS does use drag and drop. When you're rearranging icons, placing them in smart folders, etc., you're dragging and dropping.
updated to reflect changes to question