For now I came out with this design/approach:
2 lists, one near another, with checkboxes on the users selection side and crosses(for quick removal) on the new building group.
I tried to search elsewhere but couldn't find other approaches, would you find it as a good UX?