Ask them to design something. Give them a domain you work in every day, and a pencil, and ask them to sketch out a simple one or two screen prototype.
Ask them to test the design, with one of your coworkers as the subject. Observe not just how the design goes, but how the testing goes.
Observe their process. Do they understand your requirements before sketching? do they iterate their design as they get user feedback? Does their design seem like it might work? Does it work when you try to actually use the prototype, or do they have to spend all their time explaining it to you? Would you be able to get your work done using this design?
Check their portfolio if you are interested in their skill at visual design. Have them show you something pretty they made, and explain why and how they made it that way. What worked? What didn't? It's hard to do visual design on the spot with only a pencil, but you could also ask them to do something bounded, like logo design.