@MattK Finding the perfect designer is tough as unlike programmer--there’s no way specific way to go about it. But prior to that - first list down what are the main tasks you want your designer to perform.
For example - you are looking for great icon designer who can do a bit of illustrations, if required. The key is to figure out 60% -80% of candidate's day to day task and then plan on how you can test those skills.
If you want a Designer who has multiple roles to perform say - coding, visual design, content writing etc. then have a checklist for each category. Example -
Coding - For ex- Designer should be well versed with HTML5 and CSS3 with a bit of jquery knowledge - Ask him/her to design a page and then do a code review. Easy one.
Visual Design - accounts for look and feel of the interface. - Show your product/company page and talk about how it can be visually enhanced and what is the thought process behind choosing the specific colors elements etc.
Content Writing - Ask for his/her fav product headline and how your site headline/content can be improved.
Apart from that few other questions can be :
• Walk me through your best and worst work in detail.
• Have you worked in team before? Who do you interact with most and how do you
tackle communication issues or opinion differences etc.?
•What is the most amazing thing you have seen on the Internet this month? [Passionate check]
•What phase or type of design you like most and least?