James Foster of Exis Web did an interesting A/B test on the hamburger icon:
Tests on mobile showed a difference, though not all that significant, when the icon was used with a border (so it looks like a button):
Perhaps more interestingly, the A/B test seemed to more clearly indicate that desktop users don't understand the icon:
I tested 4 variations of the content inside a blue button:
(Baseline) The word “MENU”
The word “MORE”
Hamburger icon + “MENU”
Hamburger icon + “MORE”
The results were far more significant.
Tests 2, 3, and 4 performed far worse (18%, 31%, and 43% worse respectively).
And interestingly, the icon + label performed worse than just the label (which is usually pretty easy to integrate into a desktop page/app). So it would appear for mobile the icon works serviceably, as long as it looks like a button, and on desktop it's best to not use it at all.
James Foster then conducted a second AB test, this time looking at how the hamburger menu tests when used in combination with the word MENU and with a button border and combinations thereof.
Based on this and my previous AB test, a flat hamburger icon may not be ideal on a responsive website (remember this is a website not an app). Using the word MENU (and making it look like a button) could be more helpful for visitors.
This does not mean that users do not understand the hamburger/sandwich – it could be that the word MENU draws more attention.
This also supports findings recently published by NNGroup
Clear labels help users make decisions faster: they give good information scent about what will come next. Labels should still be used for newer icons, such as the three-line menu icon (or “hamburger” icon)...
...Users are still unfamiliar with newer icons, including the three-line menu icon