Actually I have come across toilets in a motorway rest area that has doors that open outwards.
It was a disabled toilet - one that was separated from the main restroom area, so the cubicle door was the only door. I used them occasionally when my kids were smaller and I needed to take a pushchair in with me, they have the larger maneuvering space required for wheelchairs.
When I came out, I nearly knocked out several people who were passing by.
"Hmmm, that's dangerous," I thought.
See also Naked Gun clip on YouTube.
Single room toilets and stall doors for disabled users
It definitely makes sense for doors to open outwards and the associated danger for others fades into the background in comparison with a disabled person needing the loo - and more to the point, potentially getting into trouble while in the cubicle. The reason for outwards opening doors is that in an emergency the occupant may be lying on the floor. The door can also open inwards if space permits, but bearing in mind the additional need for at least a 1.5m maneuvering circle.
Main restroom doors
If considering the door between the larger restroom area itself and the the outside, as opposed to the stall/cubicle doors themselves. I think generally the same principle applies. Certainly people inside a room are in the context of the facility and its purpose, whereas people outside are not necessarily in that mindset (yet) and maybe have no intention of going in but just standing chatting, waiting or passing by. People inside are rather less likely to be just standing there and more likely to be on their way out.
The solution for that particular problem, (where space allows) would be to have a L-bend or U-bend corridor or similar so that there are no doors and no-one can see round the bend. You'll frequently find exactly that in places of high footfall like airports and shopping centers.
Fire regulations
According to the US National Fire Protection Association FAQs there are circumstance that must drive the direction of swing. Quoting from the FAQs:
Except for a special form of horizontal sliding door and a couple of
other exceptions, the Life Safety Code requires doors in a means of
egress to be of the side-hinged or pivoted swinging type. Further, the
Code requires the swinging doors to open in the direction of egress
travel under any of four conditions: One, when the door serves a room
or area with an occupant load of 50 or more persons; Two, when the
door is used in an exit enclosure (such as the door into an enclosed
exit stair); Three, when the door serves a high hazard contents area;
and Four, when the door is in a horizontal exit. If none of the four
conditions applies, then the door is permitted to swing back into the
room or space.
The rule involving 50 people is related to a panicking crowd of people in an emergency pushing against a door that if opening inward would be impossible to open.
So if the restroom area counts as one of those four methods of exit then it has to swing outwards (in the US at least). If not, then logic, space and general safety should prevail.
Conclusion
At this point, it's obvious (and unsurprising) that there are many factors which affect whether a door should swing inwards or outwards. I'm sure that top of the list for installers are matters of safety, regulations, accountability. Consequentially matters of hygiene are lower on the list for many commercial and residential installations, but of course of high importance in medical or 'clean room' applications.
Some workarounds are available like the StepNPull and similar but these are retro-fit applications and certainly not suitable for use on their own.
A final tip
Personally, and depending on the establishment I tend to pull the handle at a point that is least natural to others, and perhaps tend to assume that others who do the same are those that also wash their hands properly.