Will many users use a mouse in VR or is this be disconcerting and cause motion-sickness? Or is it just nonsensical to use a mouse in VR.

How would you go about porting the UI/UX of a drag and drop game to VR? Will I have to wait for VR gloves to become a standardised commodity? It would be awkward to use an Xbox controller, that'd be like playing an RTS (e.g. Starcraft) on a Super Nintendo.

  • 1
    Not sure this can be answered here, but you don't need sensing gloves to track hands. Check out LeapMotion — they have a mount for their device that lets you use it on a HMD (head-mounted display) and track hand movement.
    – Drew Beck
    Mar 12, 2015 at 2:25
  • 3
    Also this youtube vid does a great job of sketching out what still needs to be figured out in VR: youtube.com/watch?v=n3b8hZ5NV2E
    – Drew Beck
    Mar 12, 2015 at 2:28
  • 1
    @DrewBeck That's a great video, thanks for pointing it out. Mar 12, 2015 at 10:28

2 Answers 2


This question is quite broad for a single answer to cover everything, but I'll give it a go!

First of all, motion sickness. In actual fact it's not even motion sickness, it's the opposite - it's the brain telling you you're moving when other senses tell you you're not.

And in any case, it's not the manipulation of scene elements itself that is directly related to the problem of sickness (or v-sickness, simulator sickness, Rift Sickness, or whatever you want to call it).

I worked as a developer on VR games and simulation experiences at the beginning of the 1990's - the times of Virtuality and Sega VR. At that time we had a lot of trouble with sickness. In a typical end-user environment, games were kept to about 2 minutes of playtime because after that we found people quickly started feeling unwell. Experiences tended to be slower and less frantic than games so you could get more time, but still the problem persisted.

Getting students in to test our equipment (which they happily did!) often turned out to be a err.. messy business because of the longer testing times.

The causes of sickness in VR then were mostly down to the lag between motion of the body, especially the head, and the resulting visuals in the headset feeding back to the eyes. The more lag, the sooner you became disoriented and felt ill.

The headsets were heavy, the refresh rate on the display was relatively low compared to modern VR equipment, and the trackers themselves had relatively high latency and sample rates. All this has had a lot of money thrown at it in recent years in order to reduce the lag. Tracker sample rates are now up to 1kHz and the motion lag is down to just a couple of milliseconds, drastically reducing the risk of graphics entering the 'barfogenic zone'. Displays have changed (from LCD to OLED) in order to reduce smearing and increase sharpness and refresh rates. And the ability to fine-tune or calibrate the headsets for individual users is much better now compared to the 'one ring to rule them all' approach that has been used in the past which essentially meant that no user ever had it right!

Overwhelming the senses is another thing that can add to the sickness feeling. The games of the 1990's were played in tremendously loud bustling hot environments like London's Trocadero. Reducing brightness, volume, and environmental effects allows you to take back some control of your senses to 'manage' the situation better without being overwhelmed.

So my point this far is that the feeling of sickness comes from the experience as a whole. If you tried to use a mouse as a VR input device that would not in itself be a cause.

The mouse is not an appropriate interaction mechanism for VR, other than the fact that it has scrolly type wheel which can be useful.

The natural input method for VR is your hands (putting accessibility aside - just for the moment), not least because they are always with you, and you can't put them down, but also because you don't have to think about using them. A mouse essentially needs a flat surface which is a mismatch from manipulating a 3D environment.

In fact, even talk of using similar hand held controllers to existing games is probably naive. Leaving the hands free to 'do their thing' is I think a critical step in making VR usable for most people. The use of gestures or particular hand or arm positions will have meaning. Sign language might be useful.

So drag and drop games would need to cater for different orientations of the body, the game, the head and the hands. Luckily we're already quite used to device touchscreens and using them at different angles so touch, drag, swipe, pinch, expand gestures are already quite well known and these would translate quite well to a drag and drop game in a VR environment allowing the user to immerse themselves and orient themselves and the scene and it's elements to a comfortable point.

This is in fact the least of the worries. Most VR development probably starts out with exactly some simple drag and drop game, but the problems come when you start dealing with real detailed, meaningful, personal data, and interfacing between the virtual and real worlds, because that's what is going to be needed - in the same way that we talk about cross-channel and multi-channel experiences for desktop, mobile and brick & morter environments.

Finally - I don't think you would ever 'port over' the UI/UX of a drag and drop game to VR. You would have to completely rethink the interactions from the ground up. You would have to strip back the concept of the game to it's bare bones and then say 'how could we do that in VR'? That's not to say there won't be games of the same name carried over from the PC or consoles, but they'll take as much investment of time and money, if not more, to get perfect for VR as they did originally.

  • What a great way of explaining that porting a drag and drop game over to a virtual reality environment is like porting a motion picture over to your microwave oven. Different environment means different experience.
    – DaveAlger
    Mar 12, 2015 at 13:45
  • Well, there's no need to exaggerate that much @DaveAlger, Drag and drop moved to the iPad pretty easily, and made them better. The same will be possible (in 3D space even!) with the right interactive user tools. I don't think 1-2 cameras (or similar) will reliably respond to a pinch gesture, So gloves would help sense that motion at least. (Gloves in the broadest sense of the word, it could just be a thimble (or something smaller) on your thumb and forefinger that responds to a touch.)
    – Andrew G
    Mar 12, 2015 at 18:59
  • A lot of people are used to the Xbox controller and it's their ideal tool for playing FPS. VR is made for first-person experiences. A gun doesn't have half the functionality of a controller. Porting is the process of making software usable in other environments, so besides the negative connotations of the word, And the redevelopment of the UI I still think the word is apt.
    – Andrew G
    Mar 12, 2015 at 19:24
  • Xbox tried mapping just a simple click using a push gesture and it's pretty awkward. I think both hands might be required to do even a simple action like a click (one hand hovers over the item while the other hand makes a fist for mouse down and opens to a flat palm for mouse up maybe)
    – DaveAlger
    Mar 12, 2015 at 19:44

The reason why a mouse is a terrible UI for VR is because when you (inevitably) take your hand off of it (to type or itch or eat etc.), then you have to find the mouse without being able to see it! That can't happen in VR. You have to be able to see the things you're interacting with or the things you're interacting with have to be either 2-handed items that you won't put down or basically impossible to put down. e.g. gloves, a gun, an xbox controller or your own hands.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.