# Describing 3D swiping hand movement as curve on a XY plane

I'm trying to describe a 3D swipe gesture (only vertical or horizontal, no diagonals) above a given flat surface using as much conventional geometry or similar non-machine-learning techniques (Hidden Markov model, artificial neural networks etc. are therefore excluded) as possible. From multiple observations of the data retrieved from the device I concluded that a swipe can "easily" be described as a curve (or in some cases as a really straight line). With this question I would like to know how a curve and curved movement can be described in simple geometric terms in a most efficient (mostly speed- but also memory-wise) way.

The post is divided into two parts - one that gives information on the data that is used and one that gives an overview of what I have come up with so far. Sorry in advance for my poor Paint skills. :D

The 3D position data

The device I'm using streams 3D points each representing the hand's position at a given point in time. I can capture and evaluate these. The following image visualizes the plot of the data from two different perspectives - top-down and isometric (more or less):

• XY plane view (on the left, aka top-down view) - for each sample only the values along the X and Y axes are taken into consideration. This view represents the surface of the device above which the movement of the hand is detected
• XYZ view (on the right, aka isometric view) - for each sample all three axes are taken into consideration. This view represents the full 3D movement in a volume above the device surface which defines the space where gestures can be detected

In the next image I have added the movement of the hand as detected by the device:

The actual movement looks more like this:

Based on the observation of the actual movement and the one detected by the device I can mark almost half of the samples that the device has given me as invalid namely all border values (along each axis a position can be between 0 and 65534) which do not describe the actual movement of the hand from the perspective of the user of the device (in the image below invalid data is represented as the trajectory part which is covered by a polygon):

Of course sometimes the "valid" portion of the trajectory is rather small compared to the invalid data:

The algorithm I have described below doesn't care how much the valid data is as long as there are at least 2 samples that fulfill the requirement of not being border positions meaning X and Y are different from 0 and 65534. An issue arises from this which I will elaborate on in the next part of this post.

Describing the movement

I have given it some thought and this is what I came up with:

1. Extract only the set of valid samples that is exclude all which have a border position

2. For each sample generate a local XY coordinate system which is aligned with the XY coordinate system of the surface of the device (to make things easier :)):

3. Next I'm thinking of calculating the vector between the current sample and the next one (if present) and calculate the angle between that vector and the X axis (can also do that with the Y axis):

4. Using the magnitude of each angle I can determine if the movement between the current and the next sample leans more towards horizontal or vertical one and also in which direction.

This should allow me to determine the general direction of the swipe movement as well as how it is position above the surface. I have done a lot of swiping :D but since I want to describe this in a more formal way I obviously need to describe my findings hence the need to find a way to describe and classify a curve based on its properties. Perhaps calculate the curvature of the whole trajectory?

There are of course some issues with this algorithm that came to my mind:

• Since the user swipes in full 3D space and not just rub hisfinger across a surface it might happen that he makes a swipe movement with a hand and ALL the values are border values. I'm thinking of handling this case by simple introducing two cases for a swipe gesture:

• border gesture - all samples have border position. I've noticed that when doing such a swipe movement the samples are placed along only one of the borders. This makes things easier since I don't have to think about the case where all the samples have border position but are also distributed along 2 or more borders (in which case I would actually have two portions of the trajectory in a 90deg different direction to each other). This gesture is also not a curve but a straight line so evaluating it is much, much easier (note that in the isometric view the samples are all glued to the wall of the volume that is closer to the user with all Y values equal to 0):

• non-border gesture - a portion or even all samples have position different from a border position. In this case I can exclude all samples aligned at the border and extract only those that are really part of a curve

• Efficiency - since I'm doing relatively simple calculations it should be fast enough (for example a swipe gesture should be recognized withing 200-300ms in order to provide smooth interaction). Memory-wise things also look relatively good.

I've searched online before I started thinking of creating the algorithm I've described above but couldn't find anything. Even the topic of classifying curves seems to be either not that popular or the search terms I've used are too broad/restricting. The classification here is not that essential imho (unlike what follows) but it would still be nice to be able to split the resulting curves in sets each representing a swipe gesture.

The next thing I have been thinking about is curve fitting. I have read articles about this but frankly beside a couple of tasks at my university during the math course I haven't given it much thought except for Bezier curves. Can anyone tell me if curve fitting is a plausible solution for my case? Since it's curve fitting one would be right to guess that we need some initial curve which we want to do our fitting against. This would require gathering swipe movements and then extracting a possible optimal curve which is something of an "average" of all curves for a given swipe. I can use the first algorithm I have described above to get compact description of a curve and then store and analyze multiple curves for a given swipe in order to get the "perfect" curve. How does one proceed when handling curve classification?

• Since the position of the swipe is irrelevant, the first thing I'd definitely do is to replace every sample point `p(t)` with the delta vector `d(t) = p(t+1)-p(t)`. Then you can try to formulate rules on when the `d(t)` function represents a horizontal swipe. (For example if more than 90% of values have an angle between -30 and +30 degrees, or something similar.)
– biziclop
Commented Nov 30, 2016 at 15:47
• Who are you trying to communicate this to? Why are you trying to communicate it? That would help you get an answer. Commented Jun 1, 2017 at 22:19
• @SwankyLegg The author of the answer may correct me if I am wrong but this is for an academic project. So there is no real audience apart from the vastness of the academic community Commented Aug 6, 2017 at 14:32
• @rbaleksandar I think what you are trying to achieve requires machine learning. All the ideas that you describe are just ideas and should be taken into account for the final model but your gesture detection will not be very flexible if it is hardcoded by rules that you define in an algorithm. Why not have as inputs the trajectory and as targets the correct trajectories that you feel correspond to the truth. Of course you have to create lots of targets by yourself this way but this is the current best recommendation. Commented Aug 6, 2017 at 14:37
• Would this question have a better chance of getting answered in a different Stack Exchange? It is asking for an algorithm to compute a 2D projection of 3D motion onto an XY plane. It’s also gone unanswered for 2+ years. Commented Feb 23, 2019 at 15:18

Tech aspects are generally outside of UX scope, therefore following topics cannot be answered or covered here:

• Programmatic gesture representation (like bezier curve etc)
• Gesture detection algorithms
• Input filtering methods
• Programmatic ways of defining custom gestures

To generalize your question and describe it in simple terms, you are asking:

How to represent gestures so other people can understand and learn ones. where curved movement over surface is a one of possible gesture types.

This question can be divided in two parts:

• How to represent movement (gesture) with dynamic picture. (easy)
• How to represent movement with static picture (hard)

How to represent movement with dynamic picture?

If your device is capable of displaying things or you can make your users go online, I see no better way than demonstrating gesture and its effects in action:

(kudos for beautiful animation: https://uxplanet.org/brave-nui-world-rise-of-touch-less-gesture-control-882be077cdfa)

How to represent movement (gesture) with static picture?

a) one picture + direction (http://tomlythgoe.blogspot.com/2011/12/gestures.html)

b) one picture with first / last / intermediate positions

c) timeframes

(example of very complex motion) https://weighttraining.guide/exercises/cable-bench-press/

Worst case is if you need to train your user do precise movements like trainer do to you on your first gym class. There is no simple icon to describe gesture for remote surgery equipment, people are being trained for years.

Perfect case is when it is obvious how to use your device (based on user's previous experience) and you don't need any explanations at all.

As you are not making iPhone and your resources are most likely limited, easiest way would be to use simple icons because your device should handle rest as much as possible.

https://www.vectorstock.com/royalty-free-vector/hand-touchscreen-gestures-vector-19198371

https://www.vectorstock.com/royalty-free-vector/set-of-gestures-icons-for-vector-20957687

Just make the icon look that way so it is clear you expect entire device / hand do gesture, not only fingers.

This is all user must to know about swiping on your device :)

## Research paper tacking a similar issue

http://faculty.washington.edu/eliezg/publications/Gurarie2010_BMB_HelicalHeterosigma.pdf