Recommendations for hand gesture tracking

Technical Support & Q&A Basic
1

Dear Astra pros,

We’re planning to realise a hand gesture based control for the GUI of our software. Our Software is Windows C#, WPF.
We need the recognition of hand gestures to be reliable within a distance of at least 1.5m to do zooming, panning and dragging gestures on a graphical workspace. Latency should be as low as possible. Two hands have to be recognized simultaneously and at least two gestures (closed/opened hand).
I saw this video of the Persee: Products - Persee Update on Vimeo but the latency looks quite problematic to me. Is the tracking so slow or is the application causing that delay?
If it is because of limited performance of the integrated computer, would an Astra be a better choice?
I like the fact that the Persee wouldnt require computing power from the PC being controlled and also a very minimal communication by LAN or bluetooth would be an option instead of streaming heavy imaging data over USB.
But if the latency is not avoidable we might have to use the Astra?
Another Question regarding Astra:
I read the Astra S is better for lower distances. Does that mean the accuracy of the depth image is better or is that just about the RGB image? For us only the accuracy of the depth image would be crucial. On the other hand a bigger range would be a benefit. Would the Astra S have any advantages in hand tracking at a distance of 1.8 meters compared to the normal Astra? Are there any image/video samples for a comparison of the different outputs?

Thanks a lot in advance!

Best
Perlis

2

FWIW - as best as we have seen the Astra S and Astra are pretty much the same device just tuned to different focal lengths - ASTRA S is focused on the 0.4m to 2.0m range while the ASTRA is 0.6m to 8.0m

If 1.8m is your sweet spot then you would maybe be better off looking at the ASTRA - it works very well at the 1.5m-2.0m - the ASTRA S would maybe be getting near the edge of its focal range.

BUT - having said all this - currently orbbec themselves have only limited support for GESTURE tracking - they do have a deal in place with Gestoos for access to their gesture system - but the licence seems and little odd and requires re-registration every couple of months - and gestoos seem more interested in pushing people towards their retail point of sales HTML5 based system than their c++ version which has not been updated in forever.

ALSO it only supports visual studio 2013 AND to be honest from all our testing its pretty unreliable and crashed a lot - it seem runs quite slowly and is limited to 320x240 resolution.

Having said this - once you get it setup - if you dont mind that it seems to take forever to start up - and can live with low resolution - it is reasonably effective at tracking - but again its gesture recognition can be somewhat hit and miss - L and Point are ok - but open and close / grab detection seemed very hit and miss.

The body tracking system that orbbec has released themselves has I would suggest more long term potential - but right now it only tracks joints - so Head-Shoulder-Elbow-Hand - it cant track fingers and right now doesnt have any gesture tracking capabilities.

However, we have just finished a working pointer GUI for one of our game engine implementations which works quite well once you get it tuned and have some filtering on the motion data - and can track both left and right hands at the same time - of up to 3 people - based on the body tracking sdk.

In terms of tracking - the performance of the body tracking sdk on a astra is very fast - with quite low latency (25-30FPS) - BUT - in our testing the ACCURACY of the tracking is more of an issue - this means - that even when holding your BODY or HAND perfectly still - the tracking point jumps around quite a bit - - adding a filtering algorithm or a moving average window algorithm can help this considerably - BUT you then start to increase the latency - by the length of the window.

AND well finally - right now its ONLY c++ - we have looked at couple of options around this - either a wrapper or a virtual joystick implementation which might suit your needs.

Happy to have a chat offline if you like.

Westa

1 Like
3

Thanks a lot Westa!
Regarding the Persee you didn’t use that one, right?
In view of your experience I tend to look more towards the Intel devices (SR300 and the upcomung D400 series) as a hardware choice since the D400 probably delivers the better signal basis for whatever algorithms we will use in the end. So we could start to develop for the SR300 and then switch to the D400 when available.
From the software side I’m still in the dark. I want to keep the implementation effort as low as possible by using the best available existing SDKs. But it seems hard to find the right one. What did you use for your pointer GUI finally?

Best
Tomas

1 Like
4

FWIW - the Intel 2.0 SDK has pretty much dropped support for middle ware applications such as gesture tracking and the like - as such - the Cursor SDK that was an option on the SR300 if you accepted the limitations in terms of depth tracking distance - is not supported by the SR4xxx series cameras at this time - and Intel seem to be implying that it is unlikely to return - though its often hard to get a straight answer out of intel on this sort of thing.

In terms of SDK - we are currently using the Orbbec Body Tracking SDK with a Orbbec Astra at up around 2meters with good results in terms of location tracking - ( and rolling our own gesture system on top for now ) - IMHO the gestoos system was not a good option for our needs.

Just my two cents

Westa

1 Like
6

What are your own gesture system based on? Any SDK or your own?

//Thomas