Color to depth stream registrations, Astra Embedded S

[AntoineW]

Hello,

I recently bought an Astra Embedded S camera and was successful in using it in my own scripts.
I am using the OpenNi library. My final goal is to be able to measure distances using the depth stream, but selecting the points to measure the distance on the RGB feed. To do such a thing, I need that the two feeds are perfectly aligned (registered).

I was able to calibrate the two modules (IR and RGB) by myself, as Orbbec do not provide any compatible tool to do it with this specific camera.

However the builtin registration mode does not work at all. When setting the registration to true, I have shifts along the two axis with the two images as showed in the attached picture. As you can observe the depth feed is 0-padded and their FoV are not the same.
Does anyone could register the two stream to perform pixel to pixel correspondence ? In fact, without this feature the camera modules are unusable.

Cheers

[BogdanC]

The fix I found for this (with Python under Windows 64bit) was to go all the way to the folder that has the OpenNI dll, where there will be a configuration file called Orbbec.ini. Assuming you've downloaded the OpenNI SDK from the Orbbec website, then it would be under OpenNI_2.3.0.55\Windows\Astra OpenNI2 Development Instruction(x64)_V1.3\OpenNI2\OpenNI-Windows-x64-2.3.0.55\Redist\OpenNI2\Drivers.
In the configuration file, there is a line that is commented, something like
;Registration=0
Uncommenting by removing the semicolon and changing Registration to 1 fixed it for me. However, there will still be quite a bit of padding mind you, but pixel (x,y) in the RGB image will correspond to the same (x,y) pixel in the depth image.
If this doesn't fully fix it, you may also have to change the Resolution setting (make sure you go to the one corresponding to the depth stream), since the depth stream resolution is not 640x480 but 640x400 (or 1280x800). The number corresponding to these non-standard resolutions is unfortunately not shown in the comments, but I found somewhere online (can't remember where) that 17 corresponds to 640x400.

Hope this helps!

[AntoineW]

Hello Bogdan and thanks for your help.

Unfortunately this is something I already tried and there is still some distortion...

[BogdanC]

Well that's strange .. but just in case it helps: what I was doing was using the "Depth Stream using Python and OpenCV" example from https://astra-wiki.readthedocs.io/en/latest/examples.html (where I provide the path to the Redist folder as an argument to openni2.initialize) and the setting that I mentioned - and I do get two very well aligned images.

[BogdanC]

Oh .. you are right, I looked a bit closer and I am, in fact, still getting some offset between depth and RGB. What is even stranger is that when using AstraSDK (recompiling the SimpleStreamViewer example for 640x400 depth and turning on registration) it gets even worse.
I wonder if the issues mentioned 4 years ago in https://3dclub.orbbec3d.com/t/mapping-depth-to-color-registration-offset/94/7 are still there. Has anyone managed to register the two streams successfully and reliably..? Would doing it manually help I wonder?

[AntoineW]

Yes, even is the images get a better alignment when setting this boolean to true, there is still not a perfect registration between the two streams. I tried investigating if the remaining shift was constant along the 2 axis but it appears that it is not.
I admit that I more or less gave up with this camera. It is crazy that such a crucial feature being the registration of the two streams cannot even be supported yet.

[asifkhan]

Do you get RGB stream from Astra usinig OpenCV? How do you register a RGB frame captured by OpenCV to depth (or ir) frame captured by openni?

[asifkhan]

The depth resolution is 640x400 (using openni2) and the RGB resolution is 640x480 (opencv). If you delete the lower 80 rows of RGB image, making the frame 640x400, then you can solve half of the problem.