I am in close to the finish line of integrating the Orbbec Astra into my app. Using the SDK is pretty straight forward, so good job on that note.
I am now stuck with a problem I am not able to solve. I need a depth frame that is mapped to the color frame. I may be wrong here, but I assume that the call to “depthStream.enable_registration(true);” should give me just that. Yet the resulting depth frame has an horizontal offset of about 19px.
I also checked the sample “SimpleStreamViewer-SFML”. I enable the registration using the “R” key and captured the screen. I tried aligning the frames with Photoshop which also results in the same offset.
I have the camera Model: ASTRA F, using the win64 SDK 0.4.0, in VS2013.
Additionally I would need access to some camera parameters. I may not yet have found the right methods to do just that:
focal length of color/depth cam in pixels (x,y)
principal point color/depth cam
A cosmetic thing I noticed that you might hate me for even mentioning it
astra::ColorFrame::resolutionX(); why are these get methods not const?
Glad you are finding it relatively easy to get started.
You are correct in that registration is enabled via enable_registration(bool).
The registered images are uncropped at the moment in the v0.4.0 SDK, so what I think you are seeing is the parallax between the color and ir sensors. Cropping is something that were a looking to add before a v1.0 release. In the mean time, let me investigate a bit further and see if we can triage this.
Regarding the camera parameters: This has been a common request so we are working on a solution. At the moment it is not possible, but I will make some noise when it hits the SDK. EDIT: I should clarify. The intrinsics are not available from current Astra models (you will need to do your own calibration), however, we are working to expose the extrinsics as a 3x3 homography matrix.
You are totally right about the const thing. Thanks for bringing this to our attention.
Also, we have released the source to the SDK on github.
It’s under heavy development, and fixes / features will hit the develop branch before official releases. If you are willing to compile the SDK, you can have access to the bleeding edge. To add to that, the dev team is not one to let hubris get in the way of suggestions, constructive criticism or anything that will make the code better.
I can verify that the sample (“SimpleStreamViewer-SFML”) demonstrates the registration problem @Xcessity described. Here is what I see when I press “O” in the sample (“toggle overlay color stream on depth stream”)
But as he mentioned, sliding the depth frame to the right about 20px gives you a much nicer overlay. (I did this in photoshop from the two separate streams)
We are also encountering registration errors which is why we did some simple experiments as well on our end.
It looks like the registration errors are dependent on depth (errors are larger when things are closer to the sensor), which suggests that the registration piece gets something wrong about the color-to-IR translation. At high distances (4+ meters) the registration errors aren’t noticeable.
Here’s a question: Since registration requires knowledge about the camera intrinsic parameters (focal lengths, optical centers, etc…), does Orbbec do a calibration of each camera in order to have these parameters for the registration? Or is the registration using default parameters?
We’re looking at improving the depth-to-color registration done in the sensor hardware for future versions of our cameras. We’re also looking at software improvements to registration for current cameras. For now, there is a sweet spot range where the registration works well, and closer and farther than that unfortunately there will be an offset.
@Rafael The field of view for the depth, color, and IR streams are accessible through the Astra SDK but other intrinsic parameters are not. (You can calculate focal length from field of view and the image size in pixels.)
Thanks.
So if we want accurate registration I guess that means we’re on our own for now (i.e. perform complete IR/color calibration then do the registration in software)?
Can you tell us if there is any shift between the IR image and the computed depth image? We know that Primesense had such a shift due to the correlation window they were using when they decoded depth from observed speckles. So can we assume there is a similar shift in the Astra sensor?
Thanks. Can you let us know how large the shift between depth and IR is? We could find out ourselves but you could help us save some time here if you already know this.
