Body mask is a array of integers the same size as the depth frame … Each point holds a value representing the id of any current body detected at that location in the depth frame.
Floor mask is similar … But only has a value representing the floor being detected at a point in the depth frame.
As to how accurate or reliable the math is. … sadly It’s not very good IMHO
I have used Kinect in my application to detect Floor plane and it was very reliable. I need to have that info for my application. Have you Tried NUITRACK. Is it better than Orbbec body Tracking SDK?
Can I convert Body Mask frame to OpenCV Mat frame and change the color of the area where the body is detected?
I would have to say that nuitrack SDK produces better results for the most part … But … Some of the limiting factors seem to be more related to technical issues with the current orbbec sensor hardware/ firmwwre … And in accurate depth reporting.
The masks are not colour as in RGB … They are single byte per depth location memory blocks … How you use these for visualisation is up to you… The samples in the SDK demonstrate a couple of techniques for displaying the masks.
Are you using polling or event based processing of the frame?
I assume you have setup the frame pump and are getting a valid body track?
What is the value of: bodyFrame.bodies().size():
//////////////////////////////////////
here is some sample code
// get depth frame metrics
int width = depthFrame.width();
int height = depthFrame.height();
const int depthMapLength = width * height;
// get depthData
const int16_t* depthData = depthFrame.data();
// get bodyMask
const auto& bodyMask = bodyFrame.body_mask();
const auto* bodyData = bodyMask.data();
bodyCount = bodyFrame.bodies().size();
std::uint8_t value = 0;
// we are mapping directly to a bitmap
// first 12 bytes are allocated for bitmap image type and sizing
int rgbaOffset = 12;
for (int i = 0; i < depthMapLength; i++)
{
const auto bodyId = bodyData[i];
const auto depth = depthData[i];
// quick modulo of depth mapping to put gradient of depth mask on screen
value = (depth % (255*4) / 2 );
colorRGBA thisColor = { value , value , value , 255 };
if ( bodyCount > 0 )
{
// lazy map a couple of colors for bodies - use a lookup table normally
if ( bodyId == 1 )
{
thisColor = { 255 , 0 , 0 , 255 };
}
else if ( bodyId == 2 )
{
thisColor = { 0 , 255 , 0 , 255 };
}
}
// then map color
memBlkPtr[rgbaOffset] = thisColor.r;
memBlkPtr[rgbaOffset + 1] = thisColor.g;
memBlkPtr[rgbaOffset + 2] = thisColor.b;
memBlkPtr[rgbaOffset + 3] = thisColor.a;
// increment offset ready for next value
rgbaOffset = rgbaOffset + 4;
}
/// then we map bitmap to texture etc
I got it working. The mistake I was making was converting to OpenCV Mat format.
Here is the working code;
Mat imgr(480, 640, CV_8UC1, Scalar(255, 255, 255));
const auto bodymask = bodyFrame.body_mask();
auto bb = bodymask.data();
int h = bodymask.height();
int w = bodymask.width();
for (int x = 0; x < w; x++)
{
for (int y = 0; y < h; y++)
{
if (bb[(x + (y * w))] > 0)
{
imgr.at<uchar>(y, x) = 0;
}
else
{
imgr.at<uchar>(y, x) = 255;
}
}
}
The solution is a basic 3D geometry calculation, the original coordinate system located on a camera (world coordinates), the destination coordinate system located on a floor, you need to calculate transformation from original to destination and apply it on each joint per each frame
I found that it was easier to counter-rotate the “JointRoot” object to account for the rotation of the floor plane. The JointRoot is the parent of all generated skeleton joints/nodes. This was much easier than trying to apply a translation/rotation matrix to each joint, individually.
Keep in mind, the Astra’s floor plane also includes the height of the camera (or, rather, the distance to the floor from the camera). So, you can set the JointRoot’s y-position, and then counter-rotate it to account for the slope of the floor plane. When doing so, you must remember that the Astra is giving you joints with the x-position data that is mirrored, so it looks good in the demo…the floor plane data is also “x-position inverted”.