Astra with OpenCV+OpenNI2

I have compiled OpenCV 3.1 (latest commit) with OpenNI2 (downloaded from Orbbec) and I am able to run a sample code with other OpenNI supported cameras such as Xtion PRO. When trying to use Astra, however, the program gets stuck in the grab() method, it simply does not get the image or show any errors. When using OpenNI’s SimpleViewer or NiViewer the depth stream seems to work just fine, but I cannot see any color stream.

The sample code I am using can be seen here:

#include <iostream>
#include <opencv2/opencv.hpp>

using namespace std;
using namespace cv;

int main()
{
    VideoCapture cam(CAP_OPENNI2);

    Mat color, depth, depC;

    while(true)
    {
        //Grab frames
        cout << cam.grab();
        cam.retrieve( depth, CV_CAP_OPENNI_DEPTH_MAP );
        cam.retrieve( color, CV_CAP_OPENNI_BGR_IMAGE );


        //Process depth -> colormap
        double min, max;
        minMaxIdx(depth, &min, &max);
        depth.convertTo(depth, CV_8UC1, 255./(max-min), -min);
        applyColorMap(depth, depC, COLORMAP_HOT);

        //Show
        imshow("Depth", depC);
        imshow("Color", color);

        char c = waitKey(30);
        if (c == 'q')
            break;
    }
}

I enabled OpenNI2 verbose mode and wrote the output to a file, which can be seen here. Note that it seems like depth frames are getting delivered at almost 30fps, but OpenCV can seem to get them.

Is anyone else experiencing this issue?

I have done some tests and it seems the problem is that even though Astra does not support a Color stream (through OpenNI interface), it is listed as valid. Basically, the problem occurs here. I have commented the lines where it grabs the color stream and it worked fine.

I do not think OpenCV implementation is wrong, but that you should change your driver so that the color stream gets listed as invalid. Because AFAIK the color stream is only available through V4L interface.

So my request is to update your driver listing Astra’s color stream as invalid so I can use it with OpenCV and OpenNI bindings. If possible, I would like to have the color stream also to be delivered within the OpenNI interface.

1 Like

@eduardo.arnold Thanks for reporting this. We will look into updating the driver.

@eduardo.arnold : i tried open cv with orbbec in Qt creator to get depth image, but it fail. here is the code :

#include “mainwindow.h”
#include “QApplication”
#include “opencv2/core/core.hpp”
#include “opencv2/highgui/highgui.hpp”
#include “opencv2/imgproc/imgproc.hpp”
#include “iostream”
#include “stdio.h”

int main(int argc, char *argv[])
{
//QApplication a(argc, argv);
//MainWindow w;
//w.show();
//return a.exec();

std::cout <<"opening device"<<std::endl;
cv::VideoCapture sensor();
cv::Mat depthFrame;
sensor.open(cv::CAP_OPENNI2);
if(!sensor.isOpened()){
    std::cout<<"sensor cannot open!"<<std::endl;
    return -1;
}

for(;;){

    if(!sensor.grab()){
        std::cout<<"sensor couldnt grab image"<<std::endl;
        return -1;
    }else if(sensor.retrieve(depthFrame, cv::CAP_OPENNI_DEPTH_MAP)){
        cv::imshow("depth image",depthFrame);
    }
    if(cv::waitKey(30) >= 0)
        break;
}


return 0;

}

it always stuck in opening device. VideoCapture.isOpened() always return false.
i already set path include and library, this is some of .pro file :

INCLUDEPATH += -I/usr/local/include/opencv
LIBS += -L/usr/local/lib -lopencv_core -lopencv_videoio -lopencv_imgcodecs -lopencv_highgui -lopencv_objdetect -lopencv_imgproc

INCLUDEPATH += -I/usr/include/openni2
LIBS += -L/usr/lib/OpenNI2/Drivers -lDummyDevice -lOniFile -lOpenNI2 -lORBBEC -lPS1080 -lPSLink

the default of openni2 lib from structur.io doesnt work with orbbec so i already replace the library with the one in orbbec sdk folder. can tell what i doing wrong ?

@motionagus What version of OpenCV are you using? Are you sure you are using a compiled version that was compiled with OpenNI support, ie, compiled with the WITH_OPENNI2 flag? Otherwise you will not be able to use OPENNI2 bindings for OpenCV.

Opencv 2.4.9 does not have OPENNI2 flag, does it?

I’m not able to compile it

Not the binaries they provide on the official website. You will have to compile it yourself with that flag enabled, which I must say can be quite cumbersome, specially if you are not familiar with CMake compilation procedures. Depending on your application it would be easier to get the video stream another way (perhaps OpenNI2 record application) and then just process it using OpenCV.

Hello,
I am using Orbbec Persee.
I have built OpenCV 3.3 with OpenNI2 from makefile using CMake.
However when i run the code below:

`
#include <opencv2/core/core.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>
#include
#include

using namespace cv;
using namespace std;

int main(int argc, char** argv)
{
while (true)
{
VideoCapture cap0(0);
VideoCapture cap1(1600); //for CV_CAP_OPENNI

	namedWindow("RGB Stream", 1);
	namedWindow("Depth Stream", 1);
	if (!cap1.isOpened())  // check if we succeeded
		{
			cout << "Depth Open Failed" <<endl;
			return -1;
		}
	for (;;)	
	{
		Mat intensity, range;
		cap0 >> intensity;
		if(!cap1.grab()){
 		   cout << "sensor couldn't grab depth frame" << endl;
 		   return -1;
		}
		cap1 >> range;	//**this is where the code gets stuck.**
		imshow("Depth Stream", range);
		imshow("RGB Stream", intensity);
					
		if (waitKey(30) >= 0) break;	//to exit
	}
	return 0;
}
return 0;

} `
On execution, the code gets stuck at the point where it fetches a depth frame (OpenNI), and does not even return any error.

Has anyone found a fix for this?

Also, since OpenNI wasn’t exactly built, rather only the path for libraries is specified in the environment, can that be a problem while compiling/execution?
I tried running the export paths before compiling, but there was no change.