Latest SDK v2.0.7 - Help find any example or instructions of using c# wrapper

shmlex · December 21, 2017, 5:06pm

The new version includes NET/C# wrapper, can not find any example or instructions to use it in visual studio.

victorsbd · December 21, 2017, 6:39pm

Hi, im looking for the same code examples.
I cannot find it.

andrew · December 28, 2017, 10:27am

Hello @shmlex and @victorsbd,

Here you can find sample code:

Some notes about using of C# wrapper:

Use binaries from SDK for VS 2015. In other case you’ll have issues. See Issues with initialization and deinitialization of C# wrapper -- SOLVED for details. In the above sample project all required binaries are already included.
Call Astra.Context.Initialize() to initialize library before use. And don’t forget to call Astra.Context.Terminate() before exit from application (without this call application will crash on exit). See App.xaml.cs in the sample code.
To start working with sensor, call Astra.StreamSet.Open(). If you want to work with multiple sensors, then call Astra.StreamSet.Open("device/sensorN"), where N is zero-based index of sensor. See MainWindow.xaml.cs in the sample code.
Then, using this opened streamSet create stream reader by calling streamSet.CreateReader(). See constructor in SensorViewModel.cs in the sample code.
To work with depth stream, use streamReader.GetStream<Astra.DepthStream>(). To work with color stream, use streamReader.GetStream<Astra.ColorStream>(). Etc. See constructor in SensorViewModel.cs file in the sample code.
Set required mode for depth (and/or) color stream using AvailableModes property and SetMode(mode) method. See SetDepthMode() and TrySetColorMode() methods in SensorViewModel.cs file in the sample code.
Be aware that not all modes returned by AvailableModes are actually supported! This issue was reported here: Can't get 60 fps - #3 by andrew
To start streaming of data call Start() method for appropriate steam.
To receive frames from stream(s) you have to organize loop (it is better to do it in some background thread to keep responsibility of your UI). In this loop call streamReader.TryOpenFrame() and then use out frame to access data. Don’t forget to free frame by calling frame.Dispose() or by using using (frame) clause. See BackgroundProcessingLoop() method in SensorViewModel.cs file in the sample code.
Alternatively, you can call Astra.Context.Update() method in this loop and subscribe to streamReader.FrameReady event. But it is not so convenient in case of working with multiple sensors.
Don’t forget to dispose streamSet and streamReader when you stop working with sensor. See Dispose() method in SensorViewModel.cs file in the sample code.

Have fun!

chrisib · January 2, 2018, 11:27pm

Thanks for the sample code! I’m trying to make a stand-alone camera class that will deliver OpenCV Mat objects, but I’m getting an error when I try to set the stream mode: every time I try to access depthStream.AvailableModes I get an “Astra.AstraException: astra_imagestream_request_modes: Invalid Operation” error. (My code to set the mode is copied and pasted from your sample code.)

I’ve got copies of all of the relevant DLLs (same versions as your example project), and I’m initializing the Astra context in my application startup event. Have you run into this error before? If so, what did you do to fix it?

andrew · January 3, 2018, 9:19am

Hello @chrisib,

This error is very general. It erases in case of any troubles inside Astra SDK. To have additional information, you can run your application under Debugger (F5 in MS VS) and check content of Output window in Visual Studio. It should contain log messages from Astra SDK.

And, please, make sure that you don’t have AstraDotNet.dll file in your output directory (Debug or Release). This file must be only in subdirectories Astra\amd64 and Astra\x86. If you have this file at the same level as your main executable file, then please select reference to AstraDotNet assembly in Project Explorer inside Visual Studio and set Copy Local = False in Properties panel.

Also, have you tried samples from bin directory of Astra SDK? Do they work for you?

chrisib · January 3, 2018, 4:18pm

The sample programs run fine, and I can compile and run your WPF demo just fine. So obviously I’m doing something wrong, but I have no idea what.

In my output directory (at the root level) I have the 64-bit astra.dll, astra_core.dll, astra_core_api.dll, and AstraDotNet.dll, along with my main executable. My build target is x86_64, so it’s not an architecture mismatch error.

Here’s the output that gets logged to the console:
2018-01-03 10:12:41,094 WARN [context] Hold on to yer butts 2018-01-03 10:12:41,094 INFO [context] configuration path: E:\code\Prism3\VisionServer\bin\x64\Debug\astra.toml 2018-01-03 10:12:41,094 INFO [context] log file path: astra.log 2018-01-03 10:12:41,094 INFO [context] astra_core library version: v2.0.7-beta-5f803a5e24 API Level: 1 2018-01-03 10:12:41,094 INFO [context] astra_core_api library version: v2.0.7-beta-5f803a5e24 API Level: 1 2018-01-03 10:12:41,094 INFO [context] plugin search paths E:\code\Prism3\VisionServer\bin\x64\Debug\Plugins\;E:\code\Prism3\VisionServer\bin\x64\Debug\ 2018-01-03 10:12:41,105 WARN [context] Astra found no plugins. Is there a Plugins folder? Is the working directory correct? oopsoopsoopsoopsoopsoopsoopsoops2018-01-03 10:12:43,158 INFO [context] client opening streamset: device/sensor0 2018-01-03 10:12:43,164 INFO [astra.streamset] connecting to (1,0) on device/sensor0 Exception thrown: 'Astra.AstraException' in AstraDotNet.dll An unhandled exception of type 'Astra.AstraException' occurred in AstraDotNet.dll astra_imagestream_request_modes: Invalid Operation

andrew · January 3, 2018, 4:57pm

In my output directory (at the root level) I have the 64-bit astra.dll, astra_core.dll, astra_core_api.dll, and AstraDotNet.dll, along with my main executable. My build target is x86_64, so it’s not an architecture mismatch error.

It explains a lot. And you have hint in log:

2018-01-03 10:12:41,105 WARN [context] Astra found no plugins. Is there a Plugins folder? Is the working directory correct?

The thing is that Astra SDK requires some plugins to operate with sensor. Please, copy all required dlls along with astra.dll, astra_core.dll, astra_core_api.dll, and AstraDotNet.dll.

Minimal set of required libraries is the following:

astra.dll
astra_core.dll
astra_core_api.dll
AstraDotNet.dll
OpenNI.ini
OpenNI2.dll
OpenNI2
- Drivers
  - orbbec.dll
  - orbbec.ini
Plugins
- openni_sensor.dll

chrisib · January 3, 2018, 5:01pm

I tried adding the plugins folder and it resulted in the same error. I then added the whole Astra directory from your example, along with the AssemblyResolver class you made, and I’m getting the same problem.

My application also uses OpenNI2 loaded through NuGet, so perhaps there’s a version incompatibility with the OpenNI dlls?

andrew · January 3, 2018, 5:10pm

I tried adding the plugins folder and it resulted in the same error.

You also have to add OpenNI2 directory. Also, please check log in Output window for new details about error.

I then added the whole Astra directory from your example, along with the AssemblyResolver class you made, and I’m getting the same problem.

Most likely, you didn’t remove AstraDotNet.dll assembly from directory with your exe-file. As a result, AssemblyResolver don’t work, because AstraDotNet.dll can be loaded from directory with exe-file by default assembly loading mechanism.

My application also uses OpenNI2 loaded through NuGet, so perhaps there’s a version incompatibility with the OpenNI dlls?

I don’t know for sure. You can compare versions of dll-s from Astra SDK bin directory and from NuGet package.

chrisib · January 3, 2018, 5:43pm

I did definitely remove the root-level libraries when I added the Astra directory.

Part of my problem is that I’m trying to add a new camera class to an existing (massive) application, so the dependencies are giving me a headache. I’ve made a new, really streamlined program that, in theory, should just grab depth frames from the camera and turn them into OpenCV Mat objects.

It compiles just fine, but on startup I’m getting “System.DllNotFoundException: 'Unable to load DLL ‘astra_core’: The specified module could not be found,” even though I have the name resolver initialized, and the DLL is definitely located in the Astra directory in the output folder.

The code is here: Dropbox - File Deleted
I didn’t include the NuGet packages, so you’ll need to restore them through Visual Studio before it will compile.

Any insight you can provide would be greatly appreciated. My big application is currently using OpenNI via the NuGet wrapper to control the Astra, but I’m hoping to replace that with the new .NET wrapper to cut down on the third-party dependencies, but the fact that I can’t even get a simple demo application working is making me feel like a bit of an idiot right now…

andrew · January 3, 2018, 7:10pm

This application don’t work because of bug in my AstraDotNetAssemblyResolver: line 55

var path = Path.Combine(appDomain.BaseDirectory, DirName, Environment.Is64BitProcess ? SubDir64bit : SubDir64bit);

should be changed to

var path = Path.Combine(appDomain.BaseDirectory, DirName, Environment.Is64BitProcess ? SubDir64bit : SubDir32bit);

The thing is that your application actually runs in 32-bit mode due to ticked Prefer 32-bit checkbox in Build settings of AstraSdkTest project. And as you can see there was a miss-print in my code that affected applications in 32-bit mode.

ADDITION: Also, please set Copy Local = False for reference to AstraDotNet.

ZIP-archive with fixed application:
https://drive.google.com/open?id=1F0p0vq4TmeUdN0MVFO_bfd5J6csjj0m

chrisib · January 3, 2018, 10:08pm

Thank-you! I had completely forgotten about the “Copy Local” option in Visual Studio’s references. Not the first time that’s tripped me up, and probably not the last unfortunately. And I definitely hadn’t noticed the 32/64 bit copy-and-paste mistake. Now that my OpenCV breakpoint seems to be working I can go back and see if I can get the full application working.

Have you had a chance to play around with the body-tracking part of the API yet? It’s further down on my to-do list. But with the recent announcement that the PC adapter kit for the Kinect is not available in a lot of places anymore, the Astra is quickly looking like the go-to sensor with skeletal tracking support.

vicviper · January 9, 2018, 12:18pm

Two questions:

Q1: I’ve overlooked the bibigone/AstraDotNet demo, in the external / Plugins directory, the OrbbecBodyTracking DLL is missing… that means the C# wrapper does not support body tracking? or if I copy OrbbecBodyTracking.dll to the plugins directory, I would “automatically” be able to query for body streams?

Q2: What about packaging everything in a Nuget package?

andrew · January 9, 2018, 3:26pm

Q1: I’ve overlooked the bibigone/AstraDotNet demo, in the external / Plugins directory, the OrbbecBodyTracking DLL is missing… that means the C# wrapper does not support body tracking? or if I copy OrbbecBodyTracking.dll to the plugins directory, I would “automatically” be able to query for body streams?

This question is answered in readme (AstraDotNetDemo/readme.md at master · bibigone/AstraDotNetDemo · GitHub):

Notes:

Directory externals\AstraSDK.bin does not contain plugin for body tracking (to save space). This why if you’re going to use BodyStream, then you have to add OrbbecBodyTracking.dll dll-s to Plugins subdirs (both, x86 and amd64 versions) and add them to project AstraTestWpf as links with the following properties:

Build Action = Content
Copy to Output Directory = Copy if newer

Q2: What about packaging everything in a Nuget package?

I suppose, this question should be redirected to authors of SDK and C# wrapper.

shmlex · January 12, 2018, 4:06pm

@andrew
Very grateful for your sample code and support.

andrew · January 15, 2018, 6:06pm

Because of many questions about body tracking, I’d like to add the following:

Body tracking is included in v2.0.7. And it is supported by C# wrapper. But in my sample I excluded body tracking plugin from project because this plugin’s DLLs are very large (more than 45 MB for each architecture). I described this fact in readme:

Notes:

Directory externals\AstraSDK.bin does not contain plugin for body tracking (to save space). This why if you’re going to use BodyStream, then you have to add OrbbecBodyTracking.dll dll-s to Plugins subdirs (both, x86 and amd64 versions) and add them to project AstraTestWpf as links with the following properties:

Build Action = Content
Copy to Output Directory = Copy if newer

How to use body tracking API:

get BodyStream from streamReader just like you get DepthStream and ColorStream:
// here, streamReader is object of StreamReader
var bodyStream = streamReader.GetStream<BodyStream>();
call bodyStream.Start() among with depthStream.Start() to start body tracking
when you processing received frame, you can ask BodyFrame to access current body data (again, just like your access DepthFrame):
// here, frame is object of ReaderFrame received from Astra SDK
var bodyFrame = frame.GetFrame<BodyFrame>();
then you can access the following information:

bodyFrame.FloorInfo - information about floor;
bodyFrame.BodyMask - which pixels on depth map belong to which body (map of indices)
var bodies = new Body[6]; bodyFrame.CopyBodyData(ref bodies); - information about bodies (Astra SDK can track up to 6 bodies simultaneously)

for each body you have several properties like:

Status - NotTracking/Lost/TrackingStarted/Tracking
Joints - array of body joints

Each joint, in turn, has its own properties:
- Type - joint type (Head, ShoulderSpine, etc.)

Status - NotTracked/LowConfidence/Tracked
DepthPosition - position of joint on depth map (use this property to draw skeleton over depth image)
WorldPosition - 3D position of joint (use it for 3D visualization and analysis)
Orientation - matrix of bone rotation in 3D

shmlex · January 15, 2018, 6:28pm

@andrew after the definition and start of bodyStream the frame could not open.

Look at BackgroundProcessingLoop(),
streamReader.TryOpenFrame(10, out var frame) always returns false.

I’ve tried to rise the response time, change streaming modes, eliminate colorStream … but nothings helps

Any advice?

andrew · January 16, 2018, 8:17am

Hello @shmlex,

Did you add OrbbecBodyTracking.dll to Plugins subfolder? Do you have this DLL in appropriate subdirectory of output directory? If yes, then please run your application under debugger (F5 key in Visual Studio) and check content of Output Window in Visual Studio. It should contain log messages from Astra SDK. They can help in understanding of your issue.

jantian · January 17, 2018, 4:10am

Hi Andrew,

I have the same problem. I can’t find OrbbecBodyTracking.dll anywhere. It’s not in your sample project. And it’s not in Astra V2.0.7 SDK either. Do you know where I can download it from?

Thanks again!

andrew · January 17, 2018, 8:59am

Hello,

I’ve just updated sample to include body tracking: