Depth Sensors and SDKs Comparison

Ah it’s been a while since my last post. During my intern which was ended last moth i used ASUS XTion Pro and now I’ve come back to my campus and is currently porting all of my code to C++ using a Kinect for Xbox 360.

Kinect for Xbox 360 is quite different when you compare it with XTion. i would like to say ASUS XTion Pro is far better than Kinect Xbox 360. It only needs 40-50 cm for nearest distance for depth sensor. Kinect Xbox 360 needs 80-90 cm.  XTion Pro is also lighter and smaller. Yup, Kinect is bigger, heavier  and bulky i think.

Now let’s compare OpenNI and Kinect for Windows SDK  (it’s really long to type the name so let’s make it shorter, Kinect SDK). If you talk about the SDK  features,  Kinect SDK is almost the winner,  it has a lot of features (see release notes here: http://msdn.microsoft.com/en-us/library/jj572479). OpenNI so far is really behind (except if you want to discuss about portability) .

When you’re developing  depth-sensor based software, recording is really useful when debugging. You don’t want to dance every debugging, right? (moving your body, and get tired). In OpenNI the recording file (.ONI) has smaller size even in long duration and also after you recording, you can plug-off your depth sensor and play with your recorded data. OK, what about recording in Kinect SDK? It sucks! for several minutes, the recording file will be gigabytes is size, and the one i really don’t like is  you still need your device is plugged in you PC during playback.

PS: i never use Kinect for Windows so don’t ask me okay :D

Kinect Usermap Filtering

Finally, my post after 3 months long. I’ve been really busy lately :(

Chiang Kai-shek Memorial Hall Point Cloud

This is point cloud of Chiang Kai-shek Memorial Hall (中正紀念堂) Point Cloud. It has 559219 points and loaded using PCL.

How to Compile OpenCV

In this article i will explain to how compile C++ library (OpenCV as an example), how to include the headers, linking the .lib files, and put the .dlls

You can read these two links to know the differences between Static Library (http://en.wikipedia.org/wiki/Static_library) and Dynamic Library (http://en.wikipedia.org/wiki/Dynamic-link_library)

This article assumes you have some knowledges in C/C++.  For short, compiling OpenCV and another C++ libraries is always similar and pretty easy after you know how. This tutorial will be in Visual Studio environment (2008 or 2010) .

1. The first step is to install CMake 2.8.x on your computer (http://www.cmake.org/). What is this CMake program? It simply prepares the C++ codes to be ready compiled from specified compiler (you know for windows, there are several compilers such as Cygwin, MinGW, Visual C++, etc.). This CMake is really useful to distribute C++ source code so it can be independent from other people’s Compiler and IDE.

2. Then download OpenCV library from sourceforge, the latest version is OpenCV 2.4.0 (http://sourceforge.net/projects/opencvlibrary/files/opencv-win/2.4.0/)

3. Since CUDA is getting popular and OpenCV also implements some of their libraries using CUDA, you can try to install CUDA.  But this is optional since not all computers have NVIDIA GPU. If you need to install CUDA, you can find it here (http://developer.nvidia.com/cuda-downloads)

PCL Greedy Projection Triangulation

I was experimenting surface reconstruction on point cloud. I used Greedy Triangulation Projection implemented in PCL. Here’s the input point cloud (i used dataset from here http://svn.pointclouds.org/data/tosca_hires/) :

The result was saved in .PLY file format and displayed using MeshLab (http://meshlab.sourceforge.net/)

The result is not really good since the points are not dense enough. On the sparse areas it can easily produces holes. But I should admit this algorithm is really fast. I think I’ll try another algorithm such as poisson surface reconstruction to see whether it has better result or not :)

Compiling Gtk+ App on Windows

Currently i need to obtain depth map using stereo images and  Block Matching Algorithm in OpenCV painfully requires many parameter adjustments.  Then i found out Stereo BM Tuner by Martin Perris (http://blog.martinperis.com/2011/08/opencv-stereo-matching.html) is quite handy and useful. The only problem he developed this tool using Linux and Gtk+ (Ok, i am Visual Studio user XD ) so i need to port it to Windows.

Actually the porting was not that difficult, and after 2 hours of googling i found the solution:

First you need to download Gtk+ fow Windows 32bit and you should download the all-in-one bundle version (http://www.gtk.org/download/win32.php, dependencies included), zipped version, contains all dll, lib, and include files.

Open your Project through Visual Studio and right click–> Project Properties –> C/C++ section –> Additional Include Directories:

{your gtk+ bundle directory}\include;
{your gtk+ bundle directory}\include\gtk-2.0;
{your gtk+ bundle directory}\include\atk-1.0;
{your gtk+ bundle directory}\include\freetype2;
{your gtk+ bundle directory}\include\gail-1.0;
{your gtk+ bundle directory}\include\gdk-pixbuf-2.0;
{your gtk+ bundle directory}\include\gio-win32-2.0;
{your gtk+ bundle directory}\include\glib-2.0;
{your gtk+ bundle directory}\include\pango-1.0;
{your gtk+ bundle directory}\lib\glib-2.0\include;
{your gtk+ bundle directory}\include\cairo;
{your gtk+ bundle directory}\lib\gtk-2.0\include;

Project Properties –> Linker –> General –> Additional Library Directories:

{your gtk+ bundle directory}\lib

Project Properties –> Linker –> Input –> Additional Dependencies:

atk-1.0.lib
cairo.lib
expat.lib
fontconfig.lib
freetype.lib
gailutil.lib
gdk_pixbuf-2.0.lib
gdk-win32-2.0.lib
gio-2.0.lib
glib-2.0.lib
gmodule-2.0.lib
gobject-2.0.lib
gthread-2.0.lib
gtk-win32-2.0.lib
intl.lib
libpng.lib
pango-1.0.lib
pangocairo-1.0.lib
pangoft2-1.0.lib
pangowin32-1.0.lib
zdll.lib

And don’t forget Project Properties –> Linker –>System –>  SubSystem should be set as Console (/SUBSYSTEM:CONSOLE)

Then we need to modify every callback function, for example:

extern "C" __declspec(dllexport)
G_MODULE_EXPORT void on_adjustment1_value_changed( GtkAdjustment *adjustment, ChData *data )
{
    ...
}

Last step you need to copy the .glade file and all of files in {your gtk+ bundle directory}\bin (except the .exe files) on the binary folder (bin\Debug or bin\Release)

Stereo Camera Calibration

I am planning to do 3D object reconstruction using stereo camera so first i need to do stereo calibration on my 3D camera.

The stereo camera i am using is Fujifilm w3 3D Camera. This camera produces MPO file image which is JPG in stereo format. For pre processing of course it is need to be converted into pair of images which can be read by OpenCV by using Stereo Photomarker (SPM) (http://stereo.jpn.org/eng/stphmkr/).

One thing you should notice when using this SPM when converting .MPO into pair of JPG images is to reset the automatic alignment (press “Home” to reset the alignment).  SPM automatically aligns the .MPO image so it can be viewed better when you use 3D glasses, but this alignment will crop the images. I don’t want this cropped images since resulted images won’t have the same size and it’s disaster.

The stereo calibration code can be read on here: http://blog.martinperis.com/2011/01/opencv-stereo-camera-calibration.html.

if it is success you can get xml calibration file similar to this:

<?xml version="1.0"?>
<opencv_storage>
<Q type_id="opencv-matrix">
  <rows>4</rows>
  <cols>4</cols>
  <dt>d</dt>
  <data>
    1. 0. 0. -2.8327271270751953e+002 0. 1. 0. -1.5946473121643066e+002
    0. 0. 0. 1.0546290540664800e+003 0. 0. -2.3597727835463600e-001
    -1.7014427703509563e+000</data></Q>
</opencv_storage>

At first i use chess pattern with 9×6 inner corners, 11 pairs of image, 1024×768 in size. And the result is really bad distortion, as you can see in corners of the image below

Then i figure out i need to use larger chess pattern and more pair of image. Then i use 15×8 inner corners, 26 pairs of image, and 1024×768 resolution. The rectified result looks better

Point Cloud SIFT Keypoint

This week i was trying to get 3D keypoints from Point Cloud. I use PCL and the algorithm is based on 2004 David G. Lowe paper. In PCL, the algorithm is extended 2D + 1 in the spatial domain.
I was able to obtain the keypoint extraction through Difference of Gaussian. Currently i need to compute the keypoint descriptor and match it up to another point cloud. Unfortunately, since Point Cloud is 3D data, we can’t implement SIFT Descriptor instead it is replaced with PFH Descriptor.

Note: the red dots are the keypoint

Kinect Joint Orientation Demo

Right now i’m experimenting with Kinect to animate a 3D model or model skinning by using Kinect. To calculate the joint orientation, first i need to obtain the three orthonormal axes (X, Y, and Z) and create Quaternion to change every bone orientation. By this way, the noise produces by Kinect can be reduced and character movement can look smoother and independent from position. Another advantage of using joint orientation, character movement isn’t depended to body posture and model scaling.

More info and source code: http://code.google.com/p/ogrekinect/

Google Code Commit Failed

Just experienced something trivial today, i made a new SVN project on Google Code and as usual the first step was “Checkout”, but then i got error on COmmit says “Error MKACTIVITY request for ‘/svn/!svn/act/…”

How to fix it is actually very trivial, i did mistake on Checkout, i use http instead https :)