Kinect for Windows v2 Deep dive–Part 2

In my previous K4W deep dive post I draw Body and joints in WPF and overlay them on top of Color stream. In that post notice that I am using two Image control, one to render Color and other for Body info.

   1:   <Grid Margin="10 0 10 0">
   2:          <Grid.RowDefinitions>
   3:              <RowDefinition Height="Auto" />
   4:          </Grid.RowDefinitions>
   5:          <Image Source="{Binding ImageSource}" Grid.Row="0" Width="1200" Height="700" Stretch="UniformToFill" />
   6:          <Image Source="{Binding BodyImageSource}" Grid.Row="0" Stretch="UniformToFill" Width="1200" Height="700"/>
   7:  </Grid>

Also in that post I talked about a hack to position body tracking drawings ‘properly’ over the color stream.

  jointPoints[jointType] = new Point(depthSpacePoint.X, depthSpacePoint.Y - 80);

In this post however I fix the above two problems, which means I don’t need to use 80px offset hack from the previous post and use only one Image control to render both Color and Body data.

Fixing Body tracking offset – Using MapCameraPointToColorSpace

First thing first, in the previous post I used a method of the CoordinateMapper class in the Kinect for Windows v2 SDK that uses camera point and maps them to depth space

  MapCameraPointToDepthSpace

The problem is that this method does not accurately translates points to the color frame that we receive from the Kinect sensor so I can up with the hack. Fortunately the CooridnateMapper class has another method that works perfectly for this.

  MapCameraPointToColorSpace

So here’s how it looks like when the body frame has arrived

   1:  foreach (JointType jointType in joints.Keys){
   2:     ColorSpacePoint colorSpacePoint = this.coordinateMapper.MapCameraPointToColorSpace(joints[jointType].Position);
   3:     jointPoints[jointType] = new Point(colorSpacePoint.X, colorSpacePoint.Y);
   4:  }

This time I use the ColorSpacePoint and use that to get X and Y coordinates from the camera.

Merge Color and Body frames to use one Image control

Alright so I got the Body info mapped properly to the Color frames, but I have not solved one problem, how to merge these two frames and use single image control to render them.

To do that we need to use RenderTargetBitmap function to be able to render our body frame drawing data on the WriteableBitmap we use for the Color stream, here’s how I declare them

   1:   this.bitmap = new WriteableBitmap(frameDescription.Width, frameDescription.Height, 96.0, 96.0, PixelFormats.Bgr32, null);
   2:   this.drawingGroup = new DrawingGroup();
   3:  _bodySourceRTB = new RenderTargetBitmap(displayWidth, displayHeight, 96.0, 96.0, PixelFormats.Pbgra32);
   4:   rootGrid = new Grid();
   5:  _colorWriteableBitmap = BitmapFactory.New(frameDescription.Width, frameDescription.Height);
   6:  _bodyWriteableBitmap = BitmapFactory.New(frameDescription.Width, frameDescription.Height);

In the above I declare a RenderTargetBitmap for rendering our body joints and lines to a bitmap, a drawing group used to draw those joints and two WriteableBitmaps, for Color and Body streams, take the Color writeablebitmap as a destination and use the Body bitmap as source to merge. I am initializing the WriteableBitmaps with the BitmapFactory.New() method of the library I describe below.

Also declaring a dynamic Grid control which is used to hold an Image control, the image is used below to hold the body joints and lines drawings.

   1:  this.drawingGroup.ClipGeometry = new RectangleGeometry(new Rect(0.0, 0.0, this.displayWidth, this.displayHeight));
   2:  bodyImage = new Image { Source = new DrawingImage(drawingGroup), Width = this.displayWidth, Height = this.displayHeight };
   3:  rootGrid.Children.Clear();
   4:  rootGrid.Children.Add(bodyImage);
   5:  rootGrid.Measure(new Size(bodyImage.Width, bodyImage.Height));
   6:  rootGrid.Arrange(new Rect(0, 0, bodyImage.Width, bodyImage.Height));
   7:  _bodySourceRTB.Clear();
   8:  _bodySourceRTB.Render(rootGrid);
   9:  _bodySourceRTB.CopyPixels(this.bodyBytespixels, displayWidth * this.bytesPerPixel,
  10:      0);
  11:  _bodyWriteableBitmap.FromByteArray(this.bodyBytespixels);

In the code above I am using bodyImage to hold my drawings, I add them to a Grid control, add the grid to the RenderTargetBitmap control. I use RTB’s CopyPixels method to fill a byte[] bodyBytespixels. I use bodyBytespixels as a source to get a WriteableBitmap.

The above is used when the body frame is available to read. In the Color frame arrived event I use the pixels byte array that is available to me to fill the color WriteableBitmap.

   1:   _colorWriteableBitmap.FromByteArray(this.pixels);
   2:  
   3:  var rec = new Rect(0, 0, frameDescription.Width,frameDescription.Height);
   4:  using (_colorWriteableBitmap.GetBitmapContext())
   5:  {
   6:      using (_bodyWriteableBitmap.GetBitmapContext())
   7:      {
   8:          _colorWriteableBitmap.Blit(rec, _bodyWriteableBitmap, rec, WriteableBitmapExtensions.BlendMode.Additive);
   9:      }
  10:  }

Also in the above code I am merging the two writeable bmps using this helpful open source library  WriteableBitmapExtensions which has many helpful extensions but the method I am using is Blit.

In Xaml I have only one image control stretched to the full 1920×1080 pixels.

   1:  <Window x:Class="BodyColorSource.MainWindow"
   2:          xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
   3:          xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
   4:          Title="Color Basics"
   5:          Height="1080" Width="1920"
   6:          Loaded="MainWindow_Loaded"
   7:          Closing="MainWindow_Closing">
   8:      <Window.Resources>
   9:          <Style TargetType="{x:Type Image}">
  10:              <Setter Property="SnapsToDevicePixels" Value="True" />
  11:          </Style>
  12:      </Window.Resources>
  13:      <Grid Margin="0 0 10 0">
  14:          <Grid.RowDefinitions>
  15:              <RowDefinition Height="Auto" />
  16:          </Grid.RowDefinitions>
  17:          <Image Grid.Row="0" Stretch="UniformToFill" Name="Image"  />
  18:      </Grid>
  19:  </Window>

I then use the Color WTB as source of my Image control

   Image.Source = _colorWriteableBitmap;   

Here’s the video

Kinect for Windows v2 deep dive
Also see:
FacebookTwitterLinkedInEmailGoogle+
  • BucksterMcgee

    I’ve been curious about how fast Kinect 2 was and going by that video it has almost no perceptible lag. Very impressive!

    • Ming Guo

      30fps,color ,depth or skeleton ~

  • Mindaugas Balčiūnas

    hi can you share your source code of skeleton and color mapping. Its nice article but it would be nice to seen full code. I also have this new sensor and it would help a lot. if you could send me it pczombis@gmail.com

  • Shashank Bisen

    Hi Zubair,

    Do you have any idea if Kinect v2, can detect objects without using any other software like Open Graph?

    • zubairdotnet

      Though I haven’t tested it but my guess is that it can’t detect objects with out of the box SDK

      • Shashank Bisen

        thnx