KINECT for Windows v2 SDK Deep dive

Update: I just posted the 2nd part of this series

Few weeks ago I received my Kinect for Windows version 2 and private SDK so I finally got to try it out, the new Kinect ships with many improvements from v1 such as an Full HD Camera, thumb and hand open/close detection, better microphone, improved infrared and several applications can use the sensor at the same time.


Deep dive

In this blog post I will show how to read Body source and draw Bones, Hands and Joints over the Color source received from the Kinect Sensor.


“This is preliminary software and/or hardware and APIs are preliminary and subject to change.”

In the constructor of our WPF app the code reads two important sources, Body frame, Color frame and Width and Height from the Depth Sensor. It then opens both readers to start receiving frames.

   1:  FrameDescription frameDescription = this.kinectSensor.ColorFrameSource.FrameDescription;
   2:  FrameDescription bodyFrameDescription = this.kinectSensor.DepthFrameSource.FrameDescription;
   3:  this.displayWidth = bodyFrameDescription.Width;
   4:  this.displayHeight = bodyFrameDescription.Width;
   5:  this.bodies = new Body[this.kinectSensor.BodyFrameSource.BodyCount];
   6:  this.colorFrameReader = this.kinectSensor.ColorFrameSource.OpenReader();
   7:  this.reader = this.kinectSensor.BodyFrameSource.OpenReader();

The MainWindow.xaml contains a Grid with two Image elements for Color and Body data.

   1:   <Grid Margin="10 0 10 0">
   2:          <Grid.RowDefinitions>
   3:              <RowDefinition Height="Auto" />
   4:          </Grid.RowDefinitions>
   5:          <Image Source="{Binding ImageSource}" Grid.Row="0" Width="1200" Height="700" Stretch="UniformToFill" />
   6:          <Image Source="{Binding BodyImageSource}" Grid.Row="0" Stretch="UniformToFill" Width="1200" Height="700"/>
   7:  </Grid>

We subscribe to FrameArrived events of both readers in the Loaded event of our app

   1:  private void MainWindow_Loaded(object sender, RoutedEventArgs e)
   2:          {
   3:              if (this.colorFrameReader != null)
   4:              {
   5:                  this.colorFrameReader.FrameArrived += this.ColorFrameReaderFrameArrived;
   6:              }
   8:              if (this.bodyFrameReader != null)
   9:              {
  10:                  this.bodyFrameReader.FrameArrived += this.BodyFrameReaderFrameArrived;
  11:              }
  12:          }

The color frame arrived event handler acquires a frame and validates it before converting it to Byte Array and writes to WriteableBitmap which is used by Image element in our Xaml to display color stream.

   1:  private void ColorFrameReaderFrameArrived(object sender, ColorFrameArrivedEventArgs e)
   2:          {
   3:              ColorFrameReference frameReference = e.FrameReference;
   5:              try
   6:              {
   7:                  ColorFrame frame = frameReference.AcquireFrame();
   9:                  if (frame != null)
  10:                  {
  11:                      // ColorFrame is IDisposable
  12:                      using (frame)
  13:                      {
  14:                          FrameDescription frameDescription = frame.FrameDescription;
  16:                          // verify data and write the new color frame data to the display bitmap
  17:                          if ((frameDescription.Width == this.bitmap.PixelWidth) && (frameDescription.Height == this.bitmap.PixelHeight))
  18:                          {
  19:                              if (frame.RawColorImageFormat == ColorImageFormat.Bgra)
  20:                              {
  21:                                  frame.CopyRawFrameDataToArray(this.pixels);
  22:                              }
  23:                              else
  24:                              {
  25:                                  frame.CopyConvertedFrameDataToArray(this.pixels, ColorImageFormat.Bgra);
  26:                              }
  28:                              this.bitmap.WritePixels(
  29:                                  new Int32Rect(0, 0, frameDescription.Width, frameDescription.Height),
  30:                                  this.pixels,
  31:                                  frameDescription.Width * this.bytesPerPixel,
  32:                                  0);
  33:                          }
  34:                      }
  35:                  }
  36:              }
  37:              catch (Exception)
  38:              {
  39:                  // ignore if the frame is no longer available
  40:              }
  41:          }

The reader frame arrived event handler is the most interesting one, the code uses a DrawingContext to draw a rectangle, our Body frame data will be written within this rectangle. We then get the Body data from the Kinect sensor. Because the Kinect can detect up to 6 bodies at the same time, the code loops through each body object to check if it can read body joints information from the sensor before it can do something useful with it.

If the Kinect is able to track a body it loops through each Joint and uses a CoordinateMapper to give us X and Y coordinates for each joint which it then uses to draw Body and Hand joints. Note that I cheat a little bit on line 36 to fix the vertical position of my drawing by subtracting 80px from the Height.

   1:  private void BodyFrameReaderFrameArrived(object sender, BodyFrameArrivedEventArgs e)
   2:          {
   3:              BodyFrameReference frameReference = e.FrameReference;
   5:              try
   6:              {
   7:                  BodyFrame frame = frameReference.AcquireFrame();
   9:                  if (frame != null)
  10:                  {
  11:                      // BodyFrame is IDisposable
  12:                      using (frame)
  13:                      {
  14:                          using (DrawingContext dc = this.drawingGroup.Open())
  15:                          {
  16:                              // Draw a transparent background to set the render size
  18:                              dc.DrawRectangle(Brushes.Transparent, null, new Rect(0.0, 0.0, this.displayWidth, this.displayHeight));
  20:                              // The first time GetAndRefreshBodyData is called, Kinect will allocate each Body in the array.
  21:                              // As long as those body objects are not disposed and not set to null in the array,
  22:                              // those body objects will be re-used.
  23:                              frame.GetAndRefreshBodyData(this.bodies);
  25:                              foreach (Body body in this.bodies)
  26:                              {
  27:                                  if (body.IsTracked)
  28:                                  {
  29:                                      IReadOnlyDictionary<JointType, Joint> joints = body.Joints;
  31:                                      // convert the joint points to depth (display) space
  32:                                      Dictionary<JointType, Point> jointPoints = new Dictionary<JointType, Point>();
  33:                                      foreach (JointType jointType in joints.Keys)
  34:                                      {
  35:                                          DepthSpacePoint depthSpacePoint = this.coordinateMapper.MapCameraPointToDepthSpace(joints[jointType].Position);
  36:                                          jointPoints[jointType] = new Point(depthSpacePoint.X, depthSpacePoint.Y - 80);
  37:                                      }
  39:                                      this.DrawBody(joints, jointPoints, dc);
  41:                                      this.DrawHand(body.HandLeftState, jointPoints[JointType.HandLeft], dc);
  42:                                      this.DrawHand(body.HandRightState, jointPoints[JointType.HandRight], dc);
  43:                                  }
  44:                              }
  46:                              // prevent drawing outside of our render area
  47:                              this.drawingGroup.ClipGeometry = new RectangleGeometry(new Rect(0.0, 0.0, this.displayWidth, this.displayHeight));
  48:                          }
  49:                      }
  50:                  }
  51:              }
  52:              catch (Exception)
  53:              {
  54:                  // ignore if the frame is no longer available
  55:              }
  56:          }

Notice that for both hands Kinect sensor SDK gives us a state that the code uses to draw a Red/Green ellipses

   1:  private void DrawHand(HandState handState, Point handPosition, DrawingContext drawingContext)
   2:          {
   3:              switch (handState)
   4:              {
   5:                  case HandState.Closed:
   6:                      drawingContext.DrawEllipse(this.handClosedBrush, null, handPosition, HandSize, HandSize);
   7:                      break;
   9:                  case HandState.Open:
  10:                      drawingContext.DrawEllipse(this.handOpenBrush, null, handPosition, HandSize, HandSize);
  11:                      break;
  13:                  case HandState.Lasso:
  14:                      drawingContext.DrawEllipse(this.handLassoBrush, null, handPosition, HandSize, HandSize);
  15:                      break;
  16:              }
  17:          }

That’s all I had to do to draw body joints over a camera stream from the Kinect sensor.

Also see:
  • Bruno Capuano

    Great post !!!

  • guest

    hello i am student and want to start learning kinect for my final exam.
    need your advice.

    - because kinect for windows v2 is not release yet. it is ok to start learning with old kinect (before kinect v2) with kinect SDK version 1.8 ?
    - or i must wait for newest kinect ?

    ( i just afraid there will be huge change with the basic (

  • Icepowder

    In your project, I am curious how you display two images into one grid? Can you show me the code of that part? Thanks!

    • zubairdotnet

      That you will find in the follow-post I wrote

      • Icepowder

        Yes, I have read that post already but I am still interested in how to using two image in one grid and overlay them together. Since in your next post, you merge two writeable bitmaps together and use WriteableBitmapExtensions, it runs very slow in my PC, so I want to compare that method with your original method to figure out which one is faster. Could you send me the source code in this post? My email address is Thanks again ^_^

        • Icepowder

          Please ignore my post above, I knew how to do it already. I use Brushes.Black instead of Brushes.Transparent in drawing a rectangle before drawing my skeleton, which makes my final output display skeleton only because it covers color space frame. And I like this method better since I don’t need to invoke the third-party library and it is faster!

          Also, I don’t need to cheat a little bit (minus 80px) in line 36 when I use MapCameraPointToColorSpace instead of MapCameraPointToDepthSpace as what you did in next post. In this way, this is no offset between the skeleton and the moving individual.

          I like your post very much!!! Thanks for all your posts about Kinect V2!

          Happy new year!

          • help

            Please share how do overlayed! I got the brushes.transparent part but how do I stream color into the bodybasics code from the sdk?

  • harpreet bhullar

    hi, its good but i tried the same code and the drawing context is not working in my code so can you tell me the possible issue……