KINECT for Windows v2 SDK Deep dive

Update: I just posted the 2nd part of this series http://www.zubairahmed.net/?p=1682

Few weeks ago I received my Kinect for Windows version 2 and private SDK so I finally got to try it out, the new Kinect ships with many improvements from v1 such as an Full HD Camera, thumb and hand open/close detection, better microphone, improved infrared and several applications can use the sensor at the same time.

KinectOne

Deep dive

In this blog post I will show how to read Body source and draw Bones, Hands and Joints over the Color source received from the Kinect Sensor.

KinectOnev2Experiment

“This is preliminary software and/or hardware and APIs are preliminary and subject to change.”

In the constructor of our WPF app the code reads two important sources, Body frame, Color frame and Width and Height from the Depth Sensor. It then opens both readers to start receiving frames.

   1:  FrameDescription frameDescription = this.kinectSensor.ColorFrameSource.FrameDescription;
   2:  FrameDescription bodyFrameDescription = this.kinectSensor.DepthFrameSource.FrameDescription;
   3:  this.displayWidth = bodyFrameDescription.Width;
   4:  this.displayHeight = bodyFrameDescription.Width;
   5:  this.bodies = new Body[this.kinectSensor.BodyFrameSource.BodyCount];
   6:  this.colorFrameReader = this.kinectSensor.ColorFrameSource.OpenReader();
   7:  this.reader = this.kinectSensor.BodyFrameSource.OpenReader();

The MainWindow.xaml contains a Grid with two Image elements for Color and Body data.

   1:   <Grid Margin="10 0 10 0">
   2:          <Grid.RowDefinitions>
   3:              <RowDefinition Height="Auto" />
   4:          </Grid.RowDefinitions>
   5:          <Image Source="{Binding ImageSource}" Grid.Row="0" Width="1200" Height="700" Stretch="UniformToFill" />
   6:          <Image Source="{Binding BodyImageSource}" Grid.Row="0" Stretch="UniformToFill" Width="1200" Height="700"/>
   7:  </Grid>

We subscribe to FrameArrived events of both readers in the Loaded event of our app

   1:  private void MainWindow_Loaded(object sender, RoutedEventArgs e)
   2:          {
   3:              if (this.colorFrameReader != null)
   4:              {
   5:                  this.colorFrameReader.FrameArrived += this.ColorFrameReaderFrameArrived;
   6:              }
   7:  
   8:              if (this.bodyFrameReader != null)
   9:              {
  10:                  this.bodyFrameReader.FrameArrived += this.BodyFrameReaderFrameArrived;
  11:              }
  12:          }

The color frame arrived event handler acquires a frame and validates it before converting it to Byte Array and writes to WriteableBitmap which is used by Image element in our Xaml to display color stream.

   1:  private void ColorFrameReaderFrameArrived(object sender, ColorFrameArrivedEventArgs e)
   2:          {
   3:              ColorFrameReference frameReference = e.FrameReference;
   4:  
   5:              try
   6:              {
   7:                  ColorFrame frame = frameReference.AcquireFrame();
   8:  
   9:                  if (frame != null)
  10:                  {
  11:                      // ColorFrame is IDisposable
  12:                      using (frame)
  13:                      {
  14:                          FrameDescription frameDescription = frame.FrameDescription;
  15:  
  16:                          // verify data and write the new color frame data to the display bitmap
  17:                          if ((frameDescription.Width == this.bitmap.PixelWidth) && (frameDescription.Height == this.bitmap.PixelHeight))
  18:                          {
  19:                              if (frame.RawColorImageFormat == ColorImageFormat.Bgra)
  20:                              {
  21:                                  frame.CopyRawFrameDataToArray(this.pixels);
  22:                              }
  23:                              else
  24:                              {
  25:                                  frame.CopyConvertedFrameDataToArray(this.pixels, ColorImageFormat.Bgra);
  26:                              }
  27:  
  28:                              this.bitmap.WritePixels(
  29:                                  new Int32Rect(0, 0, frameDescription.Width, frameDescription.Height),
  30:                                  this.pixels,
  31:                                  frameDescription.Width * this.bytesPerPixel,
  32:                                  0);
  33:                          }
  34:                      }
  35:                  }
  36:              }
  37:              catch (Exception)
  38:              {
  39:                  // ignore if the frame is no longer available
  40:              }
  41:          }

The reader frame arrived event handler is the most interesting one, the code uses a DrawingContext to draw a rectangle, our Body frame data will be written within this rectangle. We then get the Body data from the Kinect sensor. Because the Kinect can detect up to 6 bodies at the same time, the code loops through each body object to check if it can read body joints information from the sensor before it can do something useful with it.

If the Kinect is able to track a body it loops through each Joint and uses a CoordinateMapper to give us X and Y coordinates for each joint which it then uses to draw Body and Hand joints. Note that I cheat a little bit on line 36 to fix the vertical position of my drawing by subtracting 80px from the Height.

   1:  private void BodyFrameReaderFrameArrived(object sender, BodyFrameArrivedEventArgs e)
   2:          {
   3:              BodyFrameReference frameReference = e.FrameReference;
   4:  
   5:              try
   6:              {
   7:                  BodyFrame frame = frameReference.AcquireFrame();
   8:  
   9:                  if (frame != null)
  10:                  {
  11:                      // BodyFrame is IDisposable
  12:                      using (frame)
  13:                      {
  14:                          using (DrawingContext dc = this.drawingGroup.Open())
  15:                          {
  16:                              // Draw a transparent background to set the render size
  17:  
  18:                              dc.DrawRectangle(Brushes.Transparent, null, new Rect(0.0, 0.0, this.displayWidth, this.displayHeight));
  19:  
  20:                              // The first time GetAndRefreshBodyData is called, Kinect will allocate each Body in the array.
  21:                              // As long as those body objects are not disposed and not set to null in the array,
  22:                              // those body objects will be re-used.
  23:                              frame.GetAndRefreshBodyData(this.bodies);
  24:  
  25:                              foreach (Body body in this.bodies)
  26:                              {
  27:                                  if (body.IsTracked)
  28:                                  {
  29:                                      IReadOnlyDictionary<JointType, Joint> joints = body.Joints;
  30:  
  31:                                      // convert the joint points to depth (display) space
  32:                                      Dictionary<JointType, Point> jointPoints = new Dictionary<JointType, Point>();
  33:                                      foreach (JointType jointType in joints.Keys)
  34:                                      {
  35:                                          DepthSpacePoint depthSpacePoint = this.coordinateMapper.MapCameraPointToDepthSpace(joints[jointType].Position);
  36:                                          jointPoints[jointType] = new Point(depthSpacePoint.X, depthSpacePoint.Y - 80);
  37:                                      }
  38:  
  39:                                      this.DrawBody(joints, jointPoints, dc);
  40:  
  41:                                      this.DrawHand(body.HandLeftState, jointPoints[JointType.HandLeft], dc);
  42:                                      this.DrawHand(body.HandRightState, jointPoints[JointType.HandRight], dc);
  43:                                  }
  44:                              }
  45:  
  46:                              // prevent drawing outside of our render area
  47:                              this.drawingGroup.ClipGeometry = new RectangleGeometry(new Rect(0.0, 0.0, this.displayWidth, this.displayHeight));
  48:                          }
  49:                      }
  50:                  }
  51:              }
  52:              catch (Exception)
  53:              {
  54:                  // ignore if the frame is no longer available
  55:              }
  56:          }

Notice that for both hands Kinect sensor SDK gives us a state that the code uses to draw a Red/Green ellipses

   1:  private void DrawHand(HandState handState, Point handPosition, DrawingContext drawingContext)
   2:          {
   3:              switch (handState)
   4:              {
   5:                  case HandState.Closed:
   6:                      drawingContext.DrawEllipse(this.handClosedBrush, null, handPosition, HandSize, HandSize);
   7:                      break;
   8:  
   9:                  case HandState.Open:
  10:                      drawingContext.DrawEllipse(this.handOpenBrush, null, handPosition, HandSize, HandSize);
  11:                      break;
  12:  
  13:                  case HandState.Lasso:
  14:                      drawingContext.DrawEllipse(this.handLassoBrush, null, handPosition, HandSize, HandSize);
  15:                      break;
  16:              }
  17:          }

That’s all I had to do to draw body joints over a camera stream from the Kinect sensor.

Also see:
FacebookTwitterLinkedInEmailGoogle+
  • Bruno Capuano

    Great post !!!

  • guest

    hello i am student and want to start learning kinect for my final exam.
    need your advice.

    - because kinect for windows v2 is not release yet. it is ok to start learning with old kinect (before kinect v2) with kinect SDK version 1.8 ?
    - or i must wait for newest kinect ?

    ( i just afraid there will be huge change with the basic (