Thumb Tracking (and Possibly Hand Turkeys)

September 24, 2012 · by Kim Ash · in

After our class discussion, I was thinking a lot about the various ways that one could use the finger tracking code to distinguish between the thumb and other fingers. I tried a lot of different ways to do this, and I still think that the most accurate way would be to measure the angles between fingers and assume the largest angle to be the one between the thumb and forefinger. Unfortunately, that would probably mean making some changes to the underlying finger tracking library, since it joins the contours of all the fingers into one hand span. Oh well, that will be a project for another day.

I ultimately decided that the best way to do this was to assume that users would place hands in front of the Kinect oriented right-side up instead of upside down, and that each hand would likely be contained in either half of the screen. All of this hand outlining reminds me of doing drawings of hand turkeys for Thanksgiving in kindergarten, so my final tweak to this project will be adding left and right facing turkey heads to the appropriate thumbs. For the moment, they are identified with a blue circle and a red circle.

Here is the code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110

// import the fingertracker library
// and the SimpleOpenNI library for Kinect access
import fingertracker.*;
import SimpleOpenNI.*;
 
// declare FignerTracker and SimpleOpenNI objects
FingerTracker fingers;
SimpleOpenNI kinect;
// set a default threshold distance:
// 625 corresponds to about 2-3 feet from the Kinect
int threshold = 605;
 
void setup() {
  size(640, 480);
 
//  allFingers = new ArrayList();
 
  // initialize your SimpleOpenNI object
  // and set it up to access the depth image
  kinect = new SimpleOpenNI(this);
  kinect.enableDepth();
  // mirror the depth image so that it is more natural
  kinect.setMirror(true);
 
  // initialize the FingerTracker object
  // with the height and width of the Kinect
  // depth image
  fingers = new FingerTracker(this, 640, 480);
  // the "melt factor" smooths out the contour
  // making the finger tracking more robust
  // especially at short distances
  // farther away you may want a lower number
  fingers.setMeltFactor(100);
}
 
void draw() {
  // get new depth data from the kinect
  kinect.update();
  // get a depth image and display it
  PImage depthImage = kinect.depthImage();
  image(depthImage, 0, 0);
 
  // update the depth threshold beyond which
  // we'll look for fingers
  fingers.setThreshold(threshold);
 
  // access the "depth map" from the Kinect
  // this is an array of ints with the full-resolution
  // depth data (i.e. 500-2047 instead of 0-255)
  // pass that data to our FingerTracker
  int[] depthMap = kinect.depthMap();
  fingers.update(depthMap);
 
  //PVectors for storing thumb positions
  PVector leftThumb = new PVector();
  PVector rightThumb = new PVector();
 
  // iterate over all the contours found
  // and display each of them with a green line
  stroke(0,255,0);
  for (int k = 0; k < fingers.getNumContours(); k++) {
    fingers.drawContour(k);
  }
 
  // iterate over all the fingers found
  // and draw them as a red circle
  noStroke();
  fill(255,0,0);
  for (int i = 0; i < fingers.getNumFingers(); i++) {
    PVector position = fingers.getFinger(i);
 
    if(position.x > width/2)
    {
      if(position.y > rightThumb.y)  {
        rightThumb.x = position.x;
        rightThumb.y = position.y; 
      }
    } 
    else
    {
      if(position.y > leftThumb.y)  {
        leftThumb.x = position.x;
        leftThumb.y = position.y; 
      }
    }      
  }
  noStroke();
  fill(255,0,0);
  ellipse(rightThumb.x - 5, rightThumb.y - 5, 10, 10);
 
  fill(0, 0, 255);
  ellipse(leftThumb.x - 5, leftThumb.y - 5, 10, 10);
 
  // show the threshold on the screen
  fill(255,0,0);
  text(threshold, 10, 20);
}
 
// keyPressed event:
// pressing the '-' key lowers the threshold by 10
// pressing the '+/=' key increases it by 10 
void keyPressed(){
  if(key == '-'){
    threshold -= 10;
  }
 
  if(key == '='){
    threshold += 10;
  }
}
Tags: ,