Video Paramecia

May 11, 2012 · by Kim Ash · in ,

I now understand why ITP students often incorporate multiple classes into their final projects. The obvious reason is, of course, less work. But the less obvious reason turned out to be more compelling: inspiration. I agonized for a few weeks over potential final projects for Computational Cameras and Nature of Code, but it was not until I decided to combine them that ideas really started to flow.

One of the last things we covered in Nature of Code was genetic algorithms. A genetic algorithm simulates the mating and evolution of organisms towards a certain fitness goal. I wanted to somehow reproduce this process in video format, so that the video would “evolve” over time. I found that my project evolved over time too.

To facilitate the mating process, I broke the video up into units, which I dubbed “MateBoxes.” I used the same mechanism that I used to make the blur boxes in the Voyeur project, which stores each frame of the video as a separate PImage. The images contain pixels, and each pixel is considered a “gene” in its MateBox’s “DNA.”

A traditional genetic algorithm posed a few problems when applied to video. What would I use for a fitness function (i.e. what goal would the boxes evolve toward)? When the MateBoxes create offspring, where will they go? Will I have to select boxes that will die and be replaced? To solve these problems, I decided to create an algorithm inspired by something I remembered from AP Bio in high school. Some unicellular organisms, notably paramecia, practice a form of sexual reproduction called conjugation. Most of the time they reproduce asexually, so offspring are genetic clones of the parents. Conjugation provides an infusion of new genetic material–an opportunity for evolution. I built my algorithm around this concept, and scrapped the fitness function (though I may revisit it in the future).

Main sketch code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

//Kim Ash
//Nature of Code/Computational Cameras Final 2012
//Video Paramecia - algorithm based on paramecium conjugation alters image
 
import processing.video.*;
Capture feed;
 
MateBox mb[][] = new MateBox[16][12];
int boxSide = 40;
 
ArrayList matingPool;
//int generations;
//color mutation = color(random(255), random(255), random(255));
 
void setup()
{
  size(640, 480);
  rectMode(CORNER);
 
  feed = new Capture(this, 640, 480);
  feed.start();
 
  matingPool = new ArrayList();
 
  for (int i=0; i < 16; i++) {
    for (int j=0; j < 12; j++) {
      mb[i][j] = new MateBox (i*boxSide, j*boxSide);
      matingPool.add(mb[i][j]);
    }
  }
}
 
void draw()
{
 
  if (frameCount > 30) {
 
    //mutation = color(random(255), random(255), random(255));
 
    if (feed.available() == true) {
      feed.read();
    }
 
    //image(feed, 0, 0, width, height);
 
    for (int i=0; i < 16; i++) {
      for (int j=0; j < 12; j++) {
        mb[i][j].grabVideo();
        mb[i][j].mutate(0.01);
      }
    }
 
    reproduction();
 
    for (int i=0; i < 16; i++) {
      for (int j=0; j < 12; j++) {
        mb[i][j].display();
      }
    }
  }
}
 
void reproduction()
{
  int m = int(random(matingPool.size()));
 
  int mateShift = int(pow(-1,int(random(1))));  //select mate from 2 nearest neighbors
 
  if (m == 0)
    mateShift = 1;
 
  if (m == matingPool.size()-1)
    mateShift = -1;
 
  int d = m + mateShift;
 
  MateBox mom = matingPool.get(m);
  MateBox dad = matingPool.get(d);
 
  mom.conjugate(dad);
}

MateBox class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104

 
class MateBox
{
  public PImage section;
 
  int boxSide = 40;
  int x, y;
 
  boolean once = false;
 
  MateBox (int x_, int y_)
  {
    x = x_;
    y = y_;
    section = new PImage(boxSide, boxSide);
  }
 
  MateBox (PImage img) //alternate constructor for offspring
  {
    section = img;
  }
 
  void grabVideo()
  {
    if (!once) {
      section = feed.get(x, y, boxSide, boxSide);
      once = true;
    }
  }
 
  void display()
  {
    //stroke(255, 255, 0);
    noStroke();
    noFill();
    tint(255,10);
    //section = get(x, y, boxSide, boxSide);
    image(section, x, y);
    rect(x, y, boxSide, boxSide);
  }
 
  /*MateBox crossover (MateBox partner)  
  {
    PImage newSection = createImage(boxSide, boxSide, RGB);
    section.loadPixels();
    partner.section.loadPixels();
    newSection.loadPixels();
 
    int crossover = int(random(section.pixels.length));
    for (int i=0; i < section.pixels.length; i++)
    {
      if (i > crossover) {
        newSection.pixels[i] = section.pixels[i];
      }
      else {
        newSection.pixels[i] = partner.section.pixels[i];
      }
    }
 
    newSection.updatePixels();
    MateBox offspring = new MateBox(newSection);
    return offspring; //must define location for offspring box
  }*/
 
  void conjugate(MateBox partner)  //paramecium-style conjugation
  {
    PImage store = createImage(boxSide, boxSide, RGB);
    //storage for pixel trade between mb and partner
    section.loadPixels();
    partner.section.loadPixels();
    store.loadPixels();
 
    int crossover = int(random(section.pixels.length));
    for (int i=0; i < section.pixels.length; i++)
    {
      if (i < crossover) {
        store.pixels[i] = section.pixels[i];
        partner.section.pixels[i] = store.pixels[i];
      }
      else {
        store.pixels[i] = partner.section.pixels[i];
        section.pixels[i] = store.pixels[i];
      }
    }
    section.updatePixels();
    partner.section.updatePixels();
  }
 
  void mutate(float m)
  {
    if (random(1) < m) {
      once = false;
      //tint(random(255), random(255), random(255), 10); 
    }
    /*for (int i=0; i < section.pixels.length; i++)
    {
      if (random(1) < m) {
        section.loadPixels();
        section.pixels[i] = mutation;
        section.updatePixels();
      }
    }*/
  }
}
Tags: , , ,