Building a Color Space

There are several major systems for defining color, each well-suited to different needs. RGB is the default for computer screens; each pixel is basically made up of a red, green, and blue LED. CMYK (cyan, magenta, yellow, black) are the ink colors which produce the best gamut for printers. CIE XYZ accounts for the green-dominance in human sight.

For me and Neko, hue is central. Hue is the relationship between primaries— a pale cherry red and a dark cherry red share a hue. I found this little algorithm for converting RGB is HSV (hue, saturation/white, value/black), and wanted to make a note of it for the future.

// r,g,b values are from 0 to 1
// h = [0,360], s = [0,1], v = [0,1]
//  if s == 0, then h = -1 (undefined)

void RGBtoHSV( float r, float g, float b, float *h, float *s, float *v )
{
 float min, max, delta;

 min = MIN( r, g, b );
 max = MAX( r, g, b );
 *v = max;    // v

 delta = max - min;

 if( max != 0 )
  *s = delta / max;  // s
 else {
  // r = g = b = 0  // s = 0, v is undefined
  *s = 0;
  *h = -1;
  return;
 }

 if( r == max )
  *h = ( g - b ) / delta;  // between yellow & magenta
 else if( g == max )
  *h = 2 + ( b - r ) / delta; // between cyan & yellow
 else
  *h = 4 + ( r - g ) / delta; // between magenta & cyan

 *h *= 60;    // degrees
 if( *h < 0 )
  *h += 360;

}

Once I have completed Neko's new set of pigment swatches, I will assign each pigment to a specific hue. When Neko has a target digital hue, he can refer to that database to find the closest pigments. Our color system is also growing in text references to each pigment, so word associations can be made as well.

Edit 4/23: Found this comprehensive color system website today.

Neko 2.2

Hardware

For my needs, the accelerometer and gyroscope gave both too much and too little information. I decided to scrap the locational sensors and use mild trigonometry. Using the ultrasonic sensor as a guide for keeping the brush on the canvas did work quite well, so I might reinstall that. I'm testing out a color sensor, cameras and lights now that the arm is working smoothly.

Software

I'm expanding my text-based approach after reading the wonderful novel Galatea 2.2, by Richard Powers. Instead of just looking for color words in texts about art, I want Neko to do freer association on a larger corpus. I'm still going to be selective about the texts I incorporate, but I liked the relationship Powers' narrator had with his machine.

Firmware

This is the firmware that's running in the video. This was just a piece of test code, but I really like the motion it makes. I don't want it to get too rigid and boring, treating the painting as a 2D grid. I want to find the kind of strokes Neko is good at. I'm starting to think of it like a dance: programmed sequences of graceful motion, recombined.

Click to expand

Neko's Brain Trust

On Monday, Neko and I were successfully funded on Kickstarter! It's tremendously exciting and I can't thank my supporters enough. I just put in a big motor order and look forward to a minor rebuild. As for software, there are two things I'm now teaching Neko: how to select a digital color given a set of words, and how to select pigments to represent that digital color. My new vocabulary for the day is distributional semantics. This is a practice based on the idea that words with similar distributions have similar definitions; "You shall know a word by the company it keeps." I'm building a text crawler that will look for my six core words— Red, Orange, Yellow, Green, Blue, Purple— and find words that are highly collocated with each. I'm picking out old public domain books to walk through, and would love suggestions on books that have high frequency of these words. While I'm waiting for the new motors to arrive, I'll put all the most-associated words into my database. Some of the Kickstarter funds will be allocated to a Mechanical Turk bid, though I'm not sure quite what to ask yet.

The second task is matching the pigments to the colors on-screen. For this I've ordered a color sensor which Neko can use to compare what's on the palette to the target color. I've picked a set of mixing pigments— Cadmium Red Medium, Cadmium Yellow Lemon, and Phthalo Blue— to be Neko's go-to adjusters.

Random Walk to Remember

The random walk is my Hello World— the foundation code I always write to get things up and running. If I'm using a new visualizer, I try to make it look like TV static. I'll use random colors or numbers or motions to get a broad sense of what my new system can do.

I'm pleased with Neko's code so far, it's very well-organized. The random walk is my best yet so I'll explain it for the sake of anyone who doesn't know how to read code, but would like a little puzzle to try it out.

This a screenshot of one of the functions in my program: randomWalk(). I tell the function to repeat infinitely with two parameters: the maximum distance the brush should jump, and the time to pause after each iteration. My current parameters are 50 steps and 500 milliseconds, so when the computer reads the instruction randomWalk(50, 500) it will run this piece of code with the variables jump and del set to 50 steps and 500ms, respectively.

This function uses randomness in two ways: it randomly picks one of the six motors, then it randomly picks the number of steps for that motor to move.

Neko also has a motorized shoulder which is not moved in the random walk, but instead responds to the ultrasonic sensor. This keeps the brush on the canvas. The accelerometer is read in a balanceBrush() function, but I don't use it much yet. There's also a readGyro() function, adapted from this tutorial, which prints the gyroscope readings but does nothing more; another noch-nicht. A zip file of Neko's Arduino program can be found here— a link I'll update with each new version.

Reading live data as scrolling lines of serial output can be quite meditative. I just disabled the motors and moved the brush around the canvas by hand, allowing my own neural network to get a sense of how the sensor data changes while drawing different shapes and lines. As a stepping-stone to a Kalman filter, I'll now try writing a complementary fiter to combine accelerometer and gyroscope data and determine how the arm is moving.

Beeping with Robots

The central objective of my work is to allow robots to express themselves using oil paint. But today I've been thinking about incorporating another mode of expression: beeps. My first Roomba was a castaway from my mother who, though an enthusiastic early adopter, has vacuuming standards the robot was unable to meet. Eventually I tried teaching it to paint which went well, until it went poorly. Its stair detecting feature failed (presumably paint-related) and it plummeted off a sawhorse.

Now I have a newer model Roomba and although it's better at vacuuming, the updated voice is a huge disappointment. The red model had an all-beep vocabulary. The white one has the saccharine GPS navigatress saying things like "open Roomba's brush cage" or "please charge Roomba." It's speaking in the third person, in parroted English. The Roomba only needs to vocalize three things: I'm ready to clean, I'm in distress, I've successfully cleaned. Whatever caused the distress is apparent as soon as it's got your attention.

I've been testing out some basic emotive themes on my keyboard today, and came up with some beeps to try out on Neko. http://paintbot.org/beeps/

Terra Verte

Little Neko finally got some paint tonight: Terra Verte. He's just doing a random walk, using the ultrasonic sensor to keep the brush at the canvas, and the accelerometer to stay upright. I'm hoping eventually that using a random walk and a canvas, Neko could learn how to address the canvas as an X-Y plane. Writing something like that isn't easy, but it's less tedious than hard coding all the different angles. I'm hoping I can get away without putting potentiometers at each joint.

Sensing Brush Location

Getting Neko back and forth across the country proved extremely difficult, so I've been rebuilding him at half-size. In the process I decided to add a gyroscope (measures tilt about 3 axes) to the accelerometer  (measures static and dynamic acceleration, due to gravity and motion respectively) and ultrasonic sensor (measures distance to the nearest object by emitting an inaudible ping, and timing how long it takes to bounce back). In Neko's first iteration, I had the accelerometer on the shoulder joint, telling those motors to keep the arm generally upright. I had the ping sensor on the wrist, telling the shoulder motors to keep the brush at the right distance from the canvas. The wrist motors moved the brush back and forth as the elbow motor moved from the top of the canvas to the bottom. The shoulder had continuous rotation servos, all others were standard servos.

For half-pint Neko, all the servos are continuous rotation. I was aiming for uniformity, and the ability to scale up to stepper motors if I rebuild the robot at full-size. The accelerometer will tell me pitch and roll, but I need pitch and yaw. I think a gyroscope can fill the gaps. I got some trusty-sounding advice from r/robotics on sensor placement, and now have everything mounted together. I'm still pretty confused, and scared of whatever Unscented Kalman Filtering is, but I'm making progress. My current technical difficulty is that I don't know how to use the gyroscope in Python (the Prototyping library is extremely limited, and not really meeting my motor needs either), and don't know how to log data without it (Firmata seems deprecated). 

The Robotic Gaze

I began the morning by reading a fantastic summation of the male gaze. Not the happiest way to start out, but a subject at the back of my mind. I recently watched two movies at opposite ends of a cultural spectrum: The Godfather, and Cannonball Run II. The latter was so unredeeming I left the room and was bitterly chastised for over-sensitivity. The former had obvious merit and I stayed the course. But it was so male dominated that I didn't feel offended, I felt bored. There was so little I could relate to. My interest was sustained by the beauty of the shots and Marlon Brando's acting, and by my husband's delivered promise of a good parting shot. The only woman who really speaks is Diane Keaton. She's wonderful, but in her two big scenes she's a walking incarnation of this footnote from The System of Objects:

12. 'Loud' colours are meant to strike the eye. If you wear a red suit you are more than naked — you become a pure object with no inward reality. The fact that women's tailored suits tend to be in bright colours is a reflection of the social status of women as objects. - Jean Baudrillard, 1968

It was fortuitous to read this text and see this movie for the first time within weeks of each other. I couldn't stop thinking about it. I love this section of the book; here is what surrounds the footnote:

The world of colours is opposed to the world of values, and the 'chic' invariably implies the elimination of appearances in favour of being:¹² black, white, grey — whatever registers zero on the colour scale — is correspondingly paradigmatic of dignity, repression, and moral standing.

'Natural Colour'
Colours would not celebrate their release from this anathema until very late. It would be generations before cars and typewriters came in anything but black, and even longer before refrigerators and washbasins broke with their universal whiteness. It was painting that liberated colour, but it still took a very long time for the effects to register in everyday life. - p. 30

The good news here is that robots are ungendered. They're unburdened by history, with the potential for an aesthetic unshaped by its influence. Maybe they'll show us that the female human form is objectively beautiful and we were right all along. Maybe they'll show us something different. I really love the photos my robot Nila takes while she's painting, and am trying to figure out how to expand this with Neko.

The viewer never saw the image feed from Nila's camera, because people like looking at their own image, and I wanted them to look at Nila. I think I'll change this with Neko, and am experimenting with different modes of interaction.

To cleanse the palate, here is my favorite piece of art on the subject. The camera is given its rightful place as neutral observer. Below is an interview with the artist by MoMA.

Picture for Women, Jeff Wall, 1979

Learning About Learning

I started Andrew Ng's coursera class on Machine Learning this week. It's fun so far, and I've learned some new terminology to help frame my goals. There are two domains in which I aim to use ML: 1) learning to associate colors with words through an expansive database, and 2) learning to recommend a color given a text prompt. Here's a tidy definition of ML offered by Ng, and how I think it can be applied to both of my domains:

A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P improves with experience E. - Tom Mitchell, 1998

1) In the case of Neko learning from datasets:

   E is the collocation of colors and words in a database.
   T is the clustering and re-clustering of colors with words.
   P is the score of the clusters (how well-sorted they are).

An example of k-means clustering

2) In the case of Neko learning from people:

   E is testing colors on different individuals.
   T is returning a color, given some text.
   P is the number of well-liked colors.

An example of a support vector machine

The categorical names for each are that Case 1 is unsupervised clustering, and Case 2 is supervised classification. K-means is a likely algorithm for the former, and a support vector machine for the latter. Because order is meaningful (Orange is closer to Red than Yellow), color is a regression problem with continuously valued output. But there is a sense in which colors are discrete as well, so that's what I'm mulling over now.

Neko the Modernist

Modernism is a movement in philosophy and art, concerned with self-examination. It begins by discarding all a priori knowledge, and building up a system of evidence-based understanding. The modernist uses their medium to examine the medium itself.

“[Modernist art] happens to convert theoretical possibilities into empirical ones, in doing which it tests many theories about art for their relevance to the actual practice and actual experience of art.” Clement Greenberg, Modernist Painting, 1960

Neko's database will allow him to correlate words to colors to pigments. It's my hypothesis that he'll reveal a wealth of information about color can be used in painting to represent different concepts. I consider him a color field painter.

“Through its fetishization of the base, the sculpture reaches downward to absorb the pedestal into itself and away from actual place; and through the representation of its own materials or the process of its construction, the sculpture depicts its own autonomy.” Rosalind Krauss, Sculpture in the Expanded Field, 1979

Autonomy is the goal as I reconstruct Neko. This will come through 1) a well-balanced feedback system with me, viewers, and the work; 2) a unified, articulate, transportable body and pedestal ; 3) a well-organized archive.