Neko 2.2

Hardware

For my needs, the accelerometer and gyroscope gave both too much and too little information. I decided to scrap the locational sensors and use mild trigonometry. Using the ultrasonic sensor as a guide for keeping the brush on the canvas did work quite well, so I might reinstall that. I'm testing out a color sensor, cameras and lights now that the arm is working smoothly.

Software

I'm expanding my text-based approach after reading the wonderful novel Galatea 2.2, by Richard Powers. Instead of just looking for color words in texts about art, I want Neko to do freer association on a larger corpus. I'm still going to be selective about the texts I incorporate, but I liked the relationship Powers' narrator had with his machine.

Firmware

This is the firmware that's running in the video. This was just a piece of test code, but I really like the motion it makes. I don't want it to get too rigid and boring, treating the painting as a 2D grid. I want to find the kind of strokes Neko is good at. I'm starting to think of it like a dance: programmed sequences of graceful motion, recombined.

Click to expand

Pair Programming

Tom and I did some pair programming last week: he drove (wrote the code) and I navigated (dictated instructions). What we ended up with (link to GitHub) was something that crawls through text looking for color words. It divides the text into lines, and the lines into words. Any time the words red, orange, yellow, green, blue, or purple come up, the other words in that line are filed into a matrix. Once the algorithm is finished, it returns the set of words that have more than one correlation. For example, given the text, The Loves of Krishna in Indian Painting and Poetry by W. G. Archer, the program returned this:

The only really interesting result here is that Krishna is indeed blue (the literal translation of the name is "black" or "dark" but most depictions give him blue skin). It's singular, but still very exciting. One text is too small a dataset, so I'm building up some compilations of my favorite poets, transcendentalists, aesthetic philosophers, etc. I'll also tokenize the text by sentence rather than line (except in poems), and weigh the associated words by how close they are to the color word (so that in the line, red shoes by the newsstand, shoes gets more points than newsstand). I'll also add the words color/s and colour/s.

The results improved dramatically when I added words like black, dark, white, and light. These words are used much more often, particularly in metaphor. It occurred to me to start collecting those for Nila, my black and white painting robot, and I'm thrilled with the idea.

Neko's Brain Trust

On Monday, Neko and I were successfully funded on Kickstarter! It's tremendously exciting and I can't thank my supporters enough. I just put in a big motor order and look forward to a minor rebuild. As for software, there are two things I'm now teaching Neko: how to select a digital color given a set of words, and how to select pigments to represent that digital color. My new vocabulary for the day is distributional semantics. This is a practice based on the idea that words with similar distributions have similar definitions; "You shall know a word by the company it keeps." I'm building a text crawler that will look for my six core words— Red, Orange, Yellow, Green, Blue, Purple— and find words that are highly collocated with each. I'm picking out old public domain books to walk through, and would love suggestions on books that have high frequency of these words. While I'm waiting for the new motors to arrive, I'll put all the most-associated words into my database. Some of the Kickstarter funds will be allocated to a Mechanical Turk bid, though I'm not sure quite what to ask yet.

The second task is matching the pigments to the colors on-screen. For this I've ordered a color sensor which Neko can use to compare what's on the palette to the target color. I've picked a set of mixing pigments— Cadmium Red Medium, Cadmium Yellow Lemon, and Phthalo Blue— to be Neko's go-to adjusters.

Sensing Brush Location

Getting Neko back and forth across the country proved extremely difficult, so I've been rebuilding him at half-size. In the process I decided to add a gyroscope (measures tilt about 3 axes) to the accelerometer  (measures static and dynamic acceleration, due to gravity and motion respectively) and ultrasonic sensor (measures distance to the nearest object by emitting an inaudible ping, and timing how long it takes to bounce back). In Neko's first iteration, I had the accelerometer on the shoulder joint, telling those motors to keep the arm generally upright. I had the ping sensor on the wrist, telling the shoulder motors to keep the brush at the right distance from the canvas. The wrist motors moved the brush back and forth as the elbow motor moved from the top of the canvas to the bottom. The shoulder had continuous rotation servos, all others were standard servos.

For half-pint Neko, all the servos are continuous rotation. I was aiming for uniformity, and the ability to scale up to stepper motors if I rebuild the robot at full-size. The accelerometer will tell me pitch and roll, but I need pitch and yaw. I think a gyroscope can fill the gaps. I got some trusty-sounding advice from r/robotics on sensor placement, and now have everything mounted together. I'm still pretty confused, and scared of whatever Unscented Kalman Filtering is, but I'm making progress. My current technical difficulty is that I don't know how to use the gyroscope in Python (the Prototyping library is extremely limited, and not really meeting my motor needs either), and don't know how to log data without it (Firmata seems deprecated). 

Learning About Learning

I started Andrew Ng's coursera class on Machine Learning this week. It's fun so far, and I've learned some new terminology to help frame my goals. There are two domains in which I aim to use ML: 1) learning to associate colors with words through an expansive database, and 2) learning to recommend a color given a text prompt. Here's a tidy definition of ML offered by Ng, and how I think it can be applied to both of my domains:

A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P improves with experience E. - Tom Mitchell, 1998

1) In the case of Neko learning from datasets:

   E is the collocation of colors and words in a database.
   T is the clustering and re-clustering of colors with words.
   P is the score of the clusters (how well-sorted they are).

An example of k-means clustering

2) In the case of Neko learning from people:

   E is testing colors on different individuals.
   T is returning a color, given some text.
   P is the number of well-liked colors.

An example of a support vector machine

The categorical names for each are that Case 1 is unsupervised clustering, and Case 2 is supervised classification. K-means is a likely algorithm for the former, and a support vector machine for the latter. Because order is meaningful (Orange is closer to Red than Yellow), color is a regression problem with continuously valued output. But there is a sense in which colors are discrete as well, so that's what I'm mulling over now.