Author: Charles Céleste Hutchins

Supercolliding since 2003

Live blogging nime – ircam assigning gesture to sound

they play a sound and then ask people to represent the sound as gesture and then use that gesture to control a new soud. The sound to gesture is an experimental study, which was a very good idea!
in the existing literature: tapping a beat is a well – known gesture. Body motion to music and more new: mimic instrumental performances (ex air guitar). Sound tracting is sketching a sound?
Gaver says musicl listing has a focus on acoustic properties and everyday listening focuses on cause.
categorisation of sounds involves the sound sources. People would categorise door sounds togther, even if they are very different sonically.
will subjects try to mimic the origins of causal sounds, eg mime slamming a door?
will they trace non causal sounds?
they played kitchen sounds and kitchen sounds convoluted w white noise
track subject’s hand position. Each subject gets some time to work out and practice her gesture, then record it three times. Ask the subject to watch the viseo and narrate it.
the transformed sounds are described more metaphorically. Non transformed sounds describe the object and the action is described, rather than the sound.

Live blogging nime papers – gamelan elektrika

a midi gamelan at mit
the slide shows the url Supercollider.ch
they wanted flixible tuning for the gamelan. Evan ziporyn worked on this project. They got alex rigopolus something about touring with kronos. This is going on too long about name dropping and not enough about how it works. I’m not even yet sure what the heck this is, but media lab sure is cool.
this must be a really long time slot becuase she has not yet talked about a technical issue yet.
low latency is important. I think she accidentally let slip that this uses ableton, thus revealing a technical issue.
mit people are often very pleased with themselves.
urathane bars with piezo sensors for version 1, so they switched to FSR and a different material. Didn’t catch how it senses damping.
reong has 5 sensors per pot anf FSRs for damping. Hit the middle, touch something to damp.
gongs have capacitive disks on side, piezo on the other, to sence strikes and damping
supercollider and ableton on the backend to handle tuning and sample pplaying
samsung laughes at them when approached for sponsorship but they sure showed them! Nime is thus invited to share in gloating of the amazing skillz of mit.
instrument demo fail.
question about why not use normal percussion controller? answer: to play like a regular gamelan.
the monitoring situation is a problem for them. They use 4 speakers to monitor. House speaker to play

Nime blogging- stereotypical transducers

earphones are the most common transducer.
can use doppler frequencies from headphone w a seeperate mic to detect motion. This may be ultrasonic? Need to use bandpass filter to just get relevant frequencies. Also, there needs to stuff in the audio range, so filter that w low pass to make sure it stays where it should.
demo video shows this works when a guy waves his hands around.
now we see a chart of error between movements and detections. Moving away has more error.
the earbuds are meant to be held and not worn, so it actually doesn’t need to handle normal range of sounds too.
to make system wireless, use a portable music player.
people over 18 cannot hear the ultrasonic frequencies. There are hardware limitations with nyquist. Or ability to play 18kHz

Live bloggig NIME papers – electromagnetically sustained rhodes piano

Make a rhodes piano that doesn’t decay. Can start from a strike or from the excitation. The examples sound like an ebow made for the rhodes.
Rhodes has enharmonic overtones, especially on the strike. The pickup gets mostly integer multiples of the fundamental, especially even partials.
the actuator is an electromechanical coil driven by a sinetone generator of the fundamental. The pickup also grabs the actuator, so they remove the phase inverse of it past the pickup. They can also use feedback to drive the tine. This causes out of control feedback, so they sense the output just of the actuator and subtract that out, leaving just the tine, thus getting increasingly rube goldberg.
there are pros and cons of each approach. They measure after touch for control using a pressure sensor below each key. Appropriately, all the dignal processing is analog

Kinect and OSC Human Interface Devices

To make up for the boring title of this post, lets’s start off with a video:

XYZ with Kinect a video by celesteh on Flickr.

This is a sneak preview of the system I wrote to play XYZ by Shelly Knotts. Her score calls for every player to make a drone that’s controllable by x, y, and z parameters of a gestural controller. For my controller, I’m using a kinect.

I’m using a little c++ program based on OpenNi and NITE to find my hand position and then sending out OSC messages with those coordinates. I’ve written a class for OSCHIDs in SuperCollider, which will automatically scale the values for me, based on the largest and smallest inputs it’s seen so far. In an actual performance, I would need to calibrate it by waving my arms around a bit before starting to play.

You can see that I’m selecting myself in a drop down menu as I start using those x, y and z values. If this had been a real performance, other players names would have been there also and there is a mechanism wherein we duel for controls of each other’s sounds!

We’re doing a sneak preview of this piece on campus on wednesday, which I’m not allowed to invite the public to (something about file regulations) but the proper premiere will be at NIME in Oslo, on Tuesday 31st May @ 9.00pm atChateau Neuf (Street address: Slemdalsveien 15). More information about the performance is available via BiLE’s blog.

The SuperCollider Code

I’ve blogged about this earlier, but have since updated WiiOSCClient.sc to be more immediately useful to people working with TouchOSC or OSCeleton or other weird OSC devices. I’ve also generated several helpfiles!
OSCHID allows one to describe single OSC devices and define “slots” for them.
Those are called OscSlots and are meant to be quite a lot like GeneralHIDSlots, except that OSCHIDs and their slots do not call actions while they are calibrating.
The OSC WiiMote class that uses DarWiinRemote OSC is still called WiiOSCClient and, as far as I recall, has not changed its API since I last posted.
Note that except for people using smart devices like iPhones or whatever, OSC HIDs require helper apps to actually talk to the WiiMote or the kinect. Speaking of which…

The Kinect Code

Compiling / Installing

This code is, frankly, a complete mess and this should be considered pre-alpha. I’m only sharing it because I’m hoping somebody knows how to add support to change the tilt or how to package this as a proper Mac Application. And because I like to share. As far as I know, this code should be cross-platform, but I make no promises at all.
First, there are dependencies. You have to install a lot of crap: SensorKinect, OpenNi and NITE. Find instructions here or here.
Then you need to install the OSC library. Everybody normally uses packosc because it’s easy and stuff…. except it was segfaulting for me, so bugger that. Go install libOSC++.
Ok, now you can download my source code: OscHand.zip. (Isn’t that a clever name? Anyway…) Go to your NITE folder and look for a subfolder called Samples. You need to put this into that folder. Then, go to the terminal and get into the directory and type: make. God willing and the floodwaters don’t rise, it should compile and put an executable file into the ../Bin directory.
You need to invoke the program from the terminal, so cd over to Bin and type ./OscHand and it should work.

Using

This program needs an XML file which is lurking a few directories below in ../../Data/Sample-Tracking.xml. If you leave everything where it is in Bin, you don’t need to specify anything, but if you want to move stuff around, you need to provide the path to this XML file as the first argument on the command line.
The program generates some OSC messages which are /hand/x , /hand/y and /hand/z, all of which are followed by a single floating point number. It does not bundle things together because I couldn’t get oscpack to work, so this is what it is. By default, it sends these to port 57120, because that is the port I most want to use. Theoretically, if you give it a -p followed by a number for the second and third arguments, it will set to the port that you want. Because I have not made this as lovely as possible, you MUST specify the XML file path before you specify the port number. (As this is an easy fix, it’s high on my todo list, but it’s not happening this week.)
There are some keyboard options you can do in the window while the program is running. Typing s turns smoothing on or off. Unless you’re doing very small gestures, you probably want smoothing on.
If you want to adjust the tilt, you’re SOL, as I have been unable to solve this problem. If you also download libfreenect, you can write a little program to aim the thing, which you will then have to quit before you can use this program. Which is just awesome. There are some Processing sketches which can also be used for aiming.
You should be able to figure out how to use this in SuperCollider with the classes above, but here’s a wee bit of example code to get you started:




 k = OSCHID.new.spec_((
  ax: OscSlot(realtive, '/hand/x'),
  ay: OscSlot(realtive, '/hand/y'),
  az: OscSlot(realtive, '/hand/z')
  ));

 // wave your arms a bit to calibrate

 k.calibrate = false;

 k.setAction(ax, { |val|  val.value.postln});

And more teaser

You can see the GUIs of a few other BiLE Tools in the video at the top, including the Chat client and a shared stopwatch. There’s also a network API. I’m going to do a big code release in the fall, so stay tuned.

Strategies for using tuba in live solo computer music

I had the idea of live sampling my tuba for an upcoming gig. I’ve had this idea before but never used due to two major factors. The first is the difficulty of controlling a computer and a tuba at the same time. One obvious solution is foot pedals, which I’ve yet to explore and the other idea is a one-handed, freely moving controller such as the wiimote.
The other major issue with doing tuba live-sampling is sound quality. Most dynamic mics (including the SM57, which is the mic I own) make a tuba sound like either bass kazoo or a disturbingly flatulent sound. I did some tests with the zoom H4 positioned inside the bell and it appeared to sound ok, so I was going to do my gig this way and started working on my chops.
Unfortunately, the sound quality turns out not to be consistent. The mic is prone to distortion even when it seems not to be peaking. Low frequencies are especially like to contain distortion or a rattle which seems to be caused by the mic itself vibrating from the tuba.
There are a few possible work arounds. One is to embrace the distortion as an aesthetic choice and possible emphasise it through the use of further distortion fx such as clipping, dropping the bit rate or ring modulation. I did a trial of ring modulating a recorded buffer with another part of the same buffer. This was not successful as it created a sound lurking around the uncanny valley of bad brass sounds, however a more regular waveform may work better.
At the SuperCollider symposium at Wesleyan, I saw a tubist (I seem to recall it was Sam Pluta, but I could be mistaken) deliberately sampling tuba-based rattle. The performer put a cardboard box over the bell of the tuba. Attached to the box was a piezo buzzer in a plastic encasing. The composer put a ball bearing inside the plastic enclosure and attached it to the cardboard box. The vibration of the tuba shook the box which rattled the bearing. The piezo element recorded the bearing’s rattle, which roughly followed the amplitude of the tuba, along with other factors. I thought this was a very interesting way to record a sound caused by the tuba rather than the tuba itself.
Similarly, one could use the tuba signal for feature extraction, recognising that errors in miccing the tuba will be correlated with errors in the feature extraction. Two obvious thing to attempt to extract are pitch and amplitude, the latter being somewhat more error-resistant. I’ve described before an algorithm for time-domain frequency detection for tuba. As this method relies on RMS, it also calculates amplitude. Other interesting features may be findable via FFT-based analysis such as onset detection or spectral centroid, etc using the MLCD UGens. These features could be used to control the playing of pre-prepared sounds or live software synthesis. I have not yet experimented with this method.
Of course, a very obvious solution is to buy a better microphone. It may also be that the poor sound quality stemmed from my speakers, which are a bit small for low frequencies. The advantage of exploring other approaches include cost (although a tuba is not usually cheap either) and that cheaper solutions are often more durable or at least I’d be more willing to take cheaper gear to bar gigs (see previous note about tuba cost). As I have an interest in playing in bars and making my music accessible through ‘gigability,’ a bar-ready solution is most appealing.
Finally, the last obvious solution is to not interact with the tuba’s sounds at all, thus creating a piece for tuba and tape. This has less that can go wrong, but it looses quit a lot of spontaneity and requires a great deal of advance preparation. A related possibility is that the tubist control real-time processes via the wiimote or other controller. This would also require a great deal of advanced preparation – making the wiimote into it’s own instrument requires the performer to learn to play it and the tuba at the same time, which is rather a lot to ask, especially for an avant guarde tubist who is already dealing with more performance parameters (such as voice, etc) than a typical tubist. This approach also abandons the dream of a computer-extended tuba and loses whatever possibilities for integration exist with more interactive methods. However, a controller that can somehow be integrated into the act of tuba playing may work quite well. This could include sensors mounted directly on the horn such that, for example, squeezing something in a convenient location, extra buttons near valves, etc.
I’m bummed that I won’t be playing tuba on thursday, but I will have something that’s 20 minutes long and involves tuba by September

WiiOSCClient.sc

Because there are problems with the wiimote support in SuperCollider, I wrote a class for talking to Darwiin OSC. This class has the same methods as the official wiimote classes, so, should those ever get fixed, you can just switch to them with minimal impact on your code.
Because this class takes an OSC stream from a controller and treats it like input from a joystick, this code may potentially be useful to people using TouchOSC on their iPhones.
There is no helpfile, but there is some usage information at the bottom of the file:


 // First, you create a new instance of WiiOSCClient, 
 // which starts in calibration mode
 
 
 w = WiiOSCClient.new;

 // If you have not already done so, open up DarwiinRemote OSC and get it talking to your wii.
 // Then go to preferences of that application and set the OSC port to the language port
 // Of SuperCollider.  You will see a message in the post window telling you what port
 // that is .... or you will see a lot of min and max messages, which lets you know it's
 // already callibrating
 
 // move your wiimote about as if you were playing it.  It will scale it's output accordingly
 
 
 // now that you're done callibrating, turn callibration mode off
 
 w.calibrate = false;
 
 // The WiiOSCClient is set up to behave very much like a HID client and is furthermore
 // designed for drop-in-place compatibility if anybody ever sorts out the WiiMote code
 // that SuperCollider pretends to support.
 
 // To get at a particular aspect of the data, you set an action per slot:
 
 w.setAction(ax, {|val|
  
  val.value; // is the scaled data from ax - the X axis of the accelerometre.
  // It should be between 0-1, scaled according to how you waved your arms during
  // the callibration period
 });
 
 
 
 // You can use a WiiRamp to provide some lag
 (
  r = WiiRamp (20, 200, 15);
 
  w.setAction(ax, {|val|
   var scaled, lagged;
  
   scaled = ((val.value * 2) - 1).abs;
   lagged = r.next(scaled);
  
   // now do somehting with lagged
  });
 )

Calibration

this class is self-calibrating. It scales the wiimote input against the largest and smallest numbers that it’s seen thus far. While calibration is set to true, it does not call any of its action methods, as it assumes the calibrated numbers are bogus. After to set calibration to false, it does start calling the actions, but it still changes the scale if it sees a bigger or smaller number than previously.

WiiRamp

The WiiRamp class attempts to deal with the oddness of using accelerometers, but it does not just do a differentiation, as that would be too easy. The accelerometers give you major peaks and valleys, all centred around a middle, so just using the raw values often is a bit boring. In the example, you see that we scale the incoming data first: ((val.value * 2) – 1) changes the data range from 0 to 1 into -1 to 1. The puts the centre on 0. Then, because we care more about the height of peaks and depth of valleys than we care about whether they’re positive or negative, we take the absolute value, moving the scale back to 0 to 1.
When you shake your wiimote, the ramp keeps track of your largest gesture. It takes N steps to reach that max (updating if a larger max is found before it gets there), then holds at the number for M steps and then scoots back down towards the current input level. You can change those rates with upslope, hold and downslope.

OscSlot

This class is the one that might be useful to iPhone users. It creates an OSCResponderNode and then calls an action function when it gets something. It also optionally sends data to a Bus and has JIT support with a .kr method. It is modelled after some of the HID code. It also supports callibration. How to deploy it with TouchOSC is an exercise left to the reader.
http://www.berkeleynoise.com/celesteh/code/WiiOSCClient.sc

First BiLE Performance

BiLE, the Birmingham Laptop Ensemble, had it’s first gig on Thursday, just six or eight weeks after being formed. We played at the Hare and Hounds in Birmingham, which is a well-known venue for rock bands, as a part of the Sound Kitchen series. There were two pieces on the bill, one called 15 Minutes for BiLE by BiLE member Jorge Garcia Moncada and we did a cover of Stucknote by Scot Gresham-Lancaster, which was a piece played by The Hub.
As a first performance, I thought it went rather well. There were the usual issues where everything sounds completely different on stage and the few minutes of sound checking does not give anybody enough time to get used to the monitor speakers. And time moves completely differently in front of an audience, where suddenly every minute gets much longer. But there were also the performing-with-a-computer issues: computers get terrible stage fright and are much more prone to crash. A few people did have their sound engines crash, so the first piece had a high pitched squeal for a few minutes, while messages flew on the chat window, reminding people to be quiet during the quiet parts. Actually, there was quite a lot of panic in the chat window and I wish I’d kept a log of it. (Later the audience said we all looked panicked from time to time. I always look panicked on stage, but it’s not cool.) In the second piece, I forgot to tell my programme to commence sound-making for a bout the first three minutes. I haven’t heard the recording yet, but I bet things sounded ok. Considering that most of us had never done live laptop performance at all before and how quickly we went from our first planning meeting to our first gig, I think we got a good result.
Jorge’s piece was complicated but Stucknote seems deceptively simple, so we did not try running through it until the day before the gig. In retrospect, this was clearly an error, because the piece, like all structured improvisation, does require some practice to get the flow down. Of course, we’d all spent the requisite time working on our sound generation and I’d coded up some faders for me and the other SuperCollider user, with Ron Kuivila’s Conductor quark, which is a very quick and dirty was of making useful GUIs. I’d tried out my part at home and it worked well and the sound I got was interesting, so I felt confident in it until I got to the practice and it crashed very quickly. I restarted SuperCollider and it crashed again. And again. And again. Half the time, it brought down the other SC user’s computer also. And it was clobbering the network, causing the MAX users a bunch of error messages and a few moments of network congestion. MAX, usefully, just throws away network messages when there are too many of them, whereas SC does not seem to.
I could not figure out where the bug was and so, after the practice, I sat down to sort it out. And there was no sign of it. Everything was fine again.
Fortunately, this provided enough of a clue that I was able to figure out that I had created an infinite loop between the two SuperCollider programmes. When I moved a slider in the GUI, that sent a message to the network which effected the sound on the target machine and also caused Shelly’s programme to update the GUI. However, the Conductor class always informs listeners when it’s updated, no matter who updated it or how, so it sent a message back to the network informing everybody of it’s new value, which caused my GUI to update, which sent a message to the network, ad infintum until I crashed.
I came up with a fix using a flag and semaphores:

                   Task({
                             semaphore.wait;
                             should_call_action = false;
                             cv = con[contag];
                             cv.input = input;
                             should_call_action = true; 
                             semaphore.signal;
                     }).play;

While this fix mostly works, it does bring up some interesting questions about data management across this kind of network. If we’re all updating the data at once, is there a master copy of it somewhere? Who owns the master copy if one exists? In this case, as one person is making sound from it, that person would seem to be the owner of the data. But what if we were all sharing and using the sliders? Then we all own it and may all have different ideas of what it might actually be.
I’m writing a class for managing shared resources which holds a value and notifies listeners when it changes. The object that’s changing it passes itself along to the method, so when listeners are notified, the changer is not. I haven’t finished the class yet, so I don’t have sample code, but I’m pondering some related issues.
Like, should there be a client version of this class for a local copy held on the local machine and a master version for the canonical copy on the network that everybody else is updating? Should a master copy of some data advertise itself on the network via the API and automatically listen for updates? Should they specify a way to scale values so it can also accepted changed inputs from 0-1 and scale them appropriately? If it does accept inputs/values in a specified range, should there be a switch for the clients to automagically build a GUI containing sliders for every master variable on the network? I think that would be quite cool, but I may not have time to code it soon, as our next gig, where we’ll be playing a piece of mine, is coming up very soon on 29 of April and then there’s a gig in May and then I suspect probably one in June and one in July (although not scheduled yet) and in August, we’re going to NIME in Oslo, which is very exciting. Bright days ahead.

Semaphores are awesomesauce

Imagine, if you will, that you are a programmer and somebody has asked you to write an application that counts cars in intersections. You have webcams mounted on top of the traffic lights and it sends you a message when it sees a car. You have saved somewhere a count of all the cars so far. So, when it tells you it sees a car, you go find that number, add one to it and write down the new number. There is more than one camera in the intersections, though and while some cars are travelling north, others are travelling south at the same time. What if two cameras see different cars at the same time?

  1. Camera one sees a car and the programme listening to it goes and finds the count of cars so far, which is 75.
  2. Camera two sees a car and the programme listening to it goes and finds the count of cars so far, which is 75.
  3. Camera one’s programme adds one to the total and gets 76.
  4. Camera two’s programme adds one to the total and gets 76.
  5. Camera one’s programme saves it’s new total.
  6. Camera two’s programme saves it’s new total.
  7. You go to look how many cars have been through the intersection and the number recorded is 76.

Camera one and camera two are operating separately from each other at the same time. They are sperate threads. when two of them are trying to change the same resource at the same time, you get something called a race condition. Will the first thread finish before the second thread clobbers it’s changes? The race is on!
Fortunately, there is a solution to this problem: semaphores! Lets’ say you are trying to update your traffic count with SuperCollider:

(
 var traffic_count, camera1, camera2, semaphore, action;

 traffic_count = 0;
 semaphore = Semaphore.new(1);

 camera1 = TrafficCounterCamera(north);
 camera2 = TrafficCounterCamera(south);

 action = { // this will be called when a car is seen
  Task({
   semaphore.wait; // only one thread can get past this point at a time
   traffic_count = traffic_count +1;
   semaphore.signal; // relinquish control of the semaphore
   traffic_count.postln;
  }).play;
 };

 camera1.action = action;
 camera2.action = action;
)

You need to make a new Semaphore before you use it. By default, they allow one thread through at a time, but you can change the argument to 2 or 3 or whatever number you want.
When your code encounters a semaphore.wait, the thread will pause and wait until it’s turn to procede. Only one thread will be allowed past that line at a time. If both cameras update at the exact same time, one of them will have to wait until the other says it’s good to go ahead.
semaphore.signal is how that thread says it’s good to go ahead. The code in between those lines can only be accessed by a single thread at a time. the traffic_count.postln line is outside the seamphore because it’s not making a change to anything, so it’s safe to read it outside of the semaphore.
So when you have two seperate threads trying to change something at the same time, semaphors can help you out! This sort of situation can arrive with GUI objects, or with OSC messages, HID objects or anything with an action method.
Be thread-safe!