Domifare

Laptop and Tuba

This post is taken from the lightening talk I gave at AMRO

Abstract

I have decided to try to solve a problem that I’m sure we’ve all had – it’s very difficult to play a tuba and program a computer at the same time. A tuba can be played one-handed but the form factor makes typing difficult. Of course, it’s also possible to make a tuba into an augmented instrument, but most players can only really cope with two sensors and it’s hard to attach them without changing the acoustics of the instrument.

The solution to this classic conundrum is to unplug the keyboard and ditch the sensors. Use the tuba itself to input code.

Languages

Constructed languages are human languages that were intentionally invented rather than developing via the normal evolutionary processes. One of the most famous constructed languages is Esperanto, but modern Hebrew is also a conlang. One of the early European conlangs is Solresol, invented in 1827 by François Sudre. This is a “whistling language” in that it’s syllables are all musical pitches. They can be expressed as notes, numbers or via solfèdge.

The “universal languages” of the 19th century were invented to allow different people to speak to each other, but previously to that some philosophers also invented languages to try to remove ambiguity from human speech. These attempts were not successful, but in the 20th century, the need to invent unambiguous language re-emerged in computer languages. Programming languages are based off of human languages. This is most commonly English, although many exceptions exist, including Algol which was always multilingual.

I decided to build a programming language out of Solresol, as it’s already highly systematised and has an existing vocabulary I can use. This language, Domifare is a live coding language very strongly influenced by ixi lang, which is also written in SuperCollider. Statements are entered by playing tuba into a microphone. These can create and modify objects, all of which are loops.

Creating an object causes the interpreter to start recording immediately. The recording starts to play back as a loop as soon as the recording is complete. Loops can be started, stopped or “shaken”. The loop object contains a list of note onsets, so when it’s shaken, the notes played are re-ordered randomly. A future version may use the onsets to play synthesised drum sounds for percussion loops.

Pitch Detection

Entering code relies on pitch tracking. This is a notoriously error-prone process. Human voices and brass instruments are especially difficult to track because of the overtone content. That is to say, these sounds are extremely rich and have resonances that can confuse pitch trackers. This is especially complicated for the tuba in the low register because the overtones may be significantly louder than the fundamental frequency. This instrument design is useful for human listeners. Our brains can hear the higher frequencies in the sound and use them to identify the fundamental sound even if it’s absent because it’s obscured by another sound. For example, if a loud train partially obscures a cello sound, a listener can still tell what note was played. This also works if the fundamental frequency is lower than humans can physically hear! There are tubists who can play notes below the range of human hearing, but which people perceive through the overtones! This is fantastic for people, but somewhat challenging for most pitch detection algorithms.

I included two pitch detection algorithms, one of which is a time based system I’ve blogged about previously and the other is one built into SuperCollider using a technique called autocorrelation. Much to my surprise, the autocorrelation was the more reliable, although it still makes mistakes the majority of the time.

Other possibilities for pitch detection might include tightly tuned bandpass filters. This is the technique used by David Behrman for his piece On the Other Ocean, and was suggested by my dad (who I’ve recently learned built electronic musical instruments in 1960s or 70s!!) Experimentation is required to see if this would work.

AI

Another possible technique likely to be more reliable is AI. I anticipate this could potentially correctly identify commands more often than not, which would substantially change the experience of performance. Experimentation is needed to see if this would improve the piece or not. Use of this technique would also require pre-training variable names, so a player would have to draw on a set of pre-existing names rather than deciding names on the fly. However, in performance, I’ve had a hard time deciding on variable names on-the-fly anyway and have ended up with random strings.

Learning to play this piece already involves a neural learning process, but a physical one in my brain, as I practice and internalise the methods of the DomifareLoop class. It’s already a good idea for me to pre-decide some variable names and practice them so I have them ready. My current experience of performance is that I’m surprised when a command is recognised and play something weird for the variable name and am caught unawares again when the loop begins immediately recording. I think this experience would be improved for the performer and the listener with more preparation.

Performance Practice

The theme for AMRO, where this piece premiered was “debug”, so I included both pitch detection algorithms and left space to switch between them and adjust parameters instead of launching with the optimal setup. The performance was in Stadtwerkstadt, which is a clubby space and this nuance didn’t seem to come across. It would probably not be compelling for most audiences.

Audience feedback was entirely positive but this is a very friendly crowd, so negative feedback would not be within the community norms. Constructive criticism also may not be offered.

My plan for this piece is to perform it several more times and then record it as part of an album tentatively titled “Laptop and Tuba” which would come out in 2023 on the Other Minds record label. If you would like to book me, please get in touch. I am hoping that there is a recording of the premiere.

It works!

After many very long days, My project Domifare is working. For me. It won’t work for you because there is a bug in TuningLib. I have raised an issue, which the package maintainer will get to shortly. The package maintainer, who is me will fix it shortly. When I get back from Austria. I need to test my fix properly.

Only a subset of specified commands have been implemented, but I can record a loop and re-order the playback of a loop based on detected onsets. Hypothetically, I can also start and stop loops. In practice, pitch detection is terrible and the language is barely usable. Annoyingly, the utility of it depends on how good my tuba playing sounds.

If I want to use this as an actual tool, the way forward is playing the key phrases in as training data to an AI thing.

While writing this project, I raised three issues with the SuperCollider project over documentation and one issue with the LinuxExternals Quark over Pipewire. That will turn into a merge request. I might update the documentation for it.

If you want to hear this thing in progress, I’ll be using it on Friday. You can turn up in person to Linz, Austria or tune into the live stream. This is part of AMRO, who have a helpful schedule.

I feel like a zombie and will say something more coherent later.

Midosoldo

I put in a bid to play Domifare at AMRO, knowing it was in no state to perform, but also knowing that nothing motivates like a deadline. I thought it was likely to be accepted, so I planned to start working on it during the break between spring and summer terms.

But then I got covid and felt terrible for weeks, but also got brain fog which, to be honest, has not completely dissipated. I mean, it’s hard to tell. How could I possibly have a bassline on my mental state? I do know that my sense of taste is still messed up and if I exercise a lot I feel ill the next day, so let’s say I’m not at 100% mentally. It could be all in my head, but what difference would that make?

I wanted to finish my marking before dedicating all my time to this. I have not finished my marking, but now both are an emergency. Indeed, the list of things I have not done is kilometres long. My tuba needs a service. I haven’t played it for months and lips are completely unfit.

This is an overly-honest research update. The subject line is the Solresol word for “fear.”

This is the state of the language:

Domifare Notes — Domifare Performance Notes

The “language” has always been conceived of as a way of defining loops. So I have some syntax for recording loops as an audio recording or as a series of onsets, the ability to “shake” an onset loop, the ability to schedule shakes, and the ability to start and stop loops. These are all a series of short musical licks I should ideally memorise but at least be able to play without hesitation or split notes.

Meanwhile, the language currently has the ability to read and receive notes, which is necessarily flaky and a scaffold to hand the rest of the operations …. and a GUI to adjust thresholds because that’s necessary while playing… and that’s kind of it.

This coming weekend is a four day one which is actually a disaster because it means I can’t work during it.

Writing out what I actually have to do makes it sound fully achievable, but it will take longer than I think it will. The GUI took all of yesterday. If I spend part of every day programming and part of every day practising, I should get there. Hopefully.

I haven’t bought my train tickets yet, but I really don’t want to drop out.

I’ve got 2 weeks.

Domifare back under active development!

I’m very exciting to be submitting a proposal to do a performance in the tuba-entered live coding language Domifare.

I’ve been wanting to pick this back up for a while and it seems like the main thing that motivates me is a deadline, so now I’ve got a deadline for version 1.0.

The initial specification of the language is quite modest to implement and my teaching term ends next week, so I’m confident this will be playable by the time the gig arrives. It’s always going to be chaotic because tuba pitch tracking, but it will be a joyful chaos!

More here as I get to active developing.

Domifare – still a rough draft

Today’s diff. Everything compiles, but most things aren’t tested!

Today, I wrote the functions for most of the language structures, except the scheduling ones. And the shaking one, which I’ve just realised I’ve forgotten to include! For the variable classes, I am borrowing a lot of code from DubInstrument in AlgoRLib. Probably, this project and that should be folded into one repo or one should directly depend on the other.

For DubInstruments, asking for a random pattern can trigger the creation of a new one, but not here, as all patterns are entered by the performer.

I need some GUI, including sliders for thresholds and a text print out of entered stuff. Ideally, there should also be a record light for when the performer is entering loops.

As far as shaking goes, that’s sort of straight forward for the rhythm lines. For the melody lines, I might be looking at BufferTool. It’s designed to split up spoken text rather than played notes and relies on pauses. Another possibility is to keep onset data for melodic loops and use it to decide where cuts should be. I’ll need another trigger that gets sent by the recording synthdef when it’s gate opens, so I can relate the onset timings to the position in the buffer.

Tomorrow my wife is having a party and I’m doing marking on Monday and Tuesday, so it might be a few days before I can test properly.

Domifare Classes

Key still had some problems with transposition that related to frequency quanitsation, so those are (hopefully?) now sorted. I got rid of the gravity argument for freqToDegree because it doesn’t make sense, imo and calculating it is a tiny bit of a faff.

For Domifare, as with a spoken language, breaks between commands are articulated as pauses, so I’ve added a DetectSilence ugen. The threshold will need to be connected to a fader to actually be useful, as the margin of background noise will vary massively based on environment.

The next step is parsing. It’s been a loooong time since I’ve worried about how to do this… IxiLang uses a switch statement with string matching.

I need to draw out how this is going to work, since the repeat and the chance commands both take commands as arguments.

This might work as a statement data array:

[key, min_args, max_args, [types], function]

Types can be: \var, \number, \operator, \data. If it’s \operator, then the operator received will be the key for another statement, and the parser will listen for that too…. If it’s \data, that means start the fucntion asap….

Also, since variables are actually loop holders, I’m going to need to make a class for them.

My original plan to was to use pitch recognition to enter in midi notes, but that’s not going to work, so some commands are now defunct.

( var lang, vars, numbers; vars = (solfasire:nil, solfasisol:nil, soldosifa:nil); numbers = (redodo: 1, remimi:2, refafa: 3, resolsol: 4, relala: 5, resisi: 6, mimido: 7, mimire:8); lang = ( larelasi: [\larelasi, 2, 2, [\var, \data], nil], // func adds the name to the var array, runs the recorder dolamido: [\dolamido, 0, 1, [\var], nil], // func stops names loop or all loops domilado: [\domilado, 0, 1, [\var], nil], // func resumes named loop or all loops mifasol: [\mifasol, 0, 1, [\var], nil], // func raises an octave, which is probably impossible solfami: [\solfami, 0, 1, [\var], nil], // func lowers an octave- also impossible lamidore: [\lamidore, 2, 2, [\var, \data], nil], // add notes to existing loop dosolresi: [\dosolresi, 1, 1, [\var], nil], // shake the loop, which is possible with recordings also... misisifa: [\misisifa, 0, 1, [\var], nil], // next rhythm fasisimi: [\fasisimi, 0, 1, [\var], nil], //previous rhythm misoldola: [\misoldola, 0, 1, [\var], nil], //random rhytm refamido: [\refamido, 0, 0, [], nil], // die sifala: [\sifala, 2, 2, [\number, \operator], nil], // repeat N times (1x/bar) larefami: [\larefami, 2, 2, [\number, \operator], nil] // X in 8 chance of doing the command );

After pondering this for a bit, I decided to write some classes, because that’s how I solve all my problems. I created a github project. This is the state of the sole file today.

Tested with human voice

Testing showed that for human voice, the frequency domain onsets and pitch tracking were more accurate and faster than the time domain, which is good to know.

Once the frequency is detected, it needs to be mapped to a scale degree. I’ve added this functionality to the Tuning Lib quark. While doing this, I could the help file was confusing and badly laid out and some of the names of flags on the quantisations were not helpful, so I fixed the helpfile, documented the new method, renamed some of the flags (the old ones still work). And then I found it wasn’t handling octaves correctly – it assumed the octave ratio is always 2, which is not true for Bohlen Pierce scales, or some scales derived by Dissonance Curve. So this was good because that bug is not fixed after a mere 8 years of lurking there. HOWEVER, the more I think about it, the less I think this belongs in Key….

Pitch detecting is flaky as hell, but onsets are solid, which is going to make the creation of melodic loops difficult, unless they actually just record the tuba and do stuff with it.

This is the code that’s working with my voice:

(


s.waitForBoot({
	s.meter;
	SynthDef(\domifare_input, { arg gate=0, in=0;
		var input, env, fft_pitch, onset, chain, hasfreq;
		input = SoundIn.ar(in, 1);

		env = EnvGen.kr(Env.asr, gate, doneAction:2);
		chain = FFT(LocalBuf(2048), input);

		onset = Onsets.kr(chain, odftype:\phase);//odftype:\wphase);

		#fft_pitch, hasfreq = Pitch.kr(input);
		//send pitch

		SendTrig.kr(hasfreq, 2, fft_pitch);
		// send onsets

		SendTrig.kr(onset, 4, 1);
		//sin = SinOsc.ar(xings/2);
		//Out.ar(out, sin);
		// audio routing

		//Out.ar(out, input);
	}).add;
	k = Key(Scale.major); // A maj

	//k.change(6); // C maj - changing to c maj puts degree[0] to 6!
	b = [\Do, \Re, \Mi, \Fa, \So, \La, \Si];

	(scale:k.scale, note:k.scale.degrees[0]).play;
	OSCdef(\domifare_in, {|msg, time, addr, recvPort|

		var tag, node, id, value;
		#tag, node, id, value = msg;

		case

		{ id == 2 } {

			//value.postln;

			//c = k.freqToDegree(value.asFloat).postln;

			//b[c.asInt].postln;

			b[k.freqToDegree(value.asFloat)].postln;

		}

		{ id == 4 } { "4 freq dom onset".postln; }
	}, '/tr', s.addr);
	s.sync;
	a = Synth(\domifare_input, [\in, 0 , \out, 3, \rmswindow, 50, \gate, 1, \thresh, 0.01]);

}) )

Domifare input

Entering code requires the ability to determine pitch and entering data requires both pitch and onset. Ergo, we need a synthdef to listen for both things. There is also two ways to determine pitch, one in the time domain and the other in the frequency domain.

The frequency domain, of course, refers to FFT and is probably the best method for instruments like flute. It has a pure tone, where the loudest one is the fundamental. However, brass instruments and the human voice both have formants (loud overtones). In the case of tuba, in low notes, the overtones can be louder than the main pitch. I’ve described time-domain frequency tracking for brass and voice in an old post.

The following is completely untested sample code…. It’s my wife’s birthday and I had to go out before I could try it. It does both time and frequency domain tracking, using the fft code to trigger sending the pitch in both cases. For time domain tracking, it could -and possibly should- use the amplitude follower as a gate/trigger in combination with a frequency change of greater than some threshold. The onset cannot be used as the trigger, as the pitch doesn’t stabilise for some time after the note begins. A good player will get it within two periods, which is still rather a long time in such a low instrument. A less good player will take longer to stabilise on a pitch.

Everything in the code is default values, aside from the RMS window, so some tweaking is probably required. Presumably, every performer of this language would need to make some changes to reflect their instrument and playing technique.

(


s.waitForBoot({
	SynthDef(\domifare_input, { arg in=0, out=3, rmswindow = 200;
		var rms, xings, input, amp, peaks, sin, time_pitch, fft_pitch, onset, chain, hasfreq;
		input = SoundIn.ar(in, 1);

		amp = Amplitude.kr(input);

		rms = RunningSum.rms(input, window);

		peaks = input - rms;

		xings = ZeroCrossing.ar(peaks);

		time_pitch = xings * 2;
		chain = FFT(LocalBuf(2048), input);

		onset = Onsets.kr(chain, odftype:\wphase);

		#fft_pitch, hasfreq = Pitch.kr(input);
		//send pitch

		SendTrig.kr(hasfreq, 0, time_pitch);

		SendTrig.kr(hasfreq, 1, fft_pitch);
		// send onsets

		SendTrig.kr(onset, 2, 1);
		//sin = SinOsc.ar(xings/2);
		//Out.ar(out, sin);
		// audio routing

		//Out.ar(out, input);
	}).add;
	OSCdef(\domifare_in, {|msg, time, addr, recvPort|

		var tag, node, id, value;
		#tag, node, id, value = msg;

		case

		{ id == 0 } { "time dom pitch is %".format(value).postln; }

		{ id == 1 } { "freq dom pitch is %".format(value).postln; }

		{ id == 2 } { "onset".postln; }
	}, '/tr', s.addr);
	s.sync;
	a = Synth(\domifare_input, [\in, 0 , \out, 3, \rmswindow, 200]);

}) )

Domifare fonts

Solresol can be notated in many ways: solfedge, numbers, notes and via two competing sets of special glyphs. These glyphs are a proposed edition to the unicode standard and part of a set of glyphs known as the CSUR. They’re included in some fonts, like the amazingly ugly Unifoundry includes the more abstract glyphs

      

and Constructium which just has single characters of solfedge

       

(and this is boring, but well-rendered and easy to understand – aside from the duplication of the final syllable as both ‘si’ and ‘ti’.).

Both sets of glyphs above should render in modern web browsers, but allow some time.

Many of my compuer music projects seem to quickly get bogged down in font issues and learning a new script is probably too much to ask of performers (myself included), even if it’s only 8 glyphs. However Constructum is, essentially, a monospace font in the sence that all 4-note words will render the same length, so it’s my likely choice for a display. It is a lot more easy to do than to draw actual music notation.

Like ixilang users type into a dedicated Document window Domifare users will be provided with an auto-transcription of their input. This is enough problem to solve by itself in early versions, but ixi-lang’s page re-writing properties seem like a good plan for later ones.

Domifare sisidomi

‘Domifare sisidomi’ means ‘live code’ in solresol, which is the first ever ‘constructed language’. That is, it was the first ever language to be intentionally designed. And, as this was a new idea, the creator, François Sudre, apparently felt like new syllables were needed. He used musical tones.

This last weekend, I played at an algorave in Newcastle with tuba and algorithms. The idea was to use a foot pedal to control things, but (despite working perfectly at home), it was non-responsive when the gig started, so my set included some live coding. Live coding with one hand while holding a tuba is not terribly efficient and it’s impossible to live code and play tuba at the same time . . . unless, playing the tuba is the live coding.

And thus, I’ve now specified an ixi lang-like language, domifare sisidomi. It’s a bit sparse, but there’s only so much a player can be expected to remember.

All variables are loops. There are three built in: solfasire, solfasisol and soldosifa (low percussion, high percussion and bassline). These are entered by playing the name of the variable followed by a rhythm or melody. As there is more than one kind low or high percussion instruments, different ones can be specified by playing different pitches.

The full (rough, unimplemented) specification follows:


// Enter a loop
solfasire [rhythm] //kick & toms

solfasisol [rhythm] // higher drums

soldosifa [melody] // bassline
larelasi [4 notes = the name] [melody] // declare a new loop
// start stop and modify a loop
dolamido [name] -- silence loop

domilado [name] -- resume loop

mifasol [name] -- raise an octave

solfami [name] -- lower an octave
lamidore [name] [rhythm] -- add notes to the loop

dosolresi [name] -- randomise loop // shake in ixilang
// every time a loop is changed by playing in new notes, shaking or adding, it gets added to a list of rhythms
// moving between loops in the list
misisifa / fasisimi [optional name] - move to next or previous rhythms

misoldola [optional name] - move to a random rhythm
// if no name is given, applies to all playing loops
// control structures
refamido - die
sifala dofadore [number] [next/prev/rand/randomise/chance/octave shift] [optional name] -- repeat the command x times
larefami [number] [next/prev/rand/randomise/repeat/octave shift/die] [optional name] - x in 8 chance of doing the command
//numbers

redodo - 1 remimi - 2 refafa - 3 resolsol - 4 relala - 5 resisi - 6 mimido - 7 mimire - 8