What inspired the Joytone?
David Sharples: So the Joytone is an expressive isomorphic musical instrument. The project came out of my frustration trying to learn to play musical instruments. I was trying to teach myself piano and I was wondering why there are white keys and black keys, you know, it makes things so much harder. I learned a little bit of musical theory and came to realise that a major scale is just a well-defined musical structure; the gaps between the notes in a major scale are the same regardless of which major scale it actually is. But, because there’s white keys and black keys on a piano, you can play a C major scale just by running up all the white notes. If you want to play a C# major scale you have to throw in a whole bunch of black keys. It’s hard to remember and you have to build up this huge muscle memory. So one of the big goals with this project is to build a musical instrument that doesn’t have that bias based on the keys – so it’s isomorphic; that’s what that means.
And you’re using analogue thumbsticks?
David Glanzman: They’re Xbox joysticks…
Sharples: That’s the second big goal of the project. When I was doing research about this, I noticed there were some instruments that had these hexagonally isomorphic keyboards – a grid of hexagons – but the issue was that they were just buttons, they didn’t have the same sort of richness and depth as an actual musical instrument. When you’re playing a guitar there are a million ways to affect the sound – how hard you’re pushing on the string, how hard you pluck it, where on the fret you’re holding it, if you bend the string or not – and you can get all these rich sonic qualities. So we wanted to make it isomorphic and we wanted to make it expressive. We used these thumbsticks because you get two channels of analogue control in this really familiar little interface. One axis changes the waveform of the synthesised sound from a triangle wave (has a pure, bell-like quality) to a reverse sawtooth wave (has
a buzzy, bright sound, like a trumpet). There are two oscillators creating each note and if you push the thumbstick to the left, those oscillators are exactly in tune, making a very soft sound. If you push it all the way to the right then they’re offset by a couple of hertz, which makes a wide, rich sound. Then the amount that you rotate the joystick, regardless of direction, gives the volume. So you have like two and half dimensions of control there, which adds some depth to the sound.
What is the role of the Raspberry Pi?
Sharples: There’s kind of a two-brain system going on – we have the Raspberry Pi and then we have the Cypress PSoC 4. The Cypress PSoC 4 does all the heavy lifting with the data reading.
Glanzman: It does all the measurements for the joysticks. It’s got ADCs in it that convert analogue to digital, and then basically looks at each axis for each joystick in sequence, records it, and then waits for the Raspberry Pi to ask it for data values for each of the joysticks.
Sharples: There’s 57 thumbsticks and each one has two analogue channels, so that’s 114 analogue channels total. So what we did was we had eight 16-channel multiplexers hooked up to the PSoC and then the PSoC sends a signal to all of them that says ‘give me channel one’. Then it reads the eight channels, and then it says ‘give me channel two’ and it reads the eight channel twos. After it does that for all 16 channels it then has this full bank of current data. The Raspberry Pi periodically says ‘give me all your most recent data’, so the PSoC forwards the data to the Raspberry Pi, which then does a little bit of processing in Python and then sends commands to PureData, which is our synthesiser program.
What’s the Arduino component doing?
Sharples: Each thumbstick also has an RGB LED in its little hexagonal cell, and our intention was to use these to show which nodes are in key or out of key. We also wanted to guide the user through a scale – or even a song, showing the next note that they’re supposed to play – but we ran into some technical difficulties. The ideal setup for this is that you daisy-chain all of these lights in a big line and then you hook them up to an Arduino Micro, which is actually really good at controlling these specific lights from Adafruit, and then it can just push all of this data down the line and you should just be able to put whatever colours you want on each light individually. But we had some problems with the signal and could only get about four lights to behave.
Is it easy to learn to play the Joytone?
Sharples: The barrier to entry is much lower. It’s funny, when we first got it working, David was playing with it, wearing headphones, and he sort of stopped talking and was pushing a few of the joysticks, like ‘Wait, wait…’, and then he just played back Bach. So the key to learning it is just learning a little, tiny bit about the structures in music theory. There’s a shape you can hold your fingers in that will play a major chord; once you learn that shape, that’s it – that’s how all of the Joytone’s major chords are played.
Glanzman: When it comes to learning the Joytone, you have to attack musical instruction differently than you would with another instrument. When you learn something like the piano, you learn that this is D major, this is F# minor – you learn different things based on the note, the actual class, right? But with the Joytone, the pitch class is totally irrelevant because we hear everything in relevant terms, and you play everything in relative terms. So to learn the instrument, you don’t even have to discuss pitch classes – you just talk about relative distances. So major thirds or minor thirds, fifths, fourths – it’s distances between notes instead of the actual note values. I think if you phrase musical instruction in those terms, in terms that we experience music in rather than the terms we normally go through to create music, it becomes a much more natural interface with the Joytone because it’s built on that type of instruction, making it simple to learn.