Europe

Use Case Description
Dobot should be able to play simple songs on the piano. The user should be able to enter a song in the form of a sequence of notes (for simplicity a string). Then the program should learn this sequence using a genetic algorithm. Finally, the sequence should be played by Dobot on the piano.

Problem Statement
The robot needs to hit the desired keys in the correct order, to play music on a piano. Our genetic algorithm must learn the sequence (our music sheet), including the

1. correct notes as well as the
2. correct order of the notes.

In order to learn the sequence, there must be some kind of validation environment for feedback. To learn the sequence the robot must either know
the positions of the keys, or learn them. 

KE Concept & Validation Environment

In our experiment, we first calibrate the position of the piano. We do this, by defining two points (octave from C to C) and positioning the piano right beneath those two points.

After the positioning, we enter a target string. This is the ‘music sheet’ or sequence or the desired output, which we want the robot to play on the piano. The robot must now learn to play this sequence on the piano. 

Once the genetic algorithm has learned our desired sequence, the sequence is converted into a string and sent to Dobot via the REST-API. We are using “/sequence” from the Dobot-API. For further testing and experimentation, we have included several variables that can be changed each time the program executes. These variables include the mutation ratio and the crossover ratio. Also, the is a ‘numbers only mode’. In ‘numbers only mode’ the random generator will only generate possibilities within a certain defined alphabet (e.g. only one octave from C to C, so that the result will be found much quicker).

Ideally, the validation in this experiment would have been through the keystrokes (maybe via a midi controller). Alternatively, a microphone could have been used, meaning that a correct keystroke would have been validated by the according frequency of the note. However, we didn’t have the hardware available to realise this. Therefore, notes are already assigned to a key and the approximation occurs “virtually” by matching two strings (the input/original song and the output/learned song). This means we validate the notes by entering a ‘target string’ when the program starts. The genetic algorithm tries to generate a sequence that is as close as possible to the given target string. The genetic algorithm uses this target string in its fitness function, to see how good the current solution is.

The fitness function returns an output based on how big the difference between the current solution and the target string is. A high output of the fitness function represents a big difference, a low output (close to zero) means that the difference between current solution and target string is small. Our genetic algorithm tries to minimize the fitness function. Once the fitness function returns zero, the genetic algorithm has learned the target string
perfectly.

The experiment worked, Dobot can play simple songs on a piano now! To see the results of our project, please have a look at the attached source code and the attached video.

Since we used “/sequence”, the API was very easy to use. Most of our efforts went into the program, the genetic algorithm as well as the calibration (height, width etc. of the keyboard).

It is worth noting that we used an Open-Source MIT licensed project to get us started with genetic algorithms (github.com/jsvazic/GAHelloWorld).