To view all of the code, please visit this link. All of this code was written in the Arduino IDE (C++) for an Arduino Uno or an Arduino Pro Mini. Our code does three main things: calibrates each sensor to the values of each user, connects via Bluetooth to a basic Android app called S2 Terminal for Bluetooth, and determines what letter the user is signing.
The flex sensors are essentially large resistors, and bending them in different places will change the voltage read by the Arduino. The accelerometer picks up motion and position based on the direction of earth’s gravity on the accelerometer. It outputs this as 3 voltages for each axis. Each of these values are calculated every 0.2 seconds, sent to the Arduino and averaged into a 1x8 array such as [123, 149, 142, 400, 502, 231, 670, 658]. These reading are immediately compared to a larger array that is 28x8 in length. (The number 28 comes from the 26 letters and then the chosen positions for space and duplicate letter). Additionally, this larger array is regenerated by each user during every session in order to account for different signing habits.
As the arduino receives these shorter arrays, it runs through a loop checking the first value of the first row in the calibration arrays until a matching value is found, within a certain tolerance. Then, the loop checks the second value in that row to see if it is close enough to the array it received. If it is, it moves on to the third value, if not, it breaks out of that inner loop and begins checking the first value of each row again. Once the letter corresponding to the matched array is identified and is printed as an output. There are also three different types of outputs: a letter, a space, and a duplicated letter. As expected, any letter signed will always output itself, but you can also calibrate a space (usually an open hand) in order to create more understandable sentences and a double letter function (usually only the ring finger up) in order to tell the code to repeat the letter that you just signed. By default, the code is set not to repeat letters, otherwise you will get sentences that look like this: "hhhhheeeeeelllllllllllloooo ttttthhhhhhheeeeerrrreee."
In order to tailor the readings to each individual user, we set up a calibration system to capture the unique hand positions for all 26 letters. The user is prompted to go through the motions for each letter. Each letter is recorded as an array composed of 5 readings from the flex sensors and 3 readings from the accelerometer. The set of 28 letters are then saved as a matrix. This matrix can then be used to identify incoming readings as explained above.
To display our outputs, we used bluetooth to connect the arduino to an Android app. The app we used is S2 Terminal for Bluetooth. The bluetooth is connected to the arduino based on instructions found on 42 Bots. We used the public domain code provided in the arduino library tutorial “Software Serial Library” as a base for our own code. As the arduino reads the data and outputs the character, the Android app displays the characters in text form. The app functions similarly to Serial in the Arduino IDE.