Keyboard Glove
Midterm // October 24, 2002
Dennis Crowley


[zoom in]

Ingredients
1 Pair of Gloves
3 Flex Sensors
1 Emerging Display 16 x 4 LCD panel, 12 Pin (#ED10020TRU9511)
1 74HC595 Serial Shift Register (16 Pin)
1 speaker taken from broken Game Boy Advance (note: intentionally broken by ex-girlfriend)
1 RadioShack switch
1 330 Ohm resistor
2 4.7K resistors
6 100 Ohm resistors

How it Works
I'm not sure where the inspiration from this project came from, but I think it was left over from our tech research project on LCDs. When our LCD wasn't working with two days to go to present out tech research project to the class, we started brainstorming other ideas. A lot of the topics we were discussing were old Nintendo products - the lightgun, the Power Pad, and of course the Power Glove.

Without an idea of what I would actually do with a makeshift Power Glove, I started working on throwing on together. I already had one flex sensor which I was previously experimenting -- attaching to my jacket on the arm and shoulder -- which I then attached to a pair of knit gloves purchased at the Broadway Open Air Bazaar for $3.

The flex sensor behaved in a fairly predicatable pattern, and when connected to ground via two 100 Ohm resistors gave me a range of 0-20. When the sensor was attached to the glove, I found that when the hand was "at rest" (flat with finger spread out), the average flex was around 15. I soon starting writing BasicX code which would check the current flex reading and compare it to this average to determine whether the finger was raised or lowered.

After playing with the values returned by the flex sensor attached to the glove, I went to work reconnecting the LCD used in our tech research project. I couldn't help but we reminded of some Minority Report-esque functionality when playing with the readings returned by the flex sensor, so naturally I started working on trying to control the position of characters on my LCD screen by moving my fingers and increasing/decreasing the value returned by the flex sensor.

The initial results were disappointing - mostly because the LCD is very slow to refresh the characters it is currently displaying. As mentioned in our tech research project, our 16 character x 4 line LCD display refreshes one line, one character at a time. Also, while the screen is refreshing, a cursor moves horizontally across the line from left to right. While I originally wanted to make a Pong/Arkanoid style game using the LCD, I quickly realized that this would not be possible due to the slow refresh rate of the LCD.

My next idea was to use values returned by the flex sensor to display letters of the alphabet on the LCD. It didn't take very long to write code that was able to scroll forward through the alphabet when my finger was down and then to scroll backwards through the alphabet when my finger was moved up. Thus, the LCD would start with the letter "A" and as long as I kept my index finger pointing down the LCD would display one letter at a time scrolling through the entire alphabet and looping back to the beginning after the letter Z.

I then decided it may be interesting to try to use the LCD to spell words. Realizing that I would need another source of input (e.g. a "return" key), I added another flex sensor to the middle finger of my glove. This flex sensor was originally wired to act as a submit button when the finger was moved downwards (by submit, I mean confirming the current character was one that the user wanted to display on the screen and then moving the cursor over one space to the right) and then to reset the screen when the user's finger was pointed upwards.

After wiring the glove and writing the code, I found the interface to work awkwardly. Try holding you hand out and wigging your middle finger up while keeping your index finger straight. It's fairly difficult to do and I was finding that as the user would try to make these motions, the movement of the index finger would be tracked instead, thus leading to a less then intuitive interface.

To correct this, I attached another flex sensor to the thumb and used this as a return key. The thumb is much easier to move independently of the other fingers, thus the user can use his/her index finger to scroll through characters and move their thumb when the want to accept the current character and move the cursor over. Note that the thumb did not work on the same 0-20 flex scale as the other two fingers. This is partly due how much shorter the thumb is (and how much more of the flex sensor is attached to the wrist instead of the finger) and also due to the fact that the flex sensor on the thumb was damaged due to multiple attachments/detachments to the glove using electrical and packing tape.

Note: When using flex sensors, do not apply tape to the side with the squares. Instead apply it to the side with the "patent" notice. When the tape is removed, so will be the squares and it is the squares that drive the flex reading from the sensors. If the squares have been removed by an adhesive then the reading from the flex sensor will be far less than what it should be. (I was getting a reading of 0-5 using the same resistors as the other flex sensors. It took me about 20 minutes to debug this scenario).

With the thumb now working as the return key, I altered the role of the middle finger to act as a delete key. Everytime the user clicks their middle finger down, the word currently displayed on the LCD erases the last character. I originally had planned on writing code that allowed the user to erase one character after the next, but realizing that this would require string comparison, I decided to back off and leave the user with the ability to only delete the last character added.

Now, while it may seem that everything is working fine so far, the project only works about 60% of the time. Why? The LCD is flaky. I'm not sure if it's the way I have it connected, a glitch in my programming, or perhaps some crumbs in my breadboard but 4 out of 10 times I power up my app the LCD screen will not react properly - it'll either go dead, fail to initialize or display garbage across the screen. There really doesn't seem to be any rhyme or reason to it's behavior and whether it works or not is really hit or miss.

Realizing I may potentially be presenting my midterm project on one of my LCD's "off days", I decided to build a Plan B midterm relying on sound instead of the LCD. I was able to salvage some code I wrote for my old "bounce meter" project (the lights and sounds that worked in conjunction with a flex sensor plugged into the BX). What made the existing code more interesting this time around was the fact that I had three input devices to play with: index finger, middle finger and thumb.

I quickly started playing around with multiple variations of how the user may be able to control the sound from the speaker with their fingers. I quickly nixed the idea of playing incremental frequencies (e.g. starting at 100 and counting +1) and instead opted to use the frequencies that corresponded with certain notes on the musical scale. I started with 220 and then increased (or decreased) the frequency by 14 (based on what I found to the average difference in frequency between notes) depending on whether the user's index finger was pointing up or down.

The middle finger was used to control the octave. If the user's middle finger was bent down then the octave would start at 5 (frequency = 220 * 5) and increase as the finger was bent upwards (x6, x7, x8, x9). The thumb is used to control the duration of the frequency. The default value is 50 ms, but as the user moves his/her thumb in closer to the palm of one's hand, the value is multiplied by 2. Remember, the readings on the thumb are limited to between 0-5, thus preventing the delay from becoming too long.

While the audio functionality works with the glove, it is not very intuitive and I really struggled with configuring the reconfiguring the algorithm that generates the frequency in an attempt to make it more intuitive. As it now stands, if you put the glove on and close your eyes you'll very easily note the change in frequency and duration of each sound, but you don't really feel like you have any real control over it. Sure, you can make it stop and start and produce a sound that is faster or slower, but in terms of the notes generated, I don't feel like the user has a really solid understanding of how their actions are tied to the sounds their hearing. Perhaps I will be able to further develop this as the semester continues.

Finally, one of the last additions to the project was a simple Radio Shack switch. As I write this at 1:25am early Thursday morning, I have no idea if the LCD is going to work 8 hours from now when it comes time to present in class. If it does work - cool - the keyboard app is pretty slick and I look forward to showing it off. If not - oh well - I always have my little red button that turns the project from the LCD keyboard to the audio generating device (the button simply tells the Main routine which of two routines to go to - Audio() or Display().

It'll be interesting to see how things work out tomorrow morning. :)

Source Code
Click here to view