Benefits:

This project would open up a whole new world of usability benefits for GNOME users. Currently, the concept of keyboard shortcuts has only one meaning: the attachment of a sequence of keys to an event. However, this project will expand on this idea and allow keyboard shortcuts to also signify the attachment of voice commands to a sequence of keys or to an event. The current gnome-voice-control applet does not support such flexibility in its voice commands, but this flexibility is essential for the tool to become as useful as initially intended. These improvements would no doubt attract many first time users to GNOME who like the idea of a fully voice controlled desktop. In addition, this would also make GNOME much more friendly to the disabled with such ease of access. Also, there are many linux gamers out there that would gain a great advantage in multiplayer games by being able to bind voice commands to certain key strokes in order to be able to use their voice to make quicker movements. Also, there are many linux gamers out there that would gain a great advantage in multi player games by being able to bind voice commands to certain key strokes in order to be able to use their voice for making quicker movements.

Implementation:

This would involve either creating a specialized right click keyboard shortcut menu for the gnome-voice-control applet, or editing the current GNOME keyboard shortcut menu to allow for insertion of voice commands. This new feature allows the user to enter in the command or key sequence, as well as provide record/stop buttons to permit the user to record the voice command. This recording step will probably need to be repeated a few times for each new command in order to properly train the recognition software. Afterwards, the user will be asked to repeat the voice command once more to verify that the software can correctly recognize it.

A directory will need to be created, possibly ~/.gnome-voice-control/, where all of the recorded commands and necessary voice command binding configurations will be stored and appropriately named with the key/command binding to allow the user to use those recordings for other purposes if desired. The configuration file will probably have a format that specifies the audio file, and key sequence. This will allow the user to manually edit the configuration file and insert pre-recorded voice commands which would be a big help if the user installs a new OS and wants to preserve the voice bindings from an old system. This configuration file will be updated whenever the user adds or removes a voice command binding.

To match incoming voice commands with the stored voice bindings, a fast search method will need to be developed. For this, I propose a method of similarity hashing where the frequencies of voice commands are analyzed and converted into a key. If a voice command matches (or comes close to) a key corresponding to an already stored voice binding, then the stored audio file will be further analyzed to confirm if the incoming voice command is indeed the same as the stored voice binding. There are of course many different ways to approach this step of the problem and others will be further explored. This part will also utilize the sphinx library and other audio processing libraries if their tools can be used.

Time line:

- Get familiarized with gnome-voice-control/GNOME code base.
    ~ 2 weeks
- Creating functional voice command key binding menu for gnome-voice-control applet
    ~ 1 weeks
- Research and test different methods for matching audio clips, including the investigation of sphinx and other libraries
    ~ 3 weeks
- Implement final audio comparison method and address any issues that have arisen
    ~ 3 weeks
- Fix bugs and improve current features in gnome-voice-control
    ~ remaining time

Background:

I am currently an undergraduate student at the University of California, Los Angeles studying Computer Science. In addition to the knowledge I have gained throughout my studies, I have spent the past two years gaining experience through involvement with the Center for Embedded Networked Sensing here at UCLA. I have worked on multiple projects during my time at CENS, all of which have been centered around the collection, transmission, storage, and processing of image and acoustic data. This has included writing software to interface with network cameras and acoustic sensing devices, transmitting the collected data through wireless/wired networks to a central storage server, and then writing software to process the collected data in order to further compress it as well as to extract useful information such as the current state of the area being observed by each camera. Some of my work has gone into the processing of the collected audio data where I created detectors for various events such as human speech and bird chirps. I believe that I have a lot to offer to the GNOME team in many ways, including the further development of this project.