Abstract

Eavesdropping on electronic communication is usually preventedby using cryptography-basedmechanisms. However, these mechanisms do not prevent one from obtaining privateinformationthroughsidechannels,suchastheelectromagneticemissionsofmonitorsorthesoundproducedby keyboards. Whileextractingthesameinformationbywatchingsomebodytypingonakeyboardmightseemtobeaneasy task, it becomes extremely challenging if it has to be automated. However,anautomatedtoolisneededinthe caseof long-lasting surveillance procedures or long user activity, as a human being is able to reconstruct only a few charactersperminute. Thispaperpresentsanovelapproachtoautomatically recovering the text being typed on a keyboard, based solely on a video of the user typing. As part of the approach, we developed a number of novel techniques for motiontracking,sentencereconstruction,anderror correction. The approach has been implemented in a tool, called ClearShot, which has been tested in a number of realistic settings where it was able to reconstruct a substantial part ofthe typedinformation. 1

Citations