adam argo

I had some interest in computer vision, so I decided to try to learn what it's all about the only real way: by doing. My roommate was spending a lot of time playing a mini-game in Fable 2, which seemed like a perfect task to conquer using computer vision.

After acquiring a webcam, a USB relay board, and sacrificing an XBOX controller, I sat down and wrote some software using OpenCV (and, more specifically, the OpenCVDotNet library). The code is pretty straightforward -- it takes an image from the webcam, looks at a specific area on the image (which can be drawn by the user), and checks to see if the pixels match certain criteria for similarity in color. I've tried to make it highly adjustable because I've noticed that the closer the pixel matching is, the more reliably it performs. The analyzed screen area is generally small -- approximately 30 square pixels or less -- but this has proven to be adequate to accurately have the computer play the game for me.

I'm unsure, but I think the game itself limits the "chain" one can build to about 30. Visually the system appears to operate as it's supposed to, but at around the 30th "pull" it fails. There is no visual indication that it is not reacting fast enough. I'm currently working on object detection for uses both in this game and otherwise.

The biggest drawback is that it's a bit slow. Using this to control, say, a Guitar Hero game wouldn't work super-well because of the delay between capturing the video, processing the images, and then triggering the relay. I've seen other projects that used photosensors to quickly respond to games like that, but for this game it seems to work decently. I think with appropriate anticipation that workarounds could be accomplished for games like Guitar Hero or Rock Band.