Overview

Welcome to my project which is about books , paper and education hacking with computer vision. Throughout the years it seems that everyone started dropping paper books and switched to electronic versions, I know its good for trees and what not but still reading a book in your hand while sitting on that old arm chair has a feeling that cannot be compared with staring at a computer screen like a zombie.

On January 2014 I was invited to the mother ship @IOActive for a snowboarding trip at mount baker it has inspired me to come up with this idea because during my stay there when the sun sets it goes completely dark. When we arrived at our cabins first day we were handed some cool swag that had a headlamp so you can walk back and forth between your cabin and the party cabin and get lost on either way which was cool.

headlamp

After this trip I returned back to my country where the lights also cut in my neighborhood every often, suddenly it all came together and an idea struck me like a lightning bolt so I said to myself what if I can revive book reading and make it cool again. I’ve put my lab coat on and started building a contraption which yielded the following:

BPequ

Screen Shot 2014-03-17 at 2.09.38 AM

By doing that I thought I was onto something which appeared to me as an awesome concept .. Lets build a research assistant I said. The ideas started to flow through my brain.

Phase I (light conditions):

  • Check if night reading is possible with the webcam under certain conditions which was possible when adding a filter to the headlamp.

Screen Shot 2014-03-17 at 11.10.17 AM

Phase II (what do i need this to do):

•  A computer vision reading , learning assistant and miniature researcher.
• No privacy concerns.
• A pluggable framework for computer vision enthusiasts and programmers.
• A new way to encourage reading & learning.
• Hand gestures to allow you to copy text, image search , create a book summary, enable static analysis for code reviews, solve mathematical equations.
• Create a virtual drawing board for notes
• Interacts you with people via voice chat that are reading the same book. A social network based on learning.
• Create summaries of your books.
• Allow it to create course materials for the subject you are reading depending on your expertise level (beginner , intermediate , Advanced)
• Spell Checker and language translator.
• Monitors page flips against time consumption for each page you read.
• Optical character recognition engine.
• Learns what YOU teach it (The best way to learn is to actually teach).
• Speech to text and voice commands ability. (I used speakable commands , Dictation , and Automator thanks to my Mac)
Phase III (Gimp & PowerPoint the concept):
Screen Shot 2014-03-17 at 11.18.53 AM Screen Shot 2014-03-17 at 11.18.25 AM Screen Shot 2014-03-17 at 11.17.40 AM Screen Shot 2014-03-17 at 11.17.55 AM Screen Shot 2014-03-17 at 11.18.08 AM
And now to jolt down some gestures that would be useful.
• Book Lookup
• Start static analysis security code review assistant
• Copy text ( single line/word or multi-lines)
• Join Reading room
• Create learning material( search for online courses , pdf’s ,blogs , wiki ,forms…..etc)
• Share anything with (tutor, Twitter, Facebook, online book reading room, class mates .. Etc)
• Internet search(image, text, equation lookup/ solution, symbol, … Etc)
• Research (after a line/multi-line/image selection it will crawl the internet for such topic and create folders with downloaded files/links for your next      phase of learning even if its math equations you don’t understand)
• Highlighting (remembering page and book of highlight)
• Finger Drawing (to take side notes without damaging the book)
• Record Video , take a picture of a page.
• Start Mic session to record your notes and control the computer for something like opening a new word document and dictating via speech.
• Etc (limitless possibilities)
Phase IV (Futuristic ideas):
• Automatic rotatable camera
• Can flip a page and learn for you
• ECG sensors to present your brain activity and let you know if you are relaxed or tense.

Now that we have mostly everything needed to start bringing this to life I had to learn a few things which were programming C++ and understanding OpenCV.

In the next post below I’ve made a gearing up section for people wanting to learn C++ and openCV by adding links to tutorials , books and configurations to everything i have learned during this project.

Without further adieu I present you 
BrainPirate_new_march
EDUCATION LENS (EDULENS)

C++ & OpenCV resources

If you are a complete beginner at C++ (AND||OR) OpenCV then you should do this first. 

    • Books:
      • Not a bad book, very short (56 pages), covers installations of OpenCV on various operating systems , integrations with IDE and very basic image and video manipulations.

      • These books take you to the next step of OpenCV.

                   

  • Setup
    • I’m using OpenCV 2.4.8 on MacOSX 10.9.1 with Xcode 5.0.1, the following is my setup:
      •  Installing opencv

Screen Shot 2014-03-16 at 5.45.06 PM

    • Creating an OpenCV project with Xcode

Screen Shot 2014-03-16 at 5.41.10 PM    Screen Shot 2014-03-16 at 5.42.00 PMScreen Shot 2014-03-16 at 5.35.38 PM    Screen Shot 2014-03-16 at 5.36.02 PM

Screen Shot 2014-03-16 at 5.36.32 PM               Screen Shot 2014-03-16 at 5.51.16 PM

Screen Shot 2014-03-16 at 5.38.13 PM    Screen Shot 2014-03-16 at 5.38.29 PM

  • Optimized_compiler_settings
  • These Compiler settings are a MUST! this makes brain pirate almost realtime with no lag.

 

  • For windows users I embedded this youtube video by Kyle Hounslow that helps you configure it for Visual studio 2010 ( it worked for OpenCV 2.4.8 also)
$ g++ -O3 `pkg-config --cflags --libs opencv` main.cpp -o main

The Project

This Post will be updated frequently throughout the project

I’m currently building plugins and if you feel that you want to dive into code with me contact me for the source code.

Mail: ehab@brainpirate.net

Twitter: @__obzy__

Source code: Source Code Here

Basic Gesture Video:

Text Recognition in progress:

code review

 

I’ve been disliking how after i select a plugin my hands gets out of the frame causing some drawbacks in cropping so i thought i’d come up with something smaller than my hand and its detection should be easy.

This tool can be used as:

  • A page marker
  • Main cropping tool
  • Highlighter
  • Note taker

bp_tool