The purpose of this project was to build an Andoroid app which will automatically detect the Scrabble game board and will do an OCR on the letters. We could later use it to calculate the score, check if the words are valid or search for the optimal word given the current board state.
I used OpenCV Java bindings for Android for the image processing and Tesseract-OCR. In the repository there are two additional PC prototypes, implemented in Python and C++.
The project GitHub repository.
Overview
The main part of the project was the OCR and the Android implementation, so in order to make my life easier I used a hand-made board where the border of the board was colored in a distinct red color, which allows us to easily segment it using simple threshold.
For the thresholding, we first convert the RGB image into HSV (Hue, Saturation, Value) and then defined manually specific threshold on each of the HSV channel.
After the initial segmentation, we do a bit of morphological operations to remove small regions and then we extract all the available contours in the image using findContours. We add another filtration stage by selecting the largest contour and add some confidence measure on it’s minimal size, to avoid detecting the board in blank images. Once we have a single contour, we find the 4 corners of the contour by computing the farthest points from the center in each axis. From these 4 corners we compute the perspective transformation of the board and warp it back to get a top-view image.
Now that we have the top-view and we know the number of grid tiles, we can define a rectangular grid on the image and get a patch from each tile of the board. Using that patch we can do an OCR on a single letter for each patch. Of course that we also look on the confidence score and set a minimal threshold, which below it we mark a tile as empty.
In the last example we can see that we succeeded to detect correctly all the letters besides the I.
After we have the letters map we implemented the game engine which check the changes in each turn to find the new words, check them against a dictionary and then updates the score.