Image Not Found

Euler's Table Exhibit

We built a mechanical exhibit exploring Euler's line for Boston's Childrens Museum.

Image Not Found

MS FingerPaint at Hack Holyoke IV

I teamed up with Olin students Alexandre Hoppe, Lauren Gulland and Nathan Yee at Hack Holyoke IV to make the next advancement in MS Paint technology simply dubbed: MS FingerPaint. Using the Leap Motion, you can use all of your favorite MS Paint functionalities (specifically: drawing, clearing, changing color and changing size) with simple hand gestures. At last, you can eat you favorite branded chips and draw at the same time. What I'm saying is that the future is here.
Ridiculous pitch aside, MS FingerPaint was an exercise in using a technology none of us had played around with before, writing something functional quickly, and most importantly for me, designing an experience with different possibilities and constraints.


In preparing for the hackathon, our team had already decided that we wanted to do somthing with a Leap Motion, knowing that Hack Holyoke had a couple available. When hacking started, we threw some ideas around including a sign language translator (which another team ended up doing), a hand gesture based calculator, Tweet Motion (great name, no specific idea attached to it), gesture based password and of course, MS FingerPaint. We didn't just choose the latter because of the great name, but also because it straightforward enough that we could get a minimum viable product early on, and iterate on it throughout the hackathon (as opposed to scrabbling to get anything working).


As I said, one of my main focuses for this project was to design the user experience. The first, fairly straightforward, decision I made was to restrict all interactions to hand motion. I didn't want to mix in keyboard and mouse because then it's difficult to think outside of the MS Paint box. Pinching with the right hand (sorry lefties) to draw seemed very natural interaction, and also happened to work very well with the existing Leap Motion library.

To change the color, we decided to look at the left hand's orientation. The main benefit is the ability to change the color while drawing (your move MS Paint). The drawbacks are relatively small, but there are a few. First of all, by adding a new hand, the canvas is effectively reduced by left hand occlusion. One solution would be to re-map the drawing space to only the right half, but that diminishes resolution and assumes that the user always wants to be changing color while drawing (otherwise, they’d either keep their hand there or remove it, making half of available space useless). If the user wants to draw single color at a time, they could keep their left hand there (not a great experience), or remove it. However, when they remove their hand the Leap Motion often senses a change in orientation, changing the color. Our solution is to add pinch-to-lock. So once you chose your color, you can pinch to lock the color you chose, and safely move your hand away. Not an optimal experience because it’s another step removed from simplicity, but good enough to solve our problem.
Since the mapping from hand rotation to colorspace is arbitrary, we also added visual feedback where MS Paint usually shows color selection (more on that later). Because rotating one’s hand 360 degrees is very mildly comfortable, we mapped the color space to 270 degrees from palm up palm to the left.

To clear the screen, the user makes a fist with their right hand, as if they were crumpling the paper. In retrospect, there are a few reason why that was a bad decision. First of all, the natural motion for deleting would have been swiping. In fact, that was my first thought, but between 3 and 4am, while the rest of my team was fast asleep, I made an executive decision against that. For some reason, I liked the idea of crumpling paper, but the decision really came from implementation details. Specifically, I really did not want the user to accidentally delete their drawing by, for example, drawing too quickly. A less sleepy self could have figured out a few safeguards to prevent that, or go so far as to add a confirmation box (swipe again to delete, pinch to keep drawing for example). Instead, I went for the fist clenching, which caused its own problems. As it turns out, pinching and making a fist can be identified as being pretty similar by Leap Motion. Prioritizing pinching ensured that the drawing didn’t get deleted accidentally, but it also made it difficult to delete. In conclusion, making a fist is just not as good as swiping. Though I feel like it was a useful mistake to make and I’ll be able to look back on it in the future when I have similar decisions to make.

Currently, MS FingerPaint is aimed at right-handed people for no other reason than lack of time during the hackathon. But since each hand’s motions are really distinct, nothing is stopping us from detecting dominant hand based on gestures.


We wrote MS FingerPaint in python since it was collectively our best language. We used Leap Motion's python library and PyGame for the graphics. You might have expected some crazy stupid implementation with MS Paint but nope, it's just a screenshot. We could have restricted the ability to draw on Paint's canvas, but we found the ability to draw on the UI really funny.

Closing Notes

Above all, this was a really fun project to work on. Our team was really effective at getting work done, making decisions and communicating. I'm also really happy with how our project turned out given the 24 hour constraint. That being said, there are clear improvements to be made. First off, and I know I've discussed this extensively, clearing the drawing should be a swipe (Yes it's really gotten to me). Each point that is drawn should be connected, either by a line or a bezier curve. I didn't touch too much on changing line thickness (mostly because I wasn't in charge of that), but instead of moving left to right, it should be an up-down motion to reduce occlusion. There should be visual feedback on the "UI" for the chosen line thickness. Most importantly, there needs to be a better way for the user to learn how to use the program, which is in itself a fairly difficult UX problem.