Project Iteration 1

Objective

The objective of our project is to propose a tool that helps people suffering from tremor to enjoy drawing and coloring again. Drawing and coloring are activities that help with stress or entertainment or simply expressing oneself through art. However, people with tremor struggle with these tasks. We want to remedy this by providing a haptic-enabled tool to overcome the tremor and make these people enjoy coloring again.

Motivation

Drawing and coloring are basic activities that many people enjoy. However, people with tremor struggle to hold a pen, pencil, or marker steadily enough to make the art piece that they would like. Many of us were introduced to coloring through coloring books where we needed to color inside the lines. This got us to think, what if you actually could not go past these lines? What force feedback made it difficult to color outside the lines? And what if the tool used dampened the effects of tremor so that a more predictable streak could be drawn? The haptic device we are working with, the Haply 2diy, is perfectly suited for these tasks.

Approach

Our project is separated in three iterations. We decided that the first iteration would be focused on the color palette, or how to select a color for coloring. Each team member came up with a concept and we each implemented our own design to be able to effectively compare them since a drawing makes it difficult to evaluate haptic performance and effectiveness.

We each designed our own color selection process and then implemented it, sharing our ideas along the way. However, we still kept our designs separate. I will therefore discuss my own implementation. We read several papers describing the difficulties people with tremor face when using technology. A major find was that they struggle when their arm or wrist are held up, but the tremor is greatly reduced and they struggle less when they are sliding along a surface. This is great since the Haply device slides along a plane. However, we wanted to also add a damping force to the end effector in an attempt to further attenuate the tremor.

Sketches

I started out by sketching basic interfaces that I believe could be used for selecting the color.

My first idea was to use a gradient on the side of the display. This gradient could be "closed" usually but it could be opened by moving the end effector to the top right corner which would reveal the gradient scale. From there, the user could hover over the desired color and press the spacebar to select the color. A variation of this design would discretize the gradient and replace it with "piano keys" where each key is a color, essentially providing the user with discrete colors to choose from instead of the continuous gradient. The idea here would be that due to the tremor, it would be difficult to select the same color as before so using discrete steps makes it easier. It also gives a basis for haptic feedback where the user senses when they go from one "key" to the next.

My second idea was color swatches. This is somewhat similar to the piano keys but the keys are even bigger and the user selects the base color, which then opens a second ribbon enabling the user to select the desired shade or tint of that color. The idea here is that the edges of the color palette could act as walls that the user cannot go through until they select a color. The swatches being even larger would make larger targets that would be easier to select. However, it would require two button presses instead of 0 or 1, since the user needs to select the first color and then select the variation. After a meeting, the question was raised of the circular swatches being more difficult to hit than squares so a second iteration would most likely use squares instead, or at least square hitboxes while displaying circles for aesthetics. Here, haptics could also indicate to the user when they move from one swatch to the next. As stated before, there would also be damping when using the interface, and it would be possible to increase the damping force when using the color selection interface if it leads to better results.

After some deliberation, I opted to implement the first option with the piano keys for the following reasons: the piano keys' size can be changed depending on the severity of the tremor, there is only one button press required (with a possibility of making it zero if the selected color is simply the last one that was hovered over when leaving the palette area), and it is a simpler design without a second pane. Below is a screenshot and video of that implementation.

The end effector location is the green circle. The top right corner shows the target where the effector needs to be to open the color selection pane.

When the effector is moved to the top right corner, the "piano keys" are displayed. The user can choose the color they desire. The center circle shows the previously selected color while the outer circle reflects the color that is about to be selected. The user can feel haptic feedback as they move the effector from one key to the next. The feedback is a linear slope pushing the user away from the key boundary. Therefore, there is force preventing the user from going to the next key unless they put enough force, after which they are pulled into the next key. When the desired color has been reached, the user can "leave" the palette by moving to the left. The palette automatically closes and the color is selected. This can be replaced by a button press in addition to a virtual wall preventing the user from leaving until a color has been selected. I will opt for this alternative method after some testing before the next iteration. The prototype in operation can be observed in the video below.

Lessons Learned

Through brainstorming, literature review, and building the prototype, I have learned multiple things. I have gotten more familiar with the struggles of interacting with technology when suffering from tremor. I still have more to learn there in order to refine my prototype. I have gotten more familiar with the Haply and with haptic design after experimenting with different force feedback parameters. I have also learned some tricks with processing such as using different color modes.

What's next?

For the next iteration, I will improve the prototype. We also plan on merging our respective ideas in order to come up with the best experience possible. We will also refine the haptics since the simple damping applied to my prototype is not good enough in my opinion. We will also look deeper into the struggles that people with tremor face so that we cater our tool for them as well as possible. As for new functionalities, we want to make it easy to import a blank sketch and then for the user to draw inside the lines so we will look into implementing that.

Code

Our code can be found in this repository. My iteration is found in the dev_AA branch.