Dots: A Multi-Modal Interface For Musical Expression

12th June, 2017

Through this work, I explore tangible blocks acting as “containers” for elementary musical elements, namely fixed-frequency tones and melodies. By arranging the blocks in a two-dimensional space, the user can arrange the corresponding musical elements according to his preferences. Using a novel proximity sensing based interaction paradigm, the user can utilize hand gestures to play these elements sequentially or in parallel to create musical compositions in an expressive manner.

For me, the most interesting part of this project was to explore relevant research in fields of neurosciences, psychology, color theory in order to understand as to how the users would perceive these tangible blocks or the related interactions. Based on this information, I took my design decisions. For instance, recent experiments in field of neurosciences have proved that while listening to a musical sample, humans perceive the variation of musical pitch—called melodic contour—in the same way as they perceive a visual slope. I exploited this phenomenon of crossmodal contour sensitivity to decide regarding the physical shape of my Dot modules—based on the characteristics of the music contained inside them.

From the beginning itself, I was confident about the Implementation part relying on my prior experience in electronics and prototyping (though there too, I faced many new adventures: learning a new programming framework Node.js, working with sonification, Bluetooth Low Energy , etc. ).

I prepared this work for a publication review and submitted it to IFIP Interact 2017. Though the paper was not selected, the reviews were encouraging. All three reviewers gave my paper a rating of 3 / 5 (Borderline acceptance). They appreciated the idea, research background and writing style. They suggested inclusion of results, user trials and certain improvements in the structure of the paper. On these lines, I am preparing the paper for upcoming conferences.

You can view the introduction video of the project here.


This project is important as it was my first take into HCI research. I developed this project while working at NXP semiconductors. Ultimately, this experience persuaded me to quit the job and pursue research full-time under the guidance of Dr. Aman Parnami.


table [Recovered]-01
System Overview
Dots !
Components of a Dot module
PCB Design
CAD Design for Dot Modules of different shapes (musical content) – PitchDot


first Dot is ready!