Terminology: the difference between a gesture and a manipulation

All over the web I see the word gesture used to describe every type of interaction on a natural user interface. Just because you use your finger or a stylus or an accelerometer, does not make it a “gesture.” Is this crucial? Not really to users, consumers, marketing, et al. But it is in being a good scholar and interaction designer to get your terminology straight. It also helps when speaking with other developers to have your vocabulary correct so they do not misinterpret your meaning or solutions. Let’s start with the classical, dictionary definitions:

Main Entry: ¹ges·ture
Pronunciation: \ˈjes-chər, ˈjesh-\
Function: noun
Etymology: Middle English, from Anglo-French, from Medieval Latin gestura mode of action, from Latin gestus, past participle of gerere
Date: 15th century

1 archaic : carriage, bearing
2 : a movement usually of the body or limbs that expresses or emphasizes an idea, sentiment, or attitude
3 : the use of motions of the limbs or body as a means of expression
4 : something said or done by way of formality or courtesy, as a symbol or token, or for its effect on the attitudes of others <a political gesture to draw popular support — V. L. Parrington>

Main Entry: ma·nip·u·late
Pronunciation: \mə-ˈni-pyə-ˌlāt\
Function: transitive verb
Inflected Form(s): ma·nip·u·lat·ed; ma·nip·u·lat·ing
Etymology: back-formation from manipulation, from French, from manipuler to handle an apparatus in chemistry, ultimately from Latin manipulus
Date: 1834

1 : to treat or operate with or as if with the hands or by mechanical means especially in a skillful manner
2 a : to manage or utilize skillfully b : to control or play upon by artful, unfair, or insidious means especially to one’s own advantage
3 : to change by artful or unfair means so as to serve one’s purpose : doctor

You can already start to see the differences for our purposes. One is emotional, symbolic, indirect. The other is direct or mechanical. There are 4 primary differences between the two and they are easily classified after you know them.

Manipulations

1. contextual – they only happen at specific location(s) or on specific object(s)
2. react immediately – there is a direct correlation in cause and effect between your interaction and the system (this does not include visual affordance)
3. can be single state, but are usually 3 or more states ( see Bill Buxton’s paper on Chunking and Phrasing )
4. direct (could possibly be considered indirect by way of augmenting your actual interactions with the reaction of the system) – your actions directly affect the system, object, or experience in some way

Gestures

1. not contextual – they can be anywhere in the system in location and time
2. the system waits for the series of events to complete to decide on how to react (again, this does not include visual affordance)
3. they contain at least 2 states
4. indirect – they do not affect the system directly according to your action. Your action is symbolic in some way that issues a command, statement, or state.

In Dan Saffer’s book, Designing Gestural Interfaces, (O’Reilly, 2009) on page 2 he states “for the purposes of this book, is any physical movement that a digital system can sense and respond to without the aid of traditional pointing devices such as a mouse or stylus.” That may be a simple way to define the types of interaction for his book, but generalizing them in that manner is incorrect. I think Professor Shneiderman’s seminal paper in 1983 was absolutely correct. Direct manipulation is just that, direct manipulation. When we start to discuss more complex chained movements that are commands, we need a new set of terminology. (http://en.wikipedia.org/wiki/Direct_manipulation_interface)

Manipulations are the lowest common denominator and the “catch-all.” They are the most prevalent and the most widely patterned because they are easy to design for, easy to understand, and very intuitive with expected results. Gestures are more complex and is what all designers strive to achieve. When trying to decipher if something is a manipulation or a gesture, unless it passes all 4 tests for gesture, it is a manipulation. There are very few true gestures in systems currently.

These have also been called direct gestures (manipulations) and indirect gestures (gestures). Calling them this is confusing the terms and can lead to errors in design or implementation. I leave you with a graphical representation of gestures vs manipulations.

I’m eager to hear any dissenting opinions. Please comment or drop me an email. I’ll also send a copy of this to Dan as well.

The fake Apple gestural movies, a critique by request

I’m subscribed to just about every RSS that contains the words NUI or Multi-Touch in the title. One of the better one’s is http://theclevermonkey.blogspot.com. He made a post the other day about how impressed he was with a set of videos on youtube of a mockup from an anonymous person based on what they thought Apple’s new tablet will consist of interaction-wise. It was titled “Move Over Jonathan Ive, Hear comes Jon Doe”. That grabbed my attention.

Jonathan Ives is one of the best designers working today. I think he has great talent and he got some wonderful opportunities that really allowed him to flourish. He is great, but the only reason he is great is because he has made several errors. Big errors, as we all do in our careers. Bill Buxton was talking to me about this very thing a few days ago. Great artists make errors, learn from them, and move on. Jonathan has made his share of errors, as well as his share of wins. So, no question about Jonathan’s skills, but who is this John Doe?

I made an offhand comment on theclevermonkey about how unimpressed with the gesture language that was demonstrated in the videos. He replied that I should write a constructive and helpful post to point out a few things. So here we are.

The first video in the series:

So, I will breeze over a few of the main concepts that I feel the person missed in their mockup. I only watched 2 of the videos, and will only comment on one, but here is what I gathered from it.

• In watching the gestures I see confusion between single tap and double tap. At around 4 seconds he does a single tap to change its state after he used single tap to get it into the state? Then does another double tap to zoom out? I’m not quite sure I understand the metaphor around each of the taps. If single tap is select, then how can it also be activate and unzoom/state change.  It seems here that the user is going through a pre-determined path of usability rather than real motions.
• At around 10 seconds after the OnScreenKeyboard use, he does a single tap all the way across the screen on a very small hit target. This seems problematic and does not obey Fitt’s Law.

Fitts’s law (often cited as Fitts’ law) is a model of human movement in human-computer interaction and ergonomics which predicts that the time required to rapidly move to a target area is a function of the distance to and the size of the target. Fitts’s law is used to model the act of pointing, either by physically touching an object with a hand or finger, or virtually, by pointing to an object on a computer display using a pointing device). It was proposed by Paul Fitts in 1954.

• The closing of the keyboard should be thought about in a broader term. Why have the button so small? Why have the button on the other side of the screen? Why not put buttons on both sides to accommodate ambidextrous bi-manual input. Why not change its color slightly to ease speed of use? A very thin row of black buttons is not conducive to a natural interface.
• at 17 seconds he uses double tap twice. Once to do an “execute” on the video clip, and then again on the File Explorer window to perform a maximize/full screen. Unless in this operation if you “execute” the File Explorer that also goes into a Slideshow mode? I’m hazy on this.
• at 20 seconds he does a double tap on an image in the slideshow and that propels him into a scatterview experience? Maybe that is a Mac thing that I am unaware of, but if he just performed an “execute” on the icon, why does it go into scatterview?
• at 22 seconds he uses two-fingers to draw a marquee around items which also produces a context menu. If you had a context menu pop up everytime you selected a group it would not be a good experience. Menus need to be deliberately activated, not activated as a secondary action. There would be emails sent. :)
• at 27 seconds he uses two-fingers to then move the carousel around? Why has movement now shifted chords? Two-fingers were just used to draw a marquee and do a group select, and now they are used for movement? This is a confused metaphor and not consistent.

Thats pretty much everything in the first video besides the simple movements with single touch. To really be critical I would also say that using single touch for movement on a background is a little easy don’t you think? Certain things need to be forgivable, the background being one of them.

My main critique over the set is that multi-touch means just that, Multi-Touch. That means more than just two fingers side by side, or even three fingers next to each other. It means thinking about systems in a whole new way. What about offhand manipulation? What about modifiers? Think about occlusion and ease of use. If you are playing a game do you really want to have 3 fingers on the screen to run? then keep them on the screen the whole time? As the author noted in the last movie he had doubts.

To be more helpful and instructive I’ll throw out a few ideas. Why not use the offhand as movement and the primary hand as action? So, with your left hand you place down a finger and hold it outside of the quick menu area, this establishes the center of movement, now with any other touch in a direction near your initial touch you are running. This leaves your other hand free to perform actions that might be needed. Now with another tap with your left hand your running is now ‘frozen’ forward so you can remove your hand. Your character is now running in a direction that you chose without much primary attention. Want to change directions? Use two-fingers and slide. Want to stop running? Two-fingers and slide down. Side-step? You get the idea. Using the initial two taps to set motion then two fingers at the same time to give more detailed instructions eases the attention you would need to give it. Think how fast you could do that?

That ends the lesson for tonight. Overall, great video. I was very impressed with the videos themselves actually. High quality for a mock-up. Now hopefully you understand my thoughts when I see a lot of passion in this effort, but not quite enough expertise. The thing is …. expertise can be learned. Passion cannot. I hope he continues down this path, we could always use more passionate designers out here.

Ivan Sutherland’s Sketchpad : know your history

One of the main thought threads going through work and through design as a whole is to know your history. Things have been designed, created, used, and not used over the years. It is of being a good scholar and being a good designer, to know your history. Give credit where it is due.

This is of a video of Alan Kay presenting Ivan Sutherland’s Sketchpad from 1963. Yes, let that sink in. He was using a pen on a screen in 1963, which is a full 20 years before Macintosh and about 12 before Microsoft came to be. It was a seminal paper when he wrote it, and it was way ahead of its time. Even today’s programs do not encompass all the features that he had.

Here he is on Wikipedia.
Here is a link to his paper, “Sketchpad, A Man-Machine Graphical Communication System.”

I love seeing things I have designed in use

You know, there are not that many things greater than seeing something you worked on being put to practical use. I think the only bad thing would be if they were complaining about it. Here, they are absolutely loving it. Granted, I only did a little bit of the work and there is a large team of great designers responsible. I was in on it and loved being able to contribute.

I think the visuals are stunning as well. I love seeing how fluid the animations are. Lovely!

I have been off of the Surface Team for months but I talk to the current members almost daily. I’m still doing Natural User Interface work, just not directly for them. Its so great to see what is still coming out of that great team over there.