NCU ACA AAAA
`
`US 20060026521A1
`
`as) United States
`a2) Patent Application Publication co) Pub. No.: US 2006/0026521 Al
`
` Hotelling et al. (43) Pub. Date: Feb. 2, 2006
`
`
`(54) GESTURES FOR 'TOUCH SENSITIVE INPUT
`DEVICES
`
`(73) Assignee: Apple Computer, Inc.
`
`(75)
`
`Inventors: Steve Hotelling, San Jose, CA (US);
`Joshua A. Strickon, San Jose, CA
`(US); Brian Q. Huppi, San Francisco,
`CA (US); Imran Chaudhri, San
`Francisco, CA (US); Greg Christie,
`San Jose, CA (US); Bas Ording, San
`Francisco, CA (US); Duncan Robert
`Kerr, San ['rancisco, CA (US);
`Jonathan P. Ive, San Francisco, CA
`(US)
`
`Correspondence Address:
`BEYER WEAVER & THOMAS LLP
`P.O. BOX 70250
`OAKLAND,CA 94612-0250 (US)
`
`(21) Appl. No.:
`
`10/903,964
`
`(22)
`
`Filed:
`
`Jul. 30, 2004
`
`Publication Classification
`
`G1)
`
`Int. Cl.
`(2006.01)
`GO6F 17/00
`(52) US. Che eeececeesncsstertsenseneenneenes 715/702; 715/863
`
`(57)
`
`ABSTRACT
`
`Methods and systems for processing touch inputs are dis-
`closed. The invention in one respect includes reading data
`from a multipoint sensing device such as a multipoint touch
`screen wherethe data pertains to touch input with respect to
`the multipoint sensing device, and identifying at least one
`multipoint gesture based on the data from the multipoint
`sensing device.
`
`6a
`
`GUI
`
`
`
`DISPLAY
`
`70
`
`GESTURE
`
`INPUT
`DEVICE
`
`PROCESSOR
`
`GESTURAL
`PROGRAM
`
` 68
`85
` 88
`
`
`
`80
`
`V/O DEVICE
`
`OUTPUT
`
`1
`
`APPLE 1010
`
`APPLE 1010
`
`1
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 1 of 37
`
`US 2006/0026521 Al
`
`|Old
`
`WVvY9Oud
`
`08
`
`AOIAAGOM!
`
`LAdLNO
`
`88
`
`8s
`
`9g
`
`AV1dSIG
`
`IND
`
`89
`
`OL
`
`S8
`
`LNdNI
`
`
`
`WuNLs39Ossa90¥udJ9IAad
`
`AYNLSAD
`
`2
`
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 2 of 37
`
`US 2006/0026521 Al
`
`100 ~
`
`READ IMAGE FROM TOUCHSCREEN
`
` 102
`
`CONVERT IMAGE TO COLLECTION OR
`LIST OF FEATURES
`
`104
`
`106
`
`108
`
`110
`
`114
`
`CLASSIFY AND GROUP FEATURES
`
`CALCULATE KEY PARAMETERS OF
`FEATURE GROUPS
`
`ASSOCIATE GROUP(S) TO ULI.
`ELEMENT(S)
`
`FEEDBACK
`
`112
`
`PERFORM
`ACTION
`
`PROVIDE USER
`
`FIG. 2
`
`3
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 3 of 37
`
`US 2006/0026521 Al
`
`20
`
`TOUCH1
`
`(ID,)
`
`A,
`
`TOUCH 2
`(ID,)
`
`©;
`
`Y;
`
`122A
`
`A,
`
`8;
`
`¥
`
`2
`
`x
`
`1
`
`122B
`
`FIG. 3A
`
`x
`
`Z,
`
`122A
`
`122B
`
`FIG. 3B
`
`4
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 4 of 37
`
`US 2006/0026521 Al
`
`Dave =
`
`D,+D,
`
`2
`
`130
`
`ID
`
`+
`
`uf)
`
`UD,
`
`oOo
`Dy 7 Cc
`von
`
`ID,
`
`130
`
`FIG. 4
`
`5
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 5 of 37
`
`US 2006/0026521 Al
`
`152
`
`NO
`
`158
`
`RECEIVE GROUP OF FEATURES
`
`IS THEREA
`CHANGE IN # OF
`FEATURES?
`
`PARAMETER VALUES
`
`
`
`CALCULATE INITIAL
`
`
`
`
`CALCULATE CURRENT
`PARAMETER VALUES
`
`REPORTINITIAL AND
`
`CURRENT PARAMETER VALUES
`
`160
`
`FIG. 5
`
`END
`
`6
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 6 of 37
`
`US 2006/0026521 Al
`
`yo”
`172
`
`i
`
`©™—
`
`7
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 7 of 37
`
`US 2006/0026521 Al
`
`
`
`
`FIG.6B
`
`8
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 8 of 37
`
`US 2006/0026521 Al
`
`
`FIG.6C
`
`9
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 9 of 37
`
`US 2006/0026521 Al
`
`_~
`NX
`a
`
`<x
`+toO
`2
`
`188
`
`
`FIG.6D
`
`10
`
`10
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 10 of 37
`
`US 2006/0026521 Al
`
`
`
`
`190
`
`FIG.6E
`
`11
`
`11
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 11 of 37
`
`US 2006/0026521 Al
`
`(-») Ne
`ST y
`
`190
`
`©
`
`a“
`
`[sO
`
`Ny
`
`LU
`
`T)):
`
`‘
`
`12
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 12 of 37
`
`US 2006/0026521 Al
`
`
`DETECT USER INPUT
`
`202
`
`
`
`CLASSIFY USER INPUT AS
`TRACKING OR SELECTION
`INPUT OR GESTURE INPUT
`
`
`
`
`
`
`
`206
`
`208
`
`PERFORM TRACKING
`DURING USER INPUT
`
`PERFORM GESTURE
`CONTROL ACTIONS
`
`FIG. 7
`
`300 oN
`
`OUTPUT GUI OBJECT
`
`RECEIVE GESTURE INPUT OVER
`GUI OBJECT
`
`302
`
`304
`
`306
`
`
`
`MODIFY GUI OBJECT BASED ON
`AND IN UNISON WITH THE
`GESTURE INPUT
`
`
`
`FIG. 9
`
`13
`
`13
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 13 of 37
`
`US 2006/0026521 Al
`
`CAPTURE INITIAL IMAGE
`
`252
`
`DETERMINE TOUCH MODE
`BASED ON INITIAL IMAGE
`
`260
`
`256
`
`SET NEXT IMAGE
`AS INITIAL IMAGE
`
`CAPTURE NEXT IMAGE
`
`254
`
`
` DETERMINE IF
`
`MODE CHANGED?
`
`COMPARE INITIAL AND NEXT
`IMAGES
`
`
`
`262
`
`FIG. 8
`
`14
`
`14
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 14 of 37
`
`US 2006/0026521 Al
`
`DETECT PRESENCE OF FIRST FINGER AND SECOND FINGER ON
`A TOUCH SENSITIVE SURFACE AT THE SAME TIME
`
`352
`
` COMPARE DISTANCE BETWEEN
`
`TWO FINGERS
`
`356
`
`
`
`
`GENERATE ZOOMIN
`GENERATE ZOOM OUT
`SIGNAL IF DISTANCE
`SIGNAL IF DISTANCE
`
`
`
`BETWEEN TWO FINGERS
`BETWEEN TWO FINGERS
`
`
`
`DECREASES
`INCREASES
`
`358
`
`400
`
`FIG. 10
`
`INITIAL POSITION
`
`DETECT PRESENCE OF FIRST
`FINGER AND SECOND FINGER
`ON A TOUCH SENSITIVE
`SURFACE AT THE SAME TIME
`
`MONITOR POSITION OF TWO
`OBJECTS WHEN MOVED
`TOGETHER ACROSS TOUCH
`SENSITIVE SURFACE
`
`GENERATE PAN SIGNAL WHEN
`THE POSITION OF THE FINGERS
`CHANGES RELATIVE TO AN
`
`FIG. 12
`
`15
`
`15
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 15 of 37
`
`US 2006/0026521 Al
`
`368
`
`364
`
`364
`
`FIG. 11A
`
`368
`
`366 FIG. 11B
`
`16
`
`16
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 16 of 37
`
`FIG. 11C
`
`366
`
`372
`
`364
`
`US 2006/0026521 Al
`\ N. California
`
`366
`
`FIG. 11D
`
`366
`
`17
`
`17
`
`

`

`US 2006/0026521 Al
`
`FIG. 11F
`
`“68
`
`Patent Application Publication Feb. 2, 2006 Sheet 17 of 37
`
`N. California FIG. 11E
`San Jose
`
`364
`
`18
`
`18
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 18 of 37
`
`US 2006/0026521 Al
`
`376
`
`364
`
`San \Franci
`een
`
`378
`
`366
`
`p
`
`~ syle
`“\, San Jose
`
`
`5See
`
`FIG. 11G
`
`378
`
`366
`
`.
`
`4
`
`‘
`
`\ SM Crd
`‘
`aN. (ior)
`tenes wT NN, ie 7
`‘.
`s)
`
`364
`
`So
`
`~
`
`~‘
`D
`~N
`¢ N+
`DON.
`O*
`~~
`‘ ant
`~.,
`c
`V a ~ SS
`366
`SartCarlos MES
`™
`ye ae oF J A
`a Y
`co ‘N. /
`}
`eR
`in
`380
`¢
`/ ss
`'
`
`\
`
`oS
`
`{
`
`me
`
`‘.
`
`f
`
`FIG. 11H
`
`nee
`
`19
`
`

`

`368
`
`FIG. 13A
`
`Patent Application Publication Feb. 2, 2006 Sheet 19 of 37
`
`US 2006/0026521 Al
`
`366
`
`FIG. 13B
`
`20
`
`20
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 20 of 37
`
`US 2006/0026521 Al
`
`368
`
`366
`‘\ CS
`:
`/
`
`364
`
`Q
`
`366
`
`FIG. 13C
`
`FIG. 13D
`
`21
`
`21
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 21 of 37
`
`US 2006/0026521 Al
`
`450 _\
`
`
`
`
`
`DETECT PRESENCE OF FIRST
`FINGER AND SECOND FINGER
`ON A TOUCH SENSITIVE
`SURFACE AT THE SAME TIME
`
`452
`
`
`
`
`GENERATE INITIAL ANGLE AT
`SET DOWN
`
`454
`
`456
`
`
`GENERATE ROTATE SIGNAL
`
`
`WHEN THE ANGLE CHANGES
`
`
`FIG. 14
`
`500 ~
`
`502
`
`DETECT PRESENCE AN OBJECT
`ON TOUCH SENSITIVE SURFACE
`
`RECOGNIZE OBJECT
`
`904
`
`
`
`506
`DISPLAY IMAGEIN VICINITY OF
`
`
`
`OBJECT
`
`FIG. 16
`
`22
`
`22
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 22 of 37
`
`US 2006/0026521 Al
`
`368
`
`364
`
`UP
`
`FIG.15A
`
`366 FIG. 15B
`
`368
`
`364
`
`366
`
`FIG. 15C 38
`
`23
`
`23
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 23 of 37
`
`US 2006/0026521 Al
`
`FIG.17C
`
`520
`
` FIG.17B
`17A
`
`FIG.
`
`24
`
`24
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 24 of 37
`
`US 2006/0026521 Al
`
`516
`
`
`
`520
`
`512
`
`NY
`
`FIG.17E
`
`17D
`
`FIG.
`
`25
`
`25
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 25 of 37
`
`US 2006/0026521 Al
`
`550 ™~
`
`
`DISPLAY CONTROL BOX HAVING
`CONTROL BUTTONS
`
`552
`
`ENLARGE CONTROL BOX AND CONTROL BUTTONS WHEN
`THE PRESENCE OF AN OBJECT IS DETECTED OVER THE
`CONTROL BOX
`
`554
`
`GENERATE CONTROL SIGNAL ASSOCIATED WITH A SELECTED BUTTON
`OF THE CONTROL BOX WHEN THE PRESENCE OF THE OBJECT IS
`DETECTED OVER ONE OF THE ENLARGED CONTROL BUTTONS
`
`
`
`
`FIG. 18
`
`600 ™
`
`DISPLAY PAGE FROM A GROUP
`OF PAGES
`
`
`DETECT PRESENCE OF OBJECT
`OVER PREDETERMINED REGION
`OF DISPLAYED PAGE
`
`602
`
`604
`
`
`
`
`
`
`
`
`GENERATE PAGE TURN SIGNAL WHEN THE
`DETECTED OBJECTIS SLID HORIZONTALLY ACROSS
`
`
`THE PAGE IN THE PREDETERMINED REGION OF THE
`
`PAGE
`
`
`606
`
`FIG. 20
`
`26
`
`26
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 26 of 37
`
`US 2006/0026521 Al
`980
`
`FIG. 19A
`
`FIG. 19B
`
`27
`
`27
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 27 of 37
`
`US 2006/0026521 Al
`
`
`
`
`FIG. 19C
`
`FIG. 19D
`
`28
`
`28
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 28 of 37
`
`US 2006/0026521 Al
`
`516
`
`512
`
`FIG.21B
`
`512
`
`
`
`
`FIG.21A
`
`+—
`i)
`
`29
`
`29
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 29 of 37
`
`US 2006/0026521 Al
`
`516
`
`516
`
`
`
`FIG.21D
`
`512
`
`
`FIG.21C
`
`512
`
`30
`
`30
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 30 of 37
`
`US 2006/0026521 Al
`
`650 _
`
`652
`
`DISPLAY IMAGE
`
`DETECT SCROLLING OR PANNING STROKE
`
`654
`
`DETERMINE SPEED AND DIRECTION OF SCROLLING OR PANNING
`STROKE
`
`MOVE IMAGE IN ACCORDANCE WITH THE SPEED AND DIRECTION
`OF THE SCROLLING OR PANNING STROKE
`
`SLOWING MOTION OF IMAGE IN ACCORDANCE WITH INERTIA
`
`PRINCIPALS WHEN THE PANNING OR SCROLLING STROKE IS NO LONGER DETECTED
`
`FIG. 22
`
`656
`
`58
`
`660
`
`os
`
`702
`
`
`
` 704
`
`DISPLAY KEYBOARD
`
`DETECT PRESENCE OFA FIRST OBJECT
`OVER A FIRST KEY AND A SECOND
`OBJECT OVER A SECOND KEYAT THE
`SAME TIME
`
`GENERATE SINGLE CONTROL FUNCTION WHEN THE
`FIRST OBJECT IS DETECTED OVER THE FIRST KEY
`AND WHEN THE SECOND OBJECT IS DETECTED OVER
`THE SECOND KEY
`
`706
`
`FIG. 24
`
`31
`
`31
`
`

`

`US 2006/0026521 Al
`520 FIG. 23B
`
`679
`
`512
`
`678
`
`Patent Application Publication Feb. 2, 2006 Sheet 31 of 37
`
`681
`
`32
`
`32
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 32 of 37
`
`US 2006/0026521 Al
`
`
`
`
`
`
`
` FIG. 23C
`
`
`681
`
`681
`
`J
`
`576
`
`FIG. 23D
`
`33
`
`33
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 33 of 37
`
`US 2006/0026521 Al
`
`512
`
`520
`
`Sooorvtooy |
`
`
`
`
`
`
`
`Y oro
`
`
`
`
`
`
`
`
`
`FIG. 25A
`
`Yi
`
`000
`
`WORD PROCESSOR
`
`512
`
`CO
`
`
`
`4 fIl520
`
`576A
`
`
`CL
`Oo
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`730
`
`34
`
`34
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 34 of 37
`
`US 2006/0026521 Al
`
`fnnereneeecencesrsereesterresreterseceetseenapensecnneenenssesssnecsssocsrscescconscoesssesssessseenccensensneneencenescentneeateny
`
`512
`
`©]
`yl520
`
`}i
`
`576A
`
`
`
`
`
`
`
`
`
`
`730
`
`eR
`
`576B
`
`512
`
`an
`
`Z|)520
`
`576A
`
`A
`
`
`
`
`
`!
`
`730
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`J
`
`m—
`
`35
`
`35
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 35 of 37
`
`US 2006/0026521 Al
`
`750~
`
`752
`
`754
`
`756
`
`758
`
`PRESENT VIRTUAL SCROLL
`WHEEL
`
`
`
`DETECT PRESENCE OF
`FINGER(S) ON TOUCHSCREEN
`
`SET INITIAL POSITION OF
`FINGER(S) ON VIRTUAL SCROLL
`WHEEL
`
`
`THE VIRTUAL SCROLL WHEEL
`
`GENERATE ROTATE SIGNAL
`WHEN FINGER(S) MOVES ABOUT
`
`FIG. 26
`
`36
`
`36
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 36 of 37
`
`US 2006/0026521 Al
`
`762
`
`520
`
`512
`760a)oC)
`764 [TS
`
`U__AST
`
`FIG. 27A
`
`
`
`FIG. 27B
`
`37
`
`37
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 37 of 37
`
`US 2006/0026521 Al
`
`FIG. 27C
`
`512
`
`520
`
`
`
`
`
`
`
`(—
`
`
`
`FIG. 27D
`
`38
`
`38
`
`

`

`US 2006/0026521 Al
`
`Feb. 2, 2006
`
`GESTURES FOR TOUCH SENSITIVE INPUT
`DEVICES
`
`BACKGROUNDOF THE INVENTION
`
`[0001]
`
`1. Field of the Invention
`
`[0002] The present invention relates generally to gesturing
`associated with touch sensitive devices.
`
`[0003]
`
`2. Description of the Related Art
`
`[0004] There exist today manystyles of input devices for
`performing operations in a computer system. The operations
`generally correspond to moving a cursor and makingselec-
`tions on a display screen. The operations may also include
`paging,
`scrolling, panning, zooming, etc. By way of
`example, the input devices may include buttons, switches,
`keyboards, mice, trackballs, touch pads, joy sticks, touch
`screens and the like. Each of these devices has advantages
`and disadvantages that are taken into account when design-
`ing the computer system.
`
`[0005] Buttons and switches are generally mechanical in
`nature and provide limited control with regards to the
`movement of
`the cursor and making selections. For
`example, they are generally dedicated to moving the cursor
`in a specific direction (e.g., arrow keys) or to making
`specific selections (e.g., enter, delete, number, etc.).
`
`In mice, the movement of the input pointer corre-
`[0006]
`sponds to the relative movements of the mouse as the user
`moves the mouse along a surface. In trackballs, the move-
`ment of the input pointer correspondsto the relative move-
`ments of a ball as the user movesthe ball within a housing.
`Mice and trackballs also include one or more buttons for
`making selections. Mice mayalso include scroll wheels that
`allow a user to move through the GUI by simply rolling the
`wheel forward or backward.
`
`[0007] With touch pads, the movementof the input pointer
`corresponds to the relative movements of the user’s finger
`(or stylus) as the finger is moved along a surface of the touch
`pad. Touch screens, on the other hand,are a type of display
`screen that has a touch-sensitive transparent panel covering
`the screen. When using a touch screen, a user makes a
`selection on the display screen by pointing directly to GUI
`objects on the screen (usually with a stylus or finger).
`
`In order to provide additionally functionality, ges-
`[0008]
`tures have been implemented with some of these input
`devices. By way of example, in touch pads, selections may
`be made whenone or more taps are detected on the surface
`of the touch pad. In somecases, any portion of the touch pad
`may be tapped, and in other cases a dedicated portion of the
`touch pad maybe tapped. In additionto selections, scrolling
`may be initiated by using finger motion at the edge of the
`touch pad.
`
`[0009] U.S. Pat. Nos. 5,612,719 and 5,590,219, assigned
`to Apple Computer, Inc. describe some other uses of ges-
`turing. U.S. Pat. No. 5,612,719 discloses an onscreen button
`that is responsive to at least two different button gestures
`made on the screen on or near the button. U.S. Pat. No.
`5,590,219 discloses a method for recognizing an ellipse-type
`gesture input on a display screen of a computer system.
`
`placing four fingers on the touch pad so that the scrolling
`gesture is recognized and thereafter moving these fingers on
`the touch pad to perform scrolling events. The methods for
`implementing these advanced gestures, however, has several
`drawbacks. By way of example, once the gesture isset, it
`cannot be changed until the user resets the gesture state. In
`touch pads, for example,if four fingers equals scrolling, and
`the user puts a thumb down after the four fingers are
`recognized, any action associated with the new gesture
`including four fingers and the thumb will not be performed
`until the entire hand is lifted off the touch pad and put back
`downagain (e.g., reset). Simply put, the user cannot change
`gesture states midstream. Along a similar vein, only one
`gesture may be performed at any given time. That 1s,
`multiple gestures cannot be performed simultaneously.
`
`[0011] Based on the above, there is a need for improve-
`ments in the way gestures are performed on touchsensitive
`devices.
`
`SUMMARY OF THE INVENTION
`
`to a
`in one embodiment,
`[0012] The invention relates,
`computer implemented method for processing touch inputs.
`The method includes reading data from a multipoint touch
`screen. The data pertains to touch input with respect to the
`touch screen. The method also includesidentifying at least
`one multipoint gesture based on the data from the multipoint
`touch screen.
`
`[0013] The invention relates, in another embodimentto a
`gestural method. The method includes detecting multiple
`touchesat different points on a touch sensitive surface at the
`same time. The method also includes segregating the mul-
`tiple touches into at
`least
`two separate gestural
`inputs
`occurring simultaneously. Each gestural input has a different
`function such as zooming, panning, rotating and the like.
`
`[0014] The invention relates, in another embodimentto a
`gestural method. The method includes concurrently detect-
`ing a plurality of gestures that are concurrently performed
`with reference to a touch sensing device. The method also
`includes producing different commands for each of the
`gestures that have been detected.
`
`[0015] The invention relates, in another embodimentto a
`gestural method. The method includes displaying a graphi-
`cal image on a display screen. The method also includes
`detecting a plurality of touches at the same time on a touch
`sensitive device. The method further includes linking the
`detected multiple touches to the graphical image presented
`on the display screen.
`
`(0016] The inventionrelates, in another embodiment to a
`method of invoking a user interface element on a display via
`a multipoint
`touch screen of a computing system. The
`method includes detecting and analyzing the simultaneous
`presence of two or more objects in contact with the multi-
`point touch screen. The method also includes selecting a
`user interface tool, from a plurality of available tools, to
`display on a display for interaction by a user of the com-
`puting system based at least in part
`the analyzing. The
`method further includes controlling the interface tool based
`at least in part on the further movement of the objects in
`relation to the multipoint touch screen.
`
`In recent times, more advanced gestures have been
`[0010]
`implemented. For example, scrolling may be initiated by
`
`{0017] The inventionrelates, in another embodiment, to a
`touch-based method. The method includes detecting a user
`
`39
`
`39
`
`

`

`US 2006/0026521 Al
`
`Feb. 2, 2006
`
`input that occurs over a multipoint sensing device. The user
`input includes one or more inputs. Each input has a unique
`identifier. The method also includes, during the user input,
`classifying the user input as a tracking or selecting input
`when the user input includes one unique identifier or a
`gesture input when the user input includes at
`least two
`unique identifiers. The method further includes performing
`tracking or selecting during the user input when the user
`input
`is classified as a tracking or selecting input. The
`method additionally includes performing one or more con-
`trol actions during the user input when the user input is
`classified as a gesturing input. The control actions being
`based at least in part on changes that occur between the at
`least two unique identifiers.
`
`[0018] The invention relates, in another embodiment, to a
`touch-based method. The method includes outputting a GUI
`on a display. The method also includes detecting a user input
`on a touch sensitive device. The method further includes
`
`analyzing the user input for characteristics indicative of
`tracking, selecting or a gesturing. The method additionally
`includes categorizing the user input as a tracking, selecting
`or gesturing input. The method further includes performing
`tracking or selecting in the GUI when the user input is
`categorized as a tracking or selecting input. Moreover, the
`method includes performing contro] actions in the GUI
`when the user input is categorized as a gesturing inpul, the
`actions being based on the particular gesturing input.
`
`[0019] The invention relates, in another embodiment, to a
`touch-based method. The method includes capturing an
`initial touch image. The method also includes determining
`the touch mode based on the touch image. The method
`further includes capturing the next touch image. The method
`further includes determining if the touch mode changed
`between the initial and next touch images. The method
`additionally includes, if the touch mode changed,setting the
`next touch imageas the initial touch image and determining
`the touch mode based on the new initial
`touch image.
`Moreover, the method includes, if the touch mode stayed the
`same, comparing the touch images and performing a control
`function based on the comparison.
`
`[0020] The invention relates, in another embodiment, to a
`computer implemented method for processing touch inputs.
`The method includes reading data from a touch screen. The
`data pertaining to touch input with respect to the touch
`screen, and the touch screen having a multipoint capability.
`The method also includes converting the data to a collection
`of features. The method further includes classifying the
`features and grouping the features into one or more feature
`groups. The method additionally includes calculating key
`parameters of the feature groups and associating the feature
`groups to user interface elements on a display.
`
`[0021] The invention relates, in another embodiment, to a
`computer implemented method. The method includes out-
`putting a graphical image. The methodalso includesreceiv-
`ing a multitouch gesture input over the graphical image. The
`method further includes changing the graphical image based
`on and in unison with multitouch gesture input.
`
`[0022] The invention relates, in another embodiment, to a
`touch based method. The method includes receiving a ges-
`tural input over a first region. The method also includes
`generating a first command when the gestural
`input
`is
`received over the first region. The method further includes
`
`receiving the same gestural input over a second region. The
`method additionally includes generating a second command
`when the same gestural input is received over the second
`region. The second command being different than the first
`command.
`
`[0023] The invention relates, in another embodiment,to a
`method for recognizing multiple gesture inputs. The method
`includes receiving a multitouch gestural stroke on a touch
`sensitive surface. The multitouch gestural stroke maintain-
`ing continuous contact on the touch sensitive surface. The
`method also includes recognizinga first gesture input during
`the multitouch gestural stroke. The method further includes
`recognizing a second gesture input during the multitouch
`gestural stroke.
`
`[0024] The invention relates, in another embodiment,to a
`computer
`implemented method. The method includes
`detecting a plurality of touches on a touch sensing device.
`The method also includes forming one or more touch groups
`with the plurality of touches. The method further includes
`monitoring the movement of and within each of the touch
`groups. The method additionally includes generating control
`signals when the touches within the touch groups are moved
`or when the touch groups are movedin their entirety.
`
`It should be noted that in each of the embodiments
`[0025]
`described above, the methods may be implemented using a
`touch based input device such as a touch screen or touch pad,
`moreparticularly a multipoint touch based input device, and
`even more particularly a multipoint touch screen. It should
`also be noted that
`the gestures, gesture modes, gestural
`inputs, etc. may correspondto any of those described below
`in the detailed description. For example, the gestures may be
`associated with zooming, panning,
`scrolling,
`rotating,
`enlarging,floating controls, zoomingtargets, paging, inertia,
`keyboarding, wheeling, and/or the like.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0026] The invention will be readily understood by the
`following detailed description in conjunction with the
`accompanying drawings, wherein like reference numerals
`designate like structural elements, and in which:
`
`{0027] FIG.1 isa block diagram of a computer system,in
`accordance with one embodiment of the present invention.
`
`[0028] FIG.2 is a multipoint processing method,in accor-
`dance with one embodiment of the present invention.
`
`[0029] FIGS. 3A and B illustrate an image, in accordance
`with one embodimentof the present invention.
`
`[0030] FIG.4 illustrates a group offeatures, in accordance
`with one embodimentof the present invention.
`
`[0031] FIG.5 is a parameter calculation method, in accor-
`dance with one embodiment of the present invention.
`
`[0032] FIGS. 6A-6Hillustrate a rotate gesture, in accor-
`dance with one embodimentof the present invention.
`
`[0033] FIG. 7 is a diagram of a touch-based method, in
`accordance with one embodiment of the present invention.
`
`[0034] FIG. 8 is a diagram of a touch-based method, in
`accordance with one embodiment of the present invention.
`
`[0035] FIG. 9 is a diagram of a touch-based method, in
`accordance with one embodiment of the present invention.
`
`40
`
`40
`
`

`

`US 2006/0026521 Al
`
`Feb. 2, 2006
`
`[0036] FIG. 10 is a diagram of a zoom gesture method, in
`accordance with one embodiment of the present invention.
`
`[0037] FIGS. 11A-11J illustrates a zooming sequence, in
`accordance with one embodimentof the present invention.
`
`[0038] FIG. 12 is a diagram of a pan method, in accor-
`dance with one embodiment of the present invention.
`
`[0039] FIGS. 13A-13D illustrate a panning sequence, in
`accordance with one embodimentof the present invention.
`
`in
`[0040] FIG. 14 is a diagram of a rotate method,
`accordance with one embodimentof the present invention.
`
`[0041] FIGS. 15A-15C illustrate a rotating sequence, in
`accordance with one embodimentof the present invention.
`
`FIG.16 is a diagram of a GUI operational method,
`[0042]
`in accordance with one embodimentof the present inven-
`tion.
`
`floating control
`a
`[0043] FIGS. 17A-17E illustrate
`sequence, in accordance with one embodimentofthe present
`invention.
`
`FIG.18 is a diagram of a GUI operational method,
`[0044]
`in accordance with one embodimentof the present inven-
`tion.
`
`zooming target
`a
`[0045] FIGS. 19A-19D illustrate
`sequence, in accordance with one embodimentofthe present
`invention.
`
`FIG.20 is a diagram of a GUI operational method,
`[0046]
`in accordance with one embodimentof the present inven-
`tion.
`
`[0047] FIGS. 21A-21E illustrate a page turning sequence,
`in accordance with one embodimentof the present inven-
`tion.
`
`FIG.22 is a diagram of a GUI operational method,
`[0048]
`in accordance with one embodimentof the present inven-
`tion.
`
`[0049] FIGS. 23A-23Dillustrate an inertia sequence, in
`accordance with one embodimentof the present invention.
`
`FIG.24 is a diagram of a GUI operational method,
`[0050]
`in accordance with one embodimentof the present inven-
`tion.
`
`[0051] FIGS. 25A-25D illustrates a keyboard sequence,in
`accordance with one embodimentof the present invention.
`
`FIG.26 is a diagram of a GUI operational method,
`[0052]
`in accordance with one embodimentof the present inven-
`tion.
`
`scroll wheel
`a
`27A-27D illustrates
`[0053] FIGS.
`sequence, in accordance with one embodimentofthe present
`invention.
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`
`[0054] The invention generally pertains to gestures and
`methods of implementing gestures with touch sensitive
`devices. Examples of touch sensitive devices include touch
`screens and touch pads. One aspect of the invention relates
`to recognizing at least two simultaneously occurring ges-
`tures. Another aspect of the inventionrelates to displaying a
`
`graphical image and linking different touches that occur to
`the graphical image. Another aspect of the invention relates
`to immediately recognizing gestures so that actions associ-
`ated with the gestures can be implemented at the same time.
`Another aspect of the invention relates to changing a dis-
`played image based on and in unison with a gestural input,
`1.¢., the displayed image continuously changes with changes
`in the gestural input such that the displayed image continu-
`ously follows the gestural
`input. Another aspect of the
`invention relates to implementing an input mode based on
`the numberof fingers (or other object) in contact with the
`input device. Another aspect of the invention relates to
`providing region sensitivity where gestures mean different
`things when implemented over different areas of the input
`device. Another aspect of the invention relates to changing
`an input while making continuous contact with the touch
`sensitive surface of the touch sensitive device.
`
`[0055] These and other aspects of the invention are dis-
`cussed below with reference to FIGS. 1-27. However, those
`skilled in the art will readily appreciate that the detailed
`description given herein with respect to these figures is for
`explanatory purposes as the invention extends beyond these
`limited embodiments.
`
`[0056] FIG. 1 is a block diagram of an exemplary com-
`puter system 50, in accordance with one embodimentof the
`present invention. The computer system 50 may correspond
`to a personal computer system, such as a desktops, laptops,
`tablets or handheld computer. The computer system may
`also correspond to a computing device, such as a cell phone,
`PDA,dedicated media player, consumer electronic device,
`and the like.
`
`[0057] The exemplary computer system 50 shown in FIG.
`1 includes a processor 56 configured to execute instructions
`and to carry out operations associated with the computer
`system 50. For example, using instructions retrieved for
`example from memory, the processor 56 may control the
`reception and manipulation of input and output data between
`components of the computing system 50. The processor 56
`can be implemented on a single-chip, multiple chips or
`multiple electrical components. For example, various archi-
`tectures can be used for the processor 56, including dedi-
`cated or embedded processor, single purpose processor,
`controller, ASIC, and so forth.
`
`In most cases, the processor 56 together with an
`[0058]
`operating system operates to execute computer code and
`produce and use data. Operating systemsare generally well
`knownand will not be described in greater detail. By way of
`example,
`the operating system may correspond to OS/2,
`DOS, Unix, Linux, Palm OS, and the like. The operating
`system can also be a special purpose operating system, such
`as maybe used for limited purpose appliance-type comput-
`ing devices. The operating system, other computer code and
`data may reside within a memoryblock 58that is operatively
`coupled to the processor 56. Memory block 58 generally
`provides a place to store computer code and data that are
`used by the computer system 50. By way of example, the
`memory block 58 may include Read-Only Memory (ROM),
`Random-Access Memory (RAM), hard disk drive and/orthe
`like. The information could also reside on a removable
`
`storage medium and loadedor installed onto the computer
`system 50 when needed. Removable storage mediums
`include, for example, CD-ROM, PC-CARD, memorycard,
`floppy disk, magnetic tape, and a network component.
`
`41
`
`41
`
`

`

`US 2006/0026521 Al
`
`Feb. 2, 2006
`
`[0059] The computer system 50 also includes a display
`the touch screen 70 is
`touchpads, mice, etc. For one,
`device 68 that is operatively coupled to the processor 56.
`positioned in front of the display 68 and therefore the user
`The display device 68 may bealiquid crystal display (LCD)
`can manipulate the GUI 69 directly. For example, the user
`(e.g., active matrix, passive matrix and the like). Alterna-
`can simply place their finger over an object to be controlled.
`tively, the display device 68 may be a monitor such as a
`In touch pads, there is no one-to-one relationship such as
`monochromedisplay, color graphics adapter (CGA)display,
`this. With touchpads, the touchpad is placed away from the
`enhanced graphics adapter (EGA) display, variable-graph-
`display typically in a different plane. For example,
`the
`ics-array (VGA)display, super VGA display, cathode ray
`display is typically located in a vertical plane and the
`tube (CRT), and the like. The display device may also
`touchpad is typically located in a horizontal plane. This
`makes its use less intuitive, and therefore more difficult
`correspond to a plasma display or a display implemented
`with electronic inks.
`when compared to touch screens. In addition to being a
`touch screen, the input device 70 can be a multipoint input
`device. Multipoint input devices have advantages over con-
`ventional singlepoint devices in that they can distinguish
`more than one object (finger). Singlepoint devices are sim-
`ply incapable of distinguishing multiple objects. By way of
`example, a multipoint touch screen, which can be used
`herein, is shown and described in greater detail in copending
`and commonly assigned U.S. patent application Ser. No.
`10/840,862, which is hereby incorporated herein by refer-
`ence.
`
`[0060] The display device 68 is generally configured to
`display a graphical user interface (GUI) 69 that provides an
`easy to use interface between a user of the computer system
`and the operating system or application running thereon.
`Generally speaking, the GUI 69 represents, programs,files
`and operational options with graphical images. The graphi-
`cal
`images may include windows, fields, dialog boxes,
`menus, icons, buttons, cursors, scroll bars, etc. Such images
`may be arranged in predefined layouts, or may be created
`dynamically to serve the specific actions being taken by a
`user. During operation,
`the user can select and activate
`various graphical images in order to initiate functions and
`tasks associated therewith. By way of example, a user may
`select a button that opens, closes, minimizes, or maximizes
`a window,or an icon that launches a particular program. The
`GUI 69 can additionally or alternatively display informa-
`tion, such as non interactive text and graphics, for the user
`on the display device 68.
`
`[0061] The computer system 50 also includes an input
`device 70 that is operatively coupled to the processor 56.
`The input device 70 is configured to transfer data from the
`outside world into the computer system 50. The input device
`70 may for example be used to perform tracking and to make
`selections with respect to the GUI 69 on the display 68. The
`input device 70 mayalso be used to issue commandsin the
`computer system 50. The input d