as) United States
`a2) Patent Application Publication 10) Pub. No.: US 2008/0036743 Al
`(43) Pub. Date: Feb. 14, 2008
`
`Westermanet al.
`
`US 20080036743A1
`
`(54)
`
`(75)
`
`GESTURING WITH A MULTIPOINT
`SENSING DEVICE
`
`Inventors: Wayne C. Westerman, San Francisco,
`CA (US); Steven P. Hotelling, San
`Jose, CA (US); Myra Haggerty, San
`Mateo, CA (US); Bas Ording, San
`Francisco, CA (US); Nima Parivar,
`South San Francisco, CA (US); Duncan
`Robert Kerr, San Francisco, CA (US)
`
`Correspondence Address:
`APPLE C/O MORRISON AND FOERSTER
`sLLP
`LOS ANGELES
`555 WEST FIFTH STREET SUITE 3500
`LOS ANGELES, CA 90013-1024 (US)
`
`(73)
`
`Assignee: Apple Computer, Inc.
`
`(21)
`
`Appl. No.:
`
`11/700,636
`
`(22)
`
`Filed:
`
`Jan. 31, 2007
`
`Related U.S. Application Data
`
`(63) Continuation-in-part of application No. 10/903,964,
`filed on Jul. 30, 2004.
`Continuation-in-part of application No. 11/015,434,
`filed on Dec. 17, 2004, which is a continuation of
`application No. 09/236,513, filed on Jan. 25, 1999,
`now Pat. No. 6,323,846.
`
`(60) Provisional application No. 60/072,509, filed on Jan.
`26, 1998.
`
`Publication Classification
`
`(51)
`
`Int. CL
`(2006.01)
`GO6F 3/041
`(52) U.S. CMe
`ecsccsssessssessecestensssessesestenesinseevesee 345/173
`
`ABSTRACT
`(57)
`Methods and systems for implementing gestures with sens-
`ing devices are disclosed. More particularly, methods and
`systemsrelated to gesturing with multipoint sensing devices
`are disclosed.
`
`10 —.
`
`START
`
`12
`
`PROVIDE MULTIPOINT SENSING
`DEVICE
`
`
`
`
`STANDBY
`
`YES
`
` DETERMINE CHORD ASSOCIATED
`
`WITH TOUCH
`
`
`DETERMINE GESTURE SET
`ASSOCIATING ONE OR MORE
`ACTIONS TO ONE OR MORE
`
`
`GESTURE EVENTS
`
`MONITOR TOUCH FOR GESTURE
`EVENT
`
`
`
`
`
`
`
`" PERFORM ACTION ASSOCIATED
`WITH GESTURE EVENT
`
`
`
`
`
`24
`
`1
`
`APPLE-1034
`
`1
`
`APPLE-1034
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 1 of 49
`
`US 2008/0036743 Al
`
`10 —,
`
`START
`
`
`
`PROVIDE MULTIPOINT SENSING
`DEVICE
`
`12
`
`14
`
`NO
`
`STANDBY
`
`YES
`
`
`
`DETERMINE CHORD ASSOCIATED
`WITH TOUCH
`
`
`
`
`
`DETERMINE GESTURE SET
`
`ASSOCIATING ONE OR MORE
`
`ACTIONS TO ONE OR MORE
`GESTURE EVENTS
`
`
`
`
`MONITOR TOUCH FOR GESTURE
`EVENT
`
`YES
`
`- PERFORM ACTION ASSOCIATED
`
`WITH GESTURE EVENT
`
`FIG. 1
`
`16
`
`18
`
`20
`
`24
`
`2
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 2 of 49
`
`US 2008/0036743 Al
`
`STANDBY
`
`54
`
`mo
`
`
`56
` 58
`
`
`DETERMINE GESTURE SET FOR TOUCH
`
`MONITOR TOUCH FOR GESTURE EVENT
`
`INITIATE COMMAND WHEN GESTURE EVENT
`IS PERFORMED
`
`FIG. 2
`
`60
`
`|
`
`62
`_
`RECOGNIZE ARRANGEMENT OF CONTACTS )
`
`64
`
`
` 66
`
`COMPARE ARRANGEMENTOF CONTACTS
`TO A STORED SET OF ARRANGEMENT OF
`CONTACTS
`
`LOAD GESTURE SET ASSIGNED TOINITIAL
`ARRANGEMENTOF CONTACTS
`
`FIG. 3
`
`3
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 3 of 49
`
`US 2008/0036743 Al
`
`70 —~,
`
`.
`
`DETECT TOUCH
`
`DETERMINE LOCATION OF TOUCH
`
`DETERMINE ARRANGEMENT OF CONTACTS
`
`DETERMINE ACTIVE APPLICATION
`
`.
`
`DETERMINE STATE OF ACTIVE APPLICATION
`
`a
`
`SET GESTURE SET BASED ON THE ABOVE
`
`FIG. 4
`
`72
`
`74
`
`76
`
`78
`
`80
`
`82
`
`4
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 4 of 49
`
`US 2008/0036743 Al
`
`100~~ START
`
`102
`
`106
`
`L—
`
`108
`
`110
`
`
`COMPARE RECOGNIZED ARRANGEMENTOF
`CONTACTS TO STORED ARRANGEMENT OF
`
`CONTACTS IN GESTURE MAP
`
`112
`
`116
`
`118
`
`LOAD GESTURE SET ASSOCIATED WITH
`INITIAL ARRANGEMENT OF CONTACTS
`
`MONITOR TOUCH FOR GESTURE EVENTS
`ASSOCIATED WITH THE GESTURE SET
`
`
`
`INITIATE COMMAND ASSOCIATED WITH
`
`GESTURE EVENT WHEN GESTURE EVENTIS
`PERFORMED.
`
`FIG. 5
`
`5
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 5 of 49
`
`US 2008/0036743 Al
`
`120—
`
`122
`
`DETECT TOUCH
`
`DETERMINE CHORD ASSOCIATED WITH
`TOUCH
`
` 124
`
`DETERMINE INPUT FUNCTIONALITY
`ASSOCIATED WITH CHORD
`
`126
`
`128
`
`130
`
`LOAD GESTURE SET ASSOCIATED WITH
`INPUT FUNCTIONALITY
`
`INITIATE ACTIONS ASSOCIATED WITH
`GESTURE EVENTS WHEN GESTURE EVENTS
`
`ARE PERFORMED
`
`FIG. 6
`
`6
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 6 of 49
`
`US 2008/0036743 Al
`
`142
`
`144
`
`146
`
`148
`
`150
`
`152
`
`DETECT TOUCH
`
`RECOGNIZE ARRANGEMENT OF CONTACTS
`
`DETERMINE CHORD ASSOCIATED WITH
`ARRANGEMENTOF CONTACTS
`
`LOAD GESTURE SET ASSOCIATED WITH
`CHORD
`
`MONITOR TOUCH FOR GESTURE EVENTS
`PERFORMED BY ANY. ARRANGEMENTOF
`
`
`
`CONTACTSDURINGTHETOUCH ;
`WITH GESTURE EVENT
`
`154
`
`INITIATE CONTROL/COMMAND ASSOCIATED
`
`FIG. 7
`
`7
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 7 of 49
`
`US 2008/0036743 Al
`
`_ 160
`
`START
`
`
`
`
`
`PERFORM POINTING
`
`IN ACCORDANCE
`WITH MOTION
`
`164
`
`168 PERFORM
`
`
`DRAGGING IN
`
`ACCORDANCE WITH
`MOTION
`
`
`
`
`PERFORM
`172
`
`
`THUMB +2
`SECONDARY
`
`ADJ. FINGERS
`DRAGGING IN
`
`
`
`
`ACCORDANCE WITH
`
`MOTION
`
`FIG. 8
`
`8
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 8 of 49
`
`US 2008/0036743 Al
`
`180 —
`
`sea
`
`DETERMINE BASE CHORD
`
`186
`
`182
`
`DETECT MOTION
`
`DETECT NEW
`FIRSTDIGIT
`
`DETECT NEW
`SECOND DIGIT
`
`190
`
`188
`
`194
`
`
`
`
`
`SECOND.
`PERFORM FIRST
`PERFORM COMMAND EACH)|COMMAND EACH
`
`POINTING
`TIME DIGIT IS -
`MEOT
`TIME DIGIT IS
`DETECTED
`
`192
`
`FIG. 9
`
`9
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 9 of 49
`
`US 2008/0036743 Al
`
`200~
`
`
`
`202
`
`DETECT TOUCH
`
`RECOGNIZE ARRANGEMENT
`OF CONTACTS
`
`204
`
`
`
`
`206 208
`
`
`
`RECOGNIZE CHORD ASSOCIATED WITH
`ARRANGEMENTOF CONTACTS
`
`LOAD GESTURE SET ASSOCIATED WITH
`CHORD
`
`MONITOR TOUCH FOR A GESTURE EVENT
`PERFORMED BY ANY ARRANGEMENTOF
`CONTACTS
`
`INITIATE
`CONTROL/
`COMMAND
`ASSOCIATED
`WITH GESTURE
`EVENT
`
`
`
`
`
`
`FIG. 10
`
`10
`
`10
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 10 of 49
`
`US 2008/0036743 Al
`
`START
`
`222
`
`220
`
`
`
`PERFORM POINTING
`IN ACCORDANCE
`WITH MOTION
`
`224
`
`228
`
`232
`
`226
`
`PERFORM
`DRAGGING IN
`ACCORDANCE WITH
`MOTION
`
`230
`
`FINGERS
`
`
`
`2 ADJ. NON
`
`
`
`PERFORM
`SECONDARY
`DRAGGING IN
`ACCORDANCE WITH
`MOTION
`
`3 OR 4
`FINGERS
`:
`
`423
`
`
`
`PERFORM
`SCROLLING IN
`ACCORDANCE WITH
`MOTION
`
`FIG. 11
`
`11
`
`11
`
`

`

`\ START
`242<n STANDBY
`
`
`
`
`244
`
`
`
`INITIATE POINTING
`UNTIL LIFT OFF OF
`ALL FINGERS
`
`
`
`LIFT AND TAP1
`FINGER DURING
`POINTING
`
`CLICK
`
`246
`
`200
`
`Patent Application Publication Feb. 14,2008 Sheet 11 of 49
`
`US 2008/0036743 Al
`
`240
`
`248
`
`252
`
`253
`
`
`
`T AND TOUCRS
`FINGERS DURING
`POINTING
`
`
`
`FT AND TOUCH’
`
`FINGERS DURING
`
`DRAGGING
`
`
`
`FIG. 12
`
`12
`
`12
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 12 of 49
`
`US 2008/0036743 Al
`
`260
`
`
`
`262
`
`DETECT TOUCH
`
`
`
`RECOGNIZE SIDE OF PINKY
`
`264
`
`
`
`
`
`
`DETECT SWIPE WITH SIDE OF PINKY
`
`266
`
`268
`
`
`
`
`PERFORM CONTROL/COMMAND
`
`
`ASSOCIATED WITH SWIPING PINKY
`
`
`FIG. 13
`
`13
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 13 of 49
`
`US 2008/0036743 Al
`
`START (
`
`280 .
`
`282
`
`PROVIDE GESTURE SET MAPPING UPWARD SLIDE TO
`EXPOSE AND DOWNWARD SLIDE TO DASHBOARD
`
`RECOGNIZE BASE CHORD
`
`28
`
`4
`
`UPWARD
`SLIDE
`
`OPEN
`EXPOSE
`
`DOWNWARD
`
`DOWNWARD
`
`296
`
`OPEN
`DASHBOARD
`
`298
`
`UPWARD
`SLIDE
`
`CHORD
`CHANGE
`
`CLOSE
`EXPOSE
`
`
`
`
`
`
`
`
`
`
`CLOSE
`DASHBOARD
`
`300
`
`
`
`
`PERFORM POINTING
`
`WITH ALTERED
`
`CHORD
`
`
`
`FIG. 14
`
`CHORD
`
`CHANGE
`
`14
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 14 of 49
`
`US 2008/0036743 Al
`
`310 —™
`
`
`
`
`
`DETECT TOUCH
`
`
`
`DETERMINE ARRANGEMENT OF CONTACTS
`
`
`
`PERFORM ACTION BASED ON
`ARRANGEMENT OF CONTACTS
`
`FIG. 15
`
`320 —{\
`
`312
`
`314
`
`
`
`
`
`316
`
`MONITOR TOUCH MOTION
`
`
`
`322
`
`324
`
`
`
`
`
`
`DIFFERENTIATE MOTION BETWEENA FLICK
`AND SWIPE
`
`328
`
`
`PERFORM FIRST
`PERFORM FIRST
`
`ACTION IF MOTION ISA
`ACTION IF MOTION
`
`
`IS A FLICK
`SWIPE
`
`FIG. 16
`
`15
`
`15
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 15 of 49
`
`US 2008/0036743 Al
`
`330 —‘\
`
`DETECT TOUCH
`
`
`
`IDENTIFY TWO CONTACTS
`
`
`
`
`
`
`
`
`
`
`332
`
`334
`
`336
`
`338
`
`MONITORING THE TOUCH FOR A ROTATION
`GESTURE EVENT
`
`INITIATE ROTATE COMMAND WHENA FIRST
`CONTACTIS STATIONARY AND A SECOND
`CONTACTIS ROTATED ABOUTTHE FIRST
`CONTACT
`
`FIG. 17
`
`16
`
`16
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 16 of 49
`
`US 2008/0036743 Al
`
`340 —,
`
`.
`
`PROVIDE MULTIPLE WINDOWS
`
`342
`
`DETECT POINTING MECHANISM OVERFIRST WINDOW
`
`MONITOR TOUCH TO DETERMINE IF CHORD ASSOCIATED WITH
`POINTING MECHANISM HAS BEEN PRESENTED
`
`INITIATE GESTURE SET ASSOCIATED WITH THE CHORD AND
`FIRST WINDOW
`
`
`
`PERFORM ACTIONS IN FIRST WINDOW WHEN GESTURE EVENTS
`ASSOCIATED WITH THE GESTURE SET ARE PERFORMED WHILE
`POINTING MECHANISM IS DETECTED OVER FIRST WINDOW
`
`DETECT POINTING MECHANISM OVER SECOND WINDOW
`
`344
`
`346
`
`348
`
`350
`
`352
`
`MONITOR TOUCH TO DETERMINE IF CHORD ASSOCIATED WITH
`POINTING MECHANISM HAS BEEN PRESENTED
`
`INITIATE SECOND GESTURE SET ASSOCIATED WITH THE CHORD
`
`354
`
`356
`
`
`AND SECOND WINDOW
`
`
`
`PERFORM ACTIONS IN SECOND WINDOW WHEN GESTURE EVENTS
`ASSOCIATED WITH THE SECOND GESTURE SET ARE PERFORMED
`WHILE POINTING MECHANISM IS DETECTED OVER SECOND WINDOW
`
`358
`
`FIG. 18
`
`17
`
`17
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 17 of 49
`
`US 2008/0036743 Al
`
`360 ~~
`
`362
`
`DETECT TOUCH
`
`RECOGNIZE ARRANGEMENT OF CONTACTS ASSOCIATED WITH
`FIRST HAND
`
`LOAD GESTURE SET BASED ON ARRANGEMENTOF CONTACTS
`ASSOCIATED WITH FIRST HAND
`
`RECOGNIZE SECOND ARRANGEMENTOF CONTACTS
`ASSOCIATED WITH SECOND HAND
`
`|
`
`CHANGE OR MODIFY GESTURE SET FOR FIRST ARRANGEMENTOF
`
`CONTACTS BASED ON SECOND ARRANGEMENTOF CONTACTS
`
`FIG. 19
`
`364
`
`366
`
`368
`
`370
`
`18
`
`18
`
`

`

`Patent Application Publication Feb. 14, 2008 Sheet 18 of 49
`
`US 2008/0036743 Al
`
`382
`
`384
`
`386
`
`388
`
`390
`
`402
`
` 404
`
`
`
`
`406
`
`408
`
`380i
`
`PROVIDE MOUSE AND TOUCH SURFACE ARE PROVIDED
`
`MONITOR MOUSE FOR MOUSE EVENTS
`
`SIMULTANEOUSLY MONITOR TOUCH SURFACE FOR GESTURE
`EVENTS
`
`PERFORM MOUSE OPERATIONS IN ACCORDANCE WITH
`MOUSE EVENTS
`
`SIMULTANEOUSLY PERFORM GESTURE OPERATIONS IN
`ACCORDANCE WITH GESTURE EVENTS
`
`FIG. 20
`
`400 —,
`
`|
`
`PROVIDE MOUSE AND TOUCH SURFACE ARE PROVIDED
`
`MONITOR MOUSE FOR MOUSE EVENTS
`
`SIMULTANEOUSLY MONITOR TOUCH SURFACE FOR
`ARRANGEMENT OF CONTACTS AND/OR GESTURE EVENTS
`
`CHANGE MOUSE MODE BASED ON ARRANGEMENTOF
`CONTACTS AND/OR GESTURE EVENTS
`
`
`
`
`
`FIG. 21
`
`19
`
`19
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 19 of 49
`
`US 2008/0036743 Al
`
`410~
`
`DETECT FIRST TOUCH ON MULTIPOINT SENSING SURFACE
`
`RECOGNIZE FIRST ARRANGEMENTOF CONTACTS
`(ASSOCIATED WITH FIRST HAND)
`.
`
`SIMULTANEOUSLY DETECT FIRST TOUCH ON MULTIPOINT
`SENSING SURFACE
`
`SIMULTANEOUSLY RECOGNIZE SECOND ARRANGEMENT OF
`CONTACTS (ASSOCIATED WITH SECOND HAND)
`
`LOAD GESTURE SET ASSOCIATED WITH FIRST ARRANGEMENT
`OF CONTACTS
`
`SIMULTANEOUSLY LOAD GESTURE SET ASSOCIATED WITH
`SECOND ARRANGEMENT OF CONTACTS
`
`MONITOR FIRST TOUCH FOR GESTURE EVENTS
`
`426
`
`SIMULTANEOUSLY MONITOR SECOND TOUCH FOR GESTURE
`EVENTS
`
`INITIATE ACTIONS ASSOCIATED WITH FIRST GESTURE EVENTS
`WHEN WHEN FIRST GESTURE EVENTS ARE PERFORMED
`
`430
`
`SIMULTANEOUSLYINITIATE ACTIONS ASSOCIATED WITH
`SECOND GESTURE EVENTS WHEN SECOND GESTURE EVENTS
`
`ARE PERFORMED
`
`412
`
`414
`
`416
`
`418
`
`420
`
`422
`
`424
`
`428
`
`FIG. 22
`
`20
`
`20
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 20 of 49
`
`US 2008/0036743 Al
`
`440_
`
`.
`
`DISPLAY GUI OBJECT
`
`DETECT MULTIPLE POINTERS AT DIFFERENT POINTS OF
`DISPLAYED OBJECT
`
`LOCK POINTERS TO DISPLAYED OBJECT
`
`MONITOR POSITION OF POINTERS RELATIVE TO THEIR
`LOCKED POSITION
`
`442
`
`444
`
`445
`
`446
`
`448
`
`MODIFY DISPLAYED GUI OBJECT WHEN ONE OR BOTH OF THE
`POINTERS IS MOVED RELATIVE TO THEIR LOCKED POSITION
`
`60 FIG. 23
`DETECT TOUCHpoETECTTOUCH Gey
`
`452
`
`RECOGNIZE FIRST ARRANGEMENT OF CONTACTS
`
`DETECT SECOND TOUCH
`
`SIMULTANEOUSLY RECOGNIZE SECOND ARRANGEMENT OF
`CONTACTS
`MONITOR FIRST TOUCH FOR GESTURE EVENTS
`
`SIMULT. MONITOR SECOND TOUCH FOR GESTURE EVENTS
`
`PERFORM FIRST POINTING OPERATIONS WITH FIRST
`ARRANGEMENTOF CONTACTS
`
`456
`
`458
`
`460
`462
`
`464
`
`466
`
`SIMULTANEOUSLY PERFORM SECOND POINTING OPERATIONS
`WITH SECOND ARRANGEMENTOF CONTACTS
`
`FIG. 24
`
`21
`
`21
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 21 of 49
`
`US 2008/0036743 Al
`
`470—
`
`DISPLAY GROUP OF SELECTABLE ITEMS
`
`DETECT FIRST POINTING MECHANISM OVER GROUP OF
`SELECTABLE ITEMS
`
`ACTIVATE ITEM UNDERNEATHFIRST POINTING MECHANISM
`
`SIMULTANEOUSLY DETECT SECOND POINTING MECHANISM
`OVER WINDOW
`
`APPLY SELECTED ITEM TO SECOND POINTING MECHANISM
`
`500
`
`OY
`
`FIG. 25
`
`DISPLAY GROUP OF MEDIA ITEMS
`
`DETECT POINTING MECHANISM OVER AN ITEM THEREBY
`HIGHLIGHTING THE THE ITEM
`
`DETECT TOUCH ON TOUCH SURFACE
`
`RECOGNIZE ARRANGEMENT OF CONTACTS ASSOCIATED WITH
`TOUCH
`
`INSERT METADATATIED TO THE RECOGNIZED ARRANGEMENT
`OF CONTACTSINTO THE HIGHLIGHTED ITEM
`
`472
`
`474
`
`476
`
`478
`
`480
`
`ss
`
`504
`
`506
`
`508
`
`510
`
`FIG. 26
`
`22
`
`22
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 22 of 49
`
`US 2008/0036743 Al
`
`520
`
`
`
`DISPLAY CONTROL OBJECT
`
`524
`
`DETECT TOUCH OVER CONTROL OBJECT
`
`RECOGNIZE ARRANGEMENTOF CONTACTS ASSOCIATED WITH
`TOUCH
`
`527
`
`YES
`
`530
`
`526
`
`
`IGNORE TOUCH
`
`FIG. 27
`
`532
`
`23
`
`
`MONITOR TOUCH FOR
`GESTURE EVENT
`ASSOCIATED WITH
`CONTROLLING THE
`CONTROL OBJECT
`
`
`
`
`
`
`
`
`
`APPLY ACTION
`
`
`ASSOCIATED WITH-
`GESTURE EVENT TO
`
`
`CONTROL OBJECT WHEN
`
`
`GESTURE EVENTIS
`PERFORMED
`
`
`23
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 23 of 49
`
`US 2008/0036743 Al
`
`
`
`
`
`:
`
`
`
`
`
`
`~
`
`PROVIDE LIST OF INPUT FUNCTIONS
`
`ASSIGN INPUT FUNCTIONS TO CHORDS
`
`THEREAFTER LINK INPUT FUNCTION TO CHORD WHEN CHORD
`IS RECOGNIZED
`
`542
`
`544
`
`
`
`
`
`546
`
`FIG. 28
`
`580\
`
`PROVIDE LIST OF INPUT FUNCTIONS
`
`PROVIDE COMMANDS ASSOCIATED WITH INPUT FUNCTIONS
`
`PROVIDE LIST OF CHORDS
`
`
`
`PROVIDE GESTURE EVENTS ASSOCIATED WITH CHORDS
`
`ASSIGN INPUT FUNCTION TO CHORD
`
`ASSIGN COMMANDSTO GESTURE EVENTS
`
`FIG. 30
`
`24
`
`582
` 584
`
` 586
`
`
`
` 592
`
`588
`
`590
`
`24
`
`

`

`US 2008/0036743 Al
`
`V6¢Sls
`
`SUYAONIA&
`
`
`
`NOILONNALNdNI
`
`_SUFONIA&
`
`
`
`NOILONN4LNdNI
`
`Patent Application Publication Feb. 14,2008 Sheet 24 of 49
`
`
`
`
`OSsYONoPWIN|A)PsoueapyYonopNV]IISeg
`puepuejys
`puewwoo|puewwoos|puewwosSHAD
`
` }@Sainjseg||uoloun yndu]
`
`
`
` puewwoos|puewwoo|puewwooPUBLULUOSPUBWILUOSpuewwuod
`
`
`aunysabainysebaunyseb
`
`
`ainjsebgunysebainjseb
`
`
`eainjsebainjsebainysab
`
`
`juanejusAeJUsAd
`
`
`juaAoqusnayueas
`
`
`yusAajuaneyuaaa
`
`suoloun-jyndu|josq
`
`
`jauedJO1JUOZDaun}say
`S|<8SWOSND>
`NidLNZOVPayNONZlg
`
`
`“NOWONASINANE
`NOILLONN4LONIi,SUYAONISZ
`
`
`NOILONNSLNdNI
`
`YAONIS|
`
`SPJOYD403ST)
`
`pedyoes)
`
`25
`
`25
`
`
`

`

`Patent Application Publication Feb. 14, 2008 Sheet 25 of 49
`
`US 2008/0036743 Al
`
`BIN}S9H
`
`
`
`asodxeesodxe
`
`
`
`doysepsuoneoydde
`
`\yeasodxe
`
`
`
`SYAONIALNAOVIOVNON¢@
`
`9gu0oU
`
`d6¢Sls
`
`
`
`
`
`
`
`yoyoAlepuovaspue|jO19S
`
`OSS
`
`SUON
`
`UIOd
`
`WUDAUOd
`
`]019$
`
`YOAepuosas*g||O19g
`
`Beiq9YOOAewud
`
`BeiqpueyoyArepuosas
`
`
`
`MOPUIAASACL
`
`
`
`SUOI}eJBdOJBSMOJGGar,
`
`
`
`pseoqyseqgasodxy|
`
`
`
`suonesedoBuryipy
`
`
`
`suoyeiadoail4
`
`PAJUOD
`
`
`
`onoyynysWsegpedyoes|puepueys
`
`
`
`
`
`<9Swojsny>
`
`Sp10YDJO1817
`
`WUSAUlod
`
`YAONIS+
`
`SYHAONIS2
`
`26
`
`26
`
`

`

`Patent Application Publication Feb. 14, 2008 Sheet 26 of 49
`
`US 2008/0036743 Al
`
`
`
`
`
`julod
`
`YOOAUOd
`
`||049S
`
`
`
`
`
`onopynyyoisegpedyoes)
`
`
`
`SUONb4JUODBIN}say
`
`OssYOI1DArepuosag9110195
`pJepueys
`
`
`bedpuesilMepuooes.<8SWO}SND>
`
`pieoquseg9esodxySpl0yuyjo1SIq
`suoesadO914YAONIA|suonesado
`
`
`06¢‘Sls
`
`pseoqusegpueasodxy
`
`SYAONIAF
`
`
`
`suoljeladoJesmolgGen\
`
`Bulyipy
`
`
`
`MOPUIAASACI.
`
`YOHONUlod
`
`SHAONIS2
`
`27
`
`
`
`
`
`yoljoAlepuodaspue||o10g
`
`27
`
`

`

`
`
`SUONbs}UODSIN}SaH
`
`Patent Application Publication Feb. 14, 2008 Sheet 27 of 49
`
`US 2008/0036743 Al
`
`OSS
`
`
`
`YONDAlepuodas*[019
`
`JUlod
`
`WNOAULOd
`
`O19S
`
`BeigpueyoDArepucsas
`
`
`
`MOPUIA\SAC
`
`
`
`suoneJedygJOSMolgGap
`
`
`
`suonesiedgBuiyipy
`
`psepueys
`
`pseoquseg9asodxyspsouyjo}sI]
`
`
`
`
`
`OnNoPIINWdISsegpedyoes)
`
`
`
`SYSONIALNADVrdvNON2
`
`9u0u
`
`SYFONIA&
`
`Beiqpuex09ABW
`
`pieoqusegpueasodxy
`
`SYAONIJ+
`
`“guoneiadoong
`
`
`
`MONAb+:@WAHL
`
`SYAONISé+SINHL
`
`auou
`
`28
`
`28
`
`

`

`Patent Application Publication Feb. 14, 2008 Sheet 28 of 49
`
`US 2008/0036743 Al
`
`ass
`
`yOQAlepuodag*f[1019S
`
`yulod
`
`yooAulod
`
`HO19S
`
`
`
`Beigpue¥9119Alepuoses
`
`
`
`MOPUIA\BACH
`
`
`
`suoyeladoJOsMolgGann
`
`
`
`
`
`anono1segpedyoel)
`
`
`
`QUONDAJUODBIN}JS95
`
`piepueys
`
`pueoquseq9asodxgSpJoYyDjoIst]
`
`WeyOajas
`
`d6¢Ola
`
`jaoueo
`
`pieoqyseqpueesodxy
`
`SYAONIS&
`
`YAONIS|+SANHL
`
`
`
`suonesedgajl4
`
`3U0U
`
`SYAONIS&+GAINHL
`
`auou
`
`29
`
`29
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 29 of 49
`
`US 2008/0036743 Al
`
`OSS
`
`41DAlepuadag9||049¢
`
`BeiqpueyaI1QArepuosas
`
`
`
`MOPUI/\ASAO/
`
`
`
`_.SUOHEIOGOJOSMOIGGAM:Xx:
`
`
`
`suoljeisdoBunipy
`
`
`
`SUONDIJUOD341Nn}s9s5)
`
`wlod
`
`AUNDAUICd
`
`lOs3$
`
`
`
`nonsIsegpedyoes|
`
`prepueys
`ofig|Tub1odsdowsSuloUSYHAONIS¢+GINNHL
`
`
`
`
`yewyoogysoujolpulSo
`pIeMIO}pwyoeqvesoeooSUOUTee
`Be1a19WalidAleuld=|cesuisno>
`
`pueoquseq9esodxySpJOYD40SI)
`
`suojesadQailsSYAONIS
`>_.-SYAONIE+BIWNHL”C)SpueWILUOZBuiipy
`ajij4|YSONISb+GANHL
`
`46¢“Old
`
`auou
`
`,
`
`SHAONIS&+GIINHL
`
`pieoquseqpueesodxg
`
`suojesado
`
`30
`
`30
`
`
`

`

`
`
`
`
`ossyOugAWepucsas¥|]OI0S
`
`I1049$
`
`bIJUOD
`
`Patent Application Publication Feb. 14, 2008 Sheet 30 of 49
`
`aiN}sag
`
`pieoqyuseqg9esodxgSPJOYDJOHS!)
`Beig9yOIDAlewud<}8SWO}SND>
`suojeledge2ajl4SYIFONIS¢suonesadgo
`
`
`
`
`yoUDAUIOdOnopNyoIsegpedyoes|puepueys
`
`US 2008/0036743 Al
`
`962‘Sls
`
`6eigpueyopArepucses
`
`
`
`MOPUIAASACI
`
`
`
`suoneJadgQJOSMOlgGa;vv
`
`Buyipy
`
`pueoquseqpueasodxy
`
`
`
`suolesado9ajl4
`
`YONI+GINNHL
`
`SYAONIA2+GINHL
`
`spuewwoyBulipy
`
`
`
`SYAONIS€+EGANHL
`
`auoUu
`
`
`
` we2BUOU-‘SUAONId7+ANHL-¢
`
`31
`
`31
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 31 of 49
`
`US 2008/0036743 Al
`
`600
`
`——
`
`,
`
`|
`
`PLACE SYSTEM IN GESTURE MAPPING MODE
`
`DETECT TOUCH
`
`STORE ARRANGEMENT OF CONTACTS ASSOCIATED WITH
`TOUCH
`
`MONITOR TOUCH FOR GESTURE EVENTS
`
`STORE GESTURE EVENTAND LINK TO ARRANGEMENT OF
`CONTACTS
`
`ASSIGN COMMAND TO THE ARRANGEMENT OF CONTACTS/
`GESTURE EVENT
`
`620
`
`FIG. 31
`OPEN COMMANDMENU
`
`DISPLAY LIST OF COMMANDS
`
`HIGHLIGHT A COMMAND WITHIN COMMAND MENU
`
`DETECT TOUCH
`
`STORE ARRANGEMENTOF CONTACTS ASSOCIATED WITH
`TOUCH
`
`MONITOR TOUCH FOR GESTURE EVENT
`.
`
`ASSIGN INITIAL ARRANGEMENT OF CONTACTS/GESTURE
`EVENTS TO HIGHLIGHTED COMMAND
`
`602
`
`604
`
`606
`
`608
`
`610
`
`612
`
`622
`504
`
`626
`
`628
`
`630
`
`632
`
`634
`
`FIG. 32
`
`32
`
`32
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 32 of 49
`
`US 2008/0036743 Al
`
`640_~
`
`DETECT TOUCH
`
`RECOGNIZE ARRANGEMENT OF CONTACTS
`
`SET GESTURE SET BASED AT LEAST IN PART ON
`ARRANGEMENT OF CONTACTS
`
`,
`
`DETECT PAUSE
`
`DISPLAY GESTURE SET SO THAT USER CAN VIEW COMMANDS
`AND GESTURE EVENTS ASSOCIATED THEREWITH
`
`REMOVE DISPLAYED GESTURE SET WHEN A GESTURE EVENT
`IS STARTED OR AFTER A PRESETTIME OUT
`
`660
`
`FIG. 33
`
`642
`
`644
`
`646
`
`648
`
`650
`
`652
`
`662
`
`
`
`
`COMMAND2
`COMMAND3
`667 ;COMMAND1
`
`
`C)
`>
`oe | COMMAND9
`
`
`
`664
`
`COMMAND 4 COMMAND 5
`
`COMMAND 6
`
`FIG. 34
`
`33
`
`33
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 33 of 49
`
`US 2008/0036743 Al
`
`680of
`
`ACTIVATE TOUCH SCREEN KEYBOARD
`
`DURING TYPING OPERATION, MONITOR KEY ENTRIES OVER
`KEYBOARD
`
`DURING TYPING OPERATIONS, MONITOR SURFACE FOR
`
`682
`
`684
`
`686
`
`FINGER SWIPES 690
`
`SPACE
`
`694
`
`698
`
`ENTER/RETURN
`
`702
`
`COMPLETE WORD
`
`<>
`
`<>
`o>
`
`34
`
`34
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 34 of 49
`
`US 2008/0036743 Al
`
`700 —\
`
`RATE EASIEST CHORD
`
`
`
`RATE MOST FREQUENT OPERATIONS
`
`702
`
`704
`
`
`
`
`
`
`
`
`GROUP OR MATCH EASIEST CHORDS WITH
`
`MOST FREQUENT OPERATIONS AND GROUP
`
`OR MATCH HARDEST CHORDS WITH LESS
`FREQUENT OPERATIONS
`
`706
`
`
`
`
`
`FINE TUNE EACH GROUP
`
`
`708
`
`FIG. 36
`
`35
`
`35
`
`

`

`2|4|4|8 3 SPEED
`
`[RELIABILITY|COMFORT|EASE[OVERALLcsys|5|5|5 aeeeeeeeeeeee 3|4|3s|5 3
`
`
` 2.75
`eeee
`2p2[|
`4.25[375_|3.25
`33[4a4.
`a
` [—SubEmap[>|s|2|[rap+72|=
`
`
`
`
`FINGERCOMBOGESTURESPACE
`2FING.DOWN|*|
`sube[| TAP
`
`om
`Ww
`©
`Cie le le le lei le |e |olsH
`LO La
`pL P ) ) Gy Le |
`OOOO /O/Os/O/O/OjOle
`2 |Z |2|2 |Z |Z |Z |2 |Z |Z |+
`Le
`fle
`fh ph PL Ly
`TIMI ININININ SO ST ors
`
`P|
`
`SPREAD
`
`36
`
`Patent Application Publication Feb. 14,2008 Sheet 35 of 49
`
`US 2008/0036743 Al
`
`FIG.37
`
`—a
`
`aE
`
`36
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 36 of 49
`
`US 2008/0036743 Al
`
`OPERATION
`
`EXPOSE
`
`37
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 37 of 49
`
`US 2008/0036743 Al
`
`
`
`OPEN FIRST PROGRAM
`
`ADJUST BASE GESTURE MAP IN ACCORDANCEWITH
`FIRST PROGRAM
`
`740 —,
`:.
`? PROVIDE BASE GESTURE MAP
`
`
`
`SECOND PROGRAM
`
`742
`
`744
`
`746
`
`748
`
`750
`
`OPEN SECOND PROGRAM
`
`ADJUST BASE GESTURE MAP IN ACCORDANCE WITH
`
`FIG. 39
`
`38
`
`38
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 38 of 49
`
`US 2008/0036743 Al
`
`760 _~
`
`START
`
`764
`
`762
`
`
`
`INITIATE EXPOSE
`FUNCTION 1
`
`768
`
`766
`
`
`
`INITIATE EXPOSE
`FUNCTION2
`
`772
`
`
`INITIATE EXPOSE
`FUNCTION 3
`
`DJ
`
`UPWARD SWIPE
`
`774
`
`776
`
` DOWNWARD
`SWIPE INITIATE DASHBOARD
`
`FIG. 40
`
`39
`
`39
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 39 of 49
`
`US 2008/0036743 Al
`
`780 =~
`
`
`
`DETECT BASE CHORD
`
`782
`
`784
`
`786
`
`DETECT PAUSE
`
`OPEN CONTEXTUAL
`MENU
`
`
`
`
`
`
` DETECT UP/
`
`DOWNSLIDE
`
`790
`
`
`
`MOVE HIGHLIGHT BAR
`THROUGH
`CONTEXTUAL MENU
`
`794
`
`
`
`IMPLEMENT:
`FUNCTIONALITY OF
`
`HIGHLIGHTED ITEM
`
`FIG. 41
`
`40
`
`40
`
`

`

`Patent Application Publication Feb. 14, 2008 Sheet 40 of 49
`
`US 2008/0036743 Al
`
`
`
`
`
`
`
`primary
`
`
`
`
`
`
`
`
`
`
`Secondary drag
`Scroll/pan
`
`=e
`
`Slide left
`
`
`
`Slide up
`Slide down
`
`
`
`
`
`Slide right
`tab
`
`
`
`bktab
`Slide left
`
`
`Slide up
`undo
`Slide down
`redo
`
`
`
`
`Rotate CCW
`cancel
`
`
`Rotate CW
`Selectall
`
`
`spread
`paste
`cut
`contract
`
`
`
`
` tap
`copy
`
`
`Rotate CCW
`primary
`Thumb +2
`
`
`
`Rotate CW
`
`spread
`
`save
`contact
`
`
`
`
`[primary[|Thumb+4|motion|point,
`
`
`
`Position/motion
`
`Position/motion
`
`Position/motion
`
`point
`
`drag
`
`
`
`
`primary
`
`Thumb +1
`
`
`
`FIG. 42
`
`41
`
`41
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 41 of 49
`
`US 2008/0036743 Al
`
`
`
`
`
`click|Primary|Position/motion point
`
`
`
`
`
`Rotate CW
`secondary
`
`Hand
`primary
`
`# offingers
`1
`
`gesture event
`tap
`
`behavior
`
`primary
`
`2
`
`primary
`3
`
`primary
`4
`
`primary
`5
`primary
`Thumb +1
`
`primary
`
`Thumb +2
`
`primary
`
`Thumb +3
`
`secondary
`
`secondary
`
`secondary
`
`secondary
`
`click
`drag
`Move window
`Scroll/pan
`point
`undo
`redo
`paste
`cut
`copy
`Expose 1
`Expose 2
`Expose 3
`dashboard
`replace
`find
`back
`forward
`open
`close
`new
`save
`
`Arrow up (text cursor)
`Arrow down
`Arrow right
`Arrow left
`Apply alt/option to other hand
`Select up (e.g., <shift> arrow)
`Select down
`Selectright
`Selectleft
`Moverresize windowin
`Conjunction with opposite hand
`
`Page up
`Page down
`begin
`endline
`Apply shift to opposite hand
`Zoom out screen
`Zoom in screen
`Rotate CCW
`Rotate CW
`
`tap
`Position/motion
`motion
`motion
`motion
`Slide up
`Slide down
`spread
`contract
`tap
`Slide teft
`Slide right
`Slide up
`Slide down
`spread
`contract
`Slide left
`Slide right
`Rotate CCW
`Rotate CW
`spread
`contact
`
`Slide up
`Slide down
`Slide right
`Slide left
`
`pause
`Slide up
`Slide down
`Slide right
`Slide left
`motion
`
`Slide up
`Slide down
`Slide right
`Slide left
`
`pause
`spread
`contact
`Rotate CCW
`
`FIG. 43
`
`42
`
`42
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 42 of 49
`
`US 2008/0036743 Al
`
`Hand__|# of fingers
`
`
`
`primary
`tap
`click
`
`Position/motion
`point
`
`
`tap
`Secondary click (rightclick)
`
`spread
`Enlarge object
`
`
`contract
`shrink object
`
`
`pan
`slide
`
`
`
`Rotate object
`rotate
`
`Primary drag
`motion
`Scroll pan
`
`tap
`Secondary click (right click)
`
`|___—motion|point,
`primary
`Thumb +
`Secondary drag
`Middle +
`ring
`Thumb +
`index +
`pinky
`L3+R3
`
`
`
`
`primary
`
`2
`
`primary
`
`primary
`
`primary
`
`
`
`expose
`Window drag, resize
`Expose 3
`Maximize/restore
`
`tap
`motion
`Slide up
`tap
`
`FIG. 44
`
`43
`
`43
`
`

`

`Patent Application Publication Feb. 14, 2008 Sheet 43 of 49
`
`US 2008/0036743 Al
`
`et|eKe]©|gae
`eae|P
`
`FIG. 46
`
`FIG. 47
`
`FIG. 48
`
`44
`
`44
`
`
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 44 of 49
`
`US 2008/0036743 Al
`
`800N
`
`802
`
`804
`
`806
`
`808
`
`810
`
`DETECT FIRST FINGER
`
`DETERMINE STATE OF FIRST FINGER
`
`
`
`DETECT ADDITIONAL FINGER(S)
`
`DETERMINE STATE OF ADDITIONAL
`FINGER(S)
`
`
`
`
`IMPLEMENT DIFFERENT INPUT MODES
`BASED ON TIMING OF STATES OF FIRST AND
`ADDITIONAL FINGER(S) RELATIVE TO ONE
`ANOTHER
`
`FIG. 49
`
`45
`
`45
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 45 of 49
`
`US 2008/0036743 Al
`
`820—{\
`
`822
`
`824
`
`826
`
`828
`
`830
`
`DETECT FIRST FINGER
`
`
`
`DETERMINE STATE OF FIRST FINGER
`
`IMPLEMENT FIRST MODE WITH FIRST
`FINGER WHEN FIRST FINGER IS IN FIRST
`STATE
`
`MONITOR FOR MODIFIER FINGER EVENT
`WHEN FIRST FINGER IN SECOND STATE
`
`
`
`
`IMPLEMENT SECOND MODEWITH FIRST
`FINGER WHEN FIRST FINGER IS IN SECOND
`STATE AND MODIFIER FINGER EVENT IS
`DETECTED
`
`FIG. 50
`
`46
`
`46
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 46 of 49
`
`US 2008/0036743 Al
`
`840—\
`
`842
`
`844
`
`846
`
`848
`
`850
`
`INITIALLY DETECT FIRST FINGER
`
`IMPLEMENT FIRST MODE WITH FIRST
`FINGER WHEN FIRST FINGER IS MOVED
`
`
`
`DETECT SECOND FINGER AFTER
`DETECTING FIRST FINGER
`
`SWITCH FROM FIRST MODE TO SECOND
`MODE WHEN FIRST FINGER IS STATIONARY
`AND SECOND FINGER IS DETECTED ~
`
`
`MOVED
`
`IMPLEMENT SECOND MODE WITH AT LEAST
`FIRST FINGER WHEN FIRST FINGERIS
`
`FIG. 51
`
`47
`
`47
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 47 of 49
`
`US 2008/0036743 Al
`
`860 a.
`
`DETECT FIRST
`FINGER
`
`862
`
`864
`
`NO
`
`STANDBY
`
`866
`
`START
`
`
`
`
`Sas
`
`
`
`PERFORM POINTING
`FIRST FINGER “YES|OPERATION WITH
`
`MOVED
`FIRST FINGER
`
`MOVEMENT
`
`
`
`.
`
`
`
`NO
`
`868
`
`NO
`
`870
`
`YES
`
`FIRST FINGER
`MOVED
`
`NO
`
`YES
`
`872
`
`
`
`PERFORM DRAGGING
`
`
`OPERATION WITH
`FIRST FINGER
`
`MOVEMENT
`
`FIG. 52
`
`48
`
`48
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 48 of 49
`
`US 2008/0036743 Al
`
`880 a.
`
`882
`
`START
`
`
`
`POINTING MODE
`
`DETECT FIRST
`FINGER
`
`YES
`
`IMPLEMENT
`
`NO
`
`STANDBY
`
`884
`
`
`886
`
` STATIONARY
`
`MOVING
`
`
`
` IGNORE MODIFIER
`LOOK FOR MODIFIER
`EVENT
`
`EVENT
`
`
`
`
`
`
`
`PERFORM POINTING
`DETECT
`OPERATIONS NO
`
`ODIFIER EVENT,
`
`OPERATIONS
`
`896
`
`YES
`
`890
`
`398
`
`SWITCH FROM
`POINTING MODE TO
`DRAGGING MODE
`
`PERFORM DRAGGING
`
`FIG. 53
`
`49
`
`49
`
`

`

`Patent Application Publication Feb. 14,2008 Sheet 49 of 49
`
`US 2008/0036743 Al
`
`FIG.54
`
`0c
`
`e
`oO
`
`958
`
`
`
`
`GESTURAL
`PROGRAM
`
`
`PROCESSOR
`
`DEVICE
`INPUT
`
`DISPLAY
`] GESTURE
`69 GUI
`
`50
`
`50
`
`

`

`US 2008/0036743 Al
`
`Feb. 14, 2008
`
`GESTURING WITH A MULTIPOINT SENSING
`DEVICE
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`[0001] This application claims priority benefit of U.S.
`Provisional Patent Application No. 60/763,605, filed Jan. 30,
`2006, and entitled “GESTURING WITH A MULTIPOINT
`SENSING DEVICE,”which is herein incorporated by ref-
`erence.
`
`[0002] This application is a Continuation-in-Part of U.S.
`patent application Ser. No. 10/903,964, filed on Jul. 30,
`2004, entitled “GESTURES FOR TOUCH SENSITIVE
`INPUT DEVICES,” and which is herein incorporated by
`reference.
`
`[0003] This application is a Continuation-in-Part of U.S.
`patent application Ser. No. 11/015,434, filed on Dec. 17,
`2004, entitled “METHOD AND APPARATUSFOR INTE-
`GRATING MANUAL INPUT,” which claims priority to
`USS. Pat. No. 6,323,846, issued on Nov. 27, 2001, which
`claims priority to U.S. Provisional Patent Application No.
`60/072,509 filed Jan. 26, 1998, all of which are herein
`incorporated by reference.
`
`[0004] This application is also related to the following
`patent applications, each of which is herein incorporated by
`reference:
`
`[0005] U.S. patent application Ser. No. 10/654,108, filed
`Sep. 2, 2003, entitled “AMBIDEXTROUS MOUSE;”
`
`[0006] U.S. patent application Ser. No. 10/789,676, filed
`Feb. 27, 2004, entitled “SHAPE DETECTING INPUT
`DEVICE;”
`
`[0007] U.S. patent application Ser. No. 10/840,862, filed
`May 6, 2004, entitled “MULTIPOINT TOUCHSCREEN;”
`
`[0008] U.S. patent application Ser. No. 11/115,539, filed
`Apr. 26, 2005, entitled “HAND HELD ELECTRONIC
`DEVICE WITH MULTIPLE
`TOUCH SENSING
`DEVICES;”
`
`[0009] U.S. patent application Ser. No. 11/241,839, filed
`Jul. 30, 2004, entitled “PROXIMITY DETECTOR IN
`HANDHELD DEVICE;”
`
`[0010] U.S. Provisional Patent Application No. 60/658,
`777, filed Mar. 4, 2005, entitled “MULTI-FUNCTIONAL
`HAND-HELD DEVICE;”
`
`[0011] U.S. patent application Ser. No. 10/903,964, filed
`Jul. 30, 2004, entitled “GESTURES FOR TOUCH SENSI-
`TIVE INPUT DEVICES;”
`
`[0012] U.S. patent application Ser. No. 11/038,590, filed
`Jan. 18, 2005, entitled “MODE-BASED GRAPHICAL
`USER INTERFACES FOR TOUCH SENSITIVE INPUT
`DEVICES;”
`
`[0013] U.S. patent application Ser. No. 11/048,264, filed
`Jan. 31, 2005, entitled “GESTURES FOR TOUCH SENSI-
`TIVE INPUT DEVICES;”
`
`[0014] U.S. patent application Ser. No. 11/228,737, filed
`Sep. 16, 2005, entitled “ACTIVATING VIRTUAL KEYS
`OF A TOUCH-SCREEN VIRTUAL KEYBOARD;”
`
`[0015] U.S. patent application Ser. No. 11/228,758, filed
`Sep.
`16, 2005, entitled “VIRTUAL INPUT DEVICE
`PLACEMENT ON A TOUCH SCREEN USER INTER-
`FACE;”
`
`[0016] U.S. patent application Ser. No. 11/228,700, filed
`Sep. 16, 2005, entitled “OPERATION OF A COMPUTER
`WITH TOUCH SCREEN INTERFACE;”
`
`[0017] U.S. patent application Ser. No. 10/927,925, filed
`Aug. 26, 2004, entitled “VISUAL EXPANDER;”
`
`[0018] U.S. patent application Ser. No. 10/927,575, filed
`Aug. 25, 2004, entitled “WIDE TOUCHPAD ON A POR-
`TABLE COMPUTER;”
`
`
`
`
`
`
`
`[0019] U.S. patent application Ser. No. 11/619,553, filed
`Jan. 3, 2007, entitled “MULTI-TOUCH GESTURE DIC-
`TIONARY;”and
`
`[0020] U.S. patent application Ser. No. 11/619,571, filed
`Jan. 3, 2007, entitled “MULTI-TOUCH GESTURE DIC-
`TIONARY.”
`
`BACKGROUND OF THE INVENTION
`
`[0021]
`
`1. Field of the Invention
`
`[0022] The present invention relates generally to gesturing
`associated with multipoint sensing devices.
`
`[0023]
`
`2. Description of the Related Art
`
`[0024] There exist today many styles of input devices for
`performing operations in a computer system. The operations
`generally correspond to moving a cursor and making selec-
`tions on a display screen. The operations may also include
`paging,
`scrolling, panning, zooming, etc. By way of
`example, the input devices may include buttons, switches,
`keyboards, mice, trackballs, touch pads, joy sticks, touch
`screens and the like. Each of these devices has advantages
`and disadvantages that are taken into account when design-
`ing the computer system.
`
`[0025] With touch pads, the movementof the input pointer
`corresponds to the relative movements of the user’s finger
`(or stylus) as the finger is moved along a surface of the touch
`pad. Touch screens, on the other hand, are a type of display
`screen that has a touch-sensitive transparent panel covering
`the screen. When using a touch screen, a user makes a
`selection on the display screen by pointing directly to GUI
`objects on the screen (usually with a stylus or finger). In
`general, the touch device recognizes the touch and position
`of the touch and the computer system interprets the touch
`and thereafter performs an action based on the touch event.
`
`In order to provide additionally functionality, ges-
`[0026]
`tures have been implemented with some of these input
`devices. By way of example, in touch pads, selections may
`be made whenone or more taps are detected on the surface
`of the touch pad. In somecases, any portion of the touch pad
`may be tapped, and in other cases a dedicated portion of the
`touch pad may be tapped. In additionto selections, scrolling
`maybe initiated by using finger motion at the edge of the
`touch pad.
`
`[0027] Unfortunately, gesturing is severely limited by the
`fact that most touch technologies are only capable of report-
`ing a single point even when multiple objects are placed on
`the sensing surface. That is, they lack the ability to track
`
`51
`
`51
`
`

`

`US 2008/0036743 Al
`
`Feb. 14, 2008
`
`multiple points of contact simultaneously. In resistive and
`capacitive technologies, an average of all simultaneously
`occurring touch points are determined and a single point
`which falls somewhere between the touch points is reported.
`In surface waveandinfrared technologies, it is impossible to
`discern the exact position of multiple touch points that fall
`on the same horizontal or vertical lines due to masking. In
`either case, faulty results are generated.
`
`[0028] Based on the above, there is a need for multipoint
`sensing devices and methods of implementing gestures with
`multipoint sensing devices.
`
`SUMMARY OF THE INVENTION
`
`[0029] The invention relates, in one embodiment, to an
`electronic system. The electronic system includes a multi-
`point sensing device that provides a multipoint sensing area
`for receiving inputs from one or more objects. The electronic
`system also includes a gesture module configured to deter-
`minea gesture set for a given input arrangementreceived by
`the multipoint sensing area of the multipoint sensing device,
`to monitor the given input arrangement for one or more
`gesture events included in the gesture set, and to initiate
`input actions associated with a gesture event when the
`gesture event is performed with the input arrangement. The
`input arrangement may for example be an arrangement of
`fingers and/or other parts of the hand.
`
`[0030] The invention relates, in another embodiment, to a
`gestural control method. The method includes detecting
`multiple points within a sensing area at the same time. The
`methodalso includes determining a chord when one or more
`points are detected within the sensing area. The chord is a
`specific arrangement of points within the sensing area. The
`method further includes determining a gesture set associat-
`ing commands to one or more gesture events. The method
`additionally includes monitoring points for gesture events.
`Moreover, the method includes performing command asso-
`ciated with gesture event if a gesture event is recognized.
`
`[0031] The invention relates, in another embodiment, to a
`control operation. The control operations includes detecting
`a touch or near touch. The operations also includes deter-
`mining a gesture set for the touch. The gesture set includes
`one or more gesture events for provoking or i