a2) United States Patent
`US 8,830,181 B1
`(0) Patent No.:
`Sep. 9, 2014
`(45) Date of Patent:
`Clarket al.
`
`US008830181B1
`
`GESTURE RECOGNITION SYSTEM FOR A
`TOUCH-SENSING SURFACE
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`(54)
`
`(75)
`
`Inventors: Paul Clark, Belleuve, WA (US);
`Edward Grivna, Brooklyn Park, MN
`(US); Tony Park, Everett, WA (US);
`Patrick Prendergast, Clinton, WA (US);
`Gabe Rowe, Seattle, WA (US); Ryan
`Seguine, Seattle, WA (US)
`
`(73)
`
`Assignee: Cypress Semiconductor Corporation,
`San Jose, CA (US)
`
`(*)
`
`Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1280 days.
`
`(21)
`
`Appl. No.: 12/476,240
`
`(22)
`
`Filed:
`
`Jun. 1, 2009
`
`2002/0097229 Al*
`7/2002 Roseetal. ccc 345/173
`2006/0125803 Al*
`6/2006 Westerman etal.
`.. 345/173
`2006/0274046 Al* 12/2006 Hillisetal. ou... 345/173
`2008/0309632 Al*
`12/2008 Westerman etal.
`beteesnees 345/173
`2008/0316183 Al
`12/2008 Westerman etal.
`2009/0051660 Al*
`2/2009 Felandetal. ou... 345/173
`2009/0284478 Al
`11/2009 De Laetal.
`2009/0289902 AL* 11/2009 Carlviketal. oo... 345/173
`2010/0149115 Al
`6/2010 Park etal.
`OTHER PUBLICATIONS
`
`USPTOFinal Rejection for U.S. Appl. No. 13/449, 179 dated Sep. 30,
`2013; 12 pages.
`USPTO Non-Final Rejection for U.S. Appl. No. 12/337,411 dated
`Nov.23, 2011; 12 pages.
`USPTO Non-Final Rejection for U.S. Appl. No. 13/449,179 dated
`Mar.29, 2013; 10 pages.
`USPTO Notice of Allowance for U.S. Appl. No. 12/337,411 dated
`Mar.22, 2012; 9 pages.
`USPTO Notice of Allowance for U.S. Appl. No. 13/449,179 dated
`Nov.8, 2013; 11 pages.
`
`(60)
`
`(51)
`
`(52)
`
`(58)
`
`Related U.S. Application Data
`
`Provisional application No. 61/057,867, filed on Jun.
`1, 2008.
`
`* cited by examiner
`
`Primary Examiner — Regina Liang
`
`Int. Cl.
`
`(2006.01)
`
`G09G 5/00
`U.S. Cl.
`USPC coececccccccccccecesteeessceceseeeeeees 345/173; 715/863
`Field of Classification Search
`CPC weeeeceecccee GO6F 3/041; GO6F 3/0416; GO6F
`2203/04104; GOG6F 3/0488; GO6F 3/04883
`USPC coececcceccccccssccesseeeteeeeees 345/156—-179; 715/863
`See application file for complete search history.
`
`ABSTRACT
`(57)
`An apparatus and methodfor identifying gestures performed
`on a touch-sensing surface. In one embodiment, a gesture
`recognition unit processes an input signal to determine input
`metrics associated with contacts at the touch-sensing surface.
`The gesture recognition unit identifies a gesture based on
`comparing at least one of the input metrics with a threshold
`value associated with the gesture.
`
`18 Claims, 14 Drawing Sheets
`
`_~ dlectronic syshor
`109
`
`touch-senging surice
`116
`
`fo
`
`anglog bus
`
`gesture
`i
`t wecogRition unit
`
`capacitance
`sensor
`
`hast 340
`
`decision logic
`
`applicutions
`
`
`
`
`é
`12ECL)- PLHNY
`
`APPLE-1035
`
`1
`
`APPLE-1035
`
`

`

`U.S. Patent
`
`Sep. 9, 2014
`
`Sheet 1 of 14
`
`US 8,830,181 B1
`
`tiptstonnordd:
`
`OES1
`YO
`
`TST
`
`
`
`H01}wumsAsanole—7
`
`|aanDld
`
`
`
`HED.UCIOROO|aaniead
`
`CAPE
`
`epeeaourpondns
`
`SEPIERA-OU
`
`soeSusass
`
`JOsuag
`
`Tor
`
`
`
`OTTanaapFursssoard
`
`aABLEOSU
`
`aSi;snqZopsur4OLE
`
`
`
`
`
`SILOSHupsuas~Gono}
`
`2
`
`
`

`

`U.S. Patent
`
`Sep. 9, 2014
`
`Sheet 2 of 14
`
`US 8,830,181 B1
`
`yay
`
`sopnseued
`
`PEE
`
`éTUNOL
`
`$34
`
`JOSS
`
`conrmiromelee
`
`penaipopSore
`ROUTEEAE
`taepussiesgg
`
`eM
`
`Oh.
`
`eg
`
`HERE fp 4
`
`aSfea7JOHAS
`
`ApoaasaPRAT
`
`
`SBI:SODOFISJAI
`
`APO
`
`PEAS
`
`3
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Sep. 9, 2014
`
`Sheet 3 of 14
`
`US 8,830,181 B1
`
`Start thre
`303
`
`end time
`305
`
`.
`doration
`304
`contact
`BORFPccesnstuesssesanesusesseeeees
`S
`ae
`™
`
`contact
`302
`é‘“
`
`a
`
`i
`ii
`;
`:
`
`i
`;
`
`;{z
`
`temporal
`overlap
`386
`
`is
`
`28
`
`As
`
`as
`
`3s
`
`&s
`
`7
`
`$s
`
`Os
`
`FIGURE 3
`
`4
`
`

`

`U.S. Patent
`
`Sep. 9, 2014
`
`Sheet 4 of 14
`
`US 8,830,181 B1
`
`timer
`timer mer
`AOS
`404
`4as
`wt,
`deeeeeeedPN ceeeeesete Paananeem
`
`COHCT Ae
`avy
`
`COMRCT Ae
`
`FIGURE 4A
`
`402
`Adi
`
`wo. tOuDR Sensor
`surface
`40
`
`contact
`
`FIGURE 4B
`
`5
`
`

`

`U.S. Patent
`
`Sep. 9, 2014
`
`Sheet 5 of 14
`
`US 8,830,181 B1
`
`firmer
`804
`-a,coseeee
`
`Liner
`505
`ath,
`
`
`
`is
`
`}
`i
`}
`}
`}
`}
`}
`}
`}
`}
`}
`}
`}
`}
`}}}}
`i}}}
`}}}}}}}}}
`
`i}
`j
`j
`HananFrovnnand
`.
`
`Contact=gie,
`301
`
`veyed gee.
`COMMA- ie
`302
`
`enenentenenenencacacaranfeens
`
`}}}}}}}}}
`
`4
`{
`;
`;
`;
`;
`;
`;
`;
`;
`;
`;
`;
`;
`}
`
`,
`
`anenenenenenened
`
`
`
`
`
`i}}
`
`FIGURE 5A
`
`ww tWech sensor
`surface
`500
`
`f
`
`S82
`
`contact
`
`contact
`
`FIGURE3B
`
`6
`
`

`

`U.S. Patent
`
`Sep. 9, 2014
`
`Sheet 6 of 14
`
`US 8,830,181 B1
`
`timer
`
`timer
`
`contact ce
`601
`
`oe
`
`|
`
`OL 2
`bessesnsreresrsresesesreresesseseseey
`
`61 olcenechiceee. &12
`
`
`a wonwawaaremuaeanan
`pom timer
`
`
`coptget oe.
`G2
`
`contact eo.
`SOR
`
`en
`
`
`
`mer oy timer
`
`FIGURE 64
`
`fo surface
`
`touch sensor
`
`600
`
`j
`
`” contact
`Contact
`
`602
`
`mvad
`
`OPPO
`
`vomtset
`
`C3
`
`604
`
`604
`
`FIGURE6B
`
`7
`
`

`

`U.S. Patent
`
`Sep. 9, 2014
`
`Sheet 7 of 14
`
`US 8,830,181 B1
`
`
`direction vector
`730
`
`sepment
`
`
`
`jocation
`
`Pid
`whet
`
`
`
`
`,” ae~ ee
`
`:
`fo
`
`
`final: 0
`focation.<
`
`
`712
`RARER YEE EE Se
`
`
`
`
`initial
`location
`
`segnient
`74
`us
`
`.
`vertical
`active
`cisiance
`720
`
`segment
`
`
`
`segment
`703
`
`
`horwontal
`active
`distance
`73}
`
`FIGURE7
`
`8
`
`

`

`U.S. Patent
`
`Sep. 9, 2014
`
`Sheet 8 of 14
`
`US 8,830,181 B1
`
`finer
`804
`
` contact ape.
`
`3
`
`£
`
`BOY
`
`contact oe
`802
`
`
`
`FIGURE 8A
`
`o
`
`_. ianuich saasor
`surface
`$00
`
`coMtact
`
`contact
`
`802
`
`FIGURE 8B
`
`9
`
`

`

`U.S. Patent
`
`Sep. 9, 2014
`
`Sheet 9 of 14
`
`US 8,830,181 B1
`
`direction
`vector
`Q
`
`location
`302 ~~,
`
`j
`
`ou Ny 4 / /
`y
`¥ “
`¢ et
`
`*%
`
`N
`
`“
`
`f
`
`2
`
`anne
`
`ae
`
`4
`
`‘
`
`v
`
`4
`
`/ \
`. %
`3
`
`“
`
`~~
`
`x
`‘
`\
`
`locati
` dovation
`03 a
`%
`\
`}
`\
`/<a
`‘ .
`4 \ 2
`a
`aan
`an<
`wn
`a
`a <
`3
`
`oe
`
`»,
`
`fy
`
`.
`
`location
`90] >
`
`N
`
`N
`
`4
`
`XN
`
`’
`
`’
`
`/
`
`va
`
`x
`
`en
`7
`\
`-epee
`¢
`‘
`4 eC?
`4
`N
`vo
`a
`“og Ss
`‘
`Q
`
`al
`
`!
`
`Bral
`location
`Es
`904
`
`\
`
`‘
`\
`‘
`\,
`
`location
`OO
`904
`N
`“)
`\
`a
`* 4
`fy a
`So
`
`4
`
`ra
`
`o
`
`4
`co
`vo”.
`
`%
`
`s
`
`vo
`
`3
`
`8
`
`sy
`
`FIGURE 9
`
`10
`
`10
`
`

`

`U.S. Patent
`
`Sep. 9, 2014
`
`Sheet 10 of 14
`
`US 8,830,181 B1
`
`Bontact
`SOnEACT
`_a» touch sensor
`1001
`1002
`i
`surface
`
`
`‘
`1000
`
`
`contact
`108
`1603
`
`COnVErEENce
`port
`1006
`
`
`
`poomnenes COMBRCE
`a4
`
`contact
`L605
`
`FIGURE10
`
`contact
`
`a.
`/
`_» touchsense
`oe
`surface
`f
`1100
`
`contact
`.
`1002
`
`f
`
`
`
`
`
`mn TERY?
`
`COME
`
`1005
`
`contact
`1004
`
`sighionary
`COREL om
`
`£105
`
`BIGURE 11
`
`11
`
`11
`
`

`

`U.S. Patent
`
`Sep. 9, 2014
`
`Sheet 11 of 14
`
`US 8,830,181 B1
`
`_ touch sensor
`,
`surface
`
`1202
`
`P00 contacts
`
`
`_ fouch sensor
`<
`surface
`PStH3
`
`
`
`contacts
`
`130]
`
`FIGURE 12
`
`FIGURE13
`
`12
`
`12
`
`

`

`U.S. Patent
`
`Sep. 9, 2014
`
`Sheet 12 of 14
`
`US 8,830,181 B1
`
`gesture
`recognition
`PROCSSS
`1400
`jé
`ot
`
`receive taput stenal fron touch
`sonsar
`
`330%
`eieeieeiieinienieteiiee
`
`determing input metres frank ip
`signal
`1403
`
`1405
`
`compare input nuwirics io threshold
`values for cachmetric
`£408
`
`ichoaaty gesture based ox
`compariios of inpat racinics will
`threshold vahaes
`1407
`
`oxenulc istruofions associated seith.
`the idoniified eosture
`
`FIGURE14
`
`13
`
`13
`
`

`

`U.S. Patent
`
`Sep. 9, 2014
`
`Sheet 13 of 14
`
`US 8,830,181 B1
`
`ae Bosture recogation process
`1500
`
`
`
`Incation|
`
`* B
`
`O
`
`i beuin recording segment |
`i sequence for comtant at
`i
`
`Leeamtact
`
`_o*detectedatlocation
`
`
`<" 1 for longer than minioxans
`“0
`daeation?
`
`
`
`¥eK
`
`metan af>.
`contact detected’
`
`C
`
`a
`
`“
`
`
`.
`
`~
`
`
`
`
`
`"contact~
`
`
`ad ence
`y
`soe
`y
`.
`%
`
`focatinn |
`a
`a
`[BLEED duration 12
` Re
`5Q6
`pos
`°" gontaet at
`
`incation | hited?
`{343
`
`
`
`
`
`
`_vSeterted exceeding
`_ ginimanm duetiog ne
`“s.
`location 27
`
`
`
`
`Pernt entetnentetnr ere
`
`mister atapfioach
`at location 2
`
`|
`i
`
`location 2
`
`
`contact fifted’??
`oa
`LAEs
`
`
`
`Loo” sontactat
`
`fooation 2 ifted?
`
`
`
`
`
`Ves
`aa
`ea
`o
`
`
`oe pontact at,
`_-vénother contact~..
`
`lovation 3 exceeding
`or
`
`
`
` ” detected at location 1?
`
`at location 1
`
`
`
`™.minignm duration’
`. 19240ee
`
`
`yes
`S
`
`to
`
`
`1336
`
`
`identify cascade gesture
`{S22
`
`identify walking aesture
`1526
`
`identify drumming gesture
`
`FIGURE I5A
`
`14
`
`14
`
`

`

`U.S. Patent
`
`Sep. 9, 2014
`
`Sheet 14 of 14
`
`US 8,830,181 B1
`
`coufact ended?
`
`
`2 motion of
`
`
`"
`comfact accelerates?
`
`
`pation ab
`
`Me
`
`contact at
`
`™.
`
`
`
`location 2 exceeding
`
`“s.. mninimem duration?
`
`
`identify brush gesture
`is7o
`
`
`
`
`
`"path of contact
`starthig at location 1
`*
`
`
`“™..
`follows aware? 07
`
`
`
`atacis axe
`“ay 566a
`
`ms
`Lo" moving towards BO
`
`
`“.
`COMMON convergence
`“point?
`'
`{See
`
`or COBEACE ab
`
`Jeeation 2 in motion, “>
`”
`
`
`
`. oxceads active distance?
`
`wa
`i844
`
`
`
`
` Second location Hied~.
`
`{without exceeding max
`
`begin recording
`segment sequence for
`contact at location 2
`
`Be
`
`
`oo CORRAMS ares
`“" moving towards a ~
`slapenary contict? _-
`
`
`
`
`
`
`
`
`
`
`
`athe of contacts ~~~
`are parallel?
`1548 Within 3 snaximam
`
`
`
`dently motion and tap gesiure
`
`
`
`__ distance of eack other? _
`
`
`
`FIGURE15B
`
`15
`
`15
`
`

`

`US 8,830,181 B1
`
`1
`GESTURE RECOGNITION SYSTEM FOR A
`TOUCH-SENSING SURFACE
`
`RELATED APPLICATIONS
`
`This application claims the benefit of U.S. Provisional
`Application No. 61/057,867, filed on Jun. 1, 2008.
`
`TECHNICAL FIELD
`
`This disclosurerelates to the field of user interface devices
`
`and, in particular, to gesture recognition on devices that have
`a touch-sensing surface.
`
`BACKGROUND
`
`Computing devices, such as notebook computers, personal
`data assistants (PDAs), kiosks, and mobile handsets, have
`user interface devices, which are also known as human inter-
`face devices (HID). One user interface device that has
`become more commonis a touch-sensorpad (also commonly
`referredto as a touchpad). A basic notebook computer touch-
`sensor pad emulates the function of a personal computer (PC)
`mouse. A touch-sensorpad is typically embedded into a PC
`notebook for built-in portability. A touch-sensor padrepli-
`cates mouse X/Y movementby using two defined axes which
`contain a collection of sensor elements that detect the position
`of one or more conductive objects, such as fingers. Mouse
`right/left button clicks can be replicated by two mechanical
`buttons, located in the vicinity of the touchpad,or by tapping
`commands on the touch-sensor paditself. The touch-sensor
`pad provides a user interface device for performing such
`functions as positioning a pointer, or selecting an item on a
`display. These touch-sensor pads may include multi-dimen-
`sional sensorarrays for detecting movementin multiple axes.
`The sensor array may include a one-dimensionalsensorarray,
`detecting movementin one axis. The sensorarray mayalso be
`two dimensional, detecting movements in two axes.
`Anotheruser interface device that has become more com-
`mon is a touch screen. Touch screens, also known as touch-
`screens, touch panels, or touchscreen panels are display over-
`lays which are typically either pressure-sensitive (resistive),
`electrically-sensitive
`(capacitive),
`acoustically-sensitive
`(surface acoustic wave (SAW))or photo-sensitive (infra-red).
`The effect of such overlays allows a display to be used as an
`input device, removing the keyboard and/or the mouseas the
`primary input device for interacting with the display’s con-
`tent. Such displays can be attached to computersor, as termi-
`nals, to networks. There are a numberoftypes oftouch screen
`technologies, such as optical
`imaging, resistive, surface
`acoustical wave, capacitive,
`infrared, dispersive signal,
`piezoelectric, and strain gauge technologies. Touch screens
`have becomefamiliar in retail settings, on point-of-sale sys-
`tems, on ATMs, on mobile handsets, on kiosks, on game
`consoles, and on PDAswherea stylus is sometimes used to
`manipulate the graphical user interface (GUI) andto enter
`data. A user can touch a touch screen or a touch-sensor pad to
`manipulate data. For example, a user can apply a touch, by
`using a finger to press the surface of a touch screen, to select
`an item from a menu.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present disclosure is illustrated by way of example,
`and not by wayoflimitation, in the figures of the accompa-
`nying drawings.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`55
`
`60
`
`65
`
`2
`FIG.1 illustrates a block diagram of one embodimentof an
`electronic system having a processing device for recognizing
`a gesture.
`FIG.2 illustrates a block diagram of one embodimentof a
`device for detecting an input and a location of the input ona
`touch sensorsurface.
`FIG.3 illustrates a timing diagram offingers, touches, or
`other materials contacting a touch sensor surface, according
`to one embodiment.
`FIG.4A illustrates a timing diagram of contacts perform-
`ing a drumminggesture on a touch sensorsurface, according
`to one embodiment.
`
`FIG.4Billustrates possible locations of contacts perform-
`ing a drumming gesture on a surface of a touch sensor,
`according to one embodiment.
`FIG. 5A illustrates a timing diagram of contacts perform-
`ing a walking gesture on a touch sensorsurface, according to
`one embodiment.
`FIG.5Billustrates possible locations of contacts perform-
`ing a walking gesture on a touch sensorsurface, according to
`one embodiment.
`
`FIG.6A illustrates a timing diagram of contacts perform-
`ing a cascade gesture on a touch sensorsurface, according to
`one embodiment.
`
`FIG.6Billustrates possible locations of contacts perform-
`ing a cascade gesture on a touch sensorsurface, according to
`one embodiment.
`FIG.7 illustrates active distances and segments around an
`initial contact location, according to one embodiment.
`FIG.8A illustrates a timing diagram of contacts perform-
`ing a motion and tap gesture on a touch sensor surface,
`according to one embodiment.
`FIG.8Billustrates possible locations of contacts perform-
`ing a motion and tap gesture on a touch sensor surface,
`according to one embodiment.
`FIG.9 illustrates a path of a contact performing a rotation
`gesture, according to one embodiment.
`FIG.10 illustrates possible locations of contacts perform-
`ing a grab gesture on a touch sensor surface, according to one
`embodiment.
`
`FIG.11 illustrates possible locations of contacts perform-
`ing a sweep gesture on a touch sensor surface, according to
`one embodiment.
`
`FIG.12 illustrates possible locations of contacts perform-
`ing a strum gesture on a touch sensor surface, according to
`one embodiment.
`FIG.13 illustrates possible locations of contacts perform-
`ing a brush gesture ona touchsensorsurface, according to one
`embodiment.
`FIG.14 is a block diagram illustrating a gesture recogni-
`tion process, according to one embodiment.
`FIGS. 15A and 15Bare block diagramsillustrating a ges-
`ture recognition process, according to one embodiment.
`
`DETAILED DESCRIPTION
`
`Described herein is an apparatus and methodfor recogniz-
`ing gestures input on a touch sensor, such as a touch-sensing
`surface. In one embodiment,the gestures are performed using
`single or multiple contacts on the touch-sensing surface,
`wherein each contact indicates physical or proximate contact
`betweena usersfinger, hand, palm, or other object detected by
`the specific touch sensor. In one embodiment, a gesture rec-
`ognition system identifies gestures based on characteristics
`such as the duration of the contact, start and end times of the
`contact, acceleration of motion of the contact, direction of
`motion of the contact, or a path of motion of the contact. In
`
`16
`
`16
`
`

`

`US 8,830,181 B1
`
`3
`4
`gesture and executes instructions associated with the gesture
`the apparatus applies a segmentation
`one embodiment,
`if the segment sequences match.
`schemeto determine the path of the contact.
`In one embodiment, the gesture recognition unit identifies
`The following description sets forth numerous specific
`a gesture based on whether multiple contacts overlap in time.
`details such as examples of specific systems, components,
`This temporal overlap may occur between a first and a second
`methods, andso forth, in order to provide a good understand-
`contact, for example, if the touch sensor detects both thefirst
`ing of several embodimentsofthe present invention.It will be
`contact and the second contact at the same time. In contrast,
`apparentto one skilled in the art, however, that at least some
`no temporal overlap occurs if the first contact is detected as
`embodimentsofthe present invention maybe practiced with-
`lifted before the second contact begins.
`out these specific details. In other instances, well-known
`In one embodiment, the gesture recognition unit identifies
`components or methods are not described in detail or are
`
`presented ina simple block diagram format. Thus, the specific a gesture based onavelocity or an acceleration of a contactat
`details set forth are merely examples. Particular implementa-
`the touch sensor. For example, for a touch sensor that samples
`tions may vary from these exemplary details and still be
`the location of a contact at regular intervals, velocity of the
`contemplated to be within the spirit and scope ofthe present
`contact can be determined by the distance between sampled
`invention.
`locations of the contact. Likewise, a change in acceleration
`In one embodiment, a gesture recognition unit receives an
`may be determined by comparing the velocity between con-
`input signal from a touch sensor. For example, the touch
`secutive sampled locations. In one embodiment, a gesture
`sensor may be a touch-sensing surface that includes an array
`maybeidentified based on whetherthe detected acceleration
`of the contact exceeds a threshold acceleration or falls within
`of capacitive sensors arrangedin a grid. Such a touch-sensing
`surface may generate a signal containing information about
`the contacts present at or upon the touch-sensingarray, such
`as the presence of a conductive object touching or in close
`proximity to the surface of the touch-sensing array. Said
`signal, as presented to the gesture recognition unit, may also
`include location information for each of the contacts.
`In one embodiment, the gesture recognition unit receives
`and processes the input signal to determine a set of input
`metrics, which may include information about each of the
`contacts at the touch sensor. For example, the input metrics
`may include durations, locations, directions of movement,
`paths, velocities, accelerations, and numbers of contacts. In
`one embodiment, where the input metrics include a direction
`of motion of a contact, the gesture recognition unit deter-
`mines a direction of motion between a first location of a
`contact and a secondlocation ofa contact. Thefirst location of
`
`15
`
`25
`
`a range of acceleration values.
`FIG.1 illustrates a block diagram of one embodimentof an
`electronic system 100 that includes a gesture recognition unit
`102 that is configured to identify gestures input at a touch-
`sensing surface 116. The electronic device 100 includes a
`touch-sensing surface 116 (e.g., a touchscreen, or a touch
`pad) coupled to a processing device 110 and a host 150. In one
`embodiment, the touch-sensing surface 116 is a two-dimen-
`sional user interface that uses a sensor array 121 to detect
`touches on the surface 116.
`In one embodiment, the sensor array 121 includes sensor
`elements 121(1)-121(N) (where N is a positive integer) that
`are disposed as a two-dimensional matrix (also referred to as
`an XY matrix). The sensor array 121 is coupled to pins 113
`(1)-113(N) ofthe processing device 110 via an analog bus 115
`transporting multiple signals. In this embodiment, each sen-
`sor element 121(1)-121(N)is represented as a capacitor. The
`capacitance ofeach sensorin the sensorarray 121 is measured
`by a capacitance sensor 101 in the processing device 110.
`In one embodiment,
`the capacitance sensor 101 may
`include a relaxation oscillator or other means to convert a
`
`capacitance into a measured value. The capacitance sensor
`101 may also include a counteror timer to measure the oscil-
`lator output. The capacitance sensor 101 may further include
`software componentsto convert the count value (e.g., capaci-
`tance value) into a sensor element detection decision (also
`referred to as switch detection decision) or relative magni-
`tude. It should be noted that there are various known methods
`
`for measuring capacitance, such as current versus voltage
`phase shift measurement, resistor-capacitor charge timing,
`capacitive bridge divider, charge transfer, successive approxi-
`mation, sigma-delta modulators, charge-accumulation cir-
`cuits, field effect, mutual capacitance, frequency shift, or
`other capacitance measurement algorithms. It should be
`noted however, instead ofevaluating the raw countsrelative to
`a threshold, the capacitance sensor 101 may be evaluating
`other measurements to determine the user interaction. For
`
`example, in the capacitance sensor 101 having a sigma-delta
`modulator, the capacitance sensor 101 is evaluating the ratio
`of pulse widths of the output, instead of the raw counts being
`over a certain threshold.
`In one embodiment, the processing device 110 further
`includes a gesture recognition unit 102. Operations of the
`gesture recognition unit 102 may be implemented in firm-
`ware; alternatively, it may be implemented in hardware or
`software. The gesture recognition unit 102 stores parameters
`that define the location (e.g., XY coordinates) and granularity
`(e.g., ahalf, a quarter, /, or any percentage with respect to the
`
`40
`
`45
`
`50
`
`the contact may be, for example, a location ofan initial touch
`of a detected object, while the second location may be a
`location where said object is removed from the touch sensor.
`Alternatively, in embodiments where the location of the con-
`tact is sampled overtime,thefirst location may be the location
`of the contact at the time of a first sample, while the second
`location may be the location of the contact at the time of a
`subsequent sample.
`In one embodiment, the gesture recognition unit deter-
`mines a direction vector between the first location and the
`
`second location of the contact. The gesture recognition unit
`may then apply a segmentation schemeto classify the direc-
`tion of movementof the contact. In one embodiment, a seg-
`mentis a range ofdirectionsrelative to the first location ofthe
`contact. For example, a possible segmentation scheme may
`divide the possible 360 degrees of direction around a location
`into four equal segments, with each segment covering a range
`of 90 degrees. In this example, the gesture recognition unit
`would determine the segment correspondingto the direction
`vector according to the range of direction in which the direc-
`tion vector falls.
`
`In one embodiment, a sequence of segments can be deter-
`minedin this mannerfor a set of locations ofa single contact.
`For example, these locations may be acquired by sampling
`the location of the contact over time. In this example, a
`sequence of segments may correspond to the sequence of
`direction vectors between each of the sampled locations.
`In one embodiment, the gesture recognition unit can derive
`a segment sequence from a set of locations of a moving
`contact, and compare the derived segment sequence with a
`predefined segment sequence associated with a gesture. In
`one embodiment, the gesture recognition unit identifies a
`
`17
`
`17
`
`

`

`US 8,830,181 B1
`
`5
`size of the touch-sensing surface 116) of each logical zone
`(defined herein), and a set of rules that define the gestures to
`be recognized. The gesture recognition unit 102 receives sig-
`nals from the capacitance sensor 101, and determinesthe state
`of the sensor array 121, such as whether an object (e.g., a
`finger) is detected on or in proximity to the sensor array 121
`(e.g., determining the presence of the object), where the
`object is detected on the sensorarray (e.g., determining one or
`more logical zones in which the object is detected), tracking
`the motion of the object
`(e.g., determining a temporal
`sequence of logical zones of the touch-sensing surface in
`which the movement of the object is detected), or or other
`information related to an object detected at the touch sensor.
`In another embodiment, instead of performing the opera-
`tions of the gesture recognition unit 102 in the processing
`device 110, the processing device 101 may sendthe raw data
`or partially-processed data to the host 150. The host 150, as
`illustrated in FIG. 1, may include decision logic 151 that
`performs someorall of the operations of the gesture recog-
`nition unit 102. Operations of the decision logic 151 may be
`implementedin firmware, hardware, software, or a combina-
`tion thereof. The host 150 may include a high-level Applica-
`tion Programming Interface (API) in applications 152 that
`perform routines on the received data, such as compensating
`for sensitivity differences, other compensation algorithms,
`baseline update routines, start-up and/or initialization rou-
`tines, interpolation operations, or scaling operations. The
`operations described with respect to the gesture recognition
`unit 102 may be implementedin the decision logic 151, the
`applications 152, or in other hardware, software, and/or firm-
`ware external to the processing device 110. In some other
`embodiments, the processing device 110 is the host 150.
`In another embodiment, the processing device 110 may
`also include a non-capacitance sensing actions block 103.
`This block 103 may be used to process and/or receive/trans-
`mit data to and from the host 150. For example, additional
`components may be implemented to operate with the process-
`ing device 110 along with the sensor array 121 (e.g., key-
`board, keypad, mouse, trackball, LEDs, displays, or other
`peripheral devices).
`The processing device 110 may reside on a commoncarrier
`substrate such as, for example, an integrated circuit (IC) die
`substrate, or a multi-chip module substrate. Alternatively, the
`componentsofthe processing device 110 may be one or more
`separate integrated circuits and/or discrete components. In
`one embodiment, the processing device 110 may be the Pro-
`grammable System on a Chip (PSoC™) processing device,
`developed by Cypress Semiconductor Corporation, San Jose,
`Calif. Alternatively, the processing device 110 may be one or
`more other processing devices known bythose of ordinary
`skill in the art, such as a microprocessoror central processing
`unit, a controller, special-purpose processor, digital signal
`processor (DSP), an application specific integrated circuit
`(ASIC), a field programmable gate array (FPGA), or other
`programmable device. In an alternative embodiment, for
`example, the processing device 110 may be a network pro-
`cessor having multiple processors including a core unit and
`multiple micro-engines. Additionally, the processing device
`110 may include any combination of general-purpose pro-
`cessing device(s) and special-purpose processing device(s).
`In one embodiment, the electronic system 100 is imple-
`mented in a device that includes the touch-sensing surface
`116 as the user interface, such as handheld electronics, por-
`table telephones, cellular telephones, notebook computers,
`personal computers, personal data assistants (PDAs), kiosks,
`keyboards, televisions, remote controls, monitors, handheld
`multi-media devices, handheld video players, gaming
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`devices, control panels of a household or industrial appli-
`ances, or other computer peripheral or input devices. Alter-
`natively, the electronic system 100 may be used in other types
`of devices. It should be noted that the components of elec-
`tronic system 100 may includeall the components described
`above.Alternatively, electronic system 100 may include only
`some of the components described above, or include addi-
`tional components not listed herein.
`FIG.2 is a block diagram illustrating one embodimentof a
`capacitive touch sensor array 121 and a capacitance sensor
`101 that converts measured capacitances to coordinates. In
`one embodiment, sensor array 121 and capacitance sensor
`101 are implemented in a system such as electronic system
`100. Sensor array 220 includes a matrix 225 of NxM elec-
`trodes (N receive electrodes and M transmit electrodes),
`which further includes transmit (TX) electrode 222 and
`receive (RX) electrode 223. Each of the electrodes in matrix
`225 is connected with capacitance sensing circuit 201
`through multiplexers 212 and 213.
`Capacitance sensor 101 includes multiplexer control 211,
`multiplexers 212 and 213, clock generator 214, signal gen-
`erator 215, demodulation circuit 216, and analog to digital
`converter (ADC) 217. ADC 217is further coupled with touch
`coordinate converter 218. Touch coordinate converter 218
`
`outputs a signal to gesture recognition unit 102.
`Thetransmit andreceive electrodes in the electrode matrix
`
`225 may be arranged so that each ofthe transmit electrodes
`overlap and cross each of the receive electrodes such as to
`form an intersection, while maintaining galvanic isolation
`from each other. Thus, each transmit electrode may be capaci-
`tively coupled with each of the receive electrodes. For
`example, transmit electrode 222 is capacitively coupled with
`receive electrode 223 at the point where transmit electrode
`222 andreceive electrode 223 intersect.
`Clock generator 214 supplies a dock signal to signal gen-
`erator 215, which produces a TX signal 224 to be supplied to
`the transmit electrodes of touch sensor 220. In one embodi-
`ment, the signal generator 215 includesa set of switches that
`operate according to the clock signal from clock generator
`214. The switches may generate a TX signal 224 by periodi-
`cally connecting the output of signal generator 215 to a first
`voltage and then to a second voltage, wherein said first and
`second voltages are different.
`The output of signal generator 215 is connected with mul-
`tiplexer 212, which allows the TX signal 224 to be applied to
`any of the M transmit electrodes of touch sensor 220. In one
`embodiment, multiplexer control 211 controls multiplexer
`212 so that the TX signal 224 is applied to each transmit
`electrode in a controlled sequence. Multiplexer 212 may also
`be used to ground,float, or connect an alternate signal to the
`other transmit electrodes to which the TX signal 224 is not
`currently being applied.
`Because of the capacitive coupling between the transmit
`and receive electrodes, the TX signal 224 applied to each
`transmit electrode induces a current within each ofthe receive
`electrodes. For instance, when the TX signal 224 is applied to
`transmit electrode 222 through multiplexer 212, the TX sig-
`nal 224 induces an RX signal 227 on the receive electrodes in
`matrix 225. The RX signal 227 on each ofthe receive elec-
`trodes can then be measured in sequence by using multiplexer
`213 to connect each of the N receive electrodes to demodu-
`
`lation circuit 216 in sequence.
`The capacitance associated with each intersection between
`a TX electrode and an RX electrode can be sensedbyselect-
`ing every available combination of TX electrode and an RX
`electrode using multiplexers 212 and 213. To improve perfor-
`mance, multiplexer 213 may also be segmentedto allow more
`
`18
`
`18
`
`

`

`US 8,830,181 B1
`
`7
`than one ofthe receive electrodes in matrix 225 to be routed
`
`to additional demodulation circuits 216. In an optimized con-
`figuration, wherein there is a 1-to-1 correspondence of
`instances of demodulation circuit 216 with receive elec-
`
`trodes, multiplexer 213 may notbe present in the system.
`Whenan object, such as a finger, approachesthe electrode
`matrix 225, the object causes a decrease in the mutual capaci-
`tance between only someofthe electrodes. For example, if a
`finger is placed nearthe intersection oftransmit electrode 222
`and receive electrode 223, the presence of the finger will
`decrease the mutual capacitance between the two electrodes
`222 and 223. Thus, the location of the finger on the touchpad
`can be determined by identifying the one or more receive
`electrodes having a decreased mutual capacitance in addition
`to identifying the transmit electrode to which the TX signal
`224 was applied at the time the decreased mutual capacitance
`was measured on the one or morereceive electrodes.
`
`By determining the mutual capacitances associated with
`each intersection of electrodes in the matrix 225 the locations
`
`20
`
`8
`In one embodiment, the detection of gestures is based on
`characteristics of the input at a touch sensor, such as the
`presence of a contact (such as a touch of a finger or other
`conductive object), a location of the contact on the touch
`sensor, and the start and end times of a contact. In embodi-
`ments where multi-touch gestures are supported, gestures
`may beidentified based on the presence, location, and start
`and end times of each of a numberof contacts at the touch
`sensor. In one embodiment, this information is communi-
`cated to the gesture recognition unit 102 via an input signal
`generated by the capacitance sensor 101.
`In one embodiment, the gesture recognition unit 102 pro-
`cesses the input signal from the capacitance sensor 101 to
`determine a set of input metrics. These input metrics may
`include, for example, a duration of a contact, a direction of
`movement ofa contact over time, a temporal overlap between
`two or more contacts, a velocity of a contact, an acceleration
`of a contact, paths of one or more contacts, a numbers of
`contacts present, or other information describing a motion, a
`location, or a presence of a contact present at a touch sensor.
`In one embodiment, the gestu