as) United States
`a2) Patent Application Publication co) Pub. No.: US 2010/0156818 Al
`
`(43) Pub. Date: Jun. 24, 2010
`Burroughetal.
`
`US 20100156818A1
`
`(54)
`
`MULTI TOUCH WITH MULTI HAPTICS
`
`Publication Classification
`
`(75)
`
`Inventors:
`
`Bobby Burrough, Cupertino, CA
`(US); Benjamin J. Pope, Newton,
`MA (US)
`
`(51)
`
`Int. Cl.
`GO6F 3/041
`GO8B 6/00
`
`(2006.01)
`(2006.01)
`
`Correspondence Address:
`BEYER LAW GROUP LLP/APPLE INC.
`P.O. BOX 1687
`CUPERTINO, CA 95015-1687 (US)
`
`(73)
`
`Assignee:
`
`APPLE INC., Cupertino, CA (US)
`
`(21)
`
`Appl. No.:
`
`12/419,174
`
`(22)
`
`Filed:
`
`Apr. 6, 2009
`
`Related U.S. Application Data
`
`(60)
`
`Provisional application No. 61/140,519, filed on Dec.
`23, 2008.
`
`(52) US. CMe ceeecccssssssssteessesssssseeeseesen 345/173; 340/407.2
`
`(57)
`
`ABSTRACT
`
`Methods and systems for processing touch inputs are dis-
`closed. The invention in one respect includes reading data
`from a multi-touch sensing device such as a multi-touch touch
`screen where the data pertains to touch input with respect to
`the multi-touch sensing device, and identifying at least one
`multi-touch gesture based on the data from the multi-touch
`sensing device and providing an appropriate multi-haptic
`response.
`
`Acoustic
`
`Haptic feedback
`H(d)
`
`Uppersurface
`sor
`
`
`126
`
`Touchevent
`
`Protective layer 120
`Touch sense layer 122
`Display device 112
`
`
`| 124 |
`
`136 /
`a
`lnaptic
`
`
`
`APPLE 1007
`
`
`
`
`
` _ Haptic actuator ~
`
` 121
` Processor 106 .
`
`1340|pigeon
`
`
`
`
`Operational components
`104
`
`Housing 102
`
`APPLE 1007
`
`1
`
`

`

`Patent Application Publication
`
`Jun. 24, 2010 Sheet 1 of 21
`
`US 2010/0156818 Al
`
`WLBiz
`
`OO}
`
`POL
`
`pauaneieda
`
`CUCIOSIS
`
`s}usu0dwios
`
`vOL
`
`
`BOeSIUJesh
`OLLAejdsiq
`
`
`
`ZLLa01nag
`
`2
`
`
`
`

`

`Patent Application Publication
`
`Jun. 24,2010 Sheet 2 of 21
`
`US 2010/0156818 Al
`
`Haptic feedback
`H(d)
`
`Uppersurface
`126
`
`
`
`Touch event
`T
`
`Operational components
`104
`
`Fig. 1B
`
`Acoustic
`isolator
`494
`
`Housing 102
`
`
`
`
`
`Protective layer 120
`
`Touch sense layer 122
`
`Display device 112
`
` _ Haptic actuator| | Haptic actuator
`
`
`
`
`121
`N40
`136 /
`ae
`S$;
`
`haptic
`fen eee ee ee ee ee ee ee ee ee ee ee ee ep ep
`
`
`
`i.Hapticprofilesa—
`Processor 106
`
`
`3
`
`

`

`Patent Application Publication
`
`Jun. 24,2010 Sheet 3 of 21
`
`US 2010/0156818 Al
`
`Acoustic
`lsolator
`421
`
`410
`
`Touch event
`T
`
`Hs J
`Hz
`
`
`NO
`
`Protective layer 120
`: Touch senselayer 122
`
`
`
`
`
`
`poe ee ee ee ee ee pe eeee ee ee Peee
`
`
`
`
`
`
`
`132LT
`Micro-coniroltier
`Haptic profiles «~~
`
`
`Housing 102
`
`
`Processor 106 |
`
`Operational components
`104
`
`Fig. 1C
`
`4
`
`

`

`Patent Application Publication
`
`Jun. 24,2010 Sheet 4 of 21
`
`US 2010/0156818 Al
`
`Acoustic
`Isolators os.
`
`Touch event
`
`Heompound <
`
`
`121
`
`
`
`Protective layer 120
`| Touch senselayer 122
`
`:
`
`
`
`nee ee ee ee eeefPe pee
`
`
`132‘|
`
`) Micro-controller
`
`Processor 106
`
`
`Haptic profiles u~
`
`Housing 102
`
`Operational components
`104
`
`Fig. 1D
`
`5
`
`

`

`Patent Application Publication
`
`Jun. 24,2010 Sheet 5 of 21
`
`US 2010/0156818 Al
`
`Acoustic
`tsolator
`
`
`
`Hhousing
`
`
`
`
`
`Touch event
`
`
`
`Housing 102
`
`
`
`
`Protective layer 120
`| Touch senselayer 122
`
`Display device 112
`
`Operational components
`104
`
`Fig. 1E
`
`6
`
`

`

`Patent Application Publication
`
`Jun. 24,2010 Sheet 6 of 21
`
`US 2010/0156818 Al
`
`Silent area H(X) = Hn, 4, 4) ~
`
`128
`
`7
`
`

`

`Patent Application Publication
`
`Jun. 24,2010 Sheet 7 of 21
`
`US 2010/0156818 Al
`
`
`
`
`
`Fig. 2B
`
`8
`
`

`

`Patent Application Publication
`
`Jun. 24, 2010 Sheet 8 of 21
`
`US 2010/0156818 Al
`
`7yybue|
`
`uol}Oe1]UO
`
`—><+
`
`e“big
`
`uoisuedxe
`
`<p
`
`O0€
`
`9
`
`
`

`

`Patent Application Publication
`
`Jun. 24,2010 Sheet 9 of 21
`
`US 2010/0156818 Al
`
`306
`
`hr
`
`300
`
`Fig.4
`
`
`
`
`
`AppliedforceF(pressureF/A) | lengthL
`><contraction
`
`expansion<>
`
`-/+V
`
`Q.
`>
`
`10
`
`
`
`10
`
`

`

`Patent Application Publication
`
`Jun. 24,2010 Sheet 10 of 21
`
`US 2010/0156818 Al
`
`
`
`Sense output node
`
`502
`
`Voltage sensed?
`
`Actuator in passive
`
`506
`
`
`Sensed voltage greater
`than threshold?
`
`Actuator in active
`mode
`
`Retrieve haptic
`profile
`
`510
`
`512
`
`Generate haptic response
`by actuator based on
`retrieved haptic profile
`
`514
`
`518
`
`
`
`
`
` y
`
`
`
`
`
`
`complete? n
`
`Set to passive
`mode
`
`Haptic respone
`
` Fig. 5
`
`516
`
`‘,
`500
`
`11
`
`11
`
`

`

`Patent Application Publication
`
`Jun. 24,2010 Sheet 11 of 21
`
`US 2010/0156818 Al
`
`GUI 600
`
`_N
`
`
`
`Image feature
`612
`
`112
`
`
`
`
`Device
`
`100“]
`
`[es
`
`12
`
`12
`
`

`

`Patent Application Publication
`
`Jun. 24,2010 Sheet 12 of 21
`
`US 2010/0156818 Al
`
`
`
`= 702
`
`Target
`706
`
`poco nono nno-one on on 8n-- 28-8888-8-5 5-5-5 5-5-5 ==8 +--+
`
`_ Region 4
`
`
`
`_-—~Region1
`veer
`oe
`ae- Region 2
`me
`om
`.
`mo
`
`—~
`
`us
`
`4
`
`Lo
`
`7
`
`704
`
`~
`
`N
`
`\ A
`
`i
`
`|
`
`Target
`
`708
`
`:
`
`13
`
`

`

`Jun. 24, 2010 Sheet 13 of 21
`
`US 2010/0156818 Al
`
`
`
`
`
`
`
`JeBuyjouosod/biebe!a36zuoibe,Ta\|uoi6edJODIE}zuolBalyebue}|uoiBaabue}
`
`
`eeee=aeTe—:=EEcmd“A|,uolbelyn}ak.
`
`
`
`
`
`
`
`
`
`
`
`
`Patent Application Publication
`
`ye6ue}
`
`yebuey|
`
`gg‘bi
`
`
`
`
`
`
`
`
`
`
`cuoies208Z08z08
`
`
`
`eye}sjusosainb}2}8}i
`
`
`
`9je}sjusosainb
`
`Ayisuaqul
`
`ondey
`
`asuodsal
`
`14
`
`14
`
`
`
`
`
`
`

`

`Patent Application Publication
`
`Jun. 24,2010 Sheet 14 of 21
`
`US 2010/0156818 Al
`
`JeBuyJouonisod
`
`V6‘6i4
`
`006>
`
`
`
`(x)a}eulpso-09YONoL
`
`
`
`
`g6‘bis
`
`"H
`
`Aysuayul
`
`ondey
`
`esuodsel
`
`15
`
`15
`
`

`

`Patent Application Publication
`
`Jun. 24, 2010 Sheet 15 of 21
`
`US 2010/0156818 Al
`
`dor“6l4~aJoyeoipul
`
`
`
`
`
`eBpeJoyesipulebpes
`
`Ayisuequl
`
`ondey
`
`esuodsal
`
`
`
`
`
`
`
`
`
`(x)a]BuIpso-09yono
`
`Je6uyJouonisod
`
`
`
`
`
`
`
`ainyes]joyabpe
`
`ae
`
`
`
`Vor‘Biz
`
`16
`
`
`
`
`
`
`
`Z/L9sunjeayabeuj
`
`16
`
`

`

`Patent Application Publication
`
`Jun. 24, 2010 Sheet 16 of 21
`
`US 2010/0156818 Al
`
`1100
`
`the presence ofat
`
`least a first finger
`and a second finger.
`are detected
`
`1102
`
`
`
`e. presence of the
`
`
`two fingers represents ..
`a gesture
`
`
`1104
`
`Fig. 11
`
`haptic. devices nearest the”
`touch point are set to active
`mode in order to provide a
`vibrotactile
`
`1106
`
`distance betweenat least the
`.-two fingers is compared
`
`1108
`
`- Distanceincreasing? —
`
`
`1110
`
`
`Generate zoom in signal
`
`Generate zoomout signal
`
`Generate zoom in haptic ~-
`signal
`
`Generate zoom out haptic
`signal
`
`
`
`1114
`
`1118
`
`17
`
`17
`
`

`

`Patent Application Publication
`
`1202
`
`US 2010/0156818 Al
`
`Jun. 24,2010 Sheet 17 of 21
`
`
`
`18
`
`18
`
`

`

`Patent Application Publication
`
`Jun. 24,2010 Sheet 18 of 21
`
`US 2010/0156818 Al
`
`1204
`
`|
`
`
`
`421000 N | |
`
`California\\ SS —|
`
`
`
`yoo2
`
`4204
`
`
` /-N. California
`
`\
`
`1204. 42D.
`
`19
`
`19
`
`

`

`Patent Application Publication
`
`Jun. 24,2010 Sheet 19 of 21
`
`US 2010/0156818 Al
`
`1204 |
`
`1204 |
`
` —N. California
`
`
`
`
`
`20
`
`20
`
`

`

`Patent Application Publication
`
`Jun. 24,2010 Sheet 20 of 21
`
`US 2010/0156818 Al
`
`
`
`age
`
`\ 4. Sar@arlos |eo “ye
`
` Ye
`
`21
`
`21
`
`

`

`Patent Application Publication
`
`Jun. 24, 2010 Sheet 21 of 21
`
`US 2010/0156818 Al
`
` virtual scroll wheel
`
`is presented on the
`display:
`
`1300
`
`1302
`
`
`
`
`
`
`| thepresenceofthe
`
`two fingers represents
`
`a gesture
`
`1304
`
`the initial positionof the
`fingers on the virtual scroll
`wheel is. set.
`
`rotate signal 1s generated
`when the angle of the fingers
`change relative to the
`reference point
`
`rotate signal can be used to
`-rotate the virtual scroll
`“wheel in the direction of
`finger rotation
`
`1306
`
`1308
`
`1310
`
`Generate haptic click signal
`
`1312
`
`Fig. 13
`
`22
`
`22
`
`

`

`US 2010/0156818 Al
`
`Jun. 24, 2010
`
`MULTI TOUCH WITH MULTI HAPTICS
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`[0001] This patent application takes priority under 35 US.
`C. 119(e) to U.S. Provisional Application Ser. No. 61/140,
`519 entitled MULTI TOUCH WITH MULTI HAPTICSby
`Burroughetal., filed Dec. 23, 2008 which is incorporated by
`reference in its entirety for all purposes.
`
`BACKGROUNDOF THE INVENTION
`
`1. Field of the Invention
`[0002]
`[0003] The present invention relates generally to providing
`multi-touch/multi-haptic systems and methods.
`[0004]
`2. Description of the Related Art
`[0005] Multi-touch devices have advantages over conven-
`tional single point sensing touch devices in that they can
`distinguish more than one object(finger) in contrast to single
`point devices that are simply incapable ofdistinguishing mul-
`tiple objects. In most cases, multi-touch devices monitor a
`sensing surface for a touch or near touch, and when a touch
`occurs determines the distinct areas of contact and identifies
`
`the contacts via their geometric features and geometric
`arrangement. Once identified or classified, the contacts are
`monitored for various motions, actions or events. The con-
`tacts and motions thereof are then converted into inputs for
`controlling some aspect of an electronic device.
`[0006] Multi-touch devices can be embodied in various
`forms including but not limit to standard touch pads, large
`extended palm pads, touch screens, touch sensitive housings,
`etc. Furthermore, multi-touch devices can be placed in vari-
`ous electronic devices including but not limited to computers
`such as tablet computers, laptop computers, desktop comput-
`ers as well as handheld computing devices such as media
`players (e.g., music, video, games), PDAs, cell phones, cam-
`eras, remote controls, and/orthe like. The multi-touch devices
`can also be placed on dedicated input devices such as touch
`screen monitors, keyboards, navigation pads, tablets, mice,
`and the like. Essentially, multi-touch devices can be applied
`to any surface, and can be found in any consumerelectronic
`product that requires inputs.
`[0007]
`Since multi-touch devices provide a number of
`inputting operationsat a single location (input surface), input-
`ting with multi-touch devices can be very efficient. The user
`can maintain their hand(s) at the multi-touch surface without
`having to movetheir hand(s) to address other input devices.
`For example, conventional systems typically include a key-
`board and a separate mouse. In order to use the mouse, the
`user must move their hand from the keyboard and onto the
`mouse. In order to keyboardefficiently (both hands), the user
`must movetheir hand from the mouseto the keyboard. This
`inputting sequence is very inefficient. For one, only one
`device can be used effectively at a given time. For another,
`there is wasted time between each inputting step. In contrast,
`with multi-touch surfaces the user can generate both static
`commands(e.g., keyboarding) and manipulative commands
`(e.g., tracking) from the samelocation and at the same time.
`The user therefore does not have to move their hands to
`
`perform different inputting tasks. The user simply provides
`different chords or finger motions to generate a numberof
`inputs either sequentially or simultaneously. In one example,
`the user can provide key commands with taps at specific
`
`locations of the multi-touch surface while allowing tracking
`from all locations of the multi-touch surface.
`
`Ilowever, research has shown that providing the
`[0008]
`multi-touch surface with the ability to provide physical (hap-
`tic) feedback makes the multi-touch experience even more
`efficient andrealistic to the user. For example, physical key-
`boards provide a physical indication (a bump, for example)
`indicative of the homekey. This physical sensation can not be
`provided by a conventional multi-touch system thereby forc-
`ing the user to visually locate the home key thereby making
`keyboard use less efficient and fatiguing. However, by pro-
`viding a physical facsimile of the home key bump using an
`actuator that provides a physical sensation to the user provid-
`ing an approximate representation of the bump, the user’s
`experience of the MT keyboard (and any multi-touch system
`for that matter) can be morerealistic and therefore more
`enjoyable.
`[0009] Therefore, a system that enhances the multi-touch
`experience by incorporating a corresponding physical
`response(s) is described.
`
`SUMMARYOF THE INVENTION
`
`[0010] The invention relates, in one embodiment, to an
`apparatus and methodfor providing multi-touch haptic feed-
`back. The apparatus includes, at least, a touch pad having a
`touch sensitive surface arranged to receive a user provided
`multi-touch event associated with at least two different loca-
`tions on the touch sensitive surface, a multi-touch detection
`mechanismoperatively coupledto the touchsensitive surface
`that detects the multi-touch event and generates a correspond-
`ing a multi-touch signal, and a plurality of haptic feedback
`devices operatively coupled to the multi-touch detection
`mechanism and the touch sensitive surface cooperatively
`arranged to concurrently provide tactile feedback at each of
`the at least two different locations on the touch sensitive
`surface in response to the multi-touch signal wherein the
`tactile feedback at each of the at least two different locations
`
`are discreet from one another. When the multi-touch signal
`indicates that the multi-touch event is a dynamic multi-touch
`event indicating a change in the multi-touch event, then the
`tactile feedback at each of the at least two different locations
`
`is updatedto reflect the change in the multi-touch event.
`[0011]
`It should be noted that in some cases the tactile
`feedback event can be different for each of the at least two
`different locations.
`
`[0012] The invention relates, in another embodiment, to a
`multi-touch haptic mechanism. The multi-touch haptic
`mechanism includes, at least, a touch pad having a touch
`sensitive surface arranged to detect a user touch event at
`substantially any location on the touch sensitive surface and a
`plurality ofindependenthaptic devices operatively coupled to
`the touch sensitive surface each providing a corresponding
`type oftactile feedback thereby providing a tactile feedback
`at substantially any location on the touch sensitive surface at
`whichthe user touch event has occurred, wherein each of the
`plurality of independent haptic devices only respondsto the
`user touch event in one or more associated regions of the
`touch sensitive surface. Whenat least two ofthe plurality of
`independent haptic devices cooperate to provide a type of
`haptic response that is different than that type provided by
`either of the at least two independent haptic devices sepa-
`rately.
`[0013] The inventionrelates, in another embodiment, to an
`integrated device arranged to act as both a force sensing
`
`23
`
`23
`
`

`

`US 2010/0156818 Al
`
`Jun. 24, 2010
`
`device and a haptic feedback device. The device includes, at
`least, a touch sensitive surface, a controller unit, and a
`mechanical actuator coupled with the controller unit and the
`touch sensitive surface. The integrated device acts as the force
`sensing device by generating an output voltage in direct pro-
`portion to a force applied to the mechanicalactuator by a user
`touching the touch sensitive surface, sensing the output volt-
`age by the controller unit and converting the sensed output
`voltage to an indication of the applied force. Only when the
`sensed output voltage exceeds a voltage threshold level does
`the integrated device act as the haptic feedback device by
`halting the sensing ofthe output voltage by the controller unit
`activating the mechanical actuator by the controller unit,
`wherein the activated mechanical actuator imparts a physical
`force to the touch sensitive surface that results in a vibro-
`
`tactile response (subcutaneoustissue activated) felt by the
`user commensurate with the force applied by the user.
`[0014] The inventionrelates, in another embodiment, to an
`electronic device. The electronic device includes, at least, a
`touch pad having a touchsensitive surface arrangedto process
`a user touch event anda plurality of haptic feedback devices
`each of which is operatively coupled to the touch sensitive
`surface and each respondingto the user touch event only ina
`specific region ofthe touch sensitive surface and arranged to
`providetactile feedback singly or in combination with others
`of the plurality of haptic feedback devices in responseto the
`user touch event. When the touch sensitive regionsof at least
`two of the plurality ofhaptic devices overlap, ifthe user touch
`event occurs in the overlapping region, then the at least two
`haptic devices cooperate to provide a combinedhaptic feed-
`back response based upon the location in the overlapping
`region of the user touch event.
`[0015] The inventionrelates, in another embodiment, to an
`electronic device. The electronic device includes, at least, a
`touch pad having a touchsensitive surface arranged to receive
`a user touch event provided by a user, a controller coupled and
`in communication with the touch pad arranged to at least
`analyze the user touch event and/or a state of the touch pad
`and based uponthe analysis provide a user touch event signal
`in response to the user touch event, and at least one haptic
`device operatively coupled to the controller arranged to
`receive the user touch event signal, wherein the at least one
`haptic device responds to the user touch event signal by
`providing an appropriate haptic feedback responseto the user
`based upon the analysis provided by the controller.
`[0016]
`In one embodiment, the touch sensitive surface is
`arranged to receive different types of user touch events each
`being characterized by an amountof pressure applied on the
`touch sensitive surface by a userandat least one haptic device
`operatively coupled to the touch sensitive surface arranged to
`provide a specific type of tactile feedback corresponding to
`the amountof pressure applied to the touch sensitive surface
`by the user.
`[0017]
`It should be noted that in each of the embodiments
`described above, the methods can be implemented using a
`touch based input device such as a touch screen or touch pad,
`moreparticularly a multi-touch touch based input device, and
`even more particularly a multi-touch touch screen. It should
`also be notedthatthe gestures, gesture modes, gesturalinputs,
`etc. can correspond to any of those described below in the
`detailed description. For example, the gestures can be asso-
`
`ciated with zooming, panning,scrolling, rotating, enlarging,
`floating controls, zooming targets, paging, inertia, keyboard-
`ing, wheeling, and/orthe like.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0018] The invention will be readily understood by the
`following detailed description in conjunction with the accom-
`panying drawings, wherein like reference numerals designate
`like structural elements, and in which:
`[0019]
`FIGS. 1A-1E are a series of block diagrams of a
`system, in accordance with one embodiment of the present
`invention.
`[0020]
`FIGS. 2A-2B shows a multi-point multi-haptic sys-
`tem having a multi-touch surface that incorporatesa plurality
`of haptic devices in accordance with an embodimentof the
`invention.
`
`FIG. 3 shows a schematic diagram of a representa-
`[0021]
`tive piezo-electric haptic assembly.
`[0022]
`FIG. 4 shows a schematic diagram of the haptic
`assembly shown in FIG. 3 configured to act as a pressure
`sensor.
`
`FIG. 5 shows a flowchart detailing a process in
`[0023]
`accordance with an embodimentof the invention.
`
`FIG. 6 showsdisplay device displaying representa-
`[0024]
`tive haptic active GUI elements in accordance with an
`embodimentof the invention.
`
`FIG. 7 showsrepresentative GUI button elements in
`[0025]
`accordance with an embodimentof the invention.
`
`FIGS. 8A-8B showsrepresentative GUIbutton ele-
`[0026]
`ment and associated haptic profile in accordance with an
`embodimentof the invention.
`
`FIGS. 9A-9B showsa representative slider element
`[0027]
`and associated haptic profile in accordance with an embodi-
`mentofthe invention.
`
`FIGS. 10A-10B showsa feature edge detection sys-
`[0028]
`tem in accordance with an embodimentof the invention.
`
`FIG. 11 isa diagram ofa zoom gesture method 1100
`[0029]
`in accordance with an embodimentofthe invention.
`
`FIGS. 12A-12H illustrates a display presenting a
`[0030]
`GUIobject in the form of a map of North America with
`embedded levels which can be zoomed.
`
`FIG. 13 isa diagram ofa GUIoperational method in
`[0031]
`accordance with one embodimentofthe present invention.
`
`DETAILED DESCRIPTION OF SELECTED
`EMBODIMENTS
`
`[0032] Reference will now be made in detail to selected
`embodiments an example ofwhich isillustrated in the accom-
`panying drawings. While the invention will be described in
`conjunction with a preferred embodiment, it will be under-
`stood that it is not intended to limit the invention to one
`preferred embodiment. Tothe contrary,it is intended to cover
`alternatives, modifications, and equivalents as can be
`included within the spirit and scope of the invention as
`defined by the appendedclaims.
`[0033] The invention relates to multi-touch haptic feed-
`back. Multi-touch haptic feedbackrefers to haptic techniques
`capable of providing multiple and discretely located haptic
`sensations across a surface. The haptic system can for
`example include a plurality of haptic nodes, each of which is
`capable of issuing vibro-tactile sensations (at the same time
`or different times and with the same intensity or different
`intensity). The haptic nodes can for example be configured in
`
`24
`
`24
`
`

`

`US 2010/0156818 Al
`
`Jun. 24, 2010
`
`a matrix or array. In one embodiment, the haptic nodes are
`mappedto touch sensing nodes. Each touch sensing node can
`be assigned one or more haptic nodes. The haptic nodes are
`typically proximate the touch sensing nodes to whichit has
`been assigned.
`[0034]
`In one embodiment, the touch sensing surface is a
`multi touch surface thus making a multi touch multi-touch
`haptic device. In so doing haptic feedback can be provided
`that indicates information about a multi touch event. For
`
`example, the surface under a moving finger can be actuated
`while the surface under the non-movingfinger remainsstatic.
`In another example, the surface under the moving finger is
`actuated concurrently with a signal being passed to the other
`finger indicating that a multi touch action is occurring. In this
`way, the signals taken together can indicate the nature of the
`underlying action being taken by the user. For example, if an
`object (such as an image) is being expanded or reduced in size
`a larger/more intense signal could be generated (either by
`increasing frequency or amplitude). It is also contemplated
`that isolated feedback can be used to provide an on-screen
`click-wheelor other such user input where the touch screenis
`used to simulate the “clicks” of the click wheel both audibly
`and via tactile feedback.
`
`[0035] The described embodiments generally pertain to
`gestures and methods of implementing gestures with associ-
`ated physical
`feedback with touch sensitive devices.
`Examples of touch sensitive devices include touch screens
`and touch pads. One aspect ofthe invention describes a touch
`sensitive input device able to recognize at least two substan-
`tially simultaneously occurring gestures using at least two
`different fingers or other objects (hereinafter referred to as a
`multi-touch event). The touch sensitive input device commu-
`nicates with an array ofhaptic feedback devices(also referred
`to as haptic actuators) each arranged to provide haptic feed-
`back accordingto a hapticprofile in response to a multi-touch
`event. In another aspect of the invention, each finger receives
`different haptic feedback (multi-haptic) depending upon the
`location on the touch sensitive input device each finger is
`placed. In anotheraspect ofthe invention, a compound haptic
`feedback can be provided that combines the output from at
`least two different haptic actuators to form the compound
`responsethatis different from that provided by the twoorigi-
`nating haptic actuators. In another embodiment, an integrated
`device is described that can act as both a force sensing device
`and a haptic feedback device. In still another embodiment, a
`handheld portable device is described having a housing and a
`user interface are acoustically isolated from each other.In this
`way, the housing anduser interface and having non-interfer-
`ing and independenthaptic responses.
`[0036] These and other aspects of the invention are dis-
`cussed below with reference to FIGS. 1-13. However, those
`skilled in the art will readily appreciate that the detailed
`description given herein with respect to these figures is for
`explanatory purposes as the invention extends beyond these
`limited embodiments.
`
`FIG. 1A-1E are block diagramsof a representative
`[0037]
`electronic device or system 100,
`in accordance with one
`embodimentofthe present invention. Electronic device 100
`can correspond to a computer (such as a desktops or laptops)
`as well as small form factor electronic devices that can
`
`include portable consumerelectronic products such as cell
`phones, PDA, media players and/orthe like. As such, portable
`electronic device 100 can be sized for one-handed operation
`and placement into small areas such as a pocket. Portable
`
`electronic device 100 can process data and moreparticularly
`media suchas audio, video, images, etc. As such,the portable
`electronic device 100 can correspond to a music player, game
`player, videoplayer, personaldigital assistant (PDA), such as,
`for example, an iPod™, an iPod Nano™,an iPod Shuffle™,
`an iPod™ Touch or an iPhone™available by Apple Inc. of
`Cupertino, Calif. In some cases, portable electronic device
`100 can communicate wirelessly (with or withoutthe aid of a
`wireless enabling accessory system) and/or via wired path-
`ways(e.g., using traditional electrical wires).
`[0038]
`Portable electronic device 100 includes a housing
`102. Housing 102 can be formed of any numberof materials
`including for example plastics, metals, ceramics andthe like.
`In one embodiment, housing 102 can be formedofstainless
`steel in order to provide an aesthetic and appealing look and
`feel as well as provide structural integrity and support forall
`sub-assemblies installed therein. Housing 102 can define a
`cavity configured to at least partially enclose any suitable
`numberof operational electronic components 104 used by
`portable electronic device 100 to carry out its intended func-
`tions. Operational electronic components 104 can include
`processor 106 that can operate (in conjunction with an oper-
`ating system) to execute computer code and produce and use
`data. Processor 106 can be implemented on a single-chip,
`multiple chips or multiple electrical components. For
`example, various architectures can be used for the processor
`106, including dedicated or embeddedprocessor, single pur-
`pose processor, controller, ASIC, and so forth. The operating
`system, other computer code and data can reside within a
`memory 108 that can be operatively coupled to processor 106.
`By way of example, memory 108 can include Read-Only
`Memory (ROM), Random-Access Memory (RAM), flash
`memory, hard disk drive and/or the like. Operational compo-
`nents 104 can also include a numberof input/output (1/O)
`devices 109. Such devices can include audio output devices
`such as headphonejacks, data ports (such as J.E.E.E. 1392
`compliant, USB, etc.), and so on.
`[0039]
`Portable electronic device 100 can also include a
`user interface 110 that can operate to both receive user inputs
`and provide information to a user. In the described embodi-
`ment, user interface 110 can include display device 112 that
`can be operatively coupled to processor 106 by way of bus
`114. Display device 112 can correspondto any knowndisplay
`technology such as a plasma, LCD,or an organic light emit-
`ting diode (OLED). It should be notedthat in the embodiment
`shown in FIGS. 1A-1E,display device 112 is integrated with
`the electronic device 100. However, display device 112 can
`also be configured as a component separate from portable
`electronic device 100 in which case display device 112 would
`be considered a peripheral device that can be coupled to
`portable electronic device 100 by way of a wired connection
`(such as a peripheral bus or cable) or a wireless connection
`such as IR, RF, Bluetooth or the like (among others).
`[0040]
`Insome cases, display device 112 presents graphical
`user interface (GUI) 116 on display device 112. GUI 116 can
`provide an easy to use interface between a user of portable
`electronic device 100 and the operating system or application
`running thereon. Generally speaking, GUI 116 iconically
`represents programs,
`files and operational options with
`graphical images. The graphical images can include win-
`dows, fields, dialog boxes, menus, icons, buttons, cursors,
`scroll bars, etc. Such images can be arranged in predefined
`layouts, or can be created dynamically to serve the specific
`actions being taken by a user. During operation, the user can
`
`25
`
`25
`
`

`

`US 2010/0156818 Al
`
`Jun. 24, 2010
`
`select and activate various graphical images in orderto ini-
`tiate functions and tasks associated therewith. By way of
`example, a user can select a button that opens, closes, mini-
`mizes, or maximizes a window, or an icon that launches a
`particular program. GUI 116 can additionally or alternatively
`display information, such as noninteractive text and graphics,
`for the user on the display device 112.
`[0041] As shown more clearly in FIG. 1B in a side view
`perspective of device 100, user interface 110 can include
`protective layer 120 disposed on top of display device 112. In
`this way, protective layer 120 can be used as protective top
`layer of transparent or semitransparent material
`(clear)
`thereby affording display device 112 protection from poten-
`tially damaging external insults causedby, for example, sharp
`objects, dropping, and so on andyet still allow any image
`presented by display device 112 to be clearly viewed by a
`user. Protective layer 120 can be formed of many well known
`transparent materials such as glass (e.g., referred to as cover
`glass), and moreparticularly highly polished glass. It should
`be appreciated, however, that other transparent materials (or
`at least translucent materials) such asclear plastic may also be
`used. In some embodiments, protective top layer 120 can be
`acoustically isolated from housing 102 using, for example,
`acoustic isolation buffers 121. By acoustically isolating hous-
`ing 102 and protective top layer 120 from each other, it is
`possible to provide separate and independent haptic
`responses, one directed at housing 102 and another directed at
`protective top layer 120 withoutinterfering with each other.
`For example, it may be desirable to provide one type ofhaptic
`response at protective layer 120 and another type haptic
`responseat housing 102 at the same timeor at a different time
`independent of each other or in some cases one being the
`result of or related to the other.
`
`[0042] User interface 110 can be touch sensitive suitable
`for receiving one or more user touch events by which infor-
`mation can be passed betweenthe user andthe portable elec-
`tronic device 100. In some cases, the one or more inputsin the
`form of user touch events can be substantially simultaneously
`received (e.g., multi-touch). In these embodiments, user
`interface 110 is rendered touch sensitive by meansof a touch
`sensing layer 122 that can be disposed below protective layer
`120 such that touch sensing layer 122 is between protective
`layer 120 and the display device 112. This arrangement can be
`accomplished by, for example, applying touch sensing layer
`122 to display device 112 or by applying touch sensing layer
`122 to protective layer 120 using any numberof attachment
`processes, such as printing, depositing,
`laminating, etc.
`Touch sensing layer 122 generally includesat least one touch
`sensing device 124 configured to detect an object in close
`proximity to or exerting pressure on an upper surface 126 of
`protective layer 120. In keeping with the wide applicability of
`the invention, sensing device 124 can be widely varied and
`can be configured to activate as the finger touches the upper
`surface 126. In the simplest case, an electrical signal is pro-
`duced each timea finger (or other appropriate object) passes
`a sensor. The numberof signals in a given time frame may
`indicate location, direction, speed and acceleration of the
`finger on the touch sensitive portion,1.e., the more signals, the
`more the user movedhis or herfinger.
`[0043] Touch sensing layer 122 can be configuredto act as
`a multi-touch input device. Multi-touch input devices have
`several advantages over conventional single point devices in
`that they can distinguish more than one object (finger). As a
`multi-touch input device, touch sensing layer 122 can distin-
`
`guish a wide range ofdifferent gestures. By way of example,
`the gestures may be single point or multi-touch gestures,
`static or dynamic gestures, continuousor segmented gestures,
`and the like. It should be notedthat single point gestures are
`those gestures that are performed with a single contact point,
`e.g., the gesture is performed with a single touch as for
`example from a single finger, a palm or a stylus. Multi-touch
`gestures are those gestures that can be performed with mul-
`tiple points, e.g., the gesture is performed with multiple
`touches as for example from multiple fingers, fingers and
`palms, a finger and a stylus, multiple sty