( * ) Notice :
`
`( 52 )
`
`( 56 )
`
`( 2006 . 01 )
`( 2013 . 01 )
`
`. . . . . .
`
`( 12 ) United States Patent
`Lewbel
`I
`( 54 ) TACTILE INTERACTION WITH CONTENT
`( 75 ) Inventor : Hannah Rebecca Lewbel , Campbell ,
`CA ( US )
`( 73 ) Assignee : Amazon Technologies , Inc . , Seattle ,
`WA ( US )
`Subject to any disclaimer , the term of this
`patent is extended or adjusted under 35
`U . S . C . 154 ( b ) by 222 days .
`( 21 ) Appl . No . : 12 / 940 , 936
`( 22 ) Filed :
`Nov . 5 , 2010
`( 51 )
`Int . CI .
`G06F 3 / 041
`G06F 3 / 0481
`U . S . CI .
`CPC . . . . . . . . . . G06F 3 / 04812 ( 2013 . 01 ) ; G06F 37041
`( 2013 . 01 )
`( 58 ) Field of Classification Search
`. . . 345 / 173
`USPC
`. . . . . . . . . . . . . . . . . .
`. . . . . . . . . . . . . . .
`7
`See application file for complete search history .
`References Cited
`U . S . PATENT DOCUMENTS
`7 , 106 , 220 B2 *
`9 / 2006 Gourgey . . . . . . . . . . . . . . GO9B 21 / 005
`341 / 27
`8 , 780 , 060 B2
`7 / 2014 Maschmeyer et al .
`2005 / 0024346 A1 *
`2 / 2005 Dupraz et al . . . . . . . . . . . . . . . 345 / 179
`2005 / 0122317 A1 *
`6 / 2005 Schaaf . . . . . . . . . . . . . . . . . . B60K 35 / 00
`345 / 173
`2007 / 0298877 A1 * 12 / 2007 Rosenberg
`. . . . . . . . . .
`463 / 30
`4 / 2009 Chang et al . . . . . . . . . . . . . . . . 345 / 173
`2009 / 0085877 A1 *
`2009 / 0085881 A1 *
`4 / 2009 Keam . . . . . .
`345 / 173
`2009 / 0160793 A1 *
`6 / 2009 Rekimoto . . . . . . . . . . . . . GO6F 3 / 0414
`345 / 173
`
`| HAO WANATHA MARIA DEL TUTORIAL AT MAI M ULT
`
`US009760241B1
`
`( 10 ) Patent No . :
`( 45 ) Date of Patent :
`
`US 9 , 760 , 241 B1
`Sep . 12 , 2017
`
`7 / 2009 Terlizzi . . . . . . . . . . . . . . . G06F 3 / 016
`2009 / 0167704 A1 *
`345 / 173
`9 / 2009 Baker , III . . . . . . . . . . . G06T 17 / 05
`2009 / 0225073 A1 *
`345 / 419
`9 / 2009 Kim . . . . . . . . . . . . . . . . . .
`GO6F 3 / 016
`2009 / 0227296 A1 *
`455 / 566
`10 / 2009 Bloebaum . . . . . . . . . . . . . . . G06F 3 / 016
`2009 / 0251421 Al *
`345 / 173
`10 / 2009 Chung et al . . . . . . . . . . . . . . . . . . 345 / 173
`2009 / 0256814 A1 *
`5 / 2010 Klinghult . . . . . . . . . . GO6F 3 / 0412
`2010 / 0123686 A1 *
`345 / 178
`6 / 2010 Duarte et al . . . . . . . . . . . . . . . 340 / 815 . 4
`2010 / 0156656 A1 *
`6 / 2010 Burrough . . . . . . . . . . . . . . . . GO6F 3016
`2010 / 0156818 Al *
`345 / 173
`2010 / 0283742 A1 * 11 / 2010 Lam . . . . . . . . . . . . . . . . . . . . GO6F 3 / 04883
`345 / 173
`2010 / 0328053 A1 * 12 / 2010 Yeh . . . . . . . . . . . .
`GO6F 3 / 041
`340 / 407 . 2
`2011 / 0043077 A1 *
`2 / 2011 Yeh et al . . . . . . . . . . . . . . .
`310 / 338
`345 / 592
`2011 / 0096087 A1 *
`4 / 2011 Chun
`G06F 3 / 016
`2011 / 0128236 A1 *
`6 / 2011 Faubert . . .
`345 / 173
`
`( Continued )
`Primary Examiner - Lisa Landis
`( 74 ) Attorney , Agent , or Firm — Lee & Hayes , PLLC
`( 57 )
`ABSTRACT
`Some implementations provide tactile interaction to a user
`during consumption of content . For instance , when a user
`accesses a content item on an electronic device having a
`display , the user may touch the display at a location corre
`sponding to a first feature in a displayed image to generate
`a first type of tactile output . Moving the user ' s finger to a
`location of a second feature of the image may result in a
`second type of tactile output , different from the first . Addi
`tionally , in some implementations , a force of a touch input
`may be determined , and the tactile output may vary based on
`the amount of force of the touch input . Further , in some
`implementations , the tactile output may vary based on a time
`duration of a touch input .
`23 Claims , 14 Drawing Sheets
`
`2007
`
`?????
`
`206
`
`210 -
`
`202
`
`204
`
`Aqui
`
`- 216
`
`000 0000 0000 9000 000 000
`
`1
`
`APPLE-1036
`
`

`

`US 9 , 760 , 241 B1
`Page 2
`
`( 56 )
`
`References Cited
`U . S . PATENT DOCUMENTS
`2011 / 0221694 A1 *
`9 / 2011 Karaoguz et al . . . . . . . . . . . . . 345 / 173
`2011 / 0275412 A1 *
`11 / 2011 Khawand . . . . . . . . . . . . GO6F 3 / 04847
`455 / 566
`2012 / 0050176 A1 *
`3 / 2012 Chin .
`345 / 173
`20121
`345 / 174
`5 / 2012 Son et al . . . . . . . . . . . . . . . . . . . . . . 345 / 174
`2012 / 0105367 A1 *
`5 / 2012 Son et al .
`* cited by examiner
`
`2
`
`

`

`U . S . Patent
`
`Sep . 12 , 2017
`
`Shet l of 14
`
`US 9 , 760 . 241 B1
`
`OUTPUT COMPONENTS 116
`
`AUDIO OUTPUT 120
`VISUAL OUTPUT 122
`VARIABLE TACTILE OUTPUT 118
`
`FEEDBACK MODULE 110
`
`TOUCH DEPENDENT OUTPUT 114
`
`FIG . 1
`
`
`
`TOUCH SENSOR 102
`
`TOUCH LOCATION 106
`
`TOUCH FORCE 108
`
`DISPLAY 104
`
`CONTENT 112
`
`100
`
`3
`
`

`

`U . S . Patent
`
`Sep . 12 , 2017
`
`Sheet 2 of 14
`
`US 9 , 760 , 241 B1
`
`
`
`Wawili hata
`
`200
`
`208 - 1
`208
`
`206
`
`210
`
`5202 7204
`
`
`
`??????? ?????
`
`2 .
`
`.
`
`7111111111111211212
`
`2
`
`111111111111111110111000111111111111111111110 * * * 1111111111111111111111 * *
`
`218
`
`1218
`
`- 218
`
`
`
`RESEARCHERSTRERTERRE CARRER
`
`E RCHER
`
`" HERCH
`
`WWWWWWWWWW
`
`000 0000 0000 0000 1000
`
`FIG . 2
`
`4
`
`

`

`U . S . Patent
`
`Sep . 12 , 2017
`
`Sheet 3 of 14
`
`US 9 , 760 , 241 B1
`
`3002
`
`PROCESSOR
`302
`
`MEMORY
`304
`
`CONTENT ITEM ( S )
`306
`
`OS AND USER
`INTERFACE MODULE
`308
`
`USER CREDENTIALS
`314
`
`OTHER DATA
`312
`
`CONTENT PRESENTATION
`APPLICATION ( S )
`310
`
`TOUCH - SENSOR CONTROLLER
`316
`
`FEEDBACK MODULE
`318
`
`TACTILE OUTPUT COMPONENT
`320
`
`SPEAKERS
`322
`
`NETWORK INTERFACE
`326
`
`POWER SOURCE
`330
`
`- - -
`-
`- -
`- -
`-
`
`TOUCH SENSOR
`204
`
`DISPLAY
`202
`
`OTHER I / O COMPONENTS
`324
`
`REMOVABLE STORAGE
`328
`
`- - -
`- -
`- - -
`
`200
`
`www
`02
`
`VELEELLE LLLLLL
`
`FOCO 0030 1191
`
`FIG . 3
`
`5
`
`

`

`U . S . Patent
`
`Sep . 12 , 2017
`
`Sheet 4 of 14
`
`US 9 , 760 , 241 B1
`
`- - - - - -
`
`-
`
`- -
`
`- -
`
`
`
`
`
`?? ???? ???? 228 ???????????????????????????? 888222122220322337227337822212232
`
`
`
`
`
`202
`
`- 002
`
`202
`
`7888888338388
`
`/
`
`404
`2014
`
`to
`
`- - 402
`
`LI
`
`I
`
`I
`
`:
`
`: :
`
`: :
`
`: : :
`
`: -
`
`:
`
`- - - - - - - - - - -
`
`- - - -
`
`7402
`
`11
`
`Il
`-
`- - -
`- -
`
`Il
`
`| 11 |
`
`11
`11
`.
`het moment
`
`11
`
`
`
`
`
`033238228 ???????????? ???????????????? 8
`
`
`
`
`
`222222203207882122232 ????? 8888888338888 >
`
`@
`
`@
`
`000
`0000
`0000
`0000
`
`*
`
`*
`
`*
`
`FIG . 4B
`
`wer
`
`38889 o
`
`9999999999999999999possonogsoosy P
`000
`000
`0000
`0000
`CLUDW
`000
`
`- -
`
`cases
`on some
`.
`
`FIG . 4A
`
`6
`
`

`

`U . S . Patent
`
`Sep . 12 , 2017
`
`Sheet 5 of 14
`
`US 9 , 760 , 241 B1
`
`Looz
`
`*
`
`*
`
`*
`
`* * *
`
`*
`
`* *
`
`* *
`
`* * *
`
`*
`
`*
`
`*
`
`*
`
`* *
`
`* *
`
`*
`
`*
`
`* *
`
`*
`
`*
`
`* * *
`
`* *
`
`*
`
`*
`
`*
`
`*
`
`* *
`
`*
`
`DCIau 10
`
`*
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ???????????????????????????????????????????????????????????????????????????????????????????
`
`-
`
`- - -
`
`- 202
`
`*
`
`*
`
`* * *
`
`* *
`
`*
`
`*
`
`*
`
`*
`
`*
`
`*
`
`* *
`
`*
`
`*
`
`*
`
`*
`
`* * *
`
`* * *
`
`*
`
`*
`
`*
`
`* *
`
`502
`
`640 _ 510 , 50875067504 ,
`5147
`
`522
`
`u
`
`- 528
`526
`
`532
`
`524
`
`4530
`0000
`0000
`0000
`000
`000
`
`*
`
`* *
`
`* * * * * *
`
`* *
`
`*
`
`*
`
`*
`
`*
`
`FIG . 5
`
`7
`
`

`

`U . S . Patent
`
`Sep . 12 , 2017
`
`Sheet 6 of 14
`
`US 9 , 760 , 241 B1
`
`FIG . 6
`
`202 ,
`
`602 16067 608
`
`200 htu
`
`8
`
`

`

`U . S . Patent
`
`Sep . 12 , 2017
`
`Sheet 7 of 14
`
`US 9 , 760 , 241 B1
`
`GOG
`
`2004
`
`YA
`7022
`
`704
`
`EEEEEEEEEEEEEEEEEEEEEEEE66666666666666666
`
`6 666666666666666666666666666FF6A
`
`, 706 > 2009
`
`Da - 714
`712
`
`706
`
`4710
`
`708
`912
`
`000 0001 0000 0000 0000 000
`
`FIG . 7
`
`STICO . . . . . . . . . .
`
`CLILO WECTURE
`
`CARATTER
`MWWWWWWWWW
`
`WAMMAMMAMMA MAGAVAMAMARAAPKAMMMM1114114166KWA
`
`9
`
`

`

`U . S . Patent
`
`Sep . 12 , 2017
`
`Sheet 8 of 14
`
`US 9 , 760 , 241 B1
`
`WW
`
`W
`
`CDD
`000
`0000
`
`804
`
`FIG . 8B
`
`* * * * * * * *
`
`* 000
`
`2 /
`
`60
`
`8102
`
`????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
`
`999999 99999999
`
`RITURII
`W
`2222222222222
`
`IMI
`*
`* *
`*
`
`*
`
`MIT
`
`* *
`
`812
`
`2018
`
`804
`
`000
`
`wborne 583ST
`
`FIG . 8A
`
`
`
`202 7204
`
`200
`
`1204
`202
`
`2007
`
`HR
`
`10
`
`

`

`U . S . Patent
`
`Sep . 12 , 2017
`
`Sheet 9 of 14
`
`US 9 , 760 , 241 B1
`
`|
`
`904
`
`ARKAKKAKAKKUA
`
`0000
`
`commenteret
`
`O
`
`902
`
`FIG . 9B
`
`912
`
`KOL6
`
`????????????????????????????????????????????????????????????????????????????????????????????????????????????????
`
`OSTO
`
`SSSSSSSSS
`
`www .
`
`SARN
`
`904
`
`9067
`
`000
`0000
`0000
`0000
`000
`
`FIG . 9A
`
`912
`
`9107
`
`:
`
`??????????????
`
`
`
`202 2204
`
`200
`
`204
`
`2004
`
`11
`
`

`

`U . S . Patent
`
`Sep . 12 , 2017
`
`Sheet 10 of 14
`
`US 9 , 760 , 241 B1
`
`200
`
`1000
`
`Trrrrr????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????AddCT
`?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
`
`1114444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444
`
`r202
`
`INTERACTIVE IMAGE 1006
`
`1010
`
`TEXT 1004
`
`PAGE 1002
`
`|
`
`000 0000 0000 000U 000 ) 000 ) 0000 0000 0000 000
`
`FIG . 10A
`
`WWW
`
`*
`
`WY
`
`12
`
`

`

`U . S . Patent
`
`Sep . 12 , 2017
`
`Sheet 11 of 14
`
`US 9 , 760 , 241 B1
`
`200
`
`1000
`
`444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444
`
`77777777777777777
`
`Z
`
`y
`
`s
`
`* * * * * *
`
`* * * * *
`
`* * *
`
`*
`
`* * *
`
`* * * *
`
`* * * * *
`
`* * * *
`
`* * * * *
`
`* * *
`
`* * * * *
`
`* * *
`
`* * * * *
`
`* * * * * * * * * *
`
`* * * * * * *
`
`* * * * * * * * * *
`
`* * * *
`
`*
`
`* *
`
`* * * * *
`
`* * * *
`
`* * * * *
`
`* * *
`
`* * * * *
`
`* *
`
`Pierre
`FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
`
`21wwwwww2222222
`
`*
`
`*
`
`*
`
`*
`
`*
`
`* *
`
`*
`
`ini
`
`* * * *
`
`*
`
`*
`
`* *
`
`* *
`
`* * *
`
`*
`
`*
`
`* * *
`
`INTERACTIVE IMAGE 1006
`
`1014
`
`lle
`1016 x :
`
`!
`
`!
`
`1018
`
`TEXT 1004
`
`PAGE 1002
`
`0000
`0000
`0000
`000
`
`000 ) 000 )
`
`0000
`0000
`0000
`000
`
`FIG . 10B
`
`13
`
`

`

`U . S . Patent
`
`Sep . 12 , 2017
`
`Sheet 12 of 14
`
`US 9 , 760 , 241 B1
`
`2007
`
`1000
`
`202
`
`
`
`- ???????????????????????
`
`1022 - 1
`
`1022 - 2
`
`1022 - 3
`
`1022 - 5
`
`1022 - 4
`
`1022 - 6
`
`1024
`
`IT . . . ! ! !
`
`1020
`
`1022 - 7
`
`1022 - 8
`1022 - 97
`
`TEXT 1004
`
`PAGE 1002
`
`11111111111
`
`111
`
`11
`
`12 11111111111111111111111111
`
`000 0000 0000 0000 000 000 0000 0000 0000 000
`
`FIG . 10C
`
`14
`
`

`

`U . S . Patent
`
`Sep . 12 , 2017
`
`Sheet 13 of 14
`
`US 9 , 760 , 241 B1
`
`200
`
`1100
`
`2222222222222222222222222222222222222222222222222
`22222222222222222222222222222222222222222222222 . . . . . . . . . . . .
`
`. aaaa
`
`a aaaaaaaaaaa ' a ' a ' a ' ' '
`
`2222222222222
`
`HH .
`
`TET
`
`FRANCIACOR
`
`CECOEEEEEEEEEEEEEEE
`
`1110
`
`202
`
`INTERACTIVE IMAGE 1106
`
`TAKE A PICTURE
`OF YOURSELF
`SMILING NOW
`
`11143
`1114
`
`TEXT 1104
`
`PAGE 1002
`
`* * *
`
`* * * * * * * * * * *
`
`* * * *
`
`* * * * * * * * * * * *
`
`* * * * * * * * * * * * * * * * * * * * *
`
`* * * * * * * *
`
`* * * * * * * *
`
`* *
`
`* * * * * *
`
`* * *
`
`* * * * * * * * * * * * *
`
`* * * * * *
`
`* * * * * * * * * * * * * * * * * * * * *
`
`* * * *
`
`*
`
`* * * * * * * * * * *
`
`* * * * * * * * *
`
`* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
`
`0000
`0000
`0000
`0000
`000 ) 0000
`0000
`000
`000
`000
`
`*
`
`* *
`
`FIG . 11
`
`15
`
`

`

`U . S . Patent
`
`Sep . 12 , 2017
`
`Sheet 14 of 14
`
`US 9 , 760 , 241 B1
`
`1200
`
`OPTIONALLY , LOG IN USER
`1202
`
`OPTIONALLY , CALIBRATE TOUCH SENSOR
`1204
`
`DISPLAY CONTENT ITEM
`1206
`
`DETECT TOUCH INPUT
`1208
`
`CORRELATE LOCATION AND / OR FORCE AND / OR DURATION OF TOUCH
`INPUT WITH DISPLAYED CONTENT ITEM
`1210
`
`DETERMINE TACTILE OUTPUT LOCATION AND TYPE
`1212
`
`PROVIDE TACTILE OUTPUT OF DETERMINED TYPE AT
`DETERMINED LOCATION
`1214
`
`PROVIDE AUDIO AND / OR VISUAL OUTPUT , IF ANY
`1216
`1216
`
`FIG . 12
`
`16
`
`

`

`BACKGROUND
`
`US 9 , 760 , 241 B1
`TACTILE INTERACTION WITH CONTENT
`electronic device . In some implementations , the level of
`tactile output provided may vary depending on a location on
`the electronic device that a user touches , such as different
`locations on a touch - sensitive display . For instance , when a
`More and more people are using electronic devices for 5 user touches the display , the electronic device may provide
`tactile output of different types , such as different intensities ,
`entertainment through the consumption of digital media
`based on a location of the user ' s finger relative to one or
`content items , such as images , movies , music , electronic
`books ( eBooks ) , audio books , and so forth . Examples of
`more features or contours of a displayed image .
`these electronic devices may include eBook reader devices ,
`In some implementations , as a user slides a finger along
`cellular telephones , tablet computers , portable media play - 10 the display surface , from one location to the next , the
`intensity of the tactile output may vary , and the variation in
`ers , netbooks , personal digital assistants ( PDAs ) , and the
`like . However , user interfaces for such electronic devices
`the intensity may be in correlation with an image currently
`traditionally engage a limited number of human senses . For
`rendered on the display . For example , if the image displays
`contours or features of different heights , the higher contours
`example , users are able to view images presented on dis
`plays and hear sounds generated by speakers , but tactile or 15 or features may be associated with a different type of tactile
`haptic output has typically been limited to simple shakes or
`output than the lower contours or features . Consequently , in
`buzzes . As digital content continues to increase in popular
`some implementations , a user may feel contours or shapes in
`ity , and electronic devices for consuming that digital content
`a displayed image based on varying intensities of the tactile
`become more ubiquitous , providing ways to enhance the
`output . Additionally , in various implementations , the device
`20 may output sound and / or visual feedback that may also vary
`user experience continues to be a priority .
`with a touch input location and / or movement of a user ' s
`finger .
`BRIEF DESCRIPTION OF THE DRAWINGS
`In some implementations , the haptic or tactile output may
`The detailed description is set forth with reference to the
`also vary according to an amount of force applied by a user
`accompanying figures . In the figures , the left - most digit ( s ) of 25 to a touch sensor associated with the display . For example ,
`a reference number identifies the figure in which the refer -
`when the user touches the touch sensor using a first amount
`ence number first appears . The use of the same reference
`of force , a first type or level of tactile output may be
`numbers in different figures indicates similar or identical
`provided . When the user touches the touch sensor using a
`second amount of force , a different type or level of tactile
`items or features .
`FIG . 1 is a block diagram of a framework for user 30 output may be provided . In some implementations , these
`interaction with content according to some implementations .
`different output types based on detected force may also
`FIG . 2 is an example of an electronic device able to
`include variations in sound or visual feedback provided to
`provide tactile interaction with content according to some
`the user . Thus , a user may touch a feature in a displayed
`implementations .
`image with a first level of force and the feature may become
`FIG . 3 is a block diagram illustrating selected modules 35 animated or change appearance . Additionally , the electronic
`device may produce a sound related to the feature and
`and components of an electronic device able to provide
`provide tactile output , such as a localized vibration of a first
`tactile interaction with content .
`FIGS . 4A - 4B illustrates several examples of electronic
`intensity . If the user touches the feature with a different level
`device configurations for generating tactile output .
`of force , the feature may be animated in a different manner ,
`FIG . 5 illustrates an example of tactile interaction with 40 a different sound may be produced , and tactile output of a
`different level of intensity may be generated .
`content .
`FIG . 6 illustrates another example of tactile interaction
`Haptic or tactile output provided by most electronic
`devices has been limited . Traditional haptic output compo
`with content .
`FIG . 7 illustrates another example of tactile interaction
`nents provide coarse haptic output such as buzzes or vibra
`45 tions . For example , a typical haptic output component may
`with content .
`FIGS . 8A - 8B illustrate another example of tactile inter -
`consist of a vibrator with a rotary motor coupled to an
`action with content , including variation in tactile output
`eccentric weight that , when spun , generates a vibration . On
`the other hand , according to some implementations herein ,
`based on variation in a force of an input .
`FIGS . 9A - 9B illustrate another example of tactile inter -
`the tactile output may be provided by one or more tactile
`action with content , including variation in tactile output 50 output components . In some implementations , the tactile
`output is provided in
`a localized manner , such as at a
`based on variation in a duration of an input .
`FIGS . 10A - 10C illustrate examples of tactile interaction
`location on a display that a user ' s finger is touching . For
`example , a plurality of haptic actuators may be located in
`with an eBook .
`FIG . 11 illustrates an example of using a camera for
`contact with the display for providing a localized vibration
`55 or other physical feedback of a controllable intensity . Thus ,
`interaction with an eBook .
`FIG . 12 is a flow diagram showing a process of outputting
`some implementations may incorporate a variety of near
`varying levels of tactile output in response to receiving a
`surface - located piezoelectric actuators for providing tactile
`output at a location that the user is touching .
`touch input .
`In other implementations , an electrical stimulus may
`60 serve as the tactile output . For example , an electrically
`DETAILED DESCRIPTION
`conductive grid may be provided on the surface of the
`Tactile Interaction
`display for delivering different levels of mild electrical
`stimulus based on a location on the display that the user is
`This disclosure describes techniques for enabling a user to
`touching . Accordingly , some implementations may provide
`interact with content on an electronic device . In some 65 localized tactile output at various different levels of intensity
`implementations , tactile output or feedback may be provided
`for enhancing interaction of a user with content displayed or
`for enabling a user to interact physically with content on an
`played by the electronic device .
`
`17
`
`

`

`US 9 , 760 , 241 B1
`
`ing to some implementations . Framework 100 includes a
`This disclosure also describes a framework , architecture
`touch - sensor 102 and a display 104 . In some implementa
`and techniques for providing tactile output and interaction to
`tions , touch sensor 102 and display 104 may be combined in
`a user . For instance , when a user views a map or reads an
`a single unit as a touch - sensitive display . However , in other
`eBook , the techniques herein may be applied to enable
`tactile recognition of various displayed features . In one 5 implementations , the touch sensor 102 may be separate from
`example , the techniques may be applied to provide an
`at least a portion of the display 104 . The touch sensor 102
`electronic emulation of a pop - up - type book by providing a
`may identify at least one of a touch location 106 or a touch
`combination of tactile , audio and visual interaction to a user .
`force 108 and provide this information to a feedback module
`In another example , the techniques herein may be applied
`110 .
`for providing a raised - content experience to a user , such as 10
`Feedback module 110 assesses content 112 currently
`for reading and physically interacting with maps and other
`being rendered on display 104 in conjunction with the touch
`images having varying topologies , contours , steps , levels of
`location 106 and / or touch force 108 received from the touch
`sensor for determining a touch - dependent output 114 . For
`different heights , and the like .
`In some implementations , the content items may include
`example , in some implementations , the content 112 may
`metadata for specifying tactile output to be provided when 15 include metadata that specifies a particular tactile output to
`the content item is consumed on the electronic device . For
`be provided based on one or more of touch location 106 or
`example , in the case of an eBook , the publisher may specify
`touch force 108 at a location of a particular displayed
`tactile output to be generated in association with one or more
`feature . In other implementations , the feedback module 110
`images contained in the eBook . In other implementations ,
`may use a recognition process for determining a suitable
`for some types of content items , a recognition technology 20 output for a particular feature of the displayed content based
`may be used by the electronic device for determining
`on the input from the touch sensor 102 . For example , the
`appropriate tactile output levels . For example , in the case of
`feedback module 110 may correlate the detected touch
`a map , the electronic device may recognize different eleva
`location 106 with the location of a feature of the displayed
`tions represented , and automatically provide varying levels
`content , determine which feature was touched by the user ' s
`of tactile output based on the differences in elevation . 25 touch input , and determine the touch - dependent output 114
`Similarly , in the case of a digital photograph , the electronic
`to be provided at the location that the user is touching .
`device may distinguish foreground objects from background
`Feedback module 110 , upon determining the touch - de
`objects . For example , when a user touches the foreground
`pendent output 114 , may activate or may provide instruc
`objects , a higher level of intensity of tactile output may be
`tions for activating one or more output components 116 . For
`generated than when the user touches the background por - 30 example , output components 116 may provide a variable
`tion of the image . In addition , some implementations may
`tactile output 118 , an audio output 120 , and / or a visual
`include additional user interaction with content items , such
`output 122 . As mentioned above , the variable tactile output
`as incorporating a user - generated image into a content item
`118 may vary , such as varying the intensity of the tactile
`to further engage the user with the content item . For
`output based on the touch location 106 and / or the touch
`example , the user may specify tactile output for portions of 35 force 108 . Furthermore , based on the detected touch location
`the image , or the electronic device may specify tactile
`106 or touch force 108 , various different audio and / or visual
`outputs automatically .
`outputs may also be provided . The framework 100 is one
`The techniques described herein may be applied in con -
`example of a framework for some implementations herein ,
`and is not intended to limit the implementations herein ,
`nection with both two - dimensional ( 2D ) and three - dimen
`sional ( 3D ) display technology . For instance , the in the case 40 which may also be applied under other frameworks , archi
`of a 2D display , the techniques herein can provide a tactile
`tectures , and the like .
`3D effect to aid in viewing the 2D image . Additionally , in the
`Example Electronic Device
`case of a 3D display , the techniques herein can vary tactile
`sensations in correlation with a displayed 3D image for
`FIG . 2 depicts an example of an electronic device 200 for
`enhancing user interaction with the 3D image .
`The techniques described herein may apply to a variety of
`providing tactile interaction according to some implemen
`different types of content items , such as maps , eBooks , still
`tations . Examples of electronic device 200 may include
`images , videos , audio books , music , and so on . Furthermore ,
`eBook readers , cellular phones , portable media players ,
`the techniques may apply to a variety of different electronic
`navigation devices , tablet computers , personal computers ,
`devices , such as eBook reader devices , navigation devices , 50 netbooks , laptops , cash registers , kiosks , touch screens , and
`personal computers , cellular telephones , personal digital
`so forth . In some implementations , the electronic device 200
`assistants ( PDAs ) , portable media players , tablet computers ,
`may , but need not necessarily , implement the framework 100
`of FIG . 1 described above .
`netbooks , cash registers , and the like .
`The foregoing discussion is provided for the reader ' s
`The electronic device 200 includes a display 202 config
`convenience and is not intended to limit the scope of the 55 ured to present information to a user and a touch sensor 204
`claims or the disclosure herein . Furthermore , the techniques
`for receiving user touch input . In some implementations , the
`described above and below may be implemented in
`a
`touch sensor 204 may be adjacent to or integrated with the
`number of ways and in
`a number of contexts . Several
`display 202 to form a touchscreen . For example , display 202
`example implementations and contexts are provided with
`may display a content item 206 , which in the illustrated
`reference to the figures , as described below in more detail . 60 example is
`a topographic image , such as a map . For
`However , the following implementations and contexts are
`example , content item 206 may include lines 208 , or the like ,
`but a few of many .
`that represent variations in known properties of the content
`item . Thus , in the case of a topographic map , the lines 208
`Example Framework
`may be contour lines that represent variations in elevation of
`65 the features represented in the map . As another example , in
`FIG . 1 is a block diagram of an example framework 100
`the case of a weather map , the lines may be isobars repre
`for providing interactive and variable tactile output accord
`senting variations in temperature . Numerous other types of
`ord
`
`45
`
`18
`
`

`

`US 9 , 760 , 241 B1
`
`tories , maps , web pages , and so forth . Accordingly , the terms
`topographic images may be displayed , with the foregoing
`" book " and / or " eBook ” may include any readable or view
`being just several examples . Further , the techniques herein
`able content that is in electronic or digital form .
`may be applied to other types of images , videos , and so forth
`The electronic device 200 may further include various
`that do not include displayed lines or topology . For example ,
`an image that includes a hierarchical representation may be 5 external controls and input devices in addition to touch
`represented having varying tactile output from one hierar -
`sensor 204 . For example , some implementations of elec
`chical level to the next . As another example , an image of a
`tronic device 200 may include a virtual or physical keyboard
`face may include variations in tactile output as a user moves
`214 , a pointing stick , touchpad , trackball or joystick type
`a finger from the ears , to a cheek , eyes , nose , mouth , etc .
`mechanism 216 , and various other controls 218 for perform
`Thus , implementations herein are not limited to use with a
`10 ing various desired functions ,
`particular type of image .
`FIG . 3 is a block diagram 300 of select example compo
`Tactile output 210 is represented by broken lines and may
`nents of the electronic device 200 according to some imple
`be produced using various types of tactile output compo -
`mentations . In a very basic configuration , the electronic
`nents , as described additionally below . For example , a user
`device 200 includes or accesses components such as at least
`may touch a finger 212 or other body part to the touch sensor 15 one processor 302 and a memory 304 . Each processor 302
`204 , and the electronic device may produce tactile output
`may itself comprise one or more processors or cores .
`210 felt by the finger 212 at the location of the touch input .
`Depending on the configuration of the electronic device 200 ,
`In some implementations , to produce the tactile output , the
`the memory 304 ( and other memories described throughout
`electronic device 200 may include haptic output and feed
`this document ) may be an example of non - transitory com
`back components , as described below with reference to 20 puter storage media and may include volatile and nonvola
`FIGS . 3 and 4A - 4B . For example , some implementations of
`tile memory . Thus , the memory 304 may include , but is not
`electronic device 200 may include integrated haptic actua -
`limited to , RAM , ROM , EEPROM , flash memory , or other
`tors . In other implementations , the tactile output may be
`memory technology , or any other medium that can be used
`provided as an inductive stimulus or other low - level elec -
`to store media items or applications and data which can be
`trical feedback to the user ' s finger 212 . Other techniques for 25 accessed by the electronic device 200 .
`delivering localized tactile output are also contemplated , as
`The memory 304 may be used to store any number of
`will be apparent to those of skill in the art in view of the
`functional components that are executable on the processor
`302 , as well as data and content items that are rendered by
`disclosure herein .
`In some implementations , the tactile output 210 may
`the electronic device 200 . Thus , the memory 304 may store
`provide a varying level of tactile feedback based on a 30 one or more content items 306 , such as eBooks , audio books ,
`location on the touch sensor 204 that the user ' s finger 212 is
`songs , videos , still images , and the like . The memory 304
`touching . For example , as the user ' s finger moves over the
`may further include a memory portion designated as an
`surface of the touch sensor 204 , from one portion of image
`immediate page memory ( not shown ) to temporarily store
`206 to another , the user ' s finger 212 may cross over one or
`one or more pages of an electronic book or other content
`more of lines 208 . As the user ' s finger is detected crossing 35 item 306 . The pages held by the immediate page memory are
`each line 208 , the level of intensity of the tactile output may
`placed therein a short period before a next page request is
`increase or decrease . For instance , in the example of an
`expected .
`elevation map , the tactile output may increase in intensity as
`The term " page , ” as used herein , refers to a collection of
`a user ' s finger moves from a lower elevation to a higher
`content that is presented at one time in a display of the
`elevation . Similarly , in the case of a temperature map , the 40 electronic device 200 . Thus , a " page ” may be understood as
`tactile output may increase as the user ' s finger moves from
`a virtual frame of the content , or a visual display window
`a lower temperature to a higher temperature , or vice versa .
`presenting the content to the user . Thus , " pages ” as
`As another example , in the case of an image having fore -
`described herein are not fixed permanently , in contrast to the
`ground portions and background portions , moving the user ' s
`pages of published “ hard ” books . Instead , pages described
`finger from a background portion to a foreground portion 45 herein may be redefined or repaginated when , for example ,
`may produce a change in the level of tactile output provided .
`the user chooses a different font for displaying the content in
`The electronic device 200 may be configured with func -
`the first display . In addition to pages , the terms “ page
`tionality to enable consumption of one or more content items
`views ” , " screen views ” , and the like are also intended to
`of numerous different types or formats including , for
`mean a virtual frame of content .
`example , electronic maps , electronic texts ( e . g . , documents 50
`The memory 304 may store an operating system ( OS ) and
`of any format , electronic periodicals , such as digital maga
`user interface module 308 that is executable by the processor
`zines and newspapers , etc . ) , digital audio ( e . g . , music ,
`302 . For example , the