a2) United States Patent
`US 8,954,887 B1
`(0) Patent No.:
`Feb. 10, 2015
`(45) Date of Patent:
`Tsenget al.
`
`US008954887B1
`
`(54) LONG PRESS INTERFACE INTERACTIONS
`
`(75)
`
`(73)
`
`Assignee: Google Inc., Mountain View, CA (US)
`
`(*)
`
`Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 714 days.
`
`(21)
`
`Appl. No.:
`
`12/367,842
`
`(22)
`
`Filed:
`
`Feb. 9, 2009
`
`10/1998 Solimeneet al.
`5,828,376 A
`1/1999 Bier veces 712/1
`5,862,395 A *
`7/2000 Buxtonetal. ou. 715/863
`6,094,197 A *
`Inventors: Erick Tseng, San Francisco, CA (US);
`10/2002 Kino et al.
`6,469,719 Bl
`Daniel Johansson, Malmo (SE);
`4/2003 Selker
`6,549,219 B2
`Michael A. Cleron, Menlo Park, CA
`6/2003 Hoeket al. wu. 715/822
`6,583,798 B1*
`6,661,437 Bl=12/2003 Miller etal.
`(US); Indika C. Mendis, Mountain
`6,836,723 B2* 12/2004 de Silva woe 701/200
`View, CA (US)
`6,877,138 B2*
`4/2005 Fitzpatrick et al. ........ 715/769
`7,036,090 Bl
`4/2006 Nguyen
`7,317,449 B2*
`1/2008 Robbins etal. 0.0.0... 345/169
`7,369,102 B2*
`5/2008 Lukeetal. oe 345/9
`7,512,898 B2
`3/2009 Jenningset al.
`7,600,194 B2
`10/2009 DeMaioetal.
`7,600,197 B2
`10/2009 Gourdol etal.
`7,823,081 B2
`10/2010 Sato
`7,930,650 B2
`4/2011 Brodersenet al.
`7,966,575 Bl
`6/2011 Jetha et al.
`8,136,045 B2
`3/2012 Miura etal.
`8,255,060 B2
`8/2012 Goetz et al.
`8,321,808 B2
`11/2012 Goetz etal.
`8,347,226 B2
`1/2013 Windl
`8,365,230 B2
`1/2013 Chane etal.
`8,527,896 B2
`9/2013 Matthewsetal.
`8,589,812 B2
`11/2013 Pasqualeet al.
`8,606,516 B2
`12/2013. Vertelneyetal.
`2004/0183819 Al*
`9/2004 Henry wees 345/660
`
`(60)
`
`(51)
`
`(52)
`
`(58)
`
`(56)
`
`Related U.S. Application Data
`
`Provisional application No. 61/027,386, filed on Feb.
`8, 2008.
`
`Int. Cl.
`GO6F 3/048
`GO6F 3/0481
`GO6F 3/0487
`US. Cl.
`
`(2013.01)
`(2013.01)
`(2013.01)
`
`CPC oo... GO6F 3/04812 (2013.01); GO6F 3/0487
`(2013.01)
`USPC woe cece sete ssenetseeseeeeenees 715/823; 715/808
`Field of Classification Search
`CPC woe cee GO6F 3/04812; GO6F 3/0487
`USPC woe cceseeseeseneeseneeeeeecneteeseeaes 715/808, 823
`See application file for complete search history.
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`4,896,291 A
`5,760,776 A
`
`1/1990 Gestet al.
`6/1998 McGurrinet al.
`
`(Continued)
`
`Primary Examiner — Patrick Riegler
`(74) Attorney, Agent, or Firm — Fish & Richardson P.C.
`
`(57)
`
`ABSTRACT
`
`Acomputer-implementeduser interface method includes dis-
`playing one or more display elements in a graphical user
`interface, including elements that correspond to pop up con-
`trols, receiving a selection of one of the one or more display
`elements, determining whether the selected element corre-
`sponds to a pop up control, and displaying the selected ele-
`mentin a highlighted format upon receiving the selection if
`the element correspondsto a pop up control, and displaying a
`pop up control after displaying the elementin a highlighted
`formatif the selection persists for a long press period.
`
`19 Claims, 12 Drawing Sheets
`
`Receive Selection
`£02
`
`
`
`
`Show Elementin
`
`
`Long Press?
`Highlighted Form
`
`
`
`405
`£04
` Yes
`
`
`
`Display Contextual
`Menu
` gog
`
`
`
`Receive Selection
`on Contextual
`
`
`Menu 41z¢
`
`
`Carry Out Selected
`Function 42
`
`
`1
`
`APPLE-1024
`
`1
`
`APPLE-1024
`
`

`

`US 8,954,887 B1
`
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`ccccccccccccccccccceee 705/7
`2004/0204970 A1* 10/2004 Boden etal.
`2004/0217947 A1* 11/2004 Fitzmauriceetal.
`345/183
`
`1/2006 Aaltonen et al. 00... 715/850
`2006/0020904 Al*
`2/2006 Friedrich wo... 701/211
`2006/0031010 AL*
`379/433 .06
`2006/0062382 Al*
`3/2006 Ronkainen
`
`2006/0112350 Al*
`5/2006 Kato wc.
`wae TLS/TBL
`
`2006/0267966 Al* 11/2006 Grossman etal.
`wee 345/179
`2007/0022378 Al*
`1/2007 Hamada ...
`we TIS/TIG
`2007/0139386 Al*
`6/2007 Martinetal. oo... 345/173
`
`2007/0162937 Al*
`7/2007 Casementetal.
`......0... 725/58
`2008/0040678 Al*
`2/2008 Crump..........
`715/763
`
`2008/0066610 Al*
`3/2008 Chuetal. v..cccccee 84/609
`2008/0092081 Al*
`4/2008 Jongetal. ween 715/840
`2008/0109751 Al*
`5/2008 Fitzmaurice etal.
`......... 715/793
`2008/0163053 Al*
`w 715/702
`7/2008 Hwangetal. ....
`
`2008/0176604 A1*
`7/2008 Kim ou 455/566
`2008/0263022 A1* 10/2008 Kostorizos etal. .
`ve 7107/5
`
`715/834
`2009/0083665 Al*
`3/2009 Anttila etal.
`....
`
`... 715/780
`2009/0187846 Al*
`7/2009 Paasovaara ...
`2010/0214235 Al*
`8/2010 Shmayaetal. occ... 345/173
`
`* cited by examiner
`
`2
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 1 of 12
`
`US 8,954,887 B1
`
`ly0}Aiday&a
`isteSET
`Teeeee
`puewio,Beste
`PoyRCTGyeetBrcnacy
`
`UIPaTSeUIOL©)ceot
`
`89190FINYea
`AidaySESE
`Benes
`LIPPERSSL
`
`
`
`ieee)
`
`LSls
`
`USIZIBIULMSEPLEN@.GEOL
`UOSSJEISND)13g
`EOCORSeCRTELEC
`
`eeeyeritseesB-ttCl|-4
`PicsBLCsAbs}Bley)
`
`
`HELCre.Basalt
`
`
`PahcabetdedeeCathEDEeeay PeonoelBcce|
`
`
`
`uossjeIsnyJag
`
`[13Surozug
`
`
`PeraneeiPCCce(esseeRe
`
`
`
`Retiree)|erieCrCebBeeicietifsoNmhoaeas
`
`PLOneestieet
`
`
`
`U9s8131UIY,SEPIN@SEC!
`
`PeeneCAMCla}
`
`riceRe)BT
`
`eReLet
`
`
`
`>COL
`
`3
`
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 2 of 12
`
`US 8,954,887 B1
`
`802
`
`VeOld
`
`
`
`AS.ecacace
`
`“aia5S7
`
`4
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 3 of 12
`
`US 8,954,887 B1
`
`
`
`fens
`
`Hsgasoebatadeta
`
`it:
`
`deOld
`
`5
`
`
`
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 4 of 12
`
`US 8,954,887 B1
`
`
`
`QIJO1gJESPIE
`
`
`
`abelojssdeyy
`
`STE
`
`STE
`
`€Old
`
`SSOOIIAA
`
`BOepayU|
`
`(s)uoneoiddy
`
`clé
`
`Aejdsiq
`
`jebeueyy
`
`yndu
`
`jaBeuey
`
`O0E£
`
`POE
`
`WIe3
`
`
`
`ZUONDIJOJOR)SIEyOIg
`
`
`
`6
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 5 of 12
`
`US8,954,887 B1
`
`m4
`
`Receive Selection
`402
`
`
`
`
`Show Element in
`
`
` Long Press?
`
`Highlighted Form |
`
`
`Display Contextual
`Menu
`gag
`
`Receive Selection
`on Contextua!
`Menu 470
`
`
`
`
`
`Carry Out Selected
`Function
`412
`
`FIG. 4A
`
`7
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet6 of 12
`
`US 8,954,887 B1
`
`416¢
`
`Receive Long Press
`Selection on Map
`
`418
`
`Record Selection
`Location
`
`Display Zoom
`Control
`
`422
`
`Control
`
`Receive Manipulation
`of Zoom Control
`
`424
`
`Zoom Map Around
`Selection Location
`
`426
`
`Remove Zoom
`
`FIG. 4B
`
`8
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 7 of 12
`
`US8,954,887 B1
`
`432
`
`Yes
`
`Yes
`
`Change Zoom
`Level
`
`436
`
`Move Map to Pan
`Location
`
`440
`
`
`Display Map With
`Zoom Control
`
`
`
`
`
`Zoom
`Manipulation?
`
`
`
`434
`
`
`
`
`
`
`
`on
`
` Pan
`Manipulation?
`438
`
`
`Determine and
`Transmit Map Center
`Point
`442
`
`
`
`
`
`Receive Results
`
`
`Directed to Map
`Location
`444
`
`
`Display Results
`
`
`446
`
`FIG. 4C
`
`9
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 8 of 12
`
`US8,954,887 B1
`
`“501
`
`SaAOOOOLS
`
`A
`
`G
`
`iH
`
`Hu
`
`JK
`
`JIL
`
`fico
`
`AAVTs) JL le)
`
`
`laut} loas|le}
`|e |e
` 502
`
`==
`
`509
`
`FIG. 5
`
`10
`
`10
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 9 of 12
`
`US 8,954,887 B1
`
`609
`
`GNNOS
`
`SOVAYSLNI
`
`ANOHdATSL
`
`WALSASSNS
`
`ONILNIOd so9
`
`VNNALNV
`
`SOIASG
`
`SOVAYALNI
`
`QHYVOGASY
`
`SOVAYSLNI
`
`Awildsid
`
`SOVAYSALNI
`
`Ndo
`
`vOg
`
`c09
`
`2 o
`
`O
`
`809
`
`2409
`
`909
`
`VesWvo
`
`MYOMLAN
`
`SOVAYMALNI
`
`AOVAYSLNI
`NOILOANNOO
`SOVAYSLN!
`
`029
`
`6LO249
`
`LLO019
`
`9Sls
`
`AOYNOSYAMOdNOLLVOIAWN
`
`a1ndow
`
`
`
`
`
`ZL9
`
`vL9
`
`S1L9
`
`9OL9
`
`
`
`WNIGSW39VeOLS
`
`
`
`WSLSASONILWaadO
`
`
`
`SWVdDOddNOILVONIddV
`
`saidVLva
`
`NIVIN 7
`AYOWSW
`
`11
`
`11
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 10 of 12
`
`US 8,954,887 B1
`
`
`
`adA|aal4alojqamqy
`
`291s
`
`
`
`jeueyxnulyOLL60L
`
`202
`
`
`
`
`
`
`
`elJamobyolpny141Molpediayois
` (esnoe|Ssedyegsnjeig(GuiBessoy_)
`
`084suoneoddy~OLsadlAIegWa}SAS
`
`
`
`
`
`(dd|)JapuigSWUd}SAS-|\|yjoojanigeleweyAeidsiq
`
`juawabeueySISALJaAuqJaauq
`
`
`
`
`
`
`JaauqJaauqJaauqJaaugJaAuq
`euIyoe|[ENLIAHAIeCsalnquedo‘ONGSCH
`
`
`
`Sauesgi]evera10419SaoeInsWOMBLE]
`
`
`suuUNYploipuysouesqr
`
`
`
`ig
`
`ddiW
`
`euwUNY
`
`BHTOS
`
`}uaJUOD
`
`SIOPIAOld
`
`gounosey
`
`Jabeuep)
`
`922
`
`LOLGcL
`
`bel
`
`Jabeuey
`
`
`
`pelEdW
`
`12
`
`Auoydeya|
`Jafeuey|
`
`Jabeueyy
`abeyoed
`
`MOPUIM
`
`Jabeueyy
`
`UORESUHON
`
`Jebeuey;
`
`ae
`
`Jebeuey
`AMAR
`
`wT—
`o
`
`12
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 11 of 12
`
`US 8,954,887 B1
`
`sde;w
`
`SS800/q
`
`Jesmoig
`
`$S8001q
`
`Jajeiq
`
`SSa001d
`
`$$900/q
`
`60808
`
`
`JebeueyyJeyoune
`
`
`
`
`
`soueiqr]everal09
`
`
`
`jouuayxnulq
`
`8Sls
`
`c8
`
`ZV8
`
`
`
`‘949'0q/7
`
`
`
`‘949'0Q)7
`
`3
`
`
`
`‘919'0q/
`
`SuIYoeIN[ENLIAyIAIeQ
`
`
`
`
`SUIYORW[ENLIAHIAIEQ
`SOUBIGI]BAPB05
`
`SODIAIOSWa}shS
`geec8cBcls+8
`
`sexenneneraco|||souesaneeracyeen|||gem||teen
`
`
`JasmoigJajeiqaBexoeddyJegsniels
`
`9€8
`
`oes
`
`p28
`
`re
`
`‘18‘0q77
`
`BODIEMAN]
`
`
`aulyoeyFENUIAHIAIeG
`
`
`
`
`
`aUIUOBY[ENUIAYIAEC
`
`¥L8
`
`Jabeueyy
`
`13
`
`13
`
`

`

`U.S. Patent
`
`Feb. 10, 2015
`
`Sheet 12 of 12
`
`US8,954,887 B1
`
`
`
`fW
`
`T|
`
`
`
`
`
`FIG.9
`
`
`
`14
`
`

`

`US 8,954,887 B1
`
`1
`LONG PRESS INTERFACE INTERACTIONS
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`2
`element representing one message in the list ofmessages may
`cause a contextual menu to pop up that itself contains ele-
`ments for handling the messages (e.g., reply, reply to all,
`forward, delete, etc.).
`In certain implementations, such systems and technique
`mayprovide one or more advantages. For example, as men-
`tioned above,a user of a device that displays, almost imme-
`diately upon a selection of an on-screen control, that the
`control exhibits long press behavior, may permit a person to
`have a quick and intuitive understanding of actions they can
`take with the control. Without the fast feedback, the user
`would not know about the additional capabilities of the ele-
`ment until the actual contextual menu or other item was
`displayed, and the user would then need to removethe dis-
`played item before continuing.
`In one implementation, a computer-implemented user
`interface method is disclosed. The method comprises display-
`People spend hours at a time with their electronic
`ing one or more display elementsin a graphicaluser interface,
`devices—computers, telephones, music players, andthelike.
`including elementsthat correspondto pop up controls, receiv-
`Theylike best those devices thatare intuitive to use and whose
`ing a selection of one of the one or more display elements,
`interactions best meet
`their expectations regarding how
`determining whether the selected element corresponds to a
`machines
`should work. They interact with electronics
`pop up control, and displaying the selected element in a
`through inputs and outputs from the devices, where the out-
`highlighted format uponreceiving the selection ifthe element
`puts generally are provided audibly and/or onaflat graphical
`corresponds to a pop up control, and displaying a pop up
`display screen, and the inputs may occur via touch screens,
`control after displaying the elementin a highlighted formatif
`joysticks, mice, 4-directional keypads, and other such input
`mechanisms.
`the selection persists for a long press period. The pop up
`Fine distinctions in user interfaces can make a world of
`control can comprise a contextual menu. Also, the display
`element may bedisplayed in a highlighted format that com-
`difference in a user’s experience with a device. User interface
`prises: (1) changing a color of one or more features of the
`options may be reduced for mobile devices, where full-sized
`display element; (2) thickening one or more features of the
`keyboards or no keyboardsat all are available. In such situa-
`display element; and/or (3) increasing the size of the display
`tions, intuitive interaction may be at a premium. The impor-
`element.
`tance of such interaction may be even more pronounced with
`In certain aspects, the one or more display elements com-
`mobile devices, where the user needsto interact with a device
`prise selectable textual elements in a displayedlist. Also, the
`using a single hand.
`selection can be received by a touch screen or selectable track
`ball.
`
`This application claimspriority under 35 U.S.C. §119(e)to
`US. Provisional Application No. 61/027,386, filed Feb. 8,
`2008 whichis incorporated by reference in entirety hereto.
`
`TECHNICAL FIELD
`
`This documentrelates to systems and techniques for gen-
`erating graphical display elements and controls.
`
`BACKGROUND
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`SUMMARY
`
`40
`
`45
`
`This documentdescribes systems and techniques that may
`be used to interact with a user of a computing device like a
`mobile telephone having a touch screen user interface. In
`general, the techniques mayreact in particular waysto a long
`press of an input mechanism such as a touch screen display or
`a clickable trackball. A long press is a selection that is main-
`tained by a user for a time that is noticeably longer than a
`simple single press or click selection, andthat, in appropriate
`circumstances, produces an interaction with the user that
`differs from that of a corresponding simple, or short, press or
`click.
`In certain situations, it may be helpful to signal to a user
`whena particular selectable element that is part of a user
`interface is an elementthat will exhibit such long press activ-
`ity. For example, a user may not be willing to read a user’s
`manual or otherwise wander about a system trying to deter-
`mine which elements exhibit long press behavior—finding
`some that respond and somethat do not. To help a user may
`more readily understand the operation of a device, the tech-
`niques discussed here highlight an elementthat exhibits long
`press behavior, whenit is first selected. In this manner, a user
`may quickly see that the element has additional functionality.
`In yet another implementation, an article comprising a
`After the elementhas beenselected for a long press period,
`computer-readable data storage medium storing program
`the additional functionality is revealed—generally by a pop
`code operable to cause one or more machines to perform
`up control that a user may then interact with, and that is
`operations is disclosed. Such operations can include display-
`directedto the current contextofthe display. For example, the
`ing one or more display elementsin a graphicaluser interface,
`long press behavior maybeactivity that produces a pop up
`contextual menuthat relates to an element in the display. As
`including elementsthat correspondto pop up controls, receiv-
`
`one example, a display may showalist of messages, such as ing a selection of one of the one or more display elements,
`voice mail, text, and e-mail messages, and long pressing on an
`determining whether the selected element corresponds to a
`
`In another implementation, computer-implemented user
`interface method is disclosed that comprises displaying a
`zoomable feature in a graphical user interface, receiving a
`selection on the zoomable feature, determining whether the
`selection is a long pressselection, and ifthe selection is a long
`press selection, displaying a zooming control on the graphical
`user interface, and redisplaying the zoomable feature in
`response to user interaction with the zooming control. The
`zoomable feature can comprise a geographic map. Also, the
`zooming control can comprise a graphical slider that is
`responsiveto user inputs, and the control can not be visible on
`the graphical user interface.
`In certain aspects, redisplaying the zoomable feature com-
`prises zooming the feature accordingto user interaction with
`the zooming control, and panning the feature according to
`user interaction with a second control. Also, the second con-
`trol can be located outside the graphical user interface. In
`some aspects, the zooming control comprises a graphical
`slider and the second control comprises finger dragging on
`the zoomable feature. Moreover, redisplaying the zoomable
`feature can comprise centering the graphical user interface on
`a location of the zoomable feature correspondingto the loca-
`tion of the selection.
`
`50
`
`55
`
`60
`
`65
`
`15
`
`15
`
`

`

`US 8,954,887 B1
`
`3
`pop up control, and displaying the selected element in a
`highlighted format uponreceiving the selection ifthe element
`corresponds to a pop up control, and displaying a pop up
`control after displaying the elementin a highlighted formatif
`the selection persists for a long press period. The pop up
`control can comprise a contextual menu, and displaying the
`display elementin a highlighted format can comprise chang-
`ing a color of one or more features of the display element.
`In someaspects, displaying the display element in a high-
`lighted format comprises thickening one or more features of
`the display element; and/or increasing the size of the display
`element. Also, the one or more display elements can comprise
`selectable textual elements in a displayedlist.
`In different implementation, anotherarticle having a com-
`puter-readable data storage medium is disclosed. The opera-
`tions performed by code on the article includes displaying a
`zoomable feature in a graphical user interface, receiving a
`selection on the zoomable feature, determining whether the
`selection is a long press selection, and ifthe selection is a long
`press selection, displaying a zooming control on the graphical
`user interface, and redisplaying the zoomable feature in
`responseto user interaction with the zooming control.
`In another implementation, a computer-implemented user
`interface system is disclosed. The system comprises a graphi-
`cal user interface module to display user selectable elements,
`including elements that correspond to pop up controls, a user
`input mechanism to receive user selections in coordination
`with the display of user selectable elements, and a long touch
`interface processor to determine when a userselection is a
`selection of an element that correspondsto a pop up control,
`to cause the elementto be highlighted uponits selection, and
`to cause the pop up control to be displayed after a long touch
`period. The pop up control can comprise a contextual menu.
`The system may also comprise a mapping application,
`wherein the pop up control comprises a zoom control for the
`mapping application. In addition, the long touch processor
`can be programmedto cause the elementto be highlighted by
`thickening one or more features of the display element. Also,
`the user input mechanism can comprise a touch screen or
`trackball.
`Another disclosed computer-implemented user interface
`system includes a graphical user interface module to display
`user selectable zoomable content, a user input mechanism to
`receive user selections in coordination with the user select-
`
`able zoomable content, and a long touch interface processor
`to determine whena userselection is a long press selection, to
`display a zooming control on the graphical
`interface in
`responseto the long press selection, and to cause a redisplay
`of the zoomable content in response to user interaction with
`the coming control. The system can also comprise a mapping
`application, and the pop up control can comprise a zoom
`control for the mapping application. Also, the zooming con-
`trol can comprise a graphicalslider that is responsive to user
`inputs. In someaspects, the zooming control receives input on
`the graphical user interface but is not visible on the graphical
`user interface. In yet other aspects, redisplaying the zoomable
`feature comprises zooming the feature according to user
`interaction with the zooming control, and panning the feature
`occurs according to user interaction with a second control.
`The details of one or more embodimentsare set forth in the
`
`accompanying drawings and the description below. Other
`features and advantageswill be apparent from the description
`and drawings, and from the claims.
`
`DESCRIPTION OF DRAWINGS
`
`FIG. 1 shows several displays by a mobile device that
`exhibits long press behavior.
`
`20
`
`25
`
`40
`
`45
`
`50
`
`65
`
`4
`FIGS. 2A-2B show various displays exhibiting different
`interface mechanisms for performing a zoom function on a
`mobile device having a touch screen.
`FIG. 3 is a schematic diagram of a system that provides
`user interaction in response to touch screen inputs.
`FIGS. 4A-4C are flow charts of example processes for
`receiving user selections from graphical user interface.
`FIG. 5 is a schematic representation of an exemplary
`mobile device that implements embodimentsofthe notifica-
`tion techniques described herein.
`FIG.6 is a block diagram illustrating the internal architec-
`ture of the device of FIG.5.
`
`FIG.7 is a block diagram illustrating exemplary compo-
`nents of the operating system used by the device of FIG.5.
`FIG. 8 is a block diagram illustrating exemplary processes
`implemented by the operating system kernel of FIG.7.
`FIG. 9 shows an example of a computer device and a
`mobile computer device that can be used to implement the
`techniques described here.
`Like reference symbols in the various drawings indicate
`like elements.
`
`DETAILED DESCRIPTION
`
`This documentdescribes systems and techniques by which
`mobile devices may interact with a user of such devices. For
`example, a user may be ableto select itemsthat are part of a
`graphicaluser interface in a mannerthat quickly provides the
`user with information about
`the selected item. As one
`example, certain on-screen elements may have particular,
`hidden functionality, such as the ability to raise a context-
`sensitive menu associated with the elements. Such elements
`may react nearly immediately when pressed, whereas ele-
`ments that do not have associated menus mayreact differently
`or not at all whenfirst pressed, so that a user can understand
`that he or she has selected an element having such additional
`functionality immediately (or nearly immediately) upon
`selecting the element, but before the additional functionality
`presents itself.
`The additional activity associated with some elements may
`be the ability to generate additional graphics elements when
`selected, using, for example, along press selection technique.
`In one example, controls for a mapping application may be
`generated whena user long presses on a displayed map. Such
`controls may include a zoom control, which maybea tradi-
`tional zoom slider control that is graphically displayedafter a
`long press on a map. Other similar zooming techniques are
`also discussed here.
`FIG. 1 showsseveral displays by a mobile device 100 that
`exhibits long press behavior. In this example, the device 100
`is currently carrying a telephone conversation,as indicated by
`the “ongoing call”indication in the “ongoing activities” area
`ofthe user interface display. Separately, a numberof interface
`elements showing recent messaging activities is also dis-
`played. For example, while the user was previously using the
`device 100 for other things or not using the device atall,
`various messages may havearrived at the device. Such mes-
`sages may have beenindicatedto the user initially by scroll-
`ing information from the messages across the status bar
`shown at the top of the display, and may then have been
`indicated by icons(e.g., small envelopes or telephones) in an
`emptyarea ofthe status bar (e.g., the left edge ofthe status bar
`in the figure) to alert the user to the fact that new messages had
`arrived.
`
`The user may then have dragged his or her finger down
`from the status bar, and across the display, or selected a
`particular buttonor other form of control, to pull more details
`
`16
`
`16
`
`

`

`US 8,954,887 B1
`
`6
`ing down on itemsfor long periods of time just to determine
`whetherthey exhibit long press behaviors. Also, after the user
`has had the device for a time, they will becometrained to
`recognize immediate visual changes in items they select, and
`will know to expect a contextual menu or other such long
`press activity from the appearanceofsuch initial activity. As
`a result, the presentation of such quick feedback to the user
`may makethe graphicaluser interface of a device much more
`intuitive and easier to use.
`
`FIGS. 2A-2B show various displays exhibiting different
`interface mechanisms for performing a zoom function on a
`mobile device having a touch screen. Referring to FIG. 2A
`specifically, two displays are shown of the same mobile
`device 200, at two different points in time. An initial display
`shows amaptool for an application such as GOOGLE MAPS,
`with a user selecting a location on the map bypressing with
`their finger. Typically, such a selection may be conducted ina
`numberof different ways. For example, the user may press
`and slide their finger across the display to pan in the map and
`see nearby areas on the map. Alternatively, the user could
`press twice quickly in succession, to zoom the map in. The
`user could press the map three times in succession to zoom the
`map out.
`In this example, the user has selected a spot on the map and
`continued pressing for a long press period oftime, such as by
`leaving their finger on the map for more than several hundred
`milliseconds. Such a long press period, however, may vary.
`For example, a user may selecting how long they would like
`their long press period to be on a global basis for any opera-
`tions in a particular operating system—amuchlike the ability
`of users to select how fast a double-click will operate with a
`computer mouse.In any event, the long press period is sub-
`stantially longer thanis required fora basic clicking selection
`operation like those just discussed. Also, the long press may
`be achieved through other mechanisms, such as by pressing
`down on a clickable trackball 205.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`5
`about each of the recent events onto the display, and to move
`any prior information about ongoing applications out of the
`way and to the ongoing activities area. In such a manner, the
`user may have generated the display shownhere.
`The latest events area 102 shows such recently-received
`message information (and may show information about non-
`messagerelated events). In this example, the latest events area
`102 lists four events sorted in reverse chronological order—
`an e-mail, a chat message, a notice about a missed call, anda
`voice mail.
`
`A userselection (e.g., by pressing on a touch screeninter-
`face) of any of the messages may openthe selected message.
`A user selection of an application icon next to one of the
`messages may open the full application, such as with the
`associated message opened in the application. Such actions
`may occur upon an up press event on the item when the user
`selects the item quickly.
`The second display of the device 100 in FIG. 1 shows an
`example that may occur whenthe userselects an item—here,
`item 104. Specifically, the item, because it has associated
`additional functionality that will be described in more detail
`below, may be made to changeits appearance soonafter being
`pressed by the user. Such a change in appearance is shown
`here, by way of example, using a white lit-up box around the
`selected item. The change in appearance mayalso occur ina
`numberofdifferent manners, such as by increasing the size of
`the item slightly, changing the brightness or color(s) of the
`item (e.g., to make it temporarily brighter), by making the
`item jiggle slightly (e.g., so as to make it appear that the
`additional functionality is about to explode out of the item,
`like a chick hatching from an egg), and by other such visual
`mechanisms. The device 100 may also be madeto vibrate in
`such a situation to indicate that additional functionality is
`comingif the user holds the selection.
`Thethird, and right-most, display showsthe functionality
`generated by a long press by the user. In this example, the
`functionality involves the generation of a contextual pop up
`menu 106 associated withthe first message in the latest events
`area 102 that was long-pressed by the user. The menu is
`contextual because its content depends on the thing that was
`long pressed—here, the context involves basic operations that
`can be performed on the e-mail message that was selected
`(e.g., reply, forward, etc.). Ifa different event were selected,
`the contextual menu might have different values (e.g., it may
`not be possible to forward a missed telephonecall).
`The user maythen select an item on the contextual pop up
`menu 106 in a familiar manner. For example, the user can
`select the “reply” item on the menu 106, and a copy of the
`e-mail, addressed to the original sender, may be instantiated,
`so that the user may add sometext to the message and sendit
`back.
`Ttis also possible that certain items, such as a notification of
`a missed call, would haveso few actions associated with them
`that a contextual menu would be overkill. In such a situation,
`a user selection of those items would notresult in a white box
`being drawn around the item as the selection continues, as
`was shownin the middle display in FIG. 1. Rather, the display
`of the item might remain static, to indicate to the user that
`there is nothing additional to be gleaned from holding, or
`pressing, the item for a long time.
`Thus, by these mechanisms, a user of a mobile device may
`navigate more quickly and seamlessly through the applica-
`tions on their mobile device. They may quickly see if a selec-
`tion is associated with further information that will be dis-
`
`In the illustrated image, the user has already provided a
`long press selection, which hasresulted in the generation of a
`zoom control] 202 over the map 204. Such a selection may
`haveinitially (before the long press period had run) provided
`an indication that such a selection wouldresult in the provi-
`sion of additional functionality. For example, the zoom con-
`trol 202 may haveinitially been displayed in a “ghost” or
`outline form until the long press period was met.
`The second display, on the right, shows subsequent
`manipulation of the zoom control 202, via a slider 206 on the
`control 202. The manipulation is by a familiar technique (e.g.,
`as used with GOOGLE MAPS)ofdraggingthe slider upward
`toward a “+” sign to zoom in, and downward toward a “-”
`sign to zoom out.
`A zoom center point 208 is shown schematically over the
`display to illustrate where the display will zoom into. This
`center point 208 is set in the example to be the location at
`whichthe user previously long-pressed. (The graphic shown
`here to represent the zoom center point 208 need not be
`displayed on the device 200.)
`In this manner,a user can select a point ofinterest easily on
`a map and then can have the map zoom in on that point of
`interest automatically. Additional interaction may be pro-
`vided after the zooming. For example, the user may place a
`finger on the display and drag it in a circular motion to rotate
`the map, muchlike they wouldto spin a piece of paper(e.g.,
`65
`
`played after a long press, but without having to wait for the areal map) onaflat surface such as a desk or table. Such an
`long press period. As a result, when they are first learning a
`operation may also occur after the user has switched to a
`different view of a location, such as a street view, satellite
`new system, they will not need to move slowly around, hold-
`
`40
`
`45
`
`50
`
`55
`
`60
`
`17
`
`17
`
`

`

`US 8,954,887 B1
`
`7
`view, or airplane view (e.g., an aerial view of a geography
`taken at a slight angle so that buildings can be rotated around
`by the viewer).
`FIG. 2B shows yet another zooming interaction with a
`mobile device 230. Here, two different displays on device 230
`show operation ofthe device at two different points in time. In
`the first instance, the device 230 is displaying a map 234, and
`a user has anchored a zoom point just between Oriole Lane
`and Orchard Lane on the map 234. Theuseris in the process
`ofpressing a zoom control 232. The zoom control 232 may be
`displayed whenevera mapis displayed, or may be generated
`only after a user has selected a zoom point or at some other
`appropriate point.
`The device 230 in the right display demonstrates what
`happensafter the user selects the zoom control 232. In par-
`ticular, another zoom control having a slider control 238 has
`been displayed, and the user is beginning to manipulate the
`slider control 238 to cause zooming of the map 234. A zoom
`point 236 showsthe point into which the zoomingwill occur.
`As just described, that point 236 may be a point that was
`selected by the user before they selected the zoom control
`232. Alternatively, where the zoom control 232 is always
`displayed when a map is displayed, so that the user did not
`need to identify a zoom point, the zoom point may default to
`the center of the display or another appropriate point.
`In the various implementations shown here, the zooming
`itself may be discrete or analog. Discrete zooming occurs by
`jumping from one zoom level to a separate zoom level,
`whereas analog zooming occurs by zooming smoothly along
`a continuum. Discrete zooming maybe used, in particular,
`where a mapis generated using tiles that have been previously
`generated to show a geographyat particular zoom levels, so
`that zooming in or out causes a newset oftiles (at a closer
`zoom level) to replace a priorset of tiles. Certain zoom levels
`might simply showtheinitial tiles ina larger manner, whereas
`new tiles may be substituted when the level of zoom is such
`that the initial tiles cannot provide the needed detail. Thus, for
`example, the tiles may be replaced onlyat every third discrete
`level, or the like.
`FIG.3 is a schematic diagram ofa system 300 that provides
`user interaction in response to touch screen inputs. The sys-
`tem 300 may be implemented using a mobile device such as
`device 302. The device 302 includes various input and output
`mechanismssuch as a touch screen display 304 anda roller
`ball 306. A number of components withi