as) United States
`a2) Patent Application Publication 0) Pub. No.: US 2006/0197753 Al
`(43) Pub. Date: Sep. 7, 2006
`
`Hotelling
`
`US 20060197753A1
`
`(54) MULTI-FUNCTIONAL HAND-HELD DEVICE
`
`Publication Classification
`
`(76)
`
`Inventor: Steven P. Hotelling, San Jose, CA (US)
`
`Correspondence Address:
`WONG, CABELLO, LUTSCH, RUTHERFORD
`& BRUCCULERI LLP
`20333 SH 249
`SUITE 600
`HOUSTON, TX 77070 (US)
`
`(21) Appl. No.:
`
` 11/367,749
`
`(22)
`
`Filed:
`
`Mar.3, 2006
`
`Related U.S. Application Data
`
`(60) Provisional application No. 60/658,777, filed on Mar.
`4, 2005. Provisional application No. 60/663,345, filed
`on Mar. 16, 2005.
`
`(51)
`
`Int. CL
`(2006.01)
`G09G 5/00
`(52) U.S. CMe
`ecsccsssessssessecestensssessesestenesinseevesee 345/173
`
`(57)
`
`ABSTRACT
`
`Disclosed herein is a multi-functional hand-held device
`capable of configuring user inputs based on how the device
`is to be used. Preferably,
`the multi-functional hand-held
`device has at most only a few physical buttons, keys, or
`switches
`so that
`its display size can be substantially
`increased. The multi-functional hand-held device also incor-
`porates a variety of input mechanisms,
`including touch
`sensitive screens, touch sensitive housings, display actua-
`tors, audio input, etc. The device also incorporates a user-
`configurable GUI for each of the multiple functions of the
`devices.
`
`yw190
`
`SOFTWARE
`
`DEVICE 1 =
`
`HARDWARE
`
`SOFTWARE
`
`HARDWARE|~\
`
`lo2A
`
`\o4
`
`\Sts
`
`\oLB
`
`\o°4
`I,
`
`WO
`
`APPLE-1031
`
`1
`
`APPLE-1031
`
`

`

`Patent Application Publication
`
`Sep. 7,2006 Sheet 1 of 16
`
`US 2006/0197753 Al
`
`ye
`
`Dooadg
`
`)aC
`
`FIG. 1E
`
`FIG. 1F
`
`12
`14
`
`on
`
`14—~
`
`FIG. 1B
`12 JO
`
`
`po |
`
`
`40E
`
`poo
`
`FIG. 1C
`12 { 14
`oOpo =
`
`AOF
`
`oO0
`
`2
`
`

`

`Patent Application Publication
`
`Sep. 7,2006 Sheet 2 of 16
`
`US 2006/0197753 Al
`
`v7180
`
`DEVICE 1
`
`HARDWARE
`
`SOFTWARE
`
`_ DEVICE 2
`HARDWARE
`
`SOFTWARE
`
`“swreehwe
`
`— “MEAN ( °
`FIG. 2
`
`3
`
`

`

`Patent Application Publication
`
`Sep. 7, 2006 Sheet 3 of 16
`
`US 2006/0197753 Al
`
`22
`
`2s
`
`FIG. 4
`
`4
`
`

`

`Patent Application Publication
`
`Sep. 7,2006 Sheet 4 of 16
`
`US 2006/0197753 A1
`
`\$2
`
`FIG. 5
`
`FIG. 6
`
`St
`
`Sh
`
`FIG. 7
`
`FIG. 8
`
`5
`
`
`
`
`

`

`Patent Application Publication
`
`US 2006/0197753 A1
`
`Sep. 7,2006 Sheet 5 of 16
`
`6
`
`

`

`Patent Application Publication
`
`Sep. 7, 2006 Sheet 6 of 16
`
`US 2006/0197753 Al
`
`&
`
`=
`
`a
`
`FIG.16
`
`ise
`Sv
`
`ise
`
`FIG.15
`
`FIG.14
`
`| GO CAQUO0
`| p0@0000
` 00cog000
`
`7
`
`

`

`Patent Application Publication
`
`Sep. 7,2006 Sheet 7 of 16
`
`US 2006/0197753 Al
`
`2628
`
`L6LE
`
`Vet
`
`262 F
`
`8
`
`

`

`Patent Application Publication
`
`Sep. 7, 2006 Sheet 8 of 16
`
`US 2006/0197753 Al
`
` 34 |
`
`342
`
`FIG. 21
`
`3508
`
`9
`
`

`

`Patent Application Publication
`
`Sep. 7, 2006 Sheet 9 of 16
`
`US 2006/0197753 Al
`
`470
`
`o
`
`Fo?
`
`312.
`E—— F
`
`3714
`
`a4
`FIG. 23
`
`37b
`
`“re
`
`334
`
`38h
`
`FIG. 24
`
`10
`
`

`

`Patent Application Publication
`
`Sep. 7, 2006 Sheet 10 of 16
`
`US 2006/0197753 Al
`
`
`
`0640ONN,
`DETECT ONE OR
`
`DIFFERENTIATE WHETHER TOUCHES ARE
`
`MORE TOUCHES
`
`oF
`
`LIGHT OR HARD TOUCHES
`
` 4
`
`INITIATE
`INITIATE
`
`
`PASSIVE ACTION
`PASSIVE ACTION
`
`
`
`
`406
`
`FIG. 25
`
`g 00
`
`£
`
`DETECT ONE OR MORE TOUCHES
`
`DETERMINE Ui MODE
`
`Gar
`
`;
`
`$D4F
`
`DETERMINE WHETHER TOUCHESARE LIGHT OR HARD a
`
`DETERMINE NUMBER OF TOUCHES
`
`Sv6
`
`Sob
`
`S1D
`
`DETERMINE IF TOUCHES ARE STATIONARY OR IN MOTION
`
`DETERMINE DURATION OF TOUCHES
`
`DETERMINE LOCATION OF TOUCHES
`
`S12
`
`Sid
`
`PERFORM ACTION BASED ON THE DETERMINATIONS
`
`61 lo
`
`FIG. 26
`
`11
`
`11
`
`

`

`Patent Application Publication
`
`Sep. 7, 2006 Sheet 11 of 16
`
`US 2006/0197753 A1
`
`
`
`dn-dodnuaw
`
`Burjaqurs3s)
`
`suape]May
`
`patuejua
`
`
`
`
`
`‘nuauaanisuaspaquosdn
`
`
`
`1195‘Gulyouop‘umMop-y3n0}Ug
`
`
`
`
`
`uoGujpuadapNMOQ/dN3511
`
`
`
`
`
`asn‘paaowazuelsipNMOU/dN
`
`
`
`sapyswoypayyGyybryyy-aungsa6
`
`
`
`
`
`aBueyd‘UdaIdSaqIStAyo
`
`IE‘NMOG/dN
`
`
`6uynpvoyowLHORI/isSTI
`
`pueBausip‘seyeig9ejaut
`aroumAueAue
`nmoad/dn
`
`
`
`SIJey]Suo07WayPayYybYybyy
`
`“alqISIA
`
`uo‘Guryjouop‘uMop-1pNn0}UD
`
`
`
`(uoyng153735A.e6a|
`
`
`
`uUgau2SaJewjUe
`
`
`
`44571Suipys
`
`sapioypaquGyyBiyuado“uoyowYa}
`
`
`
`5yI)WayPazYy6rjybiyayndaxa40
`
`
`asoymAueAue
`
`
`
`UgauTsajewyue
`
`4H9TeGuys
`
`
`
`
`
`(uonounyNN3WAdeb3;2%}))
`
`UO‘Burylouop‘uMOop-YyoND}UO,
`
`sapjoy&Jno»OeQ06‘uORowWb
`auaymAueAue
`
`VLBA
`
`LHOTY
`
`4m
`
`“
`
`ypeqpaes
`
`waywoyybry
`
`Paysnay
`
`
`
`uaaosayew)jUe
`
`ugau2smauasodxa
`03JHOTYBurpys
`dapunwyJYy6yeyUMOp-431/9
`
`“yoy-unug“(Aue41)yore]
`
`PayalWs}!ay)ayWaxe
`
`351)/voRNGyoys
`
`-uO*Guiyqouop‘uMOp-YyoNo]UD
`
`yan}
`
`uo‘Bulyjouop‘umMop-yono)UD
`
`dapun(Aueji)wai!w6iyBiy‘yo-yn
`
`
`
`auaymAueAue
`
`sQ{Aeyagpasodoig
`
`uoneso}uoneing
`
`woPETIA
`
`umopysnoL
`
`tenia
`
`way
`
`UO‘Gulyjouop‘umMOp-yoNO}UD
`
`aBuey3
`
`
`
`vor]3SNwa/AVId
`uo‘Gujyyouop‘umop-ypona)uO
`
`asned/AeldWySyyB)yuMOp-DID
`
`3SNWd/Av1d*9119-uNUQ‘UED|
`3gnvdHousIWid
`
`fenpiia
`
`uoyng
`
`
`AANUSUBSpOIUOD
`-Guug‘Aejapsaypy(Aue41)yon]
`
`
`
`
`
`
`JapunwayI4y6yybiyUMop-77
`feriaNO?
`
`uoyng
`
`a179
`
`auau
`
`3NON
`
`ANON
`
`aNON
`
`14617
`
`wD
`
`#
`
`et
`
`12
`
`quawmeaow
`40Yano}
`Guunp
`ssebutg#
`pesn
`IPJoyy6y“APNISN
`soyb“A°N38
`PPsper35Joysln
`
`uogeGjaenysq215eg
`qonoyIn
`eIRIeL
`wba|-aeNISN
`ID‘AENISTT
`ID“ARENAS
`PID“ACNAST]
`wp
`10®PON
`
`12
`
`

`

`Patent Application Publication
`
`Sep. 7, 2006 Sheet 12 of 16
`
`US 2006/0197753 A1
`
`yaeqpoas
`
`
`
`sojaeyagpesodoid
`
`wayy646Iy
`
`payanoy
`
`
`
`gasajewiue
`
`uges0sMauasodxa
`0}LHOTYGulpys
`
`dapunway7461y6}yuMop-x2"13
`‘yoyo-unuQ“(Aue4})yond}
`PaxXHPWa}atyaynIexa
`
`uo“Guiyjouop‘umoap-"yaNne]UO
`
`udno}
`
`afueyd
`
`uod3SNvd/AWid
`
`
`asneg/AeigqyuG6yybjyuMap-491;9
`
`3SNvd/AWdSANP-UNUO‘UOd)
`
`uo‘Guryjouop‘uMop-YysNo]UD
`
`
`
`dn-dodnugur
`
`
`SAlSUaSPwyU0TD
`
`4apunwayWy6Yy6iyUMOP-PIP
`
`uo‘Buryjouop‘UMOp-Y3ND}UO
`
`
`
`“nualBANISUaSPIU.dn
`
`
`
`NMOO/dNSileuquntyJo3sS})
`
`
`
`a0ueIsIPNMOQ/dNUOBulpuadap
`
`
`
`uaaunsayewIUe
`
`14931Gurpits
`
`
`
`uaasayewjue
`
`
`
`LHONYSulpits
`
`
`
`
`
`aGueyd‘uaasI5JIQISIAYOSaptis
`
`S]JetBU07wazparyByyoy
`
`“QIQISIA
`
`uo*buyujauop‘uMop-Yysno]UD
`
`
`
`ays)waypayy6iyGiyaynsaxeJo
`
`
`
`(voyng9373SAde6a]
`
`ug‘Bujyjouop‘umop-tone)uD
`
`
`
`
`
`(uounyANAaAWAde69}241)
`
`
`NMOd/d/NBui[TO.0s
`SJBUQUINY]JO3S)
`
`
`
`
`JapjospayyByyBiyuado‘uoRouryo}
`
`
`waypazySyyBiyjy“aunysa6Burnp
`
`
`
`
`Japun(Aueji)wayqwéyBiy‘yo-yy
`
`poses“Guyujouop‘uMop-yone]ug
`
`
`Japjoy&noeqO68“uoNOowJYyGu
`
`
`-Buyg‘Aejapsayy“(Aue3))Yano)
`uo“buryjouop‘umop-yono}uD
`
`
`uopowAHOTY/Lsa1PaeBausip
`
`*SayRJGYEAU]Asn‘paaow
`
`
`umopuonolquaweaacnw140n
`
`auayMAUeAueLHOTYIsowimona
`
`amymhue=AueNMOG/NiTIDmaiaaud
`auaymAueAue4431T4919mataaid
`
`
`peuquuny34/uoung
`won|eee=yanogin*
`asaymAueAueMOIAIIGLyguouUWoy
`
`
`jengita)—aoypnoyP85OHRELnay,
`yoysJNONtyou)mana
`
`(guoyngPIPPID
`
`[ErWIAmojAaud
`
`uoRngmataaud
`
`asnvdHausoroyd3NONtIP“
`
`
`jeraiyaONOTSoudaNONtPIP
`
`jenaajoy
`otoud
`
`
`uonediaeu/matacudjreuunyyopoydgz
`
`
`
`iw
`
`uoyng
`
`Yl1ISB
`
`40yyfr=joy
`
`10146)oyoyd
`
`13
`
`13
`
`

`

`Patent Application Publication
`
`Sep. 7, 2006 Sheet 13 of 16
`
`US 2006/0197753 A1
`
`woeqpesd
`
`JoyAryegpesodaig
`
`uoo!ASNVd/AVId
`
`
`“aSNVWd/AVIdUMOp-421)
`
`e/u
`
`aBbuey>
`
`UO“BulyjoUop‘umop-YyaneyUO
`
`Guryjouop
`
`e/u
`
`a/u
`
`G6ujyyouop
`
`Guyyyouop
`
`
`.UBaIsQJeWwIUe
`4apjo3@ano>peq06‘uojowyu
`uo“Guiyjouop‘umop-yone}uC
`
`
`
`LHOTY6ulpis
`
`
`
`(uoipuryNNIWAreBay3y11)
`
`
`
`
`
`(guezng
`
`auaymAue
`
`araymAue
`
`araymAue
`
`aroymAue
`
`auaymAue
`
`
`uaaldsayeulue
`0306‘uonowyayuM“Gujyyou
`
`Op‘UMOP-431]91BUMOP-YyIND}UO
`
`14376uipis
`
`
`
`UsalDSsyeuNue
`
`LHOTYGurpis
`
`
`
`
`
`4x3nA5e63)9911)spe0wxau
`
`Op‘UMOP-}3])YUMOP-YyoNO}UO,
`
`
`
`*“payepvoyjowWyouo-“Guryyou
`
`(uopnq
`
`
`
`
`
`AsdaArebayayy)sen‘Aad0306
`
`(uoyng
`
`
`
`UOROW1H51y/1497pieBausippue
`
`
`
`
`
`
`
`4apl|sauNjon
`
`uoROWNMOG/dnUg‘Gulyjou
`
`
`NMOG/dNsoAow
`
`
`JWINIOAaseaap/aseanul‘payap
`
`auaymAue
`
`Op‘uMOP-43|P3UMOP-YoNo3UD
`
`umopyonoL
`
`AIP
`
`aueymAue
`
`Aue
`
`41451
`
`w3aymAue
`
`Aue
`
`LHOTY
`
`“PLL
`‘4
`
`Aue
`
`Aue
`
`Aue
`
`Aue
`
`Aue
`
`auou
`
`JNON
`
`NMOG/dn
`
`137)
`
`LHOTY
`
`T
`
`Aue
`
`NMOd/dN
`
`UO}JEI0}
`iengita)
`uoReing
`WUSWIOAOW
`40yono}
`6upnp
`siebuy#
`pesn
`aeL
`yono,
`epow
`in*
`
`wen
`
`40
`
`PIS
`
`qwby
`
`219
`
`Wor
`
`161
`
`146y
`
`MON
`
`burherg
`
`MON
`
`BulAetd
`
`MON
`
`furheig
`
`MON
`
`6uldelg
`
`MON
`
`Burkeig
`
`6urhelg
`
`MON
`
`WIP
`
`mon
`
`Gurderg
`
`WIP
`
`MON
`
`GurAelg
`
`x59
`
`MON
`
`GulAely
`
`14
`
`14
`
`

`

`Patent Application Publication
`
`Sep. 7, 2006 Sheet 14 of 16
`
`US 2006/0197753 A1
`
`yoeqpaa4
`
`e/u
`
`
`
`vos!3SNVd/AVId
`
`a6ueys
`
`e/u
`
`e/u
`
`
`
`uzansayewue
`
`LHOTeGurpas
`
`
`
`JaplisAWAOA
`
`
`
`NMOG/dNsaAow
`
`
`
`uaaudsayew|ue
`
`1437Guipis
`
`cut
`
`
`
`u3zasIsaJewiue
`
`
`
`LHOTY6ulpis
`
`(uogng
`
`
`
`GLt‘414
`
`
`
`
`43pjo3&No49eqOB‘uoNjowWYybyauaymAueAueLHOTY“fqwByuaaos
`
`AaudAseGa)oy1)|)way-nasd0]06Wa}!
`
`
`OWNIOAaseaumap/aseauou)‘a}apway
`
`
`
`
`‘payapvojowWY6Uug“BuryjouLHOTYT491]ayoymAueAueugaus
`Op‘UMOP->D1/9@UMOP-YyoNDUO:“Hing
`OP‘UMOP->91]999UMOP-49ND)UO“nnd
`Op‘UMOP-»¥31/99gUMOP-Y>NOIUD“41n4
`uO*Gujyjouop‘uMop-yoN0}UOauaymdueAueJNONtPH?=:vas
`
`UO“Bujyzouep‘uMop-yone)uO“1nd
`uoRoWNMOG/dNUO“BuryyOUeuoyadueAueNMOG/aNtIP=uaau2S
`
`
`0}06‘uogowyajug“BulyjouaraymAueAueLaT3312ugauos
`
`
`LXANAdeG8)9>1])Weyyrouway
`(uogouryNNAWA5e6a;291)way
`
`
`
`‘3SNWd/AVIdUMOp-991Dwy
`so1aeyogpasodaigvoRro]uoTeingSuunpesoBuly40an#umopypnoL
`
`
`
`
`
`
`
`uonowLHOTY/1337pue6ausippue
`
`
`GuryjouopasaymAueAueauou.worugau2s
`GuyyyouopauaymAueAue4431Tq6y)= uaans
`
`BujyyouoparaymAueAve=NMOG/dNT346uaauds
`
`
`
`(338‘MONAPeqUODo/6urs‘AepsepuE]eD‘mojadew40304)Wie{lUR8I2S-])N4
`
`
`
`
`
`quaweany=*cuore
`
`
`yenquia)Joupne,=PesP8g,OIRPEL
`
`(2uoging42119yD
`
`.way
`
`“ling
`
`way
`
`“mn
`
`LL
`
`“1g
`
`way
`
`15
`
`15
`
`

`

`Patent Application Publication
`
`Sep. 7, 2006 Sheet 15 of 16
`
`US 2006/0197753 A1
`
`
`
`
`
`
`
`
`
`
`
`12]03Bujpucadeabew)ajefsued)
`
`
`
`
`
`JOUO/JeE|sUeyPjosqUaDjeqUaWAsIU
`
`umopysnoy
`
`(gu0ynq
`
`voneing
`
`pue6ulje2sJeqUswaIDU]183030)Buipsdooe_®uaymdAue
`
`
`
`
`Peqpessuones0}
`JO]AEYOgpesodaigrenvia)
`
`
`
`
`WS}!USS1DS-[jNZUOSBIN{seaByIn0}-1nInNW;eUoNdO
`pesn=ayjyeL40lpnOW8amPssoGury40ap#rquawenomtis#*yonolnwin
`yary6uowyaweal
`yor"IK
`
`
`
`
`
`
`
`‘uoHejsuesabe)afew)(wooz)ates‘syequoa
`
`
`
`uonelaJo(uonesA6Josnjpei)peauds
`
`
`
`
`
`
`
`Bujpuoooeayewyageqos‘speyuco
`
`
`
`ALTINA
`
`
`
`SJIEIWO)JOUONEYOY[eEqUaUaUIU]0}
`
`16
`
`16
`
`

`

`Patent Application Publication
`
`Sep. 7, 2006 Sheet 16 of 16
`
`US 2006/0197753 Al
`
`a
`
`Se,roO
`
`oe ec x
`
`INDICATOR
`
`&2 .
`
`AUDIO/TACTILE
`FEEDBACK
`
`Ul4
`
`tr
`
`DISPLAY-
`
`408
`
`TOUCH
`SENSING
`ec +
`FORCE
`SENSING
`
`God
`
`v
`
`Y. ”
`
`vo
`
`CHARGING |
`
`BATTERY
`
`bsSz
`
`G5?
`
`Lor
`
`bart
`
`MEMORY
`
`34
`
`MOTION|ors|,
`
`ie
`
`ID
`
`MECHANICAL
`ACTUATORS
`
`IB
`
`MICROPHONE
`
`DISPLAY
`ACTUATOR
`bua
`
`sess f-U
`
`lo
`
`vat
`
`c40
`
`bot
`
`Fib. 22
`
`17
`
`

`

`US 2006/0197753 Al
`
`Sep. 7, 2006
`
`MULTI-FUNCTIONAL HAND-HELD DEVICE
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`[0001] This application is related to and claimsthe benefit
`of: (1) U.S. Provisional Patent Application No. 60/658,777,
`titled “Multi-Functional Hand-Held Device,” filed Mar. 4,
`2005 and (2) U.S. Provisional Patent Application No.
`60/663,345,
`titled “Multi-Functional Hand-Held Device,
`*filed Mar. 16, 2005, each of which is hereby incorporated
`by reference.
`
`[0002] This application is related to the following appli-
`cations, which are all herein incorporated by reference: (1)
`USS. patent application Ser. No. 10/188,182, title “Touch
`Pad for Handheld Device,” filed on Jul. 1, 2002; (2) U.S.
`patent application Ser. No. 10/722,948, titled “Touch Pad for
`Handheld Device,” filed on Nov. 25, 2003; (3) U.S. patent
`application Ser. No. 10/643,256,titled “Movable Touch Pad
`with Added Functionality,” filed on Aug. 18, 2003; (4) U.S.
`patent application Ser. No. 10/654,108, titled “Ambidex-
`trous Mouse,” filed on Sep. 2, 2003; (5) U.S. patent appli-
`cation Ser. No. 10/840,862,
`titled “Multipoint Touch
`Screen,” filed on May 6, 2004; (6) U.S. patent application
`Ser. No. 10/903,964, titled “Gestures for Touch Sensitive
`Input Devices,” filed on Jul. 30, 2004;
`(7) US. patent
`application Ser. No. 11/057,050, titled “Display Actuator,”
`filed on Feb. 11, 2005, (9) U.S. patent application Ser. No.
`11/115,539, titled “Hand-Held Electronic Device with Mul-
`tiple Touch Sensing Devices,”filed Apr. 26, 2005.
`
`BACKGROUND
`
`[0003] There exist today many types of hand-held elec-
`tronic devices, each of which utilizes some sort of user
`interface. The user interface typically includes and output
`device in the form of a display, such as a Liquid Crystal
`Display (LCD), and one or more input devices, which can be
`mechanically actuated (e.g. switches, buttons, keys, dials,
`joysticks, joy pads) or electrically activated (e.g., touch pads
`or touch screens). The display is typically configured to
`present visual information such as text and graphics, and the
`input devices are typically configured perform operations
`such as issuing commands, making selections or moving a
`cursoror selector in the electronic device. Each of these well
`
`known devices has considerations such as size and shape
`limitations, costs, functionality, complexity, etc. that must be
`taken into account when designing the hand-held electronic
`device. In most cases, the user interface is positioned on the
`front face (or front surface) of the hand-held device for easy
`viewing of the display and easy manipulation of the input
`devices.
`
`To elaborate, the telephone 10A typically includes
`[0005]
`a display 12 such as a character or graphical display, and
`input devices 14 such as a number pad and in somecases a
`navigation pad. The PDA 10B typically includes a display
`12 such as a graphical display, and input devices 14 such as
`a stylus based resistive touch screen and buttons. The media
`player 10C typically includes a display 12 such as a char-
`acter or graphic display and input devices 14 such as buttons
`or wheels. The iPod® media player manufactured by Apple
`Computer, Inc. of Cupertino, Calif.
`is one example of a
`media player that includes both a display and input devices
`disposed next to the display. The remote control 10D typi-
`cally includes an input device 14 such as a keypad and may
`or may not have a character display 12. The camera 10E
`typically includes a display 12 such as a graphic display and
`input devices 14 such as buttons. The GPS module 10F
`typically includes a display 12 such as graphic display and
`input devices 14 such as buttons, and in some cases a joy
`pad.
`
`traditionally separate hand-held elec-
`[0006] Recently,
`tronic devices have begun to be combinedin limited ways.
`For example, the telephone 10A has been combined with the
`PDA 10B. Oneproblem that has been encountered is in the
`way inputs are made into the device. Each of these devices
`has a particular set of input mechanismsfor providing inputs
`into the device. Someof these input mechanismsare generic
`to all the devices (e.g., power button) while others are not.
`The ones that are not generic are typically dedicated to a
`particular functionality of the device. By way of example,
`PDAstypically include four dedicated buttons while cell
`phones typically include a numeric keypad andat least two
`dedicated buttons.
`
`Thusit is a challenge to design a merged device
`[0007]
`with limited input devices without adversely affecting the
`dedicated inputs for each device. As will be appreciated, it
`is preferable, not to overload the hand-held devices with a
`large number of input mechanismsas this tends to confuse
`the user and take up valuable space,i.e., “real estate.” In the
`case of hand-held devices, space is at a premium because of
`their small size. At some point there is not enough space on
`the device to house all the necessary buttons and switches,
`etc. This is especially true when considering that all these
`devices need a display that typically takes up a large amount
`of space on its own. To increase the numberof input devices
`beyond somelevel, designers would have to decrease the
`size of the display. However, this will often leave a negative
`impression on the user because the user typically desires the
`largest display possible. Alternatively,
`to accommodate
`more input devices designers may optto increasethe size of
`the device. This, too, will often leave a negative impression
`on a user because it would make one-handed operations
`difficult, and at some point, the size of the device becomes
`so large that it is no longer considered a hand-held device.
`
`[0008] Therefore what is needed in the art is an improved
`user interface that works for multi-functional hand-held
`devices.
`
`[0004] FIGS. 1A-1F are diagrams of various hand-held
`electronic devices including for example a telephone 10A
`(FIG. 1A), a PDA 10B (FIG. 1B), a media player 10C (FIG.
`1C), a remote control 10D (FIG. 1D), a camera 10E (FIG.
`1E), and a GPS module 10F (FIG. 1F). In each of these
`devices 10, a display 12, which is secured inside the housing
`SUMMARY
`of the device 10 and which can be seen through an opening
`in the housing, is typically positioned inafirst region of the
`electronic device 10. Each of these devices also include one
`[0009] Disclosed herein is a multi-functional hand-held
`device capable of configuring user inputs based on how the
`device is to be used. Preferable, the multi-functional hand-
`held device has at most only a few physical buttons, keys, or
`
`or more input devices 14, which are typically positioned in
`a second region of the electronic device 10 next to the
`display 12.
`
`18
`
`18
`
`

`

`US 2006/0197753 Al
`
`Sep. 7, 2006
`
`its display size can be substantially
`so that
`switches
`increased. In other words, by eliminating physical buttons,
`keys, or switches from a front surface of an electronic
`device, additional surface area becomes available for a
`larger display. Ultimately this strategy would allow a sub-
`stantially full screen display. As used herein, a full screen
`display is a display that consumes, or at least dominates, a
`surface (e.g., front surface) of an electronic device.
`
`[0010] Various embodiments of a multi-functional hand-
`held device are discussed below with reference to FIGS.
`2-28. However, those skilled in the art will appreciate that
`the detailed description given herein with respect to these
`figures is exemplary and not exhaustive and that many
`variations on these embodiments are possible.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0011] The invention may best be understood by reference
`to the following description taken in conjunction with the
`accompanying drawings in which:
`
`[0012] FIGS. 1A-1F are diagrams of various electronic
`devices.
`
`[0030] FIG. 19 is a side elevation view, in cross section,
`of a hand-held device incorporating a force sensitive display.
`
`[0031] FIG. 20 illustrates an input device that combines
`touch sensing and force sensing devices to provide x, y and
`z components when touched.
`
`[0032] FIG. 21 is a side elevation view of an I/O device
`that combines a display with touch screen and a force
`sensing mechanism.
`
`[0033]
`
`FIG.22is a side elevation view of an input device.
`
`in cross section, of a
`[0034] FIG. 23 is a side view,
`hand-held device that incorporates a squeeze feature.
`
`[0035] FIG. 24 is a side view,
`hand-held electronic device.
`
`in cross section, of a
`
`[0036]
`method.
`
`FIG. 25 is a block diagram of a touch sensing
`
`[0037] FIG. 26 is a block diagram of touch sensing
`method.
`
`[0038] FIGS. 27A-Eare tables representing one example
`of a touch vocabulary associated with a music player.
`
`[0013] FIG. 2 isa simplified diagram ofa multi-functional
`hand-held device.
`
`[0039] FIG. 28 is a block diagram of an exemplary
`multi-functional hand-held device.
`
`FIG.3 is a perspective view of a substantially full
`[0014]
`screen hand-held device with a limited number of limited
`buttons.
`
`[0015] FIG. 4 is a front view of a hand-held device with
`at least one button.
`
`[0016] FIG. 5 is a diagram of a GUI separated into a
`standard region and a control region.
`
`[0017] FIG. 6 is a diagram of a GUI separated into a
`standard region and a control region.
`
`[0018] FIG. 7 is a diagram of a GUI separated into a
`standard region and a control region.
`
`[0019] FIG. 8 is a diagram of a GUI separated into a
`standard region and a control region.
`
`[0020]
`
`FIG.9 illustrates an exemplary a GUI for a PDA.
`
`[0021] FIG. 10 illustrates an exemplary GUI for a cell
`phone.
`
`[0022] FIG. 11 illustrates an exemplary GUI for a media
`player.
`
`[0023] FIG. 12 illustrates an exemplary GUI for a video
`player.
`
`[0024] FIG. 13 illustrates an exemplary GUI for a game
`player.
`
`[0025] FIG. 14 illustrates an exemplary GUIfor a camera.
`
`[0026] FIG. 15 illustrates an exemplary GUI for a GPS.
`
`[0027] FIG. 16 illustrates an exemplary GUI for a remote
`control.
`
`[0028] FIG. 17 illustrates an exemplary GUI for a hand
`top.
`
`DETAILED DESCRIPTION
`
`J. Multi-Functionality
`
`[0040] Electronic device manufacturers have discovered
`the advantages of combining separate hand-held electronic
`devices to form multi-function devices. By having a single
`multi-function device, a user is not burdened with carrying,
`purchasing, and maintaining multiple devices. Further, the
`user is not limited in the operations that can be performed,
`ie., the user can perform different operations with a single
`device that would have otherwise required the use of a
`different devices.
`
`[0041] As used herein, the term “multi-functional” is used
`to define a device that has the capabilities of two or more
`traditional devices in a single device. The multi-functional
`device may,
`for example,
`include two or more of the
`following device functionalities: PDA, cell phone, music
`player, video player, gameplayer, digital camera, handtop,
`Internet terminal, GPS or remote control. For each new
`device functionality that is added to a single device, the
`complexity and size of the device tends to increase. There-
`fore, with hand-held devices, there is typically a trade-off
`between beeping the footprint small and complexity low
`while still maximizing the functionality of the device.
`
`in
`In some cases, combing devices may result
`[0042]
`redundant hardware components, which allows components
`to be used for multiple different, device functionalities. In
`other cases, certain hardware components are distinct to
`each device and therefore additional space and connectivity
`must be madeavailable. Furthermore, each device function-
`ality typically has its own programming or application
`software and, therefore, the multifunction device must be
`designed with enough memory to accommodate all
`the
`various software components.
`
`[0029] FIG. 18 illustrates an exemplary GUI for a main
`menu of a multi-functional hand held device.
`
`[0043] A personal digital assistant (PDA) is a mobile
`hand-held device that provides computing and information
`
`19
`
`19
`
`

`

`US 2006/0197753 Al
`
`Sep. 7, 2006
`
`storage and retrieval capabilities for personal and/or busi-
`ness use. PDAs are severally capable of tracking names,
`addresses, phone numbers and appointments. They are also
`often capable of taking notes, performing calculations, pag-
`ing, data messaging, and electronic mail. PDAs may also
`include functionality for playing simple games, music, and
`other media files. Examples of PDAs include the Palm Pilot
`and Blackberry.
`
`[0044] Like most hand-held devices, PDAs typically
`include a display and various input devices. The input
`devices may includea stylus and touch screen that work in
`combination with a handwriting recognition program, key-
`pads, mini-keyboards, navigation pads, and/or soft or fixed
`function buttons.
`
`[0045] Cell phones are mobile telephones that allow a user
`to connect to other telephones using a cellular network. Cell
`phonestypically include a transceiver for transmitting and
`receiving telephone calls, controls such as a navigation pad
`for traversing through a display, a keypad for making
`numeric entries (and in somecases alphabetic entries), and
`soft or fixed function buttons. For example, in many cell
`phonesone fixed function button is used for starting a call
`and another fixed function button is used for ending a call.
`
`[0046] Media players come in a variety of forms. Music
`players are generally configured to store, process and output
`music. Music players can be based on the MP3 or AAC
`format, which is a compression system for music. Music
`Players typically include a microprocessor, memory,display,
`audio jack, data port and playback controls. The playback
`controls typically include features such as menu, play/pause,
`next, previous, volumeup, and volume down.Video players
`are similar to music players in most respects. In some cases,
`they may include a data storage device for receiving a
`removable storage medium such as a DVD. The iPod®
`media player manufactured by Apple Computer, Inc. of
`Cupertino, Calif. is one example of a media player.
`
`[0047] Handtops are general purpose computers similar to
`laptops, but in a smaller form factor. Handtops typically
`include a display and a full keyboard.
`
`[0048] FIG. 2 isa simplified diagram ofa multi-functional
`hand-held device 100. The multi-functional hand-held
`
`device 100 integrates at least two devices 102 into a single
`device. Each device 102 includes both hardware and soft-
`
`ware components 104 and 106, which are integrated into
`multi-functional hand-held device 100. It should be pointed
`out that the multi-functional hand-held device 100 is not
`
`limited to only two devices, and may in fact integrate any
`numberof devices.
`
`[0049] Multi-functional device 100 also includes switch
`110, which that allows multi-functional device 100 to be
`switched from one device operating modeto another device
`operating mode. For example, switch 110 may allow a user
`to cycle through cell phone, media player, and PDA modes
`of operation. Once a particular operating modeis selected,
`the multi-functional device 100 operates as the selected
`device. For example,
`the programming related to the
`selected device is activated for use by the multi-functional
`hand-held device. The programming may include reconfig-
`uring the UI based on the selected device so that the inputs
`madebythe usercorrelate to the device in use. For example,
`the functions of any physical buttons, switches or dials as
`
`well as soft buttons, switches or dials can be reconfigured to
`correspond to the selected device.
`
`[0050] However, the operating modes of multi-functional
`hand-held device 100 need not completely independent. In
`many cases,
`it will be desirable to allow the multiple
`functionalities to interact with each other. For Example, a
`user may look up a telephone number of a contact in the
`PDAandpass this number to the phoneto be dialed.
`II. Form Factor
`
`[0051] The form factor of a hand-held device is generally
`a device that can easily be held in one hand. A typical
`hand-held device includes a small display in an upper
`portion of the front surface of the device and input controls
`in a lower portion of the front surface of the device. The
`device may additionally include controls and ports on the
`top, bottom, and side surfaces. Prior art hand-held devices
`have typically had small displays which causes some dis-
`satisfaction for users of these devices. It is generally pre-
`ferred to have larger displays so that more information can
`be displayed or the information being displayed can be more
`easily viewable (e.g.,
`larger text).
`In the case of video
`players and gameplayers in particular, larger displays are
`much preferred over smaller ones.
`
`[0052] However, the use of large displays has been limited
`in the past because the required input controls often take up
`substantial portions of the available space. Furthermore, as
`device functionality converges, the number of controls on
`the device typically increases as each device includes its
`own dedicated controls. Therefore devices must be made
`larger or the displays must be smaller to accommodate all
`the new controls. Both of these outcomesare unsatisfactory.
`Larger devices are bulky and hard to use, and smaller
`screens are difficult to utilize for both relaying information
`and reading information from the screen.
`
`A. One-Handed vs. Two-Handed Operation
`
`[0053] A hand-held electronic device may be directed at
`one-handed operation or two-handed operation. In one-
`handed operation, a single hand is used to both support the
`device as well as to perform operations with the user
`interface during use. Cellular phones and media players are
`examples of hand-held devices are generally intended to can
`be operated solely with one hand.In the case of a cell phone,
`for example, a user may grasp the phone in one hand
`between the fingers and the palm and use the thumb to make
`entries using keys, buttons or a # joy pad.
`
`In two-handed operation, one hand is used to
`[0054]
`support the device while the other hand performs operations
`with a user interface during use or, alternatively, both hands
`support the device as well as perform operations during use.
`PDA’s and gameplayers are examples of hand-held device
`that are typically operated with two hands. In the case of a
`PDA, for example, the user may grasp the device with one
`hand and make entries using the other hand, for example,
`using a stylus. In the case of a gameplayer, the user typically
`grasps the device in both hands and makeentries using either
`or both hands while holding the device.
`
`B. Footprint/Size
`
`[0055] Hand-held devices may have a variety different
`footprints or sizes. The footprint is typically associated with
`how the device is going to be used. Devices such as PDAs
`
`20
`
`20
`
`

`

`US 2006/0197753 Al
`
`Sep. 7, 2006
`
`are typically used with both hands and thus they tend to be
`larger. Alternatively, cell phone handsets are typically used
`with only one hand, and thus they tend to be smaller. When
`integrating multiple devices,
`it is important task to deter-
`mine the appropriate footprint of the device. For example,
`some believe that PDAs are too large for phone use while
`cell phones are too small for PDA use. Designers generally
`must consider the primary use of the device and gear the
`footprint
`towards that use. Although there are different
`footprints, there are typically minimum and maximum foot-
`prints. If the footprint is too large or too small, the device
`maybe difficult to use.
`
`the smaller devices are
`[0056] Because of their size,
`typically placed in a pocket while the larger devices are not.
`A media player is one example of a hand-held device thatis
`sized for placement into a pocket of the user. By being
`pocket-sized, the user does not have to directly carry the
`device and therefore the device can be taken almost any-
`where the user travels (e.g.,
`the user is not
`limited by
`carrying a large, bulky and often heavy device,as in a laptop
`or notebook computer).
`
`[0057] Generally it is preferred, although not necessary,
`that hand-held devices of the type disclosed herein have
`dimensions of about 5 inx3 inx1 in and perhaps about 4.1
`inx2.4 inx0.75 in.
`
`C. Full Screen Display
`
`[0058] Because the footprint of a multifunctional hand-
`held device is substantially fixed by the intended primary
`use of the device,
`it becomes important for designers to
`determine an appropriate layout for the UI. For example,
`some devices may work better with a limited number of
`buttons and a large display while others may work better
`with a large number of buttons or a complete keyboard and
`a small display.
`
`[0059] A preferred multi-functional hand-held device may
`be configured with a full screen display or a near full screen
`display. A full screen display consumes substantially the
`entire front surface of the device. The display may extend
`edge to edge or mayfit within a small bezel of the housing
`at the edge of the device. The full screen display may
`consumes 90% or more of a front surface of a housing for
`a hand-held electronic device.
`
`[0060] The full screen display may have a variety of
`different configurations depending on the overall footprint of
`the device. If the device is wide, the full screen display may
`have a traditional aspect ratio of about 4:3. If the device is
`elongated, the full screen display may have an aspectratio
`that is more panoramic such as 16:9.
`D. Limited Number of Mechanical Actuators
`
`[0061] To accommodate a full screen display, the multi-
`functional hand-held device is preferably configured with a
`limited number of physical buttons. Because a limited
`number of physical buttons are provided,
`the hand-held
`device preferably uses a touch screen as the primary input
`device. Touch screens are transparent touch sensing devices
`that are positioned over displays. They typically work in
`conjunction with a GUI presented on the display. For
`example, the GUI may present an on-screen button, and the
`touch screen may detect when a user presses the on-screen
`
`button (e.g., places their finger or stylus over the on-screen
`button). Touch screens and GUI’s are described in greater
`detail below.
`
`[0062] The hand-held device may be constructed with
`only cross-functional physical buttons,
`i.e.,
`there are no
`buttons dedicated to individual devices. These type of but-
`tons may include power buttons and hold switches.
`In
`another embodiment, the hand-held device may not include
`any physical buttons at all.
`In some embodiments,
`the
`physical buttons are limited to only the sides and back
`surface of the hand-held device. In other embodiments, the
`physical buttons of the hand-held device are limited to the
`upper and lower portion of the sides so that there are no
`buttons in the regions of the sides where a user would
`physically support the device (i.e., holding region). In still
`other embodiments, the physical buttons may be located on
`the front surface, but only in the bezel region surrounding
`the display.
`In some embodiments,
`the but