as) United States
`a2) Patent Application Publication (0) Pub. No.: US 2010/0066763 A1
`
` MACDOUGALLet al. (43) Pub. Date: Mar.18, 2010
`
`
`US 20100066763A1
`
`(75)
`
`(54) ORIENTING DISPLAYED ELEMENTS
`RELATIVE TO A USER
`Inventors:
`FRANCIS MACDOUGALL,
`Ottawa (CA); EVAN HILDRETH.
`Ottawa (CA)
`Correspondence Address:
`FISH & RICHARDSONP.C.
`
`P.O. BOX 1022
`MINNEAPOLIS, MN 55440-1022 (US
`,
`(US)
`GestureTek, Inc., Sunnyvale, CA
`(US)
`
`(73) Assignee:
`
`(21) Appl. No.:
`.
`Filed:
`
`(22)
`
`12/559,147
`
`Sep. 14, 2009
`
`Related U.S. Application Data
`
`(60) Provisional application No. 61/096,367, filed on Sep.
`12, 2008.
`
`Publication Classification
`
`(61)
`
`Ine Cl
`GO9G 5/00
`
`(2006.01)
`
`(52) US. Cle cecccccssssssssesssssssssessssssesnsssensesese 345/656
`
`(57)
`ABSTRACT
`One or more elements are initially displayed on a display
`component of an electronic device. After the one or more
`elements have been displayed on the display componentof
`the electronic device, an image of a user of the electronic
`device is captured, and an orientation of the electronic device
`relative to the user is determined based on the captured image
`ofthe user ofthe electronic device. Thereafter, an orientation
`of at least one ofthe displayed elementsis adjustedrelative to
`the display componentof the electronic device based on the
`determinedorientation ofthe electronic device relative to the
`user.
`
`124 106
`
`104
`
`114
`
`AL
`
`I 108
`
`110 AS
`
`1
`
`APPLE 1014
`
`APPLE 1014
`
`1
`
`

`

`Patent Application Publication Mar. 18, 2010 Sheet 1 of 15
`
`US 2010/0066763 Al
`
`108
`
`110 Na
`108
`
`FIG.1A
`
`FIG.1B
`
`2
`
`

`

`Patent Application Publication Mar. 18, 2010 Sheet 2 of 15
`
`US 2010/0066763 Al
`
`108
`
`
`FIG.1C
`
`FIG.1D
`
`3
`
`

`

`Patent Application Publication Mar. 18, 2010 Sheet 3 of 15
`
`US 2010/0066763 Al
`
`
`
`FIG.1E
`
`4
`
`

`

`Patent Application Publication Mar. 18, 2010 Sheet 4 of 15
`
`US 2010/0066763 Al
`
`210
`
`210
`
`212
`
`
`
`
`
`
`
`
`
`
`
`5
`
`

`

`US 2010/0066763 Al
`
`312
`
`310
`
`Patent Application Publication Mar. 18, 2010 Sheet 5 of 15
`
`308
`
`a
`
`a
`
`“
`
`a
`
`ra
`
`“
`
`“
`
`7
`
`a
`
`“
`
`“
`
`7
`
`”
`
`ao
`
`302 306
`302
`
`a”
`
`wo
`
`“
`
`a”
`
`“
`
`0,
`
`~
`
`~
`
`~
`
`~
`
`~
`
`~
`
`~
`
`™~
`
`~
`
`~
`
`6
`
`

`

`Patent Application Publication Mar. 18, 2010 Sheet 6 of 15
`
`US 2010/0066763 Al
`
`400
`
`
`
`
`Display Element on a Display
`Componentof an Electronic
`
`Device
`
`402
`
`the Electronic Device
` Capture an Image of a User of|,404
`
`
`
`
`
`Process the Captured Image of
`the User of the Electronic
`Device to Determine an
`Orientation of the Electronic
`Device Relative to the User
`
`
`
`
`
`406
`
`
`
`
` Adjust Orientation of thé
`Displayed Element?
`
`
`408
`
`
`
`
`
`Adjust the Orientation of the
`Displayed Element Relative to
`
`the Display Component Based
`
`on the Orientation of the
`Electronic Device Relative to
`
`the User
`
`410
`
`FIG. 4
`
`7
`
`

`

`Patent Application Publication Mar. 18, 2010 Sheet 7 of 15
`
`US 2010/0066763 Al
`
`FIG. 5A
`
`reject
`
`eaauae
`
`face
`
`FIG. 5B
`
`8
`
`

`

`Patent Application Publication Mar. 18, 2010 Sheet 8 of 15
`
`US 2010/0066763 Al
`
`610-—Nk2 610AAF608 608
`
`
`
`
`
`
`
`
`Fig.6C
`
`Fig.6B
`
`9
`
`

`

`Patent Application Publication Mar. 18, 2010 Sheet 9 of 15
`
`US 2010/0066763 Al
`
` CHAPTER 4
`OOWOW
`[mene|
`
` 1:05 / 2:00
`
`706
`
`Fig.7B
`
`10
`
`10
`
`

`

`
`
`11
`
`

`

`J
`
`O O O 0 O L
`
`Patent Application Publication Mar. 18,2010 Sheet 11 of 15
`
`US 2010/0066763 Al
`
`O)
`
`O O O
`
`)
`CJ
`
`O C
`
`O)
`
`C O O Q
`
`l O O C
`
`l
`CO
`
`12
`
`

`

`Patent Application Publication Mar. 18,2010 Sheet 12 of 15
`
`
`
`
`
`
`
`
`
`
`
`The quick brown fox jumped
`over the lazy dog.
`The quick brown fox jumped
`over the lazy dog.
`The quick brown fox jumped
`over the lazy dog.
`The quick brown fox jumped
`
`US 2010/0066763 Al
`
`Fig. 10A
`
`The quick brown
`fox jumped over
`the lazy dog.
`The quick brown
`fox jumped over
`|.
`the lazydog.
`The quick brown:
`fox jumped over
`the lazy dog.
`
`
`
`Fig. 10B
`
`13
`
`13
`
`

`

`Patent Application Publication Mar. 18,2010 Sheet 13 of 15
`
`US 2010/0066763 Al
`
` The quick brown fox jumps
`
`
`
`
`
`over the lazy dog.
`The quick brown fox jumps
`over the lazy dog.
`The quick brown fox jumps
`over the lazy dog.
`
`Fig. 11A
`
`The quick brown fox jumps
`over the lazy dog.
`The quick brown fox jumps
`over the lazy dog.
`The quick brown fox jumps
`over the lazy dog.
`The quick brown fox jumps
`overthe lazy dog.
`The quick brown fox jumps
`over the lazy dog.
`The quick brown fox jumps
`
`overthe lazy dog.
`
`Fig. 11B
`
`14
`
`

`

`Patent Application Publication Mar. 18,2010 Sheet 14 of 15
`
`US 2010/0066763 Al
`
`The quick brown fox
`
`jumps overthe lazy
`the lazy dog.
`
`Fig.12A
`
`Dave
`
`Today
`
`10:00AM
`
`The quick brown
`fox jumps over
`
`Fig.12B
`
`15
`
`

`

`Patent Application Publication Mar. 18, 2010 Sheet 15 of 15
`
`US 2010/0066763 Al
`
`1304
`
`PHOTOS
`
`MUSIC
`
`4304
`
`1304
`
`a:0
`
`=
`
`MAIL
`
`INTERNET
`
`1302
`
`9
`
`DOWNLOADS
`1304
`0
`
`TELEVISION Fig.13A
`TELEVISION
`
`DOWNLOADS
`
`PHOTOS
`
`1304
`
`1304
`
`INTERNET
`
`16
`
`

`

`US 2010/0066763 Al
`
`Mar. 18, 2010
`
`ORIENTING DISPLAYED ELEMENTS
`RELATIVE TO A USER
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`[0001] This application claims the benefit of U.S. Provi-
`sional Application No. 61/096,367, filed Sep. 12, 2008, which
`is incorporated by reference.
`
`TECHNICAL FIELD
`
`
`
`[0002] This disclosure relates to orienting elements dis-
`played on a display ofan electronic devicerelative to a user of
`the electronic device.
`
`BACKGROUND
`
`[0003] Many electronic devices include display compo-
`nents that facilitate user interaction with the electronic
`devices by generating visual outputthat is displayed to users
`of the electronic devices and/or, in some cases, receiving
`input from users of the electronic devices.
`
`SUMMARY
`
`In one general aspect, one or more elements are
`[0004]
`initially displayed on a display componentofan electronic
`device. After the one or more elements have been displayed
`on the display componentof the electronic device, an image
`of a user of the electronic device is captured, and an orienta-
`tion ofthe electronic devicerelative to the user is determined
`based on the captured image of the user of the electronic
`device. Thereafter, an orientation of at least one of the dis-
`played elementsis adjusted relative to the display component
`of the electronic device based on the determinedorientation
`of the electronic device relative to the user.
`[0005]
`Implementations may include one or more of the
`following features. For example, a displayed element may be
`displayed with an initial orientation relative to the user.
`Thereafter, the orientation ofthe displayed elementrelative to
`the display component of the electronic device may be
`adjusted such that the orientation of the displayed element,
`relative to the user, is substantially the same as the initial
`orientation ofthe at least one displayed elementrelative to the
`user. Prior to adjusting the orientation of the displayed ele-
`ment, it may be determined that the displayed elementis not
`appropriately orientedrelative to the user, and the orientation
`of the at least one displayed elementrelative to the display
`component of the electronic device may be adjusted in
`response to determining that the at least one displayed ele-
`mentis not appropriately orientedrelative to the user. Adjust-
`ing the orientation of the least one displayed elementrelative
`to the display component of the electronic device may be
`performed automatically without user involvement.
`[0006]
`Insome implementations, the display component of
`the electronic device may support displaying elements
`accordingto at least a portrait orientation relative to the dis-
`play component and a landscape orientation relative to the
`display component. In such implementations, an element
`may be displayed on the display componentof the electronic
`device according to the landscape orientation relative to the
`display component. However, the display componentof the
`electronic device may be determinedto be in a portraitori-
`entation relative to the user. Therefore, the orientation of the
`displayed element may be changed from the landscape ori-
`entation relative to the display component to the portrait
`
`orientation relative to the display component in response to
`determining that the display component of the electronic
`device is in a portrait orientation relative to the user. Similarly,
`the display componentof the electronic device initially may
`display the element on the display componentofthe elec-
`tronic device according to the portrait orientationrelative to
`the display component. However, the display component of
`the electronic device may be determinedto be in a landscape
`orientation relative to the user. Therefore, the orientation of
`the displayed element may be changed from the portrait ori-
`entation relative to the display componentto the landscape
`orientation relative to the display component in response to
`determining that the display component of the electronic
`device is in a landscape orientation relative to the user.
`[0007]
`Insome implementations, the image that is captured
`of the user of the electronic device may include at least a
`portion of the user’s face and/or head. In such implementa-
`tions, the orientation of the user’s face and/or head in the
`captured image may be determined(e.g., using an AdaBoost
`process), and the orientation of the electronic device relative
`to the user may be determined based on the determinedori-
`entation of the user’s face and/or head in the captured image.
`For example, the image that is captured of the user of the
`electronic device may includethe user’s eyes, the orientation
`of the user’s eyes in the captured image may be determined,
`andthe orientation ofthe electronic device relative to the user
`
`may be determined based on the determinedorientation ofthe
`user’s eyes in the captured image.
`[0008] Additionally or alternatively, the image that is cap-
`tured ofthe user of the electronic device may includeatleast
`a portion of the user’s body. In such implementations, the
`orientation of the portion of the user’s body in the captured
`image may be determined, and the orientation of the elec-
`tronic device relative to the user may be determined based on
`the determinedorientation ofthe portion ofthe user’s body in
`the captured image.
`[0009]
`Insome implementations, the image that is captured
`of the user of the electronic device may include at least a
`portion ofthe user’s face. At least two different versionsofthe
`captured image, each being oriented differently, may be gen-
`erated. Thereafter, face detection processing techniques may
`be performed on each ofthe differently-oriented versions of
`the captured image, and a particular version of the captured
`image maybeselected as the version of the captured image
`that represents the orientation ofthe electronic devicerelative
`to the user based on results of the face detection processing
`techniques. The orientation of the selected version of the
`captured imagerelative to the orientation of the originally
`captured image then may be determined, andthe orientation
`ofthe electronic devicerelative to the user may be determined
`based on the determinedorientationof the selected version of
`
`the captured imagerelativeto the orientationofthe originally
`captured image.Asa result ofperformingthe face processing
`techniques on the differently-oriented versions of the cap-
`tured image, a score may be assigned to each differently-
`oriented version of the captured imagereflecting the signifi-
`cance of the presence ofa face in the version of the captured
`image. The version of the captured image that was assigned
`the score that reflects the most significant presence of a face
`then maybeselected as the version ofthe captured imagethat
`represents the orientation of the electronic devicerelative to
`the user.
`
`[0010] After determining the orientation of the electronic
`devicerelative to the user and adjusting the orientation of the
`
`17
`
`17
`
`

`

`US 2010/0066763 Al
`
`Mar. 18, 2010
`
`at least one displayed elementrelative to the display compo-
`nent of the electronic device, an image of another user of the
`electronic device may be captured, and theorientation of the
`electronic device relative to the other user may be determined.
`Then, the orientation of the at least one displayed element
`relative to the display componentofthe electronic device may
`be determined based on the determined orientation of the
`electronic device relative to the other user.
`
`In somecases, it may be determinedthat the orien-
`[0011]
`tation ofthe electronic device relative to the user has changed
`from an earlier orientation ofthe electronic devicerelative to
`the user. In such cases, the orientation of the at least one
`displayed elementrelative to the display component may be
`changed in responseto the determination that the orientation
`of the electronic device relative to the user has changed from
`the earlier orientation of the electronic device relative to the
`
`user. In other cases, it may be determinedthat the orientation
`of the electronic device relative to the user has not changed
`from an earlier orientation ofthe electronic devicerelative to
`the user. In such cases, the orientation of the at least one
`displayed elementrelative to the display component may be
`maintained in responseto the determination that the orienta-
`tion of the electronic device relative to the user has not
`
`changed from the earlier orientation of the electronic device
`relative to the user.
`
`In some implementations, the orientation of the
`[0012]
`electronic device relative to the user may be determined
`before initially displaying an elementon the display compo-
`nent of the electronic device. In such implementations, an
`initial orientation for the elementrelative to the display com-
`ponent may be determined based on the determinedinitial
`orientation ofthe electronic devicerelative to the user, and the
`element may be displayed on the display componentof the
`electronic device with the initial orientation determined for
`the at least one element.
`
`In some implementations, the orientation of the
`[0013]
`electronic device relative to the user may be determined
`exclusively from images captured of the user without also
`relying on gravity to determine the orientation of the elec-
`tronic device relative to the user.
`
`Inother implementations,a first elementthatis to be
`[0014]
`orientedrelative to a user(e.g., text) and a second elementthat
`is to be oriented relative to gravity (e.g., a video) may be
`displayed on the display componentofthe electronic device.
`In such implementations, it may be determinedthatthefirst
`elementis not appropriately orientedrelative to the user based
`on the determinedorientationofthe electronic device relative
`to the user. Therefore, the orientation of the first element
`relative to the display componentofthe electronic device may
`be adjusted in response to determining that the at least one
`displayed elementis not appropriately orientedrelative to the
`user. In addition,the orientation of the electronic devicerela-
`tive to gravity may be determined. Based on the determined
`orientation of the electronic device relative to gravity, it then
`may be determined that the second elementis not appropri-
`ately orientedrelative to gravity. Therefore, the orientation of
`the second elementrelative to the display component of the
`electronic device may be adjusted in response to determining
`that the at least one displayed element is not appropriately
`oriented relative to gravity.
`[0015]
`In another general aspect, one or more elements are
`displayed on a display component of an electronic device.
`One or more images of a user of the electronic device are
`captured, and the orientation of the electronic devicerelative
`
`to the user is monitored based on the one or more captured
`imagesofthe user of the electronic device. The display ofat
`least one ofthe displayed elements on the display component
`then is controlled based on results of monitoring the orienta-
`tion ofthe electronic device relative to the user.
`
`Implementations may include one or more of the
`[0016]
`following features. For example, a changein the orientation
`of the electronic device relative to the user from an earlier
`
`orientation ofthe electronic device relative to the user may be
`detected. In response, an appropriate changeto the orienta-
`tion of the at least one displayed element relative to the
`display component may be determined based on the detected
`changein the orientation of the electronic device relative to
`the user. Thereafter, the change to the orientation of the at
`least one displayed elementrelative to the display component
`that was determined to be appropriate based on the detected
`changein the orientation of the electronic device relative to
`the user may be effected.
`[0017]
`For example, it may be determined thatthe orienta-
`tion of the electronic device relative to the user rotated sub-
`
`stantially positive ninety degrees from the earlier orientation
`of the electronic device relative to the user. Therefore, a
`determination may be madeto rotate the orientation of the at
`least one displayed element substantially negative ninety
`degrees relative to the display component in response to
`determining that the orientation of the electronic devicerela-
`tive to the user rotated substantially positive ninety degrees
`from the earlier orientation ofthe electronic devicerelative to
`the user. Thereafter, the orientation of the at least one dis-
`played element maybe rotated substantially negative ninety
`degreesrelative to the display component.
`[0018] Later, it may be determined that the orientation of
`the electronic device relative to the user rotated another sub-
`
`stantially positive ninety degrees. Therefore, the orientation
`of the at least one displayed element may be rotated another
`substantially ninety degreesrelative to the display component
`in response to determining that the orientation of the elec-
`tronic devicerelative to the user rotated another substantially
`positive ninety degrees.
`[0019]
`In somecases, it may be determinedthat the orien-
`tation of the electronic device relative to the user rotated
`
`substantially positive one-hundred and eighty degrees from
`the earlier orientation of the electronic device relative to the
`
`user. In such cases, it may be determinedto rotate the orien-
`tation ofthe at least one displayed element substantially nega-
`tive one-hundred and eighty degrees relative to the display
`componentin response to determiningthat the orientation of
`the electronic devicerelative to the user rotated substantially
`positive one-hundred and eighty degrees from the earlier
`orientation ofthe electronic device relative to the user. There-
`after, the orientation ofthe at least one displayed element may
`be rotated substantially negative one-hundred and eighty
`degreesrelative to the display component.
`[0020]
`In other cases, it may be determinedthat the orien-
`tation of the electronic device relative to the user rotated a
`number of degrees relative to the user that is in a range
`between substantially zero degrees and substantially ninety
`degrees in a positive direction from the earlier orientation of
`the electronic device relative to the user. In such cases, a
`determination may be madeto rotate the orientation of the at
`least one displayed element substantially the number of
`degrees in the range between substantially zero degrees and
`substantially ninety degrees in a negative direction relative to
`the display component in response to determining that the
`
`18
`
`18
`
`

`

`US 2010/0066763 Al
`
`Mar. 18, 2010
`
`orientation ofthe electronic device relative to the user rotated
`
`the numberofdegreesrelative to the user in the range between
`substantially zero degrees and substantially ninety degrees.
`Thereafter, the orientation of the at least one displayed ele-
`ment mayberotated substantially the numberof degrees in
`the range between substantially zero degrees and substan-
`tially ninety degrees in a negative direction relative to the
`display component.
`[0021]
`In still other cases, it may be determined that the
`orientation ofthe electronic device relative to the user has not
`
`changed from an earlier orientation of the electronic device
`relative to the user. In such cases, the orientation oftheat least
`one displayed elementrelative to the display component may
`be maintained based on determiningthat the orientation ofthe
`electronic devicerelative to the user has not changed from an
`earlier orientation ofthe electronic devicerelative to the user.
`
`In some examples, a changein the orientation of the
`[0022]
`electronic device relative to the user from an earlier orienta-
`
`tion of the clectronic device relative to the user may be
`detected. In response to detecting the change in orientation of
`the electronic device relative to the user, at least one displayed
`element may be hidden by removingthe at least one displayed
`elementfrom the display component. In addition, in response
`to detecting the changein orientation of the electronic device
`relative to the user, a first displayed element and a second
`displayed element may be maintained on the display compo-
`nent and a position and orientation of the first displayed
`element relative to the second displayed element may be
`changed on the display component. Further, in response to
`detecting the change in orientation of the electronic device
`relative to the user, text displayed on the display component
`may be formatted. For instance, a font size of the text dis-
`played on the display component may be changedor a subset
`of the text displayed on the display component may be main-
`tained while a remainderofthe text displayed on the display
`componentthat is not included in the subset may be removed.
`[0023]
`In yet another general aspect, a portable electronic
`device includes a display component that is configured to
`display output within a field of display and a camerathat is
`configured to capture images within a field of view of the
`camera. The camerais positioned on the portable electronic
`device such that thefield of view of the camera includes at
`
`least a portion of the field of display of the display compo-
`nent. The portable electronic device also includes a process-
`ing unit that is configured to process one or more images
`captured by the camera that include a user of the portable
`electronic device, monitor orientation of the portable elec-
`tronic device relative to the user of the portable electronic
`device based on the one or more captured imagesthat include
`the userof the portable electronic device, and control display
`on the display component of at least some of the output
`displayed on the display componentbased on results ofmoni-
`toring the orientation of the electronic device relative to the
`userof the portable electronic device.
`[0024]
`Implementations may include one or more of the
`following features. For example, in some implementations,
`the camera maybe a video camera. In other implementations,
`the camera maybea still camera. The processing unit may
`control orientation relative to the display componentofat
`least someofthe output displayed on the display component
`based on results of monitoring the orientation of the elec-
`tronic device relative to the user of the portable electronic
`device.
`
`In still another implementation, an image is cap-
`[0025]
`tured of a user of an electronic device that has a display
`component. Based on the captured imageofthe user of the
`electronic device, an initial orientation of the electronic
`device relative to the user of the electronic device is deter-
`
`mined. Before displaying an element on the display compo-
`nentofthe electronic device, an initial orientation for display-
`ing the element relative to the display component of the
`electronic device is determined based on the determinedini-
`tial orientation of the electronic device relative to the user of
`
`the electronic device. Thereafter, the elementis displayed on
`the display componentofthe electronic device with the deter-
`minedinitial orientation for displaying the elementrelative to
`the display component.
`[0026] The various aspects, implementations, and features
`disclosed may be implemented using, for example, one or
`more of a method, an apparatus, a system,tool, or processing
`device for performing a method, a program or other set of
`instructions, an apparatus that includes a program ora set of
`instructions, and a computer program embodiedina tangible,
`computer-readable medium. The tangible, computer-read-
`able medium mayinclude, for example, instructions, soft-
`ware, images, and otherdata.
`[0027] The details of one or more implementationsare set
`forth in the accompanying drawings and the description
`below. Other features will be apparent from the description
`and the drawings.
`
`DESCRIPTION OF DRAWINGS
`
`FIGS. 1A-1E areillustrations of a user interacting
`[0028]
`with an example of a portable electronic device that is con-
`figured to orient displayed elements in accordance with the
`portable electronic device’s orientation relative to a user.
`[0029]
`FIGS. 2A-2Dare illustrations of two users interact-
`ing with an example of a portable electronic device that is
`configured to orient displayed elements in accordance with
`the portable electronic device’s orientation relative to a user.
`[0030] FIG.3Aisa frontal view of an example of a portable
`electronic device that is configured to orient displayed ele-
`ments in accordance with the portable electronic device’s
`orientation relative to a user.
`
`FIG. 3B is a side view of an example of a portable
`[0031]
`electronic device that is configured to orient displayed ele-
`ments in accordance with the portable electronic device’s
`orientation relative to a user.
`
`FIG. 4 is a flowchart of an example of a process for
`[0032]
`controlling the orientation of an element on a display of an
`electronic device based on the orientation of the electronic
`device relative to a user of the electronic device.
`
`FIG. 5A is an illustration of examples of four types
`[0033]
`of Haar-like scalar features and their combination.
`
`FIG. 5B is an example of a detection cascade struc-
`[0034]
`ture obtained from the Adaboost algorithm.
`[0035]
`FIGS. 6A-6C are illustrations of a user interacting
`with an example of a portable electronic device that is con-
`figured to orient some displayed elements in accordance with
`the portable electronic device’s orientation relative to a user.
`[0036]
`FIGS. 7A-7B, 8A-8B, 9A-9B, 10A-10B, 11A-11B,
`12A-12B, and 13A-13B areillustrations of an example of a
`portable electronic device that is configured to control the
`arrangement of displayed items depending on whether the
`
`19
`
`19
`
`

`

`US 2010/0066763 Al
`
`Mar. 18, 2010
`
`display of the portable electronic device is in a landscapeor a
`portrait orientationrelative to a user ofthe portable electronic
`device.
`
`DETAILED DESCRIPTION
`
`relative to the user, the portable electronic device may arrange
`elements displayed on the portable electronic device in a
`portrait orientation. At some point in time, the user of the
`portable electronic device may rotate the portable electronic
`device such that the display of the portable electronic device
`is in a landscape orientation relative to the user, and the
`[0037] Many portable electronic devices including, for
`portable electronic device may determinethat the display of
`example, mobile telephones, portable digital assistants. por-
`the portable electronic device is in a landscape orientation
`table music players (e.g., MP3 players), handheld computers,
`relative to the user by capturing and processing one or more
`palmtop computers, and laptop computers include display
`images of the user of the portable electronic device. In
`components (e.g., liquid crystal displays (LCDs), organic
`response to determiningthat the display of the portable elec-
`light emitting diode (OLED)displays, plasma display panels
`tronic device is in a landscapeorientationrelativeto the user,
`(PDPs)) that display content and facilitate user interaction
`the portable electronic device may reorient the elements dis-
`with the devices. In addition, portable electronic devices gen-
`played on the portable electronic device so that they are
`erally tend to be easily maneuverable, handheld devices that
`arranged in a landscapeorientation.
`lend themselves to being held and/or positioned (whether
`[0041]
`It will be appreciated that the relationship between
`intentionally or unintentionally) in a variety of different ori-
`entationsrelative to a user.
`the orientation of a user ofa portable electronic device and the
`orientation of the portable electronic device (or the orienta-
`[0038]
`For example, while manipulating a portable elec-
`tion of the display of the portable electronic device) may be
`tronic device, a user (intentionally or unintentionally) may
`described from the perspective of the user of the portable
`rotate his/her hand resulting in a change of the orientation of
`electronic device or from the perspective ofthe portable elec-
`the portable electronic device (or the display of the portable
`tronic device (or the display ofthe portable electronic device),
`electronic device) relative to the user. Similarly, when a user
`but that irrespective of the perspective from whichthe rela-
`retrieves a portable media device from a pocket or other
`tionship is described,
`the relationship will be the same.
`resting place, the user inadvertently may grasp the portable
`Accordingly, the phrases “orientation of the portable elec-
`device such that the conventional orientation of the portable
`tronic device (or orientation of the display of the portable
`electronic device (or the conventional orientation ofthe dis-
`electronic device) relative to the user”and “orientation of the
`play of the portable electronic device) is upside down (or
`userrelative to the portable electronic device (orthe display
`otherwise misaligned)relative to the user, potentially imped-
`of the portable electronic device)” (and similar such phrases)
`ing the user’s ability to view and easily make senseofele-
`may be used interchangeably throughoutthis disclosure.
`ments displayed on the portable electronic device’s display.
`[0042]
`In order to determine the orientation of a portable
`Likewise, when a user glancesat a portable electronic device
`electronic device (or the display of the portable electronic
`that is lying in a rest position, for example, in response to an
`device) relative to a user, a camera that is built into (or oth-
`alert of an incoming e-mail, phone call, or instant message,
`erwise accessible to) the portable electronic device may take
`the conventional orientation of the portable electronic device
`one or morestill or video imagesofthe user, and the images
`(or the conventionalorientation of the display of the portable
`maybe processed to determinethe orientation of the portable
`electronic device) may be misaligned relative to the user
`electronic device (or the orientation of the display of the
`potentially making it difficult for the user to perceive ele-
`portable electronic device) relative to the user. For example,
`ments presented on the portable electronic device’s display.
`the camera may take one or morestill or video imagesof the
`[0039]
`To improve a user’s ability to view, perceive, and
`user’s face, and the orientation of the portable electronic
`interact with elements displayed on the display of a portable
`device (orthe orientation of the display ofthe portable elec-
`electronic device, the orientation of the portable electronic
`tronic device)relative to the user may be determined based on
`device (or the orientation of the display of the portable elec-
`the orientation of the user’s face in the still or video image(s)
`tronic device) relative to the user may be determined by
`captured by the camera. As a user interacts with the portable
`capturing and processing a photographor other imageof the
`electronic device, the portable electronic device’s orientation
`user, and elements may be displayed on the portable elec-
`(or the orientation ofthe portable electronic device’s display)
`tronic device’s display in a mannerthat is appropriate given
`relative to the user may be monitored by taking and process-
`the determined orientation of the portable electronic device
`ing one or more additional still or video images on a con-
`(or the display ofthe portable electronic device) relative to the
`tinual, periodic, random,or event-triggered(e.g., in response
`user. In addition, as the orientation of the portable electronic
`to detection ofmotionofthe portable electronic device) basis.
`device (or the orientation of the display of the portable elec-
`Taking and processing one or morestill or video images of a
`tronic device) changesrelative to the user, elements displayed
`user of a portable electronic device may beaparticularly
`on the portable electronic device’s display may be reoriented
`useful technique for determining the orientation of the por-
`to accommodate the new orientation ofthe portable electronic
`table electronic device (or the display of the portable elec-
`device (or the display of the portable electronic