Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`
`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`
`META PLATFORMS, INC.,
`Petitioner
`
`v.
`
`MULLEN INDUSTRIES LLC,
`Patent Owner.
`
`
`Case IPR2025-00744
`U.S. Patent No. 12,019,791 B2
`Issue Date: June 25, 2024
`
`Title: AUGMENTED REALITY VIDEO GAME SYSTEMS
`
`
`PETITION FOR INTER PARTES REVIEW
`OF U.S. PATENT NO. 12,019,791 B2
`
`
`
`
`
`U.S. Patent No. 12,019,791 B2
`
`
`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`
`META PLATFORMS, INC.,
`Petitioner
`
`v.
`
`MULLEN INDUSTRIES LLC,
`Patent Owner.
`
`
`Case IPR2025-00744
`U.S. Patent No. 12,019,791 B2
`Issue Date: June 25, 2024
`
`Title: AUGMENTED REALITY VIDEO GAME SYSTEMS
`
`
`PETITION FOR INTER PARTES REVIEW
`OF U.S. PATENT NO. 12,019,791 B2
`
`
`
`
`
`
`Table of Contents
`
`
`Page
`
`
`
`I.
`
`MANDATORY NOTICES UNDER §42.8(A)(1) ........................................ 1
`A.
`Real Party-In-Interest under §42.8.(b)(1) ........................................... 1
`B.
`Related Matters under §42.8(b)(2) ..................................................... 1
`C.
`Lead and Back-Up Counsel under §42.8(b)(3) ................................... 2
`D.
`Service Information ........................................................................... 2
`FEE PAYMENT .......................................................................................... 3
`II.
`III. REQUIREMENTS UNDER §§ 42.104 AND 42.108 AND
`CONSIDERATIONS UNDER §§ 314(A) AND 325(D) .............................. 3
`A. Grounds for Standing ......................................................................... 3
`B.
`Identification of Challenge and Statement of Precise Relief
`Requested .......................................................................................... 3
`Considerations Under §§ 314(a) and 325(d) ....................................... 3
`C.
`IV. OVERVIEW OF THE PATENT ................................................................. 4
`A.
`Level of Ordinary Skill ...................................................................... 4
`B.
`Specification Overview ...................................................................... 5
`CLAIM CONSTRUCTION ......................................................................... 7
`V.
`VI. THE CHALLENGED CLAIMS ARE UNPATENTABLE .......................... 7
`A. Overview of Grounds ......................................................................... 7
`B.
`Prior Art Status of Relied-Upon References ....................................... 9
`C.
`Ground 1: Claims 21, 23, 37, 41, and 42 are Obvious Over
`Fager in view of Martins .................................................................. 10
`Independent Claim 21: “A system comprising:”
`
`(21[pre])................................................................................. 10
`
`
`
`
`
`-i-
`
`
`
`
`
`
`
`(a)
`
`(b)
`
`Table of Contents
`(continued)
`
`Page
`
`(ii)
`
`(iii)
`
`“a head-mounted display comprising a camera for
`obtaining images in view of said head-mounted
`display and utilizing said images to detect a
`surface of a physical object included in said
`images, wherein said head-mounted display is
`operable to display a virtual object associated with
`an application; and” (21[a]) ........................................ 11
`(i)
`“a head-mounted display comprising a
`camera for obtaining images in view of said
`head-mounted display” ...................................... 11
`“utilizing said images to detect a surface of a
`physical object included in said images” ............ 18
`“wherein said head-mounted display is
`operable to display a virtual object
`associated with an application” .......................... 24
`“a hand-held controller for providing a first three-
`dimensional control signal to move said virtual
`object and a second three-dimensional control
`signal to change the movement of said virtual
`object from the movement provided from said first
`three-dimensional control signal, wherein said first
`three-dimensional control signal moves said virtual
`object three-dimensionally over at least three axes
`of movement in a first manner and said second
`three-dimensional control signal moves said virtual
`object three-dimensionally over at least three axes
`of movement in a second manner.” (21[b]) ................. 27
`(i)
`“a hand-held controller for providing a first
`three-dimensional control signal to move
`said virtual object” ............................................. 27
`
`
`
`
`
`-ii-
`
`
`
`
`
`Table of Contents
`(continued)
`
`Page
`
`-iii-
`
`
`
`(ii)
`
`(iii)
`
`(iv)
`
`“[a hand-held controller for providing]…a
`second three-dimensional control signal to
`change the movement of said virtual object
`from the movement provided from said first
`three-dimensional control signal” ...................... 30
`“wherein said first three-dimensional control
`signal moves said virtual object three-
`dimensionally over at least three axes of
`movement in a first manner” .............................. 31
`“said second three-dimensional control
`signal moves said virtual object three-
`dimensionally over at least three axes of
`movement in a second manner” ......................... 32
`Claim 23: “The system of claim 21, wherein said
`application is a video game having a selectable playing
`area.” ..................................................................................... 32
`Claim 37: “The system of claim 21, wherein a physical
`object type of said physical object is operable to be
`determined from several physical object types, a first
`type of said several physical object types is a floor and a
`second type of said several types of physical object types
`is a wall.” ............................................................................... 35
`Claim 41: “The system of claim 21, wherein at least said
`second three-dimensional control signal moves said
`virtual object causing said virtual object to interact with
`said detected surface.” ........................................................... 38
`Claim 42: “The system of claim 21, wherein said camera
`for obtaining images in view of said head-mounted
`display comprises a video camera for obtaining video in a
`view of said head-mounted display.” ..................................... 40
`D. Ground 2: Claim 31 is Obvious Over Ground 1 Prior Art in
`Further View of Maguire ................................................................. 41
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Table of Contents
`(continued)
`
`Page
`
`
`
`
`
`Claim 31: “The system of claim 21, further comprising a
`second camera and a third camera and the views of said
`second and third camera overlap.” ......................................... 41
`VII. CONCLUSION ......................................................................................... 46
`CERTIFICATE OF SERVICE ............................................................................. 48
`
`
`
`
`
`
`-iv-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`List of Exhibits
`
`
`
`
`
`
`
`
`
`Exhibit
`Description of Document
`No.
`1001 U.S. Patent No. 12,019,791 B2 to Jeffrey David Mullen (filed Jan. 4,
`2022, issued Jun. 25, 2024) (“’791” or “’791 patent”)
`1002 Declaration of Jeremy Cooperstock, Ph.D. (“Cooperstock”)
`1003 U.S. Patent App. Pub. No. 2004/0104934 A1 to Jan G. Fager (filed Feb.
`7, 2002; published Jun. 3, 2004) (“Fager”)
`1004 U.S. Patent App. Pub. No. 2003/0179218 A1 to Fernando C. M.
`Martins (filed Mar. 22, 2002; published Sep. 25, 2003) (“Martins”)
`1005 U.S. Patent No. 6,411,266 B1 to Francis J. Maguire, Jr. (filed Nov. 17,
`1995; issued Jun. 25, 2002) (“Maguire”)
`1006 Masayuki Kanbara et al., “A stereoscopic video see-through augmented
`reality system based on real-time vision-based registration,” in
`Proceedings IEEE Virtual Reality 2000 (2000) (“Kanbara”)
`1007
`’791 Patent File History
`1008 Declaration of Ingrid Hsieh-Yee, Ph.D. (“Hsieh-Yee”)
`1009 Amended Complaint in Mullen Industries LLC v. Meta Platforms, Inc.,
`Case No. 1:24-cv-00354-DAE (W.D. Tex.) (July 17, 2024)
`
`-i-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`I. MANDATORY NOTICES UNDER §42.8(A)(1)
`A. Real Party-In-Interest under §42.8.(b)(1)
`Meta Platforms, Inc. is the real party-in-interest to this IPR petition.
`
`B. Related Matters under §42.8(b)(2)
`The ’791 patent is the subject of pending litigation involving Petitioner:
`
`Mullen Industries LLC v. Meta Platforms, Inc., Case No. 1:24-cv-00354-DAE (W.D.
`
`Tex.). Petitioner was served, via the filing of an amended complaint adding the ’791
`
`patent to the pending litigation, on July 17, 2024. (EX1009.)
`
`Petitioner is filing IPR petitions against twelve related patents asserted by
`
`Patent Owner in the pending litigation involving Petitioner:
`
`U.S. Patent No.
`8,585,476 B2
`9,744,448 B2
`10,179,277 B2
`10,828,559 B2
`10,967,270 B2
`11,033,821 B2
`11,376,493 B2
`11,904,243 B2
`11,947,716 B2
`12,019,791 B2
`9,662,582 B2
`10,974,151 B2
`
`IPR Case
`IPR2025-00737
`IPR2025-00738
`IPR2025-00739
`IPR2025-00740
`IPR2025-00702
`IPR2025-00703
`IPR2025-00741
`IPR2025-00742
`IPR2025-00743
`IPR2025-00744
`IPR2025-00745
`IPR2025-00746
`
`-1-
`
`
`
`
`
`
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`C. Lead and Back-Up Counsel under §42.8(b)(3)
`Petitioner provides the following designation of counsel.
`
`LEAD COUNSEL
`
`BACK-UP COUNSEL
`
`Heidi L. Keefe (Reg. No. 40,673)
`hkeefe@cooley.com
`
`COOLEY LLP
`ATTN: Patent Group
`1299 Pennsylvania Ave. NW, Suite 700
`Washington, DC 20004
`Tel: (650) 843-5001
`Fax: (650) 849-7400
`
`Phillip E. Morton (Reg. No. 57,835)
`pmorton@cooley.com
`
`Andrew C. Mace (Reg. No. 63,342)
`amace@cooley.com
`
`Mark R. Weinstein (Admission pro hac
`vice to be requested)
`mweinstein@cooley.com
`
`Lowell D. Mead (Admission pro hac vice
`to be requested)
`lmead@cooley.com
`
`Patrick Lauppe (Admission pro hac vice to
`be requested)
`plauppe@cooley.com
`
`COOLEY LLP
`ATTN: Patent Group
`1299 Pennsylvania Ave. NW, Suite 700
`Washington D.C. 20004
`
`
`
`Service Information
`D.
`This Petition is being served by Federal Express to the attorney of record for
`
`the ’791 patent, 32733 - JEFFREY D. MULLEN, 2212 Hassinger Lane, Glenshaw,
`
`PA 15116. Petitioner consents to electronic service at the addresses provided above
`
`for lead and back-up counsel.
`
`
`
`
`
`-2-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`FEE PAYMENT
`II.
`Petitioner requests review of six claims, with a $51,875 payment.
`
`42.104 AND
`§§
`III. REQUIREMENTS UNDER
`CONSIDERATIONS UNDER §§ 314(A) AND 325(D)
`A. Grounds for Standing
`Petitioner certifies that the ’791 patent is available for IPR and that Petitioner
`
`42.108 AND
`
`is not barred or otherwise estopped.
`
`B.
`
`Identification of Challenge and Statement of Precise Relief
`Requested
`Petitioner requests IPR institution based on:
`
`Ground
`1
`
`2
`
`Basis for Challenge under §103
`Claims
`21, 23, 37, 41, 42 Fager in view of Martins
`Ground 1 Prior Art + Maguire
`31
`
`Submitted with this Petition is the Declaration of Jeremy Cooperstock, Ph.D.
`
`(EX1002) (“Cooperstock”), a qualified technical expert. (EX1002, ¶¶1-18, App. A.)
`
`C. Considerations Under §§ 314(a) and 325(d)
`§314(a): The General Plastic factors are not relevant as this is the first and
`
`only IPR petition filed by Petitioner with respect to the ’791 patent.
`
`Nor do the Fintiv factors support discretionary denial under §314(a). The
`
`pending litigation against Petitioner is in relatively early stage, with little substantive
`
`discovery having taken place to date and claim construction proceedings ongoing.
`
`Trial is scheduled to start October 5, 2026—over eighteen months from now—and
`
`-3-
`
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`a motion to transfer the case by Petitioner to the Northern District of California is
`
`currently pending, which if granted would likely result in an even later trial date.
`
`Petitioner also intends to move to stay the litigation pending resolution of the IPRs.
`
`Additionally, to allay concerns over duplication of efforts, Petitioner represents that,
`
`in the event of institution, it will not pursue in litigation any invalidity defense that
`
`was or could have been raised in IPR.
`
`§325(d): This Petition does not raise a § 325(d) issue because none of the
`
`prior art relied upon was cited or otherwise identified during prosecution.
`
`IV. OVERVIEW OF THE PATENT
`A. Level of Ordinary Skill
`A person of ordinary skill would have possessed a bachelor’s degree in
`
`electrical engineering, computer science, or similar field, with two years combined
`
`experience in designing and/or developing interactive location-based computer
`
`systems/software, such as video games or other simulations incorporating location
`
`information (such as GPS information associated with a user’s physical location),
`
`and in designing and/or developing computer systems/software involving graphical
`
`virtual and/or augmented reality. A person could also have qualified as a person of
`
`ordinary skill in the art with some combination of (1) more formal education (such
`
`as a master’s of science degree) and less technical experience, or (2) less formal
`
`education and more technical or professional experience. (EX1002, ¶¶21-25.)
`
`
`
`
`
`-4-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`Specification Overview
`B.
`The ’791 patent states that it “relates to augmented reality.” (’791, 1:19.) The
`
`patent states that it provides “[a]ugmented reality video game systems” where
`
`“[v]ideo game indicia may be perceived to be located in a user’s physical
`
`environment through virtual indicia that are placed on a head-mounted display.”
`
`(’791, 1:23-26.)
`
`Figure 3, reproduced below, “is an illustration of an augmented reality video
`
`game system and associated video game played in an environment with a physical
`
`obstacle constructed in accordance with the principles of the present invention”:
`
`(’791, Fig. 3, 4:48-51.) Referring to Figure 3, the ’791 patent states that “the game
`
`may be played in [the user’s] actual environment.” (’791, 7:35-37.) “User 301 may
`
`
`
`control virtual object 351 and user 302 may control virtual object 352.” (’791, 7:37-
`
`
`
`
`-5-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`39.) Physical surroundings—such as object 399—can be detected and incorporated
`
`into game play:
`
`Physical surroundings (e.g., object 399) may be detected (either prior
`to a game during setup and/or during a game) and virtual objects may
`interact with such physical objects. For example, if a user places object
`399 onto a table, plane 351 may be animated to crash into object 399
`instead of animated to pass through object 399.
`
`(’791, 7:39-44.)
`
`For example, a user “may be perceived as holding a virtual object such as a
`
`virtual basketball” and may “shoot the virtual basketball towards a virtual basketball
`
`hoop the user installed in his/her living room.” (’791, 14:45-49.) “[T]he virtual
`
`basketball may react with a user’s physical environment,” such as by appearing to
`
`bounce off the user’s ceiling:
`
`For example, a user’s head-mounted display may include cameras that
`are operable to view more of a user’s environment than the user can
`perceive. The augmented reality system may determine, for example,
`the location of walls, floors, and other objects. In or near real time, for
`example, the augmented system may determine that a virtual basketball
`that was thrown by a user at a trajectory has hit a physical surface (e.g.,
`a ceiling) and the augmented system may utilize this information to
`change the course of the virtual ball (e.g., by causing the ball to be
`perceived by the user to bounce off the user’s ceiling).
`
`(’791, 17:49-61.)
`
`
`
`
`
`-6-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`V. CLAIM CONSTRUCTION
`Petitioner does not believe express claim construction is necessary at this
`
`time. The prior art cited herein, as demonstrated below, renders the challenged
`
`claims obvious under any reasonable construction. Petitioner therefore respectfully
`
`submits that, for purposes of this IPR, express constructions are not required.
`
`VI. THE CHALLENGED CLAIMS ARE UNPATENTABLE
`A. Overview of Grounds
`The challenged claims attempt to lay claim to techniques for using a head-
`
`mounted display device and hand-held controller to display and interact with a
`
`virtual object and for using a camera to detect a user’s physical surroundings—such
`
`as for a game of virtual basketball. (’791, e.g., 17:45-61.) But these techniques were
`
`well-known before the ’791 patent.
`
`The Petition primarily relies on Fager (EX1003), a strong reference that
`
`discloses the majority of the claim limitations and that pre-dates the ’791 patent by
`
`over five years. Like the ’791 patent, Fager discloses using a head-mounted display
`
`device and handheld controller for displaying and interacting with virtual objects
`
`mixed with the user’s physical environment. (Cooperstock, ¶¶54-56.) Fager even
`
`specifically describes a game of virtual tennis that is strikingly similar to the game
`
`of virtual basketball described in the ’791 patent. Figure 5 of Fager illustrates:
`
`
`
`
`
`-7-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`
`
`
`(Fager, Fig. 5 (highlighting added).) As shown in the above, Fager’s virtual tennis
`
`involves a user wearing a head-mounted display 12 and holding a controller 26.
`
`With the head-mounted display, the user sees their physical environment combined
`
`with virtual objects including tennis ball 29, tennis racket 27, tennis net 24, and court
`
`lines 25. With handheld controller 26, the user can swing virtual racket 27 to hit the
`
`virtual tennis ball 29, which can then interact with the user’s physical
`
`surroundings—for example, by bouncing off the floor or lamp 30. (Cooperstock,
`
`¶¶56-57 (citing Fager, ¶¶0079, 0081-0082, 0085).)
`
`The Petition also cites Martins (EX1004) in Ground 1 for its disclosures
`
`regarding using a camera to detect physical surfaces, such as the surfaces of floors,
`
`
`
`
`
`-8-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`walls, and ceilings, and using that information in providing an augmented reality
`
`environment to a user. (Cooperstock, ¶¶60-61.) As discussed below, Fager teaches
`
`both a head-mounted display having a camera as well as detecting physical surfaces
`
`in the user’s physical environment. The Petition cites Martins to show that it would
`
`have been obvious to use Fager’s head-mounted camera to detect the physical
`
`surfaces in the user’s physical environment.
`
`Maguire (EX1004) is cited in Ground 2 in connection with claim 31 for
`
`disclosure of a head-mounted display comprising multiple cameras.
`
`Prior Art Status of Relied-Upon References
`B.
`Pre-AIA law applies to the challenged claims, and each reference cited in the
`
`grounds listed in Part III.B qualifies as prior art to the ’791 patent. Fager, Martins,
`
`and Maguire all qualify as prior art under § 102(b) because they issued more than
`
`one year before the priority date.1
`
`
`1 The ’791 patent claims priority to a provisional application (no. 60/979,379) filed
`
`October 11, 2007. For purposes of this Petition, Petitioner has assumed a priority
`
`date of October 11, 2007. Petitioner at this time expresses no opinion on whether
`
`the ’791 patent is actually entitled to the benefit of the provisional application filing
`
`date. This issue is not relevant here because all of the prior art references relied upon
`
`are prior art under § 102(b). In the event entitlement to the provisional filing date or
`
`
`
`
`
`
`
`-9-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`C. Ground 1: Claims 21, 23, 37, 41, and 42 are Obvious Over Fager
`in view of Martins
`Independent Claim 21: “A system comprising:” (21[pre])
`
`The preamble of claim 21 recites “[a] system comprising.” As detailed
`
`below, Fager discloses an “augmented reality” system in which images of the user’s
`
`physical environment captured by a camera of a head-mounted display are combined
`
`with virtual images and presented together to the user via the head-mounted display.
`
`(Cooperstock, ¶68 (citing Fager, e.g., ¶¶0051-0052, 0069, 0084, 0092, 0094).) Fager
`
`refers to “augmented reality” using the term “milieu”:
`
`By the term “milieu” (corresponding to “augmented reality” in the
`English language), the milieu experienced by a creature is intended,
`which milieu by providing fictitious creatures, fictitious objects and/or
`fictitious phenomena to the real environment means that the creature
`experiences that the milieu has other properties than what is the case
`for the real environment without these apparent characteristics.
`
`(Fager, ¶0009; see also id., ¶0001 (“The present invention relates to a device and a
`
`method for creating a milieu for a creature in a real environment to the creature, the
`
`milieu comprising both at least one part of the real environment and at least one
`
`fictitious phenomenon, the properties of said milieu experienced by the creature
`
`
`another earlier priority date may later become relevant, Petitioner reserves the right
`
`to respond at that time.
`
`
`
`
`
`-10-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`depending on the position and/or orientation of the creature, or a part thereof, relative
`
`to the real environment….”).)2 Fager’s system supports multiple users, where each
`
`user has their own head-mounted display. (Fager, e.g., ¶0086 (“A real opponent—
`
`player—may be equipped with a similar device according to the invention…”).)
`
`
`
`As explained below, the combination of Fager and Martins renders the
`
`claimed system obvious.
`
`(a)
`
`“a head-mounted display comprising a camera for
`obtaining images in view of said head-mounted display
`and utilizing said images to detect a surface of a physical
`object included in said images, wherein said head-
`mounted display is operable to display a virtual object
`associated with an application; and” (21[a])
`The Petition will address this limitation in parts because of its length. As
`
`explained below, this limitation is disclosed by and obvious over Fager in view of
`
`Martins.
`
`(i)
`
`“a head-mounted display comprising a camera
`for obtaining images in view of said head-
`mounted display”
`Fager discloses and renders obvious “a head-mounted display comprising
`
`a camera for obtaining images in view of said head-mounted display.”
`
`(Cooperstock, ¶¶71-80.)
`
`
`2 All underlining added to quoted material unless noted.
`
`-11-
`
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`Starting first with the “head-mounted display,” Fager teaches a head-
`
`mounted device (referred to as “stimulation generating member 14”) that includes a
`
`“head-mounted display” comprising one or more of what Fager refers to as “means
`
`for transferring visual impressions 12.” (Fager, e.g., ¶¶0051-0052; Cooperstock,
`
`¶72.) Figure 1 of Fager, reproduced with highlighting below, illustrates:
`
`
`(Fager, Fig. 1 (highlighting added); see also id., ¶¶0051 (“stimulation generating
`
`
`
`member 14”; “means included in a transferring unit, for transferring visual
`
`information 12 to the creature”), 0052 (“…calculate the information which is to be
`
`communicated to the means for transferring visual impressions 12 of the transferring
`
`unit, so that generated pictures of the fictitious phenomena may be incorporated with
`
`great accuracy in the real picture information of the room 1, the milieu being realised
`
`
`
`
`
`-12-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`for the person”), 0077 (“Said device includes…a transferring unit comprising two
`
`means 12 for transferring information to the eyesight, one for each eye….”).)3
`
`
`
`Fager makes clear that the “means for transferring visual impressions” can
`
`take the form of a “head-mounted display,” such as a device comprising separate
`
`LCD displays placed over each eye of the wearer. (Fager, ¶0069 (“[A] unit 12 for
`
`transferring stimuli is further included, which comprises two stimulation means, one
`
`for each eye,…each a means in the form of a LCD-display, which has the capability
`
`to be either transparent or imperious [sic; impervious] to light as regards each pixel
`
`in the resolution of the display.”); see also id. ¶0084 (“a transparent LCD-display
`
`located in the field of vision is used as a part of the transferring means to stimulate
`
`the eyesight”); Cooperstock, ¶73.)
`
`
`
`Turning to a “a head-mounted display comprising a camera,” Fager further
`
`teaches that the “means for transferring visual impressions 12” (“head-mounted
`
`display”) can comprise a “camera”—which Fager refers to as “obtaining means for
`
`obtaining picture information.” (Fager, e.g., ¶¶0052, 0094; Cooperstock, ¶74.) For
`
`example, referring to an augmented reality “tennis example” illustrated in Figure 5
`
`(reproduced below), Fager explains how the “means 12 of the transferring unit for
`
`
`3 Fager uses the term “creature” to refer to human beings and animals. (Fager,
`
`¶0004.)
`
`
`
`
`
`-13-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`transferring visual information [‘head-mounted display’] comprises an obtaining
`
`means for obtaining picture information”:
`
`
`(Fager, Fig. 5 (highlighting added).) Fager explains:
`
`
`
`In a tennis example where the means 12 of the transferring unit for
`transferring visual information comprises an obtaining means for
`obtaining picture information, a means which blocks the light of the
`environment, and a transferring means for visual information for
`directly projection of pictures on the retina, and where the tool
`component 26 is provided with control buttons, a signal is emitted from
`a control button to the control- and calculating unit 17 so that the milieu
`
`
`
`
`
`-14-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`which the player experiences is similar to play on a clay court.
`
`(Fager, ¶0094; see also id., ¶0009 (explaining that term “milieu” “correspond[s] to
`
`‘augmented reality’ in the English language”).)
`
`Fager makes clear that the “obtaining means for obtaining picture
`
`information” can take the form of a “camera.” (Cooperstock, ¶75.) For example,
`
`referring to the “tennis example,” Fager describes using the “obtaining means” to
`
`obtain picture information of the user’s physical surroundings so that “[a] fictitious
`
`tennis ball in the realised milieu may be seen moving in a potential field in the real
`
`environment”:
`
`In a more sophisticated embodiment of the previous example, the
`transferring means for transferring stimuli to the visual organs have an
`obtaining means for picture information. Furthermore, the path for the
`light to incident directly towards the eye is blocked by a blocking means
`in the form of an impermeable shield. The information obtained from
`the obtaining means is superimposed with the picture information from
`electrical potentials simulated in the model of the environment, so that
`information about the electric properties of the environment is
`represented by different semitransparent tints seamlessly adapted to the
`real picture. The transferring unit then generates the whole picture by
`means of a transferring means for projection of the picture on the retina
`of the eye. In this case the fictitious tennis ball in the realised milieu
`may be seen moving in a potential field in the real environment.
`
`(Fager, ¶0092.) A person of ordinary skill in the art would have understood that
`
`
`
`
`
`-15-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`Fager in the passage above is describing camera functionality and, moreover, that a
`
`camera would have been an obvious choice for obtaining “picture information” of
`
`the user’s surroundings. (Cooperstock, ¶76.) In fact, Fager describes an
`
`embodiment where an “obtaining means” “include[s] IR [infrared] cameras for
`
`obtaining information about the external environment in the night and/or when fog
`
`is present.” (Fager, ¶0074.) A person of ordinary skill in the art would have
`
`therefore readily appreciated and found it obvious based on Fager’s teachings for the
`
`“obtaining means” of Fager’s head-mounted display to comprise a camera
`
`(Cooperstock, ¶76)—especially since “[c]ombining two embodiments disclosed
`
`adjacent to each other in a prior art patent does not require a leap of inventiveness.”
`
`Boston Scientific Scimed, Inc. v. Cordis Corp., 554 F.3d 982, 991 (Fed. Cir. 2009).
`
`The fact that Fager discloses and renders obvious a head-mounted display
`
`comprising a camera is further confirmed by Fager’s discussion in the passage above
`
`of a head-mounted display device where “the path for the light to incident directly
`
`towards the eye is blocked by a blocking means in the form of an impermeable
`
`shield.” (Cooperstock, ¶77 (quoting Fager, ¶0092).) A person of ordinary skill in
`
`the art would have recognized that Fager is describing what is known as a video see-
`
`through head-mounted display. (Id.) There are (and were) two types of head-
`
`mounted displays for showing a user images in which the real and virtual
`
`environments are merged: (1) optical see-through; and (2) video see-through.
`
`
`
`
`
`-16-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`(Cooperstock, ¶77 (citing EX1006, p.0010).) With optical see-through, the user can
`
`see their real environment through a
`
`half-transparent mirror. (Id.) With
`
`video see-through, the user’s direct
`
`view of their real environment is
`
`blocked—just as Fager describes—
`
`and they instead see image sequences
`
`of the real environment captured by a
`
`camera of the head-mounted display. (Id.) An example of a video see-through head-
`
`mounted display is shown at right, where “two small identical CCD cameras
`
`(Toshiba IK-UM42) are mounted on a HMD (Olympus Media Mask)[.]” (Id.
`
`(quoting EX1006, p.0014).)
`
`Therefore, a person of ordinary skill in the art would have understood and
`
`found it obvious that Fager’s head-mounted display “compris[es] a camera” as
`
`claimed. (Cooperstock, ¶78.)
`
`Fager also teaches that the camera of the head-mounted display would be “for
`
`obtaining images in view of said head-mounted display.” (Cooperstock, ¶79.)
`
`As explained above, Fager teaches how the “obtaining means” (“camera”) is used
`
`to obtain picture information of the physical environment in front of the user (“in
`
`view of said head-mounted display”) so that images combining virtual objects and
`
`
`
`
`
`-17-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`the user’s physical environment can be displayed to the user. (Fager, ¶0092 (“…the
`
`fictitious tennis ball in the realised milieu may be seen moving in a potential field in
`
`the real environment.”); see also id., ¶0052 (“…calculate the information which is
`
`to be communicated to the means for transferring visual impressions 12 of the
`
`transferring unit, so that generated pictures of the fictitious phenomena may be
`
`incorporated with great accuracy in the real picture information of the room 1, the
`
`milieu being realised for the person”).)
`
`
`
`Therefore, Fager discloses and renders obvious “a head-mounted display
`
`comprising a camera for obtaining images in view of said head-mounted
`
`display” as claimed. (Cooperstock, ¶80.)
`
`(ii)
`
`“utilizing said images to detect a surface of a
`physical object included in said images”
`This portion of claim 21[a] is obvious over Fager in view of Martins.
`
`(Cooperstock, ¶¶81-89.)
`
`Fager generally describes detecting and modeling physical objects in the real
`
`world and using that information to determine how the physical objects (such as a
`
`lamp or floor surface) affect virtual game play. (Cooperstock, ¶82.) For example,
`
`referring to Figure 5 and the augmented reality tennis example, Fager explains that
`
`a real-world suspended lamp 30 can be detected and used to affect the behavior of
`
`virtual tennis ball 29:
`
`
`
`
`
`-18-
`
`
`
`
`
`Petition for Inter Partes Review of
`U.S. Patent No. 12,019,791 B2
`
`
`
`
`
`In this example, a physical object in the form of a suspended lamp 30
`is present in the environment, but the object could be any occurring real
`object. This object is modelled in the control- and calculating unit 17.
`Furthermore, in the control- and calculating unit 17 there is stored
`information about the fictitious ball 29, a fictitious tennis net 24 and
`fictitious tennis lines 25, calculation models for how a tennis ball
`physically acts and other rules for how the game tennis is performed in
`the
Accessing this document will incur an additional charge of $.
After purchase, you can access this document again without charge.
Accept $ Charge
Still Working On It
This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.
Give it another minute or two to complete, and then try the refresh button.
A few More Minutes ... Still Working
It can take up to 5 minutes for us to download a document if the court servers are running slowly.
Thank you for your continued patience.
This document could not be displayed.
We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.
Your account does not support viewing this document.
You need a Paid Account to view this document. Click here to change your account type.
Your account does not support viewing this document.
Set your membership
status to view this document.
With a Docket Alarm membership, you'll
get a whole lot more, including:
- Up-to-date information for this case.
- Email alerts whenever there is an update.
- Full text search for other cases.
- Get email alerts whenever a new case matches your search.
One Moment Please
The filing “” is large (MB) and is being downloaded.
Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.
Your document is on its way!
If you do not receive the document in five minutes, contact support at support@docketalarm.com.
Sealed Document
We are unable to display this document, it may be under a court ordered seal.
If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.
Access Government Site
