- . _
`0
`.....
`CJI
`0\
`.....
`
`-
`
`.....
`
`0
`
`......
`......
`N
`c:::
`a...
`fl)
`' ~ }f,a request for filling a PROVISIONAL APPLICATION under 37 CFR 1.53(b )(2).
`~ ""'
`
`•
`
`Patent and T
`•
`PROVISIONAL APPLICATION COVER SHEET
`
`PTO/SB/16 (6-95)
`roved for use through 04/11/98. 0MB 0651-0037
`k Office: U.S. DEPARTMENT OF COMMERCE
`(Reproduction)
`
`A
`
`!
`
`I Type a plus sign (+) I
`
`inside this box ➔
`
`+
`
`/ lfb Dll
`
`I Docket Number I 8418/1
`
`LASTNAME
`JASZLICS
`JASZLICS
`
`FIRSTNAME
`Ivan
`Sheila
`
`MIDDLE INITIAL
`J.
`L.
`
`RESIDENCE {CTIY AND EITHER STATE OR FOREIGN COUNTRY)
`Golden, Colorado, U.S.A.
`Golden, Colorado, U.S.A.
`
`INVENTOR(s)/APPLICANT(s)
`
`SYSTEM FOR COMBINING VIRTUAL IMAGES WITH REAL-WORLD SCENES
`
`TITLE OF THE INVENTION (280 characters max)
`
`.
`
`CORRESPONDENCE ADDRESS
`Dorr, Carson, Sloan & Birney, P.C.
`3010 E. 6th Avenue
`Denver
`
`;i
`
`'""
`
`lco
`
`I ZIP CODE
`
`180206
`
`!COUNTRY
`
`I U.S.A.
`
`ENCLOSED APPLICATION PARTS (check all that apply)
`
`filing fees and credit Deposit Account Number
`
`PROVISIONAL
`FILING FEE
`AMOUNT($)
`
`$75.00
`
`I
`
`The invention was made.by an agency of the United States Govennnent or under a contract with an agency of the United States Government.
`[Zj No.
`0 Yes, the name of the U.S. Government agency and the Government contract number are:
`
`Respectfully submitted,
`SIGNATURE ~~ S ~
`
`Date
`
`11/2 6/971
`
`TYPED or PRINTED NAME Thomas S . Birney
`
`REGISTRATION NO.
`( if appropriate)
`
`30,025
`
`D Additional inventors are being named on separately numbered sheets attached hereto
`PROVISIONAL APPLICATION FILING ONLY
`
`·.·.'[g]
`,!.[g] Drawing(s)
`
`Specification
`
`"i :STATE
`'
`..
`...
`..
`
`~
`
`.;~
`---
`
`Number of Pages ~
`[g] A check or money order is enclosed to cover the Provisional filing fees
`□ The Commissioner is hereby authorized to charge I 04-1414
`
`Number of Sheets
`
`9
`
`[g) Small Entity Statement
`[g] Other(specif)t llssj gnment:
`
`I
`
`METHOD OF PAYMENT (check one)
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 001
`
`

`

`•
`
`•
`
`SYSTEM FOR COMBINING VIRTUAL IMAGES
`WITH REAL-WORLD SCENES
`
`BACKGROUND OF THE INVENTION
`
`Field of the Invention. The present invention relates generally
`to the field of virtual reality simulation, and more particularly to
`systems for combining virtual images with real-world scenes.
`
`5
`
`10
`
`15
`
`20
`
`Statement of the Problem. Virtual reality systems have
`gained wide acceptance in recent years for training and simulations in
`a number of fields, including aircraft simulators, virtual reality games,
`surgical training, and military training. Conventional virtual reality
`systems generate a field of view for the user that is either completely
`computer-generated, or may
`include
`real-world scenery as
`background. Some virtual reality systems use portions of real-world
`images {e.g., a particular object, pattern, or texture) that can be
`incorporated into a computer-generated environment. However,
`conventional virtual reality systems do no typically incorporate virtual
`images into real-world scenes due to the difficulties of integrating a
`virtual image into a real-world scene in a realistic manner. For
`example, the virtual image can mask portions of the real-world scene,
`and objects in the real-world scene can mask portions of the virtual
`
`image depending on their relative locations and sizes.
`A need exists in many types of simulations to combine virtual
`
`images with real-world scenes. For example, the U.S. Army trains its
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 002
`
`

`

`•
`
`•
`
`-2--
`
`armored vehicle crews using vir1ual reality simulators.
`
`These
`
`simulators are static and rely entirely on computer-generated images.
`The army also trains armored vehicle crews in war games using actual
`
`vehicles moving over real terrain. These vehicles can be fully
`
`5
`
`instrumented and connected to computer systems that record and
`analyze the performance of the crew and vehicle throughout the
`indicate
`The computer system can also record and
`exercise.
`
`simulated "hits" on opposing vehicles. Although such war games are
`
`much more realistic than simulators, they are also much more
`expensive and time consuming due to the requirements of operating
`
`10
`
`15
`
`20
`
`25
`
`actual vehicles and managing the manpower necessary to conduct a
`realistic exercise. The cost of instrumentation for a large number of
`vehicles is also substantial. The present invention can be used in
`these war games to generate virtual tanks that appear move about the
`terrain. Virtual vehicles and real vehicles can appear in the same
`scene. The present invention can also be used to generate virtual
`"hits" and explosions during an exercise.
`The present invention also has application in the field of aircraft
`simulation. Virtual aircraft or virtual flying conditions can be combined
`with real-world scenes during an actual flight or a simulated flight.
`
`Another possible field of use is in games, such as laser tag, so that
`virtual players, objects, and special effects can be combined with real
`players and real objects in the playing field.
`
`Solution to the Problem. The present system enables
`computer-generated virtual images to be combined with images of the
`real world. A range scanner determines the shape and distance of
`
`real-world objects within a field of regard of interest to the observer.
`Virtual masking objects, which are simplified computer models of real-
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 003
`
`

`

`•
`
`•
`
`-3-
`
`world objects, are derived from the irange data. A computer simulates
`
`virtual entities and combines these virtual images with the virtual
`
`masking objects to create masked virtual images. The masked virtual
`images show the portions of virtual entities that would be visible if the
`
`5
`
`virtual entities actually existed in the real world. The masked virtual
`images and the real-world scene are combined and displayed in such
`
`a manner that the virtual images appear to be obscured, when
`appropriate for their simulated location relative to real-world objects.
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 004
`
`

`

`•
`
`•
`
`-4--
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present invention can be more readily understood in
`conjunction with the accompanying drawings, in which:
`
`5
`
`Figures 1a through 1d are examples of a real-world image, a
`virtual image, the resulting masked virtual image, and combined
`images, respectively.
`Figure 2 is a simplified block diagram showing the major steps
`in the present method.
`Figure 3 is an example of a range map generated by the range
`scanner.
`Figure 4 is an example of the virtual masking objects
`corresponding to the range map shown in Figure 3.
`Figure 5 is a VRML listing of a virtual masking object display
`construct representing one of the grassy knolls shown in Figures 3
`and 4.
`
`Figure 6 is an example of the effect of implementing the virtual
`masking object defined in Figure fi on the virtual image of a cube
`behind the knoll and the virtual imag1e of a cone in front of the knoll.
`Figure 7 is an image combining the grassy knoll, cube, and
`
`cone.
`
`10
`
`15
`
`20
`
`Figure 8 is a listing of the equations used to implement time
`
`extrapolation for the virtual masking objects between range frame
`updates.
`
`25
`
`Figure 9 is a three-dimensional example of a field of regard
`illustrating the use of time processing algorithms.
`Figure 10 is a pseudo-code listing of an algorithm for digital
`image combination.
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 005
`
`

`

`•
`
`•
`
`-5•-
`
`Figure 11 is a simplified diagram showing one embodiment for
`
`injecting virtual images into a real-world scene.
`
`Figure 12 is a simplified diagram showing an alternative
`
`embodiment for combining a virtual iimage with a real-world image.
`
`Figure 13 is an example of a video image combining the virtual
`image from Figure 6 with a real-world scene.
`Figure 14 is an example of a combined view in the infrared as
`seen through the thermal viewer of a tank or infantry fighting vehicle.
`
`Figure 15
`
`is an example of a combined view depicting
`
`conceptual virtual entities.
`Figure 16 is a simplified block diagram of the reference
`apparatus showing one possible implementation of the present
`system.
`Figure 17 is a simplified diagram of the transmitter section of
`
`the laser range scanner.
`Figure 18 is a simplified diagram of the receiver section of the
`laser range scanner.
`Figure 19 is a diagram of the scanning pattern for the laser
`range scanner.
`Figure 20 is a simplified block diagram of the virtual masking
`object (VMO) generation software.
`Figure 21 is an example of the control frame in the computer(cid:173)
`generated image.
`Figure 22 is a simplified block diagram of the virtual simulation
`and control software.
`Figure 23 is a simplified block diagram demonstrating how an
`input laser beam can be steered using a crystal excited by an
`
`alternating electrical field.
`
`5
`
`10
`
`15
`
`20
`
`25
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 006
`
`

`

`•
`
`•
`
`-6-
`
`Figure 24 is a simplified block diagram illustrating passive
`
`optical ranging using two cameras to determine the range of an object.
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 007
`
`

`

`•
`
`•
`
`-T-
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`An example of the basic functionality of the present invention is
`
`demonstrated
`
`in Figures 1a through 1d.
`
`The present system
`
`5
`
`combines the image of a real-world scene (Figure 1 a) with a virtual
`image (Figure 1b). The invention provides a method and the
`
`10
`
`15
`
`20
`
`25
`
`description of a reference apparatus and alternative implementations,
`to sense and remove elements of a virtual image, thereby generating
`a masked virtual image (Figure 1c). The masked virtual image and
`
`the image of the real world are then combined in such a manner that
`in the resulting combined image (Figure 1 d) the virtual object will
`appear to exist and move in the real world at its virtual position
`The block diagram of Figure 2 shows the major steps of the
`present method. One implementation of the present invention is
`diagrammed in Figures 16 through ~~2.
`
`Range Scanning. Turning to Figure 2, a range scanner 101
`determines the distance of the entities present in a real-world scene
`102 within small ranging sectors. E.ach ranging sector is defined as a
`small angular sector (e.g., 1 milliradlian in height and width) and by its
`azimuth and elevation, seen from a momentary observation point 100.
`The ranging sectors are included within the field of regard 103 of the
`range scanner 101. Scanning is limited in range, for example to 1000
`meters. Range scanning may be performed by a laser range scanner
`(as in the reference apparatus shown in Figures 16 - 22) or by other
`
`means that can determine the distance of objects within the angular
`space of ranging sector. Range data 104 are generated by the range
`scanning step, and include azimuth, elevation, and range to the
`nearest object for each ranging sector.
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 008
`
`

`

`•
`
`•
`
`-8·-
`
`Range scanning is an existing technology that provides range
`data necessary for virtual masking object generation. Normally,
`
`existing ranging systems will be used. The type of scanning will
`
`depend on the range and ranging sector requirements of a particular
`implementation of the invention for a specific application. For a range
`
`5
`
`10
`
`15
`
`20
`
`25
`
`requirement of approximately 400 meters, and angular resolution of
`
`approximately 1 milliradian, a 905 nanometer eye-safe infrared laser
`scanner was sufficient in the reference apparatus.
`In lieu of laser range scanning, other types of range scanning
`can provide sufficient range data for the present system. Examples of
`other scanning methods include three-dimensional radar scanning for
`
`longer ranges (multiple kilometers), or acoustical ranging for short
`ranges (e.g., within 50 meters). Active flash ranging methods, in which
`the acquisition of range for the ranging sectors is achieved in a
`simultaneous manner, covering the whole field of regard by phase(cid:173)
`modulated laser or radar ranging flashes, or by multiple amplitude(cid:173)
`modulated laser or radar pulses, are also applicable for short ranges.
`Passive ranging methods (e.g., stereographic or parallax comparison
`of real-world scene elements) can also be used, when their angular
`resolution fits the requirements of a specific application.
`The frequency of range data updates imposed on the range
`scanner depends on the type of real-world scene considered for
`specific implementations of the invention. A 266 Hz laser pulse rate
`
`was used for scanning the static scenes considered for the reference
`apparatus in Figures 16 - 22. The field of regard was scanned from a
`static observation point with approximately 26,000 ranging elements to
`provide a range frame update rate in the order of 0.01 Hz. For
`dynamic scenes with a moving observation point and mobile elements
`
`within
`
`the
`
`real-world scene,
`
`the system will normally require
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 009
`
`

`

`•
`
`•
`
`-9·-
`
`considerably higher range scanning pulse rates, such as 30 KHz, and
`
`a range frame update rate for the field of regard in the order of 1 Hz or
`
`higher.
`
`Range, azimuth, and elevation data may be provided directly
`
`5
`
`for each ranging sector, or indirectly, by providing only the range in
`each ranging sector, the scanning row number in the case of
`
`horizontal scanning for each row, and a frame start and stop signal. In
`
`this case, the angular width and height of each ranging sector is also
`
`provided.
`
`10
`
`15
`
`Virtual Masking Object Generation. Virtual masking object
`generation 105 is a key step of the invention. The ranging data of the
`
`ranging sectors are used to define the virtual masking objects 106
`corresponding to real world objects. The range map obtained in a
`0.261 radians wide and 0.096 radians high field of regard by a 905
`
`nanometer wavelength infrared scanning laser, scanning an area of
`small grassy hillocks, a parked automobile van, and a tree is shown in
`Figure 3. The range return for each 0.98 milliradian x 0.98 milliradian
`ranging sector is encoded as a gray intensity value, longer distances
`
`20
`
`are shown in darker shades. Areas of no return are shown in white.
`Such range maps are raw data, normally requiring further processing.
`
`Rectangular virtual masking objects can be generated from the
`raw data at the measured range, having the angular width and height
`
`of each ranging sector, to mask portions of the images of such virtual
`
`25
`
`entities as may lie beyond this virtual masking object. This is one
`method within the scope of the invention for virtual masking -object
`generation. Another, alternative approach is to apply a statistical filter
`
`to the raw ranging data in order to establish coherent, larger virtual
`
`masking objects (Fig. 4). The software code used for generating
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 0010
`
`

`

`•
`
`•
`
`-10-
`
`larger virtual masking objects, as shown in Figure 4, is listed in
`
`.Appendix A. The conditions for establishing coherence of virtual
`
`masking objects in the software code of Appendix A are listed in Table
`
`1. These conditions are associated with the specific method of ranging
`
`lines,
`
`the reference apparatus, which uses horizontal scan
`in
`advancing vertically from scan line to scan line.
`Alternative rules may be used for establishing the virtual
`masking objects. An example of an alternative method is direct three(cid:173)
`dimensional filtering. With this method, individual points defined by
`their ranging sector's azimuth, elevation and range are added to a
`virtual masking object as long as new points fall, within a pre(cid:173)
`determined tolerance "t" on a common plane, and when such points
`are contiguous within the tolerance of a hole-filling algorithm.
`The above methods result in a geometrical definition of the
`virtual masking objects. These may be simple quadrangles for a single
`ranging sector, or complex agglomerations of many ranging sectors
`into a two or three-dimensional surface. Within the virtual masking
`object generation step, three additional functions are performed.
`
`Display Construct Generation. Display construct generation
`builds data constructs compatible with a specific rendering system.
`For example, the virtual masking objects shown in Figure 4 were
`prepared
`for presentation under
`the Virtual Reality Modeling
`Language (VRML), version 2.0 (International Standards Organization
`ISO/IEC CD 14772), using the "lndexedFaceSet" high-level virtual
`reality node specification. This is reflected in the last section of the
`software code in Appendix A written in the Java language. Other
`
`sections of the software code are generic and independent of the
`
`5
`
`10
`
`15
`
`20
`
`25
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 0011
`
`

`

`•
`
`•
`
`-11-
`
`actual rendering system used. The invention itself is independent of
`
`any specific rendering implementation.
`
`Display Attribute Preparation. Display attribute preparation is
`
`5
`
`the last processing function of the virtual masking object generation
`step. This function may be separate from, or coincident with the
`
`10
`
`15
`
`20
`
`25
`
`display construct generation function, depending on the specific
`rendering system and image combination system used. The virtual
`
`masking objects must be rendered with display attributes that permit
`the automatic replacement of their pixels in the image combination
`step with a video background representing the external world. This
`attribute depends on the specific image combination hardware, and is
`usually referred to as the background attribute. The VRML 2.0 display
`construct and background display attribute of one of the grassy knolls
`shown in Figures 3 and 4 is listed in Figure 5. The background
`attribute is set as the ambient intensity, diffuse color, and shininess
`values in the material node of the construct, and corresponds to
`values set for a DeltaScan Pro GL GenLock system for image
`combination. Other image combination systems may use different
`attributes, such as a specific blue background color.
`The use of any particular rendering system, such as the Virtual
`Reality Modeling Language, is not claimed as part of the invention,
`and the method described will work with any rendering system
`capable of rendering three-dimensional objects, or two-dimensional
`projections of three-dimensional objects. VRML is used for virtual
`object definition in the reference apparatus, rendered through VRML
`
`browser software.
`The effect of implementing the virtual masking object described
`in the VRML display construct of Figure 5 is shown in Figure 6. The
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 0012
`
`

`

`•
`
`•
`
`-12:-
`
`grassy knoll of Fig. 5 is approximately 45 meters from the origin of the
`
`coordinate system located at the observation point of the user of the
`
`invention. By placing the virtual image of a cube beyond the grassy
`
`knoll, at 49.5 meters from the origin, and the virtual image of a sphere
`
`5
`
`in front of the grassy knoll, at 42.5 meters from the origin, the masking
`
`1effect of Figure 6 is obtained, assuming a white background attribute.
`
`The actual virtual objects present in the scene, including the grassy
`knoll's virtual masking object, are shown in Figure 7.
`
`10
`
`Time Processing Algorithms. Time processing algorithms
`are used for dynamic scenes in the virtual mask generation step in
`
`cases when the range frame rate achieved by the range scanning is
`not sufficient to provide the illusion of continuous motion (25-30 Hz
`range frame rate) of the virtual masking objects. With single-beam
`
`15
`
`sequential scanning methods, which are normally used for long
`ranges of several hundred meters to multiple kilometers to establish a
`
`range map, the achievable range frame rate is limited by the speed of
`light.
`
`As an example, in the laser range map of Figure 3, there are
`26068 ranging sectors. A sequential ranging system, such as the laser
`
`20
`
`range scanner used in generating Figure 3, limited at 1500 meters
`range, has an out-and return time frame of 10 microseconds for the
`range scanning laser pulse for each ranging sector. In this case this
`
`results in a minimum possible frame update time of 0.26 seconds, or a
`25 maximum possible
`range
`frame
`rate of 3.83 Hz. Practical
`
`considerations of actual ranger electronics implementations make the
`
`achievable range frame rate lower than this theoretical limit. The time
`processing algorithms linearly extrapolate the advance of the vertices
`
`of existing virtual masking objects to obtain the intermediate geometry
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 0013
`
`

`

`•
`
`•
`
`-13-
`
`of these objects between range frame updates obtained by range
`
`scanning. An example of
`
`the equations
`
`implementing
`
`time
`
`extrapolation for the azimuth An, the elevation En, and range rn of a
`virtual masking object with N vertices is shown in Figure 8. In these
`
`5
`
`equations tis the time for which the extrapolation is performed, T1 is
`the time of acquisition of the latest range frame, and TO is the time of
`acquisition of the previous range frame.
`
`10
`
`15
`
`20
`
`25
`
`The time processing extrapolation algorithms will be executed
`
`normally at a lower frequency than the maximum update rate of 25 to
`30 Hz required for the illusion of continuous motion. The extrapolation
`algorithms can be invoked by control algorithms on an as-needed
`logical conditions for
`invoking and controlling
`time
`basis. The
`processing are listed in Table 2. A set of three dimensional "envelope
`points" (e.g., the ranging sectors of maximum and minimum azimuth
`and elevation) are selected at the time of acquisition of each virtual
`masking object for implementing the conditions of Table 2.
`Figure 9 illustrates the logic conditions of Table 2. It shows
`several virtual masking objects,
`including
`the apparent
`terrain
`observed by range scanning, and other virtual masking objects 131,
`which include Object A located near to the observation point, and
`Object B located at the extreme range of range scanning. The
`envelope points for each virtual masking object are also indicated as
`black dots (see legend, 132).
`The angular position of the envelope points of Object A is
`assumed to have changed between the last two frame updates as
`shown by the arrows. If the angular movement of the left-side
`envelope points of Object A exceeds two ranging sectors (or a
`
`minimum number of ranging sectors as specified for a particular
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 0014
`
`

`

`•
`
`•
`
`-14-
`
`implementation) the extrapolation code will be invoked at least for
`some of the vertices of Object A by the invocation logic.
`The angular position of all of the envelope points of Object B in
`Figure 9 is assumed to have been less than two scan sectors ( or the
`5 minimum value specified
`for the
`implementation),
`therefore no
`
`extrapolation processing takes place for Object B.
`Time processing algorithms do not need to be implemented at
`all in those applications of the invention in which the observation point
`of the user of the invention is static, and the real-world field of regard
`of interest does not contain moving ireal-world entities.
`Although the method of virtual masking object generation is an
`element of the overall method of the invention, the specific form in
`which the virtual masking objects are implemented depends on what
`existing method of virtual object generation and rendering is used in a
`particular implementation of the present system. The example of
`Figure 5 uses the Virtual Reality Modeling Language (VRML).
`Alternatively, a Distributed Interactive Simulation Standard (IEEE
`1278-1995) implementation may be utilized. When VRML is used,
`virtual masking objects are updated through nodes injected by Java
`code, shown in the computer code listings of Appendix A. When the
`Distributed
`Interactive Simulation Standard
`is used,
`the virtual
`masking objects may be generated as Entity Appearance Protocol
`Data Units, or Dynamic Terrain updates. The method of virtual
`is compatible with any virtual entity
`masking object generation
`generation and rendering method.
`
`1 O
`
`15
`
`20
`
`25
`
`Virtual Entity Simulation 107 in Figure 2 is a step of the
`method performed
`through existing
`technologies
`for computer
`simulation. Virtual entities may be simulated, by the animation and
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 0015
`
`

`

`•
`
`•
`
`-15-
`
`5
`
`10
`
`15
`
`20
`
`25
`
`control techniques available through the Virtual Reality Modeling
`
`Language (VRML). An example of this approach is shown in the listing
`of Appendix B, in which three battle tanks are animated. Alternatively,
`
`the virtual simulation techniques defined in the Distributed Interactive
`
`Simulation Standard (IEEE 1278-1995) may be utilized. Virtual entity
`simulation generates continuously virtual entity appearance data (108
`in Figure 2). With VRML, the virtual entity appearance data are
`updated through animation code implemented as ROUTE constructs,
`Script nodes, or nodes generated through Java code. When the
`Distributed Interactive Simulation Standard is used, the virtual entity
`
`appearance data are generated as Entity Appearance Protocol Data
`Units. The invention is compatible with any virtual entity simulation. It
`is also compatible with a combination of virtual and real entities
`interacting with the observer, in which case the entity appearance data
`of real-world entities are transmitted to the apparatus implementing
`the method by radio or other remote means. In this case, the virtual
`simulation generates the avatars of the real-world entities, and
`resolves interactions with the avatars in the same way interactions
`with virtual entities are resolved.
`
`Virtual Image Generation 109 in Figure 2 renders the virtual
`entity appearance data generated by the virtual entity simulation and
`the virtual masking objects. The rendering process itself depends on
`
`It can be
`the particular implementation of the present system.
`implemented using existing technologies, such as VRML, or the
`Distributed Interactive Simulation Standard (IEEE 1278-1995), or
`other, commonly available virtual entity rendering methods. The virtual
`
`entity appearance data result in virtual images rendered in their
`normal appearance attributes, that is in their color, texture, intensity
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 0016
`
`

`

`•
`
`•
`
`-16-
`
`and other attributes as defined in the virtual entity simulation. The
`
`virtual masking objects, and all background of the virtual scene, are
`rendered in the background attributes compatible with the image
`combination techniques used in the image combination step. As a
`result, the virtual image generation step 109 renders masked virtual
`images, in which those elements that would be masked by natural
`objects are missing from the images of the virtual entities rendered
`from the virtual entity appearance data, as shown in Figures 1 c and 6.
`
`5
`
`10
`
`15
`
`In this step 11 0 of Figure 2, the virtual
`Image Combination.
`images constructed in the virtual image generation step 109 are
`combined with the real-world ima9e 114 originating from the video
`camera 102. The image combination step utilizes commercially
`available hardware,
`together with commercial or specially-built
`software for combining the virtual images with real-world image. The
`video image will fill out all areas of the display 111 rendered in the
`background attribute. In the case when the real-world image is
`provided as an analog video signal, a feasible technique to implement
`this step is to use a video "Genlocl<" board, such as a DeltaScan Pro
`20 GL used in the reference apparatus, setting the background attribute
`to 0 intensity, and black color, as shown in the code example of Figure
`5. When the
`real-world image is supplied as digital video, the
`pseudo-code shown in Figure 10 will implement the appropriate image
`combination.
`Image injection is an optional method of image combination, to
`be used when the observer 113 would normally observe the real-world
`scene through an instrument, such as the gunsight of a tank. Two
`alternatives of image injection are virtual image injection (Figure 11),
`and combined image injection (Figure 12).
`In Figure 11, the virtual
`
`25
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 0017
`
`

`

`•
`
`•
`
`-17-
`
`image output 140 of virtual image generation is displayed through a
`color video projector 141. It is then focused at infinity through a
`collimator 143 and prism 144 and injected into the collimated field of
`an optical device (such as the gun sight of a battle tank) normally
`showing only the real-world image 145. The observer 146 using the
`optical device will see the virtual image over the whole field of view of
`the real-world image. In Figure 12, the combined image 112 is
`generated in a color video display projector, which provides the input
`into an optical device, such as an advanced gun sight that uses video
`input as its image source.
`
`5
`
`1 0
`
`15
`
`Display. The output of the image combination step is shown in
`the display 111 of the combined image 112 in Figure 12, presented to
`the observer 113. An example of ai combined image display, using a
`VGA monitor in a 16-bit color depth, 640 x 480 pixel display mode is
`shown in Figure 13. The virtual image of Figure 6, with the grassy
`knoll's masking object, was combined with a live video scene obtained
`as an S-video signal from a color video camera. The live scene
`includes the live original of the grassy knoll, which appears to be
`20 masking out a portion of the virtual cube, while the virtual sphere is
`rendered with no masking.
`Alternative display implementations are compatible with the
`method. These may include flat display panels, image projectors, or
`virtual reality goggles or visors worn by the observer. In cases when
`the observer would normally observe the external world through an
`optical or other instrument, the virtual images can be injected into this
`instrument, as described under image injection.
`The generation of combined images, as described in the
`method, does not have to reflect what can be seen in the visible
`
`25
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 0018
`
`

`

`•
`
`•
`
`-18-
`
`spectrum. Figure 14 shows a combined view in the infrared, as it may
`
`show up in the thermal viewer of a modern battle tank or infantry
`
`fighting vehicle. The virtual entitiE~s generated by a virtual entity
`
`simulation may not be those of a real-world object, they may represent
`
`5
`
`fictional characters or entities, or views of conceptual objects. An
`example of conceptual virtual entities is shown in Figure 15.
`
`10
`
`15
`
`20
`
`25
`
`In Figure 15, the real-world objects 170 include the terrain, a
`tree, and a battle tank, in which the observer or observers 171 of a
`combined reality view presented to them are located. The conceptual
`virtual entities in the figure are conceptual objects: a path marker 172,
`
`a unit boundary 173, a navigational marker 174, and a phase line 175.
`The present system is required to position the conceptual virtual
`entities in the display in such a manner that they appear to be present
`in the landscape visible to the observer(s).
`
`is
`the display
`Observer Actions. The observer 113 of
`assumed to be the user of the invention for various purposes, such as
`military training, or entertainment. The observer may interact with the
`apparatus implementing the invention through observer actions 114 in
`Fig. 2. Observer actions include the observer's interactions with
`virtual elements in the combined reality scene. An example of such
`
`direct interaction is the engagement of a virtual opponent from a
`combat vehicle, such as a battle tank. In this case, the observer (the
`battle tank's gunner or commander) would observe and track the
`virtual opponent embedded in the real-world scene through the tank's
`actual gun sight, and engage it by firing the tank's main gun. This
`virtual-live interaction will take place by the observer's actions being
`captured through the observer's sensors and controls 115 in Figure 2.
`Observer actions also include interactions of the observer with the real
`
`Meta Exhibit 1011
`Meta v. Mullen - Page 0019
`
`

`

`•
`
`•
`
`-19-
`
`world, such as head movement of the observer, the observer's use of
`
`the battle tank's fire control system controls, or the actual firing of the
`
`gun through a trigger. The observer's interactions with the real world
`
`5
`
`also include the movement of the observation point (in the case of a
`battle