IN THE UNITED STATES PATENT AND TRADEMARK OFFICE
`
`
`In re Patent of: Michael S. Smith
`Attorney Docket No. 50095-0171IP1
`U.S. Patent No.:
`10,936,114
`
`
`Issue Date:
`March 2, 2021
`
`Appl. Serial No.: 16/558,026
`
`August 30, 2019
`Filing Date:
`Title:
`GESTURE-EQUIPPED TOUCH SCREEN SYSTEM, METHOD,
`AND COMPUTER PROGRAM PRODUCT
`
`
`
`DECLARATION OF DR. ANDREW COCKBURN
`
`I currently hold the opinions set expressed in this declaration. But my
`
`
`
`
`
`
`
`
`analysis may continue, and I may acquire additional information and/or attain
`
`supplemental insights that may result in added observations.
`
`I hereby declare that all statements made of my own knowledge are true and
`
`that all statements made on information and belief are believed to be true. I further
`
`declare that these statements were made with the knowledge that willful false
`
`statements and the like so made are punishable by fine or imprisonment, or both,
`
`under Section 1001 of the Title 18 of the United States Code.
`
`
`
`
`
`Dated:
`
`
`
`
`
`
`26th June, 2024
`
`
`
`
`
`
`
`
`
`
`By:
`
`
`
`Andrew Cockburn
`
`
`
`
`
`1
`
`APPLE 1003
`
`

`

`I. 
`II. 
`
`TABLE OF CONTENTS
`QUALIFICATIONS AND BACKGROUND INFORMATION .................. iv 
`LEGAL PRINCIPLES ..................................................................................... 2 
`A.  Anticipation ............................................................................................... 2 
`B.  Obviousness .............................................................................................. 3 
`III.  OVERVIEW OF CONCLUSIONS FORMED ............................................... 4 
`IV.  LEVEL OF ORDINARY SKILL IN THE ART ............................................. 5 
`V. 
`INTERPRETATIONS OF THE ’114 PATENT CLAIMS AT ISSUE........... 6 
`VI.  OVERVIEW OF THE ’114 PATENT ............................................................ 6 
`A.  The ’114 Patent ......................................................................................... 6 
`B.  Prosecution History ................................................................................... 8 
`VII.  PRIOR ART ANALYSIS ................................................................................ 9 
`A.  Ground 1A: Claims 1, 27, and 32 are anticipated and/or rendered
`obvious over Kim ...................................................................................... 9 
`1.  Kim .................................................................................................. 9 
`2. 
`Claim 1 .......................................................................................... 18 
`3. 
`Claim 27 ........................................................................................ 37 
`4. 
`Claim 27 ........................................................................................ 44 
`5. 
`Claim 32 ........................................................................................ 50 
`B.  Ground 1B: Claims 34, 59, and 64 are obvious over Kim in view of
`Lang 55 
`1. 
`Lang ............................................................................................... 55 
`2. 
`The Kim-Lang Combination ......................................................... 56 
`3. 
`Claim 34 ........................................................................................ 57 
`4. 
`Claim 59 ........................................................................................ 59 
`5. 
`Claim 64 ........................................................................................ 60 
`C.  Ground 1C: Claim 30 is obvious over Kim in view of Burrough .......... 61 
`1. 
`Burrough ........................................................................................ 61 
`2. 
`The Kim-Burrough Combination .................................................. 63 
`3. 
`Claim 30 ........................................................................................ 67 
`D.  Ground 1D: Claim 62 is obvious over Kim in view of Lang and
`Burrough ................................................................................................. 72 
`1. 
`The Kim-Lang-Burrough Combination ........................................ 72 
`2. 
`Claim 62 ........................................................................................ 73 
`E.  Ground 2A: Claims 1, 27, and 32 are obvious over Kim in view of
`Bowens .................................................................................................... 74 
`1. 
`Bowens .......................................................................................... 74 
`
`2
`
`

`

`The Kim-Bowens Combination .................................................... 75 
`2. 
`Claim 1 .......................................................................................... 90 
`3. 
`Claim 27 ........................................................................................ 97 
`4. 
`Claim 32 ......................................................................................104 
`5. 
`F.  Ground 2B: Claims 34, 59, and 64 are obvious over Kim in view of
`Bowens and Lang ..................................................................................109 
`1. 
`The Kim-Bowens-Lang Combination .........................................109 
`2. 
`Claim 34 ......................................................................................109 
`3. 
`Claim 59 ......................................................................................111 
`4. 
`Claim 64 ......................................................................................112 
`G.  Ground 2C: Claim 30 is obvious over Kim in view of Bowens and
`Burrough ...............................................................................................113 
`1. 
`The Kim-Bowens-Burrough Combination ..................................113 
`2. 
`Claim 30 ......................................................................................116 
`H.  Ground 2D: Claim 62 is obvious over Kim in view of Bowens, Lang,
`and Burrough ........................................................................................119 
`1. 
`The Kim-Bowens-Lang-Burrough Combination ........................119 
`1. 
`Claim 62 ......................................................................................120 
`
`
`
`
`3
`
`

`

`I.
`QUALIFICATIONS AND BACKGROUND INFORMATION
`1. My education and experience are described more fully in the attached
`
`curriculum vitae (Appendix A). I am a Professor at the Department of Computer
`
`Science and Software Engineering at the University of Canterbury, New Zealand.
`
`I also currently head the HCI (Human-Computer Interaction) and Multi-Media
`
`research group at the University of Canterbury.
`
`2.
`
`In 1988, I was awarded a Bachelor of Science with Honors in
`
`Computer Science from the University of York, England.
`
`3.
`
`In 1993, I was awarded a Ph.D. from the University of Stirling,
`
`Scotland. My thesis was on “Computer Supported Cooperative Work” which
`
`relates to forms of group interaction supported on computers.
`
`4.
`
`In 1993, I joined the University of Canterbury as a Lecturer in the
`
`Department of Computer Science (now the Department of Computer Science and
`
`Software Engineering). I was subsequently promoted to a Senior Lecturer, and
`
`then an Associate Professor, before my appointment as a Professor in 2010. I
`
`currently hold the title of Professor.
`
`5.
`
`I have over 30 years of experience in the area of HCI. The field of
`
`HCI is generally concerned with ways of understanding and improving the
`
`interaction between humans and computers, with a view to understanding,
`
`evaluating, designing, and building new styles of interactions that improve on one
`
`4
`
`

`

`or more of the end goals of making computers faster to learn, more intuitive, more
`
`efficient to use, and more subjectively satisfying.
`
`6.
`
`Throughout my career, I have published the results of many research
`
`projects that have involved building new user interfaces or reviewing existing user
`
`interfaces for performing a particular task, and evaluating their effectiveness. This
`
`includes publications relating to:
`
`(a)
`
`investigating user experiences with web navigation interfaces, for
`
`example when using the “back” button on web browsers;
`
`(b)
`
`reviewing and improving various interface schemes for traversing
`
`through documents and lists in desktop and mobile computer applications,
`
`including zooming, scrolling, and other techniques;
`
`(c)
`
`analyzing new interfaces for text entry on mobile and touch-sensitive
`
`devices;
`
`(d)
`
`addressing the problems arising from the small form factor of mobile
`
`devices with touch-sensitive displays, including wearable technologies;
`
`(e)
`
`examining the influence of haptic feedback on user performance with
`
`mouse and touchscreen input devices;
`
`evaluating the importance of spatially stable displays in user interfaces;
`
`(f)
`
`and
`
`5
`
`

`

`(g)
`
`designing, analyzing and evaluating gesture-based input techniques for
`
`various user activities, including music data entry, web navigation, text entry,
`
`and scrolling.
`
`7.
`
`I also have extensive experience in designing and building new user
`
`interfaces and reviewing existing user interfaces. This includes a number of
`
`projects regarding the design, development and evaluation of user interfaces that I
`
`have undertaken with companies in the computing and HCI industry, such as:
`
`(a)
`
`working with Airbus SAS in 2016 on methods to assist pilots’ gesture-
`
`based interaction with touchscreens in turbulent environments;
`
`(b)
`
`working with Hewlett-Packard Research Labs from 2010-2012 on the
`
`design and evaluation of gesture-based pointing techniques for remote displays,
`
`such as interactive TVs;
`
`(c)
`
`a number of projects from 2006-2010 working with Logitech on the
`
`design, development, evaluation and improvement of user interfaces for next
`
`generation mice, including in relation to scrolling and window management
`
`tasks;
`
`(d)
`
`working with IBM Almaden Research in 2006 on the design,
`
`development and evaluation of user interfaces for gesture-based text entry on
`
`mobile devices;
`
`6
`
`

`

`(e)
`
`working with Digit Wireless in 2002 on the evaluation of user
`
`performance for user interfaces for digital text entry on mobile devices; and
`
`(f)
`
`working with Microsoft Research in 1999 on the development,
`
`evaluation and improvement of user interfaces for web browsing, in particular
`
`the mechanisms for revisiting pages (such as through the “back” button or
`
`bookmarks).
`
`8.
`
`At the University of Canterbury, I currently teach a course on HCI for
`
`final year undergraduate students and a course on research methods in HCI for
`
`postgraduate students.
`
`9.
`
`In the past, I have also taught introductory computer programming
`
`courses for first year students across disciplines, which included the
`
`implementation of graphical user interfaces.
`
`10.
`
`I also manage an active research lab with a number of Ph.D. students.
`
`I have previously supervised fourteen students to successful completion of their
`
`Ph.D.s in the field of HCI.
`
`11. The Association for Computing Machinery (ACM) is the world’s
`
`largest scientific an educational computing society. In 2015, I was elected to the
`
`ACM Special Interest Group on Computer-Human Interaction (CHI) Academy,
`
`which honors the principal leaders of the field of HCI.
`
`7
`
`

`

`12.
`
`I also annually attend conferences relating to the field of HCI and
`
`regularly read journals that cover research in the field of HCI.
`
`13. To keep abreast with industry trends, I regularly read broader journals,
`
`magazines and websites related to computer science and technology, including
`
`Communications of the ACM, ACM TechNews, BBC Technology, and New York
`
`Times Technology.
`
`14. A detailed list of my other professional activities, memberships, and
`
`speaking engagements is included in my CV.
`
`15.
`
`I have been retained on behalf of Apple Inc. to offer technical
`
`opinions relating to U.S. Patent No. 10,936,114 (“the ’114 Patent”) and prior art
`
`references relating to its subject matter. I have reviewed the ’114 Patent, and
`
`relevant excerpts of the prosecution history of the ’114 Patent. I have also
`
`reviewed the following references:
`
` APPLE-1001 U.S. Patent No. 10,936,114 (“the ’114 Patent”)
`
` APPLE-1002 U.S. Patent No. 10,936,114 File History
`
` APPLE-1004 U.S. Patent Publication No. 2010/0302281 to Kim
`(“Kim”)
`
` APPLE-1005 U.S. Patent Publication No. 2009/0273571 to Bowens
`(“Bowens”)
`
` APPLE-1006 U.S. Patent Publication No. 2006/0139320 to Lang
`(“Lang”)
`
`8
`
`

`

` APPLE-1007 U.S. Patent Publication No. 2010/0156818 to Burrough
`et al. (Burrough)
`
` APPLE-1008 U.S. Patent Publication No. 2011/0310043 to Lacroix et
`al. (Lacroix)
`
` APPLE-1009 Simon, Windows CE 2 for Dummies (1997)
`
` APPLE-1010 U.S. Patent Publication No. 2011/0221666 to Newton et
`al. (“Newton”)
`
` APPLE-1011 U.S. Patent Publication No. 2015/0324116 to Marsden et
`al. (“Marsden”)
`
` APPLE-1012 U.S. Patent Publication No. 2018/0059928 to Westerman
`et al. (“Westerman”)
`
` APPLE-1013 U.S. Patent Publication No. 2013/0212541 to Dolenc et
`al. (“Dolenc”)
`
` APPLE-1014 U.S. Patent Publication No. 2005/0168488 to Montague
`(“Montague”)
`
` APPLE-1015 U.S. Patent No. 8,954,887 to Tseng et al. (“Tseng”)
`
` APPLE-1016 Smith’s Infringement Contentions
`
` APPLE-1017 U.S. Patent Publication No. 2007/0024593 to Schroeder
`(“Schroeder”)
`
` APPLE-1018 U.S. Patent Publication No. 2009/0167508 to Fadell et al.
`(“Fadell”)
`
`9
`
`

`

`16.
`
`I have also reviewed various supporting references and other
`
`documentation as further noted in my opinions below.
`
`17. Counsel has informed me that I should consider these materials
`
`through the lens of one of ordinary skill in the art related to the ’114 Patent at the
`
`time of the earliest possible priority date of the ’114 Patent, and I have done so
`
`during my review of these materials. The ’114 Patent claims the benefit of priority
`
`to a number of provisional applications, the earliest of which was filed on August
`
`5, 2011 (“the Critical Date”). Counsel has informed me that the Critical Date
`
`represents the earliest possible priority date to which the challenged claims of ’114
`
`Patent are entitled, and I have therefore used that Critical Date in my analysis
`
`below.
`
`18.
`
`I have no financial interest in the parties or in the outcome of this
`
`proceeding. I am being compensated for my work as an expert on an hourly basis.
`
`My compensation is not dependent on the outcome of these proceedings or the
`
`content of my opinions.
`
`19.
`
`In writing this declaration, I have considered the following: my own
`
`knowledge and experience, including my work experience in the field of HCI; my
`
`experience in teaching those subjects; and my experience in working with others
`
`10
`
`

`

`involved in those fields. In addition, I have analyzed various publications and
`
`materials, in addition to other materials I cite in my declaration.
`
`20. My opinions, as explained below, are based on my education,
`
`experience, and expertise in the fields relating to the ’114 Patent. Unless otherwise
`
`stated, my testimony below refers to the knowledge of one of ordinary skill in the
`
`art as of the Critical Date, or before. Any figures that appear within this document
`
`have been prepared with the assistance of Counsel and reflect my understanding of
`
`the ’114 Patent and the prior art discussed below.
`
`II. LEGAL PRINCIPLES
`A. Anticipation
`I have been informed that a patent claim is invalid as anticipated
`
`21.
`
`under 35 U.S.C. § 102 if each and every element of a claim, as properly construed,
`
`is found either explicitly or inherently in a single prior art reference. Under the
`
`principles of inherency, if the prior art necessarily functions in accordance with, or
`
`includes the claimed limitations, it anticipates.
`
`22.
`
`I have been informed that a claim is invalid under 35 U.S.C. § 102(a)
`
`if the claimed invention was known or used by others in the U.S., or was patented
`
`or published anywhere, before the applicant’s invention. I further have been
`
`informed that a claim is invalid under 35 U.S.C. § 102(b) if the invention was
`
`patented or published anywhere, or was in public use, on sale, or offered for sale in
`
`
`
`11
`
`

`

`this country, more than one year prior to the filing date of the patent application
`
`(critical date). And a claim is invalid, as I have been informed, under 35 U.S.C. §
`
`102(e), if an invention described by that claim was described in a U.S. patent
`
`granted on an application for a patent by another that was filed in the U.S. before
`
`the date of invention for such a claim.
`
`B. Obviousness
`I have been informed that a patent claim is invalid as “obvious” under
`
`23.
`
`35 U.S.C. § 103 in light of one or more prior art references if it would have been
`
`obvious to a POSITA, taking into account (1) the scope and content of the prior art,
`
`(2) the differences between the prior art and the claims, (3) the level of ordinary
`
`skill in the art, and (4) any so called “secondary considerations” of non-
`
`obviousness, which include: (i) “long felt need” for the claimed invention, (ii)
`
`commercial success attributable to the claimed invention, (iii) unexpected results
`
`of the claimed invention, and (iv) “copying” of the claimed invention by others.
`
`For purposes of my analysis above and because I know of no indication from the
`
`patent owner or others to the contrary, I have applied a date of August 5, 2011, as
`
`the date of invention in my obviousness analyses, although in many cases the same
`
`analysis would hold true even at an earlier time than August 5, 2011.
`
`24.
`
`I have been informed that a claim can be obvious in light of a single
`
`prior art reference or multiple prior art references. To be obvious in light of a
`
`
`
`12
`
`

`

`single prior art reference or multiple prior art references, there must be a reason to
`
`modify the single prior art reference, or combine two or more references, in order
`
`to achieve the claimed invention. This reason may come from a teaching,
`
`suggestion, or motivation to combine, or may come from the reference or
`
`references themselves, the knowledge or “common sense” of one skilled in the art,
`
`or from the nature of the problem to be solved, and may be explicit or implicit
`
`from the prior art as a whole. I have been informed that the combination of
`
`familiar elements according to known methods is likely to be obvious when it does
`
`no more than yield predictable results. I also understand it is improper to rely on
`
`hindsight in making the obviousness determination.
`
`III. OVERVIEW OF CONCLUSIONS FORMED
`25. This declaration explains the conclusions that I have formed based on
`
`my analysis. To summarize those conclusions, based upon my knowledge and
`
`experience and my review of the prior art publications listed above, I believe that:
`
` Claims 1, 27, and 32 are anticipated and/or obvious over Kim
`
`(Ground 1A)
`
` Claims 34, 59, and 64 are obvious over Kim in view of Lang (Ground
`
`1B)
`
` Claim 30 is obvious over Kim in view of Burrough (Ground 1C)
`
`
`
`13
`
`

`

` Claim 62 is obvious over Kim in view of Lang and Burrough (Ground
`
`1D)
`
` Claims 1, 27, and 32 are obvious over Kim in view of Bowens
`
`(Ground 2A)
`
` Claim 34, 59, and 64 are obvious over Kim in view of Bowens and
`
`Lang (Ground 2B)
`
` Claim 30 is obvious over Kim in view of Bowens and Burrough
`
`(Ground 2C)
`
` Claim 62 is obvious over Kim in view of Bowens, Lang, and
`
`Burrough (Ground 2D)
`
`IV. LEVEL OF ORDINARY SKILL IN THE ART
`26.
`In my opinion, a person of ordinary skill in the art (“POSITA”)
`
`relating to, and at the time of, the purported invention of the ’114 Patent would
`
`have had at least a bachelor’s degree in computer science, computer engineering,
`
`electrical engineering, human-computer interaction, or a related field, and would
`
`have had at least two years of relevant work experience in the design and
`
`development of graphical user interfaces (GUIs), human-computer interfaces, or
`
`equivalents thereof. Less work experience may be compensated by a higher level
`
`of education and vice versa.
`
`14
`
`

`

`27. Based on my experiences, I have a good understanding of the
`
`capabilities of one of ordinary skill. Indeed, I have taught, participated in
`
`organizations, and worked closely with many such persons over the course of my
`
`career. Based on my knowledge, skill, and experience, I have an understanding of
`
`the capabilities of one of ordinary skill. Furthermore, I possess those capabilities
`
`myself.
`
`V.
`
`28.
`
`INTERPRETATIONS OF THE ’114 PATENT CLAIMS AT
`ISSUE
`I have been informed by Counsel and understand that the best
`
`indicator of claim meaning is its usage in the context of the patent specification as
`
`understood by one of ordinary skill. I further understand that the words of the
`
`claims should be given their plain meaning unless that meaning is inconsistent with
`
`the patent specification or the patent’s history of examination before the Patent
`
`Office. Counsel has also informed me, and I understand that, the words of the
`
`claims should be interpreted as they would have been interpreted by one of
`
`ordinary skill at the time of the invention was made (not today). I have been
`
`informed by Counsel that I should use August 5, 2011 as the point in time for
`
`claim interpretation purposes.
`
`VI. OVERVIEW OF THE ’114 PATENT
`A. The ’114 Patent
`
`
`
`15
`
`

`

`29.
`
`In a brief Background section, the ’114 Patent states that “the present
`
`invention generally relate to touch screens.” APPLE-1001, 1:37-38.
`
`30.
`
`In particular, the ’755 purports to provide an apparatus including: “at
`
`least one non-transitory memory storing instructions and a plurality of
`
`applications; a touch screen; and one or more processors in communication with
`
`the at least one non-transitory memory and the touch screen, wherein the one or
`
`more processors execute the instructions to: display, via the touch screen, an object
`
`and at least a portion of an interface; detect a gesture on the touch screen on the
`
`object; when a magnitude of a duration of the gesture being detected on the touch
`
`screen on the object is less than a first magnitude threshold, perform an operation;
`
`when the magnitude of the duration of the gesture being detected on the touch
`
`screen on the object is greater than the first magnitude threshold, perform a scaling
`
`in connection with at least the object, where the scaling includes a reduction in a
`
`size such that the size is inversely related to the magnitude of the duration of the
`
`gesture being detected on the touch screen on the object; and when the magnitude
`
`of the duration of the gesture being detected on the touch screen on the object is
`
`greater than a second magnitude threshold, perform another operation.” APPLE-
`
`1001, 1:42-61.
`
`31. Claim 27 of the ’114 Patent, for example, recites “[a]n apparatus”
`
`comprising, among other components, “a touch screen,” and “one or more
`
`
`
`16
`
`

`

`processors in communication with the at least one non-transitory memory and the
`
`touch screen, wherein the one or more processors execute the instructions to cause
`
`the apparatus to:
`
`display an object on at least a portion of an interface;
`
`detect a gesture via the touch screen on the object;
`
`perform a first function in a first touch state;
`
`perform a second function in a second touch state; and
`
`during detection of at least a portion of the gesture and based on a change in
`
`a duration of the gesture being detected via the touch screen on the object,
`
`perform a scale operation in connection with the at least portion of the
`
`interface, where the scale operation includes a reduction in a size that is
`
`inversely related to the duration of the gesture being detected via the touch
`
`screen on the object.
`
`B.
`Prosecution History
`32. Application Serial No. 16/558,026, which later issued as the ’114
`
`Patent, was filed August 30, 2019, with 28 claims, which were soon after canceled
`
`with claims 29-58 added. APPLE-1002, 911-924, 741-764.
`
`33. During prosecution, the examiner issued no Office action, and thus,
`
`no prior art rejections. See, generally, id. Instead, Applicant offered a “Proposed
`
`Amendment,” which the parties discussed in an interview, and the examiner
`
`
`
`17
`
`

`

`subsequently allowed the claims without any prior art rejection. Id., 305-358, 134,
`
`74-133.
`
`34. Notably, however, despite the hundreds of pages of references
`
`provided to the Office, the examiner did not issue a single prior art rejection before
`
`allowing the claims. This petition addresses claim features distinguished during
`
`prosecution in a fresh way using new combinations of prior art, none of which
`
`were previously considered.
`
`VII. PRIOR ART ANALYSIS
`A. Ground 1A: Claims 1, 27, and 32 are anticipated and/or
`rendered obvious over Kim
`1. Kim1
`
`35. Kim describes a mobile device that “has a touch sensitive surface such
`
`as a touch screen” that provides “a touch-based zooming control method” for
`
`“portable and handheld devices.” APPLE-1004, Abstract, [0118]. At a high level,
`
`Kim explains that, “while performing a selected mode” (i.e., “a particular mode
`
`that allows displaying some kind of display target data outputted while a specific
`
`application selected at a user’s request runs in the mobile device”), “the mobile
`
`
`1 General descriptions provided for this and other references and combinations
`
`thereof are incorporated into each subsection and mapping of the claims that
`
`includes citations to these references.
`
`
`
`18
`
`

`

`device detects the occurrence of an interrupt,” which is “a signal or instructions,
`
`inputted through a touch screen or a special function key by a user, to stop the
`
`execution of a running application.” Id., [0048]. “After detecting an interrupt, the
`
`mobile device determines whether an interrupt is a request for activating the zoom
`
`mode,” including, for example, determining “whether an interrupt is a long press
`
`input.” Id., [0049]. Kim further explains that the interrupt may be, for example, “a
`
`request for the zoom mode” or a request for “some action in connection with the
`
`interrupt,” such as “a selection input for a specific icon” or “a request for a specific
`
`menu.” Id., [0049]-[0051]. In any case, when Kim’s device “detect[s] a user
`
`gesture” by “receiving an input event from a user through the touch screen, in
`
`response, the mobile device determines the type of an input event,” including, for
`
`example, “whether an input event is one of a move event … , a hold event …, and
`
`a double tap event.” Id., [0053]. If the detected input is none of them, “the mobile
`
`device disregards the received input event.” Id.
`
`36.
`
`In Kim, while “target data is displayed, the mobile device activates a
`
`zoom mode in response to a given gesture or input event,” and “executes a
`
`zooming function such as a zoom-in and a zoom-out in response to the received
`
`gesture input.” APPLE-1004, Abstract. Figure 5 “illustrates an example of a
`
`process of activating a zoom mode of a mobile device.” Id., [0083]. Kim explains
`
`that the central point of the zoom can be determined by a tap. Id., [0066]-[0067].
`
`
`
`19
`
`

`

`
`
`APPLE-1004, FIG. 5.
`
`
`
`37. Kim explains that “a zoom mode can be activated when a user invokes
`
`a long press at a certain point (x, y) on a touch screen” and “can perform a
`
`zooming action in the following two ways.” APPLE-1004, [0139] (emphasis
`
`added). First, a user can move his finger up or down “from a pressed point (x, y)
`
`in order to perform a zoom-in or a zoom-out around that point (x, y). Id., [0140].
`
`Or second, “a user can hold one of his or her finger … without moving at a pressed
`
`point (x, y).” Id.
`
`38.
`
`In more detail, Figure 7 “illustrates a process of executing the
`
`zooming function depending on a touch-based user gesture in a mobile device.”
`
`APPLE-1004, [0094]. As illustrated in Figure 7, “the mobile device can activate
`
`the zoom mode in response to a specific user gesture as discussed earlier in FIG.
`
`5,” and after “the zoom mode is activated, a user can manipulate the mobile device
`
`
`
`20
`
`

`

`to perform a continuous zooming function (namely, a zoom-in or a zoom-out)
`
`through an appropriate gesture such as a move event or a hold event.” Id., [0095].
`
`APPLE-1004, FIG. 7.
`
`
`
`39.
`
`In the top two examples above, a user can “continuously invoke”
`
`either a move up or move down event at a selected point to cause a continuous
`
`zoom-in or zoom-out function, respectively. APPLE-1004, [0096]-[0103]. In
`
`another example, shown in the bottom path of Figure 7, “reference numbers 741,
`
`743 and 745 show a continuous zoom-in action depending on a hold event.”
`
`APPLE-1004, [0104]. Specifically, “a user can invoke the hold event at a selected
`
`point through a touch input such as tap and hold,” and “the mobile device executes
`
`
`
`21
`
`

`

`a continuous zoom-in function around the selected point at given periodic intervals
`
`while the hold event continues.” Id. That is, as shown in Figure 7, the user
`
`touches the displayed person on a bike, or just the bike (“selected point”), and the
`
`device executes a zoom-in function around that point on the bike, as evidenced by
`
`the bike (and its surrounding data) being displayed at a larger size. Id. In this way,
`
`Kim teaches or at least renders obvious that the user input is received on the object
`
`that is later scaled up (zoomed-in) or scaled down (zoomed-out). Id. Put another
`
`way, Kim teaches or renders obvious that the user touches the portion of the screen
`
`from which the zooming function is centered. Id. In each of the figures showing a
`
`zooming action, the person on the bike is clearly modified by a zooming action, so
`
`a POSITA would have understood that the user gesture was received on the
`
`displayed person on a bike. See, e.g., id., FIGS. 5, 7. Additionally, Figures 5, 6,
`
`and 7 all clearly show a user input on the person on the bike on the touch screen.
`
`40. Moreover, Kim explains that in “a zoom-in process, the mobile device
`
`continuously offers a real-time, scaled-up bitmap image on the screen in response
`
`to the duration of the hold event,” and when “the hold event is released from the
`
`screen, the mobile device displays an enlarge display target data around the
`
`selected point on the touch screen, depending on a zoom-in ratio of a bitmap
`
`image.” Id., [0105]. Kim further explains that “the hold event is assigned to a
`
`
`
`22
`
`

`

`continuous zoom-in function is an exemplary case only,” and “the hold event may
`
`be alternatively assigned to a continuous zoom-out function.” Id., [0107].
`
`41. Since “the hold event may be alternatively assigned to a continuous
`
`zoom-out function,” Kim teaches that bottom path of Figure 7 can be zoom-out
`
`instead of zoom-in, as illustrated in Figure 7 (modified), below (highlighted
`
`orange). APPLE-1004, [0107].
`
`APPLE-1004, FIG. 7 (modified)
`
`42.
`
`In addition to using a hold event, Kim further teaches that other
`
`gestures, such as press and move, can be used to control zooming. APPLE-1004,
`
`
`
`
`
`23
`
`

`

`[0094]-[0103]. Kim also teaches that, after a zoom action, “a user can invoke a
`
`double tap event on the screen,” which causes “the mobile device [to] display[]
`
`target data with a given basic magnification value, as indicated by reference
`
`number 750” in Figure 7 (modified) above (highlighted yellow). APPLE-1004,
`
`[0099], [0103], [0106], FIG. 7.
`
`43. While Figure 7 shows only a rider on a bicycle,2 Figure 8 clearly
`
`shows that scaling can be performed on objects other than the object with respect
`
`to which the user gesture is directed or detected. APPLE-1004, FIG. 8. Kim’s
`
`device, “controls a zooming action ... in response to a user gesture” where the
`
`displayed data is a portion of the web page that fits on the device screen’s
`
`interface. APPLE-1004, [0111]. While Figure 8 does not explicitly indicate where
`
`the user gesture was received, a POSITA would have understood that the gesture
`
`was received on a single object (as opposed to on the entire touch screen), and
`
`most likely, it was received on or near the car in view of the resulting enlarged
`
`display of the content 870. That is, Kim explains that the “zooming function”
`
`occurs “around a specific point on the target data,” so, given that the zoomed-in
`
`
`2 Arguably, the rider on the bicycle shown in Figure 7 could be considered at least
`
`two separate objects (e.g., a rider and a bicycle), where the gesture is detected on
`
`the bicycle but not the rider. See APPLE-1004, FIG. 7 (741).
`
`
`
`24
`
`

`