`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`______________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`______________
`
`APPLE INC.,
`
`Petitioner
`
`
`v.
`
`
`SMITH INTERFACE TECHNOLOGIES, LLC,
`Patent Owner
`
`______________
`
`Case IPR2024-01091
`Patent No 10,936,114
`______________
`
`
`DECLARATION OF RAJEEV SURATI, Ph.D.
`
`
`
`Apple Inc. v. Smith Interface Techs., LLC
`IPR2024-01091 | Smith EX2001 | Page 1 of 51
`
`

`

`
`
`I.
`II.
`III.
`IV.
`
`IX.
`
`X.
`
`TABLE OF CONTENTS
`INTRODUCTION ........................................................................................................................... 1
`PROFESSIONAL BACKGROUND ............................................................................................... 1
`COMPENSATION ........................................................................................................................ 10
`LEGAL CONSIDERATION ......................................................................................................... 10
`A.
`Anticipation ....................................................................................................................... 10
`B.
`Obviousness ....................................................................................................................... 11
`C.
`Claim Interpretation ........................................................................................................... 12
`OVERVIEW OF TASK & BASIS FOR OPINIONS .................................................................... 13
`V.
`LEVEL OF SKILL IN THE RELEVANT ART ........................................................................... 14
`VI.
`VII. OVERVIEW OF THE CITED ART .............................................................................................. 15
`A.
`U.S. Patent Pub. No. 2010/0302281 (“Kim”) .................................................................... 15
`1. User gestures detected via “interrupt” in “selected mode” (i.e., non-zoom mode) ....... 15
`2. User gestures in “zoom mode” ...................................................................................... 16
`U.S. Patent Pub. No. 2009/0273571 (“Bowens”) .............................................................. 19
`B.
`U.S. Patent Pub. No. 2010/0156818 (“Burrough”) ............................................................ 21
`C.
`VIII. OVERVIEW OF THE ’114 PATENT ........................................................................................... 22
`A.
`Claims 1, 32, 34, and 64 .................................................................................................... 23
`B.
`Claims 27 and 59 ............................................................................................................... 23
`C.
`Claims 30 and 62 ............................................................................................................... 24
`CLAIM CONSTRUCTION ........................................................................................................... 24
`A.
`“Magnitude of [a] / [the] Duration of the Gesture” ........................................................... 24
`B.
`“Touch State” ..................................................................................................................... 27
`C.
`“Inversely Related To” ...................................................................................................... 27
`CLAIMS 1, 32, 34, AND 64 (GROUNDS 1A, 1B, 2A, 2B) ......................................................... 28
`A.
`The Petition Fails to Consistently Map “The Duration” of “The Gesture” ....................... 28
`1. Rather than identify a common gesture for the claims’ “the gesture,” the Petition points
`to three distinct gestures ..................................................................................................... 28
`2. Rather than identify a common duration for the claims’ “the magnitude of the duration,”
`the Petition points to three distinct durations that can never even overlap......................... 32
`3. These “gesture” and “duration” defects are present in all grounds for all of claims 1, 32,
`34, and 64 ........................................................................................................................... 34
`The Petition Fails to Demonstrate that Performance of the Mapped “Operation / Function”
`Occurs “When” / “Based On” / “In Response To” “a Duration of the Gesture” “is Less Than
`A First Magnitude Threshold” ........................................................................................... 36
`
`B.
`
`IPR2024-01091
`
`- i -
`
`Patent No. 10,936,114
`
`Page 2 of 51
`
`

`

`C.
`
`
`
`The Petition Fails To Demonstrate That Performance of the Mapped “Another Operation /
`Function” Occurs “When” / “Based On” / “In Response To” “the Duration of the Gesture
`Being Detected” “is Greater Than a Second Magnitude Threshold” ................................. 38
`1. Ground 1 Mapping ........................................................................................................ 39
`2. Ground 2 Mapping ........................................................................................................ 41
`CLAIMS 27 AND 59 (GROUNDS 1A, 1B, 2A, 2B) .................................................................... 42
`A.
`The Petition Fails to Identify a “First” or “Second” “Touch State” As Properly Construed
` ........................................................................................................................................... 42
`CLAIMS 30 AND 62 (GROUNDS 1C, 1D, 2C, 2D) .................................................................... 45
`A.
`The Petition Fails to Identify an Evidentiary Basis to Add Haptic Feedback to Kim’s Double
`Tap Gesture ........................................................................................................................ 45
`XIII. CONCLUSION .............................................................................................................................. 48
`
`
`
`XI.
`
`XII.
`
`
`
`IPR2024-01091
`
`- ii -
`
`Patent No. 10,936,114
`
`Page 3 of 51
`
`

`

`
`
`I.
`
`INTRODUCTION
`1. My name is Dr. Rajeev Surati, Ph.D. I have been retained by Smith
`
`Interfaces Technologies, LLC as an independent expert consultant in this inter partes
`
`review (“IPR”) proceeding before the United States Patent and Trademark Office
`
`(“PTO”).
`
`II.
`
`PROFESSIONAL BACKGROUND
`All of my opinions stated in this Declaration are based on my own
`2.
`
`personal knowledge and professional judgment. In forming my opinions, I have
`
`relied on my knowledge and experience in designing, developing, and researching
`
`the technology referenced in this Declaration.
`
`3.
`
`I am over 18 years of age and, if I am called upon to do so, I would be
`
`competent to testify as to the matters set forth herein. I understand that a copy of
`
`my current curriculum vitae, which details my education and professional and
`
`academic experience, is being submitted as EX2002. The following provides a brief
`
`overview of some of my experience that is relevant to the matters set forth in this
`
`Declaration.
`
`4.
`
`I hold a Doctor of Philosophy (Ph.D.) degree in Electrical Engineering
`
`and Computer Science from the Massachusetts Institute of Technology (“MIT”)
`
`(awarded in 1999) with a Grade Point Average of 5.0/5.0. I obtained a Master of
`
`Science degree in Electrical Engineering and Computer Science from MIT (awarded
`
`IPR2024-01091
`
`- 1 -
`
`Patent No. 10,936,114
`
`Page 4 of 51
`
`

`

`
`
`in 1995) with a Grade Point Average of 5.0/5.0. I have a Bachelor of Science degree
`
`in Electrical Engineering from MIT (awarded 1992), graduating with a Grade Point
`
`Average of 4.9/5.0.
`
`5. My Ph.D. thesis was entitled “Scalable Self-Calibrating Technology for
`
`Large Scale Displays.” My master’s thesis was entitled “Practical Partial
`
`Evaluation.” My undergraduate thesis, for which I received MIT EECS’s William
`
`A. Martin thesis prize for best undergraduate thesis, was entitled “A Parallelizing
`
`Compiler based on Partial Evaluation.” I was also awarded the Department of
`
`Energy’s highly-selective Computational Science Fellowship in 1995, which funded
`
`my Ph.D. studies.
`
`6.
`
`In 1989, I began programming embedded systems and, in 1995, built
`
`one of the first publicly available Internet Coke Machines. During that time, I
`
`worked on various user interfaces. For example, one to augment the coke machine
`
`to dispense coke and another that became one of the early ecommerce mechanisms,
`
`using standards-based protocols for tracking user consumption, payments, and drink
`
`stock.
`
`7.
`
`In 1997, while at MIT pursuing my Ph.D., I started a company called
`
`Flash Communications. At Flash, I invented an instant messaging platform focused
`
`on enterprise needs. I had the idea while observing the popularity of the Zephyr
`
`Instant Messaging Service at MIT beginning in 1988. Attracted by Flash’s
`
`IPR2024-01091
`
`- 2 -
`
`Patent No. 10,936,114
`
`Page 5 of 51
`
`

`

`
`
`enterprise focus, Microsoft acquired Flash in 1998. With Microsoft, we built both a
`
`client and server product. The basic protocol we invented became the foundation
`
`for the well-known XMPP protocol that was widely used in the mid-2000s among
`
`instant messaging providers. I developed both the client and server products and
`
`dealt with many, if not all, of the issues one might face when implementing user
`
`interfaces for instant messaging systems. In fact, we helped to build Microsoft’s
`
`famous MSN product, which adopted various user interface elements we created.
`
`8.
`
`As a result of this experience, I became heavily involved with User
`
`Interface design on desktop machines, experience that I have continued to develop
`
`by building desktop, mobile, and web-based applications used by millions of people.
`
`9.
`
`After graduating, I joined Microsoft (as was required by Microsoft in
`
`its acquisition of Flash Communications), where I worked on both client and server
`
`technologies related to instant messaging. My work spanned both the Microsoft
`
`Exchange Instant Messaging product that was released in 2000 and MSN Messenger.
`
`10. From 2000 to 2007, I cofounded, led, and sold a business called
`
`photo.net to Namemedia, which is now part of GoDaddy.com. In early 2000,
`
`Photo.net was considered a top 1,000 website and it received large amounts of
`
`traffic. I built the site from its beginning as a single computer installed in a
`
`datacenter to a full rack of computers in that same data center. As a result, I became
`
`very familiar with the careful design and programming one must employ to build
`
`IPR2024-01091
`
`- 3 -
`
`Patent No. 10,936,114
`
`Page 6 of 51
`
`

`

`
`
`and maintain systems that enable both engaging user interfaces and experiences at
`
`scale.
`
`11. At photo.net, we prototyped many fundamental Internet community
`
`features, such as photo sharing, social networking, and memberships. The photo.net
`
`system was written on top of Oracle SQL Database and served up many millions of
`
`web pages each day (corresponding to millions of records) with high-volume inserts.
`
`The system also incorporated an ecommerce platform used to bill subscriptions,
`
`track users, etc. I spent seven years running the site, both writing new features driven
`
`by the Oracle SQL database and maintaining tables containing the many millions of
`
`records generated by the site. I also served as the database administrator, which
`
`provided me with extensive operational experience running such a system. I dealt
`
`with everything from database query speedup to the more mundane day-to-day
`
`issues, such as the maintenance of relational databases, including backups. In
`
`addition to the web interface, we also ran a mobile version of the product based on
`
`WAP, which I helped to develop.
`
`12. After developing one of the first photo sharing sites, we further
`
`increased user engagement by designing an innovative photo rating interface. Our
`
`photo rating system became a huge draw for the site. The rating system increased
`
`engagement by four orders of magnitude, ultimately resulting in Sergey Brin,
`
`cofounder of Google, asking photo.net to provide imagery to Google for a screen
`
`IPR2024-01091
`
`- 4 -
`
`Patent No. 10,936,114
`
`Page 7 of 51
`
`

`

`
`
`saver application. Joichi Ito, former head of the MIT Media Lab and a famous
`
`internet luminary called it “the best photo sharing site” in 2001.
`
`13.
`
`I also innovated user interface technology at Nexaweb, where I was an
`
`investor and technical advisor. Nexaweb designed User Interfaces that worked well
`
`over the web. At Nexaweb, I spent a great deal of time solving underlying issues
`
`related to implementing a UI framework both in Java and Javascript. We also
`
`innovated a mechanism using server push prior to the AJAX becoming widely
`
`adopted.
`
`14.
`
`In 2004, I started Scalable Display Technologies based on the
`
`technology in my Ph.D. thesis and the related patent (U.S. Patent No. 6,456,339)
`
`entitled “Super-resolution Display.” As part of my Ph.D. thesis, I designed a special
`
`display that allows one to create ultra-high-resolution displays composed of multiple
`
`projectors tiled with a slight overlap. The system uses camera-based feedback to
`
`create an inverse map that drives the system such that a person can only see a
`
`continuous, seamless display. There is no visible bezel or overlap. During this
`
`project, I became familiar with how and why to drive content onto the display. Thus,
`
`I also became familiar with content encoding/decoding, user interfaces for driving
`
`large displays, multimedia content storage, high resolution imagery, networks,
`
`recording, GPUs, storage of content, etc.
`
`IPR2024-01091
`
`- 5 -
`
`Patent No. 10,936,114
`
`Page 8 of 51
`
`

`

`
`
`15. At Scalable Display Technologies, I developed the core algorithms and
`
`implemented—with a team—the entire software system, which included the core
`
`user interface, machine vision and embedded components, and a multimedia
`
`playback system. Our first customer was Pixar. Today, the software is primarily
`
`used by the US military and commercial theme parks, such as Disney and Universal.
`
`Scalable is well-known in the industry for having some of the easiest and intuitive
`
`software to use.
`
`16.
`
`In parallel with my work at Scalable, I lectured at MIT on many
`
`subjects, including the Android operating system. In 2008, I worked with a group
`
`of MIT students on the development of mobile applications for Android with Rich
`
`Miner, a cofounder of Android. Since many students created applications involving
`
`Google Maps and localization, I became familiar with these technologies. I also
`
`served as a lecturer and mentor at MIT Play Labs, which was an incubator for
`
`augmented reality (AR) and virtual reality (VR) software for mobile handsets and
`
`headset applications. Notably, in addition to mobile handsets, Android runs on the
`
`Oculus headset. As part of that program, I worked with several startup companies
`
`on mobile applications, including a startup that developed applications for Telegram,
`
`which is a cross-platform, cloud-based instant messaging system.
`
`17. Since 2014, I have worked as an independent consultant for several
`
`companies including NEC, Hitachi, Hi Marley, and Estee Lauder.
`
`IPR2024-01091
`
`- 6 -
`
`Patent No. 10,936,114
`
`Page 9 of 51
`
`

`

`
`
`18.
`
`In 2018, I became a senior partner at nCent Labs. In that role, I
`
`consulted on the development of an incentive-market-based platform for block
`
`chains and cryptocurrency. Another part of my work at nCent Labs focused on the
`
`development of SMS messaging applications for the nCent platform. In that part, I
`
`developed a full stack application around the protocol.
`
`19. From 2019 to 2020, I served as a Technical Lead of the Skunkworks at
`
`Hydrow, which is a startup company that develops indoor rowing machines. In this
`
`role, I worked on special projects, including the development of a VR experience
`
`using Magic Leap and Oculus that immersed users in a team-based rowing crew on
`
`a scull using VR cinematography. I leveraged my experience with user interfaces
`
`and Human-Computer Interaction (“HCI”) to develop a system that gives people a
`
`sense of rowing on the water with others, while in reality, they were rowing on a
`
`rowing machine.
`
` We accomplished our goal through a clever use of
`
`cinematography. The result was quite compelling and increased user engagement
`
`with their exercise program.
`
`20.
`
`In 2020, I started a company called Skyline Nav AI Inc. that develops
`
`technology that uses a visual location (i.e., the skyline) to geo-locate the place a
`
`picture was taken. At Skyline, I built and now maintain a full user experience
`
`centered around a workflow for people to snap a picture and use the image’s skyline
`
`and downloaded 3D mode to localize a person. I have written applications for both
`
`IPR2024-01091
`
`- 7 -
`
`Patent No. 10,936,114
`
`Page 10 of 51
`
`

`

`
`
`Android and iOS that communicate with a backend server running the machine
`
`vision algorithms.
`
`21. Over the past decade, I have served as a technical consultant and expert
`
`witness on patent litigation matters. In the course of this work, I have provided
`
`consulting services to a wide variety of technology companies, including
`
`BritishTelecom, Apple,
`
`IBM, Philips, Shopify, Zillow, Polaris Powered
`
`Technologies, Amazon, Salesforce, Hitachi, Slack, Harris Teeter, and others.
`
`22. During the course of my career, I was the inventor of the subject matter
`
`for approximately 15 U.S. Patents. I have also received additional patents,
`
`including: U.S. Patent No. 8,817,111, entitled “System and method of calibrating a
`
`display system free of variation in system input resolution,” U.S. Patent No.
`
`8,994,757, “System and method for providing improved display quality by display
`
`adjustment and image processing using optical feedback,” U.S. Patent No.
`
`9,215,455, “System and method of calibrating a display system free of variation in
`
`system input resolution,” U.S. Patent No. 9,369,683, “System and method for
`
`calibrating a display system using manual and semi-manual techniques,” and U.S.
`
`Patent No. 9,497,447, “System and method for color and intensity calibrating of a
`
`display system for practical usage.”
`
`23.
`
`I am also a co-inventor of patented technology related to instant
`
`messaging, including U.S. Patent No. 5,943,478, which describes associated
`
`IPR2024-01091
`
`- 8 -
`
`Patent No. 10,936,114
`
`Page 11 of 51
`
`

`

`
`
`technology that I developed related to pop-up, two-way messaging over the Internet.
`
`While at Microsoft, I was an inventor on several patents, including: U.S. Patent No.
`
`6,260,148 relating to methods and systems for message forwarding and property
`
`notifications using electronic subscriptions; and U.S. Patent Nos. 6,415,318 and
`
`6,604,133 relating to inter-enterprise messaging systems using bridgehead servers.
`
`Aspects of these patents relate specifically to messaging and notification technology
`
`in telecommunications systems.
`
`24.
`
`I serve on the advisory boards of several technology companies,
`
`including Paneve, which develops general purpose ASICs coupled with compiler
`
`technology; Nexaweb, which develops real-time web application frameworks using
`
`HTTPS; Antix Labs, which develops compiler technology for a universal gaming
`
`platform; and Permabit, which develops content-addressable storage.
`
`25.
`
`In addition to my above experience, I have published papers on subjects
`
`relating to computing systems, computer network communications, databases, and
`
`other subjects within the realm of electrical and computer engineering. I have also
`
`received several awards for my contributions as an inventor and entrepreneur,
`
`including the Global Indus. Technovator Award 2009 and Laureate of 2009
`
`Computer World Honors Program. My publication and award history are listed in
`
`more detail on my CV.
`
`IPR2024-01091
`
`- 9 -
`
`Patent No. 10,936,114
`
`Page 12 of 51
`
`

`

`
`
`III. COMPENSATION
`26. My work on this matter is being billed at my customary rate of $700 an
`
`hour. Also, I am being reimbursed for reasonable expenses I incur in relation to my
`
`services. I have no pecuniary interest in the outcome of this proceeding. I
`
`understand I will be paid regardless of the outcome of any proceeding in which my
`
`work is used.
`
`IV. LEGAL CONSIDERATION
`I am not an attorney. My understanding of the law is based on
`27.
`
`information provided by counsel for Smith Interface Technologies, LLC.
`
`A. Anticipation
`I understand that a reference anticipates a claim if it discloses each and
`28.
`
`every element recited in the claim, arranged as in the claim, so as to enable a person
`
`of ordinary skill in the art (POSITA) to make and use the claimed invention without
`
`the need for undue experimentation in light of the general knowledge available in
`
`the art. I understand that a U.S. Patent document is presumed to have sufficient
`
`description to include sufficient detail for a person of ordinary skill in the art to make
`
`and use the subject matter that document describes.
`
`29.
`
`I also understand that a claim limitation may be present in an alleged
`
`prior art reference either expressly or inherently, but inherency may be established
`
`only if such limitation is necessarily present in the reference. I understand inherency
`
`cannot be established by probabilities or possibilities, and the mere fact that a certain
`
`IPR2024-01091
`
`- 10 -
`
`Patent No. 10,936,114
`
`Page 13 of 51
`
`

`

`
`
`thing may result from a given set of circumstances is not sufficient to establish
`
`inherency.
`
`B. Obviousness
`I understand that a claimed invention is obvious and, therefore, not
`30.
`
`patentable if the subject matter claimed would have been obvious to a person of
`
`ordinary skill in the art at the time that the invention of the ’114 Patent (a
`
`“POSITA”), which I have been asked to treat as December 30, 2011.
`
`31.
`
`I understand that a claim can be obvious in view of a single prior art
`
`reference (e.g., via modification of that prior art reference) or multiple prior art
`
`references (e.g., via a combination of two or more prior art references), if such a
`
`modification or combination was within the skill of a POSITA. I understand that
`
`there must be some articulated reasoning with some rational underpinning to support
`
`a conclusion of obviousness. I also understand that to establish a finding of
`
`obviousness one must show that one of ordinary skill in the art would have had a
`
`motivation to combine the prior art references to produce the claimed invention and
`
`a reasonable expectation that the combination would be successful.
`
`32.
`
` I further understand that exemplary rationales that may support a
`
`conclusion of obviousness include: (1) simply arranging old elements in a way in
`
`which each element performs the same function it was known to perform, and the
`
`arrangement yields expected results, (2) merely substituting one element for another
`
`IPR2024-01091
`
`- 11 -
`
`Patent No. 10,936,114
`
`Page 14 of 51
`
`

`

`
`
`known element in the field, if the substitution yields no more than a predictable
`
`result, (3) combining elements in a way that was “obvious to try” because of a design
`
`need or market pressure, where there was a finite number of identified, predictable
`
`solutions, (4) that design incentives or other market forces in a field would have
`
`prompted variations in a work that were predictable to a person of ordinary skill in
`
`the art, and (5) that there was some teaching, suggestion, or motivation in the prior
`
`art that would have led a POSITA to modify or combine prior art references to arrive
`
`at the claimed invention.
`
`C. Claim Interpretation
`I understand that a claim term is interpreted according to its ordinary
`33.
`
`and customary meaning as a POSITA would have understood the term in light of the
`
`surrounding claim language, other claims, the specification, and the patent’s
`
`prosecution history, which are referred to as intrinsic evidence. I also understand
`
`that prior art references cited in the patent’s prosecution history are considered
`
`intrinsic evidence. I further understand that evidence outside the patent and its
`
`prosecution history (e.g., dictionaries and technical articles), may inform the context
`
`in which a POSITA would have understood the claims of a patent. I understand this
`
`ordinary and customary meaning applies absent unique circumstances, such as
`
`where a patent clearly expresses an intent to set forth a special meaning for a term.
`
`IPR2024-01091
`
`- 12 -
`
`Patent No. 10,936,114
`
`Page 15 of 51
`
`

`

`
`
`V. OVERVIEW OF TASK & BASIS FOR OPINIONS
`I have been asked to review the ’114 Patent. I have been asked to
`34.
`
`provide opinions related to certain issues from the perspective of a person of ordinary
`
`skill, having knowledge of the relevant art, as of December 30, 2011, and—except
`
`where otherwise noted—the opinions stated in this declaration are from that
`
`perspective. The qualifications and abilities of such a person are described in
`
`paragraph 38 below.
`
`35. More particularly, I have been informed that this proceeding is
`
`currently at a preliminary stage during which the Patent Trial and Appeal Board (the
`
`Board) will decide whether the Petition establishes a reasonable likelihood of at least
`
`one claim of the ’114 Patent being invalid. I also understand that the Board may
`
`review the Petition under a stricter “compelling merits” standard that requires the
`
`Petition to make a stronger showing of invalidity; however, my analysis is not
`
`performed using this higher standard. At this preliminary stage, I have been asked
`
`to focus my analysis on the Petition’s proposed invalidity theories for the ’114
`
`Patent’s independent claims 1, 27, 30, 32, 34, 59, 62, and 64.
`
`36. My opinions are based on my education, training, and experience as
`
`well as items that I reviewed to prepare my opinions, including the Petition, the
`
`documents listed in the Exhibit List included with the Petition, and any other items
`
`that I reference in my below analysis.
`
`IPR2024-01091
`
`- 13 -
`
`Patent No. 10,936,114
`
`Page 16 of 51
`
`

`

`
`
`37.
`
`I understand that other issues may arise that require further explanation,
`
`and I will provide that explanation if appropriate. As a result, I respectfully reserve
`
`the right to update and supplement this declaration and the information and opinions
`
`provided herein.
`
`VI. LEVEL OF SKILL IN THE RELEVANT ART
`I understand that a patent must be analyzed from the perspective of a
`38.
`
`person of ordinary skill in the art (“POSITA”) as of the time of invention of the
`
`patent, which is typically considered to be the patent’s earliest filing date. While I
`
`do not necessarily agree with every aspect of Dr. Cockburn’s explanation of the level
`
`of ordinary skill in the art, I do not currently believe the aspects of that explanation
`
`I would change or clarify would impact the analysis set forth in this declaration. For
`
`purposes of this declaration, my opinions are therefore provided from the perspective
`
`of the POSITA described in paragraph 26 of Dr. Cockburn’s declaration (EX1003)
`
`as of December 30, 2011, which is produced below:
`
`at least a bachelor’s degree in computer science, computer engineering,
`
`electrical engineering, human-computer interaction, or a related field, and
`
`would have had at least two years of relevant work experience in the design
`
`and development of graphical user interfaces (GUIs), human-computer
`
`interfaces, or equivalents thereof. Less work experience may be compensated
`
`by a higher level of education and vice versa.
`
`IPR2024-01091
`
`- 14 -
`
`Patent No. 10,936,114
`
`Page 17 of 51
`
`

`

`
`
`VII. OVERVIEW OF THE CITED ART
`A. U.S. Patent Pub. No. 2010/0302281 (“Kim”)
`39. Kim discloses a mobile device and “touch-based zooming control
`
`method.” EX1004, Abstract. Kim discloses detecting gestures and taking responsive
`
`action across two modes: a “selected mode” “that allows displaying some kind of
`
`display target data outputted while a specific application selected at a user's request
`
`runs in the mobile device,” including “a web page display mode,” and a “zoom
`
`mode” in which the device can perform a “zooming function” in response to further
`
`inputs. Id. [0041], [0043]. Kim describes moving across these modes by, first, in a
`
`“selected mode,” detecting “a long press event” for “activating a zoom mode” (id.,
`
`[0031], [0049]) and transitioning to zoom mode, then, in zoom mode, detecting “a
`
`hold event” for “execut[ing] a continuous zoom-in function” (id., [0053], [0061]);
`
`and, then, while still in zoom mode, detecting “a double tap event” and “returning
`
`to an initial state with a basic display size after a zoom-in or zoom-out action in a
`
`zoom mode” (id., [0031]). Kim discloses a separate algorithm for detecting user
`
`gestures in selected mode and zoom mode.
`
`1.
`
`User gestures detected via “interrupt” in “selected mode” (i.e.,
`non-zoom mode)
`40. With reference to Figure 2a, Kim discloses detection of user gestures
`
`and responsive functions in “selected mode” upon detection of an “interrupt,” which
`
`Kim “refers to a signal or instructions, inputted through at least one of a touch screen
`
`IPR2024-01091
`
`- 15 -
`
`Patent No. 10,936,114
`
`Page 18 of 51
`
`

`

`
`
`and a special function key by a user, to stop the execution of a running application.”
`
`EX1004, [0048], FIG. 2a. First, in response to detecting “a long press input,” Kim
`
`discloses “activating the zoom mode.” Id., [0049]. Alternatively, upon detection of
`
`“an interrupt [that] is not a request for the activation of the zoom mode,”
`
`“perform[ing] some action in connection with an interrupt.” Id., [0048]-[0050]. But,
`
`to be clear, Kim discloses this as a category, not a specific user input or specific
`
`corresponding action. Id. Kim describes three examples of actions other than
`
`activation of zoom mode in response to interrupts that are not a long press:
`
`(a) “display[ing] the requested menu” in response to an unspecified input that
`
`indicates “a request for a specific menu,” (b) “launch[ing] and enabl[ing] one of an
`
`application and a function which is assigned to the selected icon” in response to an
`
`unspecified input that indicates “a selection input for a specific icon,” and
`
`(c) entering “toggle mode” in response to a “double tap.” EX1004, [0044]-[0045],
`
`[0050], [0089].
`
`User gestures in “zoom mode”
`2.
`41. Figures 2b and 7 illustrate detection of user gestures and responsive
`
`functions in zoom mode. EX1004, [0046]; see also id., [0093] (describing FIG. 7 as
`
`illustrating “a process of executing a zooming function depending on a user gesture
`
`while a zoom mode is activated”). Specifically, Kim explains that after the mobile
`
`IPR2024-01091
`
`- 16 -
`
`Patent No. 10,936,114
`
`Page 19 of 51
`
`

`

`
`
`device activates the zoom mode it “waits for a user input, especially based on a
`
`gesture.” Id., [0051]; see also id., FIG. 2a (211).
`
`42. Upon detecting a user gesture in the zoom mode, the mobile device
`
`“determines the type of an input event,” including whether the gesture is “a move
`
`event,” “a hold event,” or “a double tap event.” EX1004, [0053] (emphasis added);
`
`see id., FIG. 2b (hold event, 231; double tap, 241). That is, Kim teaches—while in a
`
`zoom mode—performing a different function depending on the type of gesture—
`
`e.g., whether the gesture is “a move event,” “a hold event,” on one hand or “a double
`
`tap event” on the other hand. See, e.g., id., FIG. 2b (231, 241), FIG. 7.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`EX1004, FIG. 2b.
`
`43. Figure 7 illustrates “perform[ing] a continuous zooming function”
`
`through “a hold event” at steps 741, 743, and 745 (id., [0061]–[0063], [0095],
`
`IPR2024-01091
`
`- 17 -
`
`Patent No. 10,936,114
`
`Page 20 of 51
`
`

`

`[0104]–[0106]); and “returning to an initial state with a basic display size” via “a
`
`double tap event” at step 750 (id., [0031]; see id., [0106]).
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`EX1004, FIG. 7.
`
`44. Separately, “thereafter, a user can invoke a double tap event on the
`
`screen.” Id., [0106]; see also id., FIG. 2b (241). “Then, in response to the double tap
`
`event, the mobile device displays target data with a given basic magnification value,
`
`as indicated by the reference number 750.” EX1004, [0106]; see also id., FIG. 2b
`
`(243), [0064].
`
`45. Finally, Kim explains that, while in zoom mode, if the detected user
`
`gesture does not “correspond[] to” a “move event,” a “hold event,” or “a double