UNITED STATES PATENT AND TRADEMARK OFFICE
`______________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`______________
`
`APPLE INC.,
`
`Petitioner
`
`v.
`
`SMITH INTERFACE TECHNOLOGIES, LLC,
`
`Patent Owner
`
`______________
`
`IPR2024-01087
`Patent No. 10,656,755
`______________
`
`DECLARATION OF RAJEEV SURATI, Ph.D.
`
`Apple Inc. v. Smith Interface Techs., LLC
`IPR2024-01087 | Smith EX2001 | Page 1 of 34
`
`

`

`TABLE OF CONTENTS
`INTRODUCTION ............................................................................................ 1
`
`PROFESSIONAL BACKGROUND ................................................................ 1
`
`I.
`
`II.
`
`III. COMPENSATION .........................................................................................10
`
`IV. LEGAL CONSIDERATION ..........................................................................10
`
`A. Obviousness ....................................................................................................10
`B. Claim Interpretation ........................................................................................11
`V. Overview of Task & Basis for Opinions ........................................................12
`
`VI. Level of Skill in the Relevant Art ...................................................................13
`
`VII. OVERVIEW OF THE ’755 PATENT............................................................14
`
`A. Independent Claims 1 and 23 .........................................................................16
`VIII. OVERVIEW OF THE PRIOR ART ..............................................................18
`
`A. U.S. Patent Application Publication No. 2011/0119610 (“Hackborn”) ........18
`B. U.S. Patent Application Publication No. 2008/0316183 (“Westerman”) ......21
`IX. Claims 1 and 23 are non-obvious over the Hackborn-Westerman Device
`(HWD) ......................................................................................................................25
`
`A. [1.j] and [23.i]: The HWD Would Not Change Hackborn’s Background In
`Accordance With A Determination That The User Input Has A Magnitude That
`Is Above a Threshold ............................................................................................26
`X. Claims 1 and 23 are non-obvious over the Hackborn2-Westerman-Jobs
`Device (H2WJD) ......................................................................................................30
`
`XI. CONCLUSION ...............................................................................................32
`
`- i -
`
`Page 2 of 34
`
`

`

`I.
`
`INTRODUCTION
`1.
`My name is Dr. Rajeev Surati, Ph.D. I have been retained by Smith
`
`Interfaces Technologies, LLC as an independent expert consultant in this inter partes
`
`review (“IPR”) proceeding before the United States Patent and Trademark Office
`
`(“PTO”).
`
`II.
`
`PROFESSIONAL BACKGROUND
`All of my opinions stated in this Declaration are based on my own
`2.
`
`personal knowledge and professional judgment. In forming my opinions, I have
`
`relied on my knowledge and experience in designing, developing, and researching
`
`the technology referenced in this Declaration.
`
`3.
`
`I am over 18 years of age and, if I am called upon to do so, I would be
`
`competent to testify as to the matters set forth herein. I understand that a copy of
`
`my current curriculum vitae, which details my education and professional and
`
`academic experience, is being submitted as EX2002. The following provides a brief
`
`overview of some of my experience that is relevant to the matters set forth in this
`
`Declaration.
`
`4.
`
`I hold a Doctor of Philosophy (Ph.D.) degree in Electrical Engineering
`
`and Computer Science from the Massachusetts Institute of Technology (“MIT”)
`
`(awarded in 1999) with a Grade Point Average of 5.0/5.0. I obtained a Master of
`
`Science degree in Electrical Engineering and Computer Science from MIT (awarded
`
`IPR2024-01087
`
` 1 -
`
`Patent No. 10,656,755
`
`Page 3 of 34
`
`

`

`in 1995) with a Grade Point Average of 5.0/5.0. I have a Bachelor of Science degree
`
`in Electrical Engineering from MIT (awarded 1992), graduating with a Grade Point
`
`Average of 4.9/5.0.
`
`5.
`
`My Ph.D. thesis was entitled “Scalable Self-Calibrating Technology for
`
`Large Scale Displays.” My master’s thesis was entitled “Practical Partial
`
`Evaluation.” My undergraduate thesis, for which I received MIT EECS’s William
`
`A. Martin thesis prize for best undergraduate thesis, was entitled “A Parallelizing
`
`Compiler based on Partial Evaluation.” I was also awarded the Department of
`
`Energy’s highly-selective Computational Science Fellowship in 1995, which funded
`
`my Ph.D. studies.
`
`6.
`
`In 1989, I began programming embedded systems and, in 1995, built
`
`one of the first publicly available Internet Coke Machines. During that time, I
`
`worked on various user interfaces. For example, one to augment the coke machine
`
`to dispense coke and another that became one of the early ecommerce mechanisms,
`
`using standards-based protocols for tracking user consumption, payments, and drink
`
`stock.
`
`7.
`
`In 1997, while at MIT pursuing my Ph.D., I started a company called
`
`Flash Communications. At Flash, I invented an instant messaging platform focused
`
`on enterprise needs. I had the idea while observing the popularity of the Zephyr
`
`Instant Messaging Service at MIT beginning in 1988. Attracted by Flash’s
`
`IPR2024-01087
`
` 2 -
`
`Patent No. 10,656,755
`
`Page 4 of 34
`
`

`

`
`
`enterprise focus, Microsoft acquired Flash in 1998. With Microsoft, we built both a
`
`client and server product. The basic protocol we invented became the foundation
`
`for the well-known XMPP protocol that was widely used in the mid-2000s among
`
`instant messaging providers. I developed both the client and server products and
`
`dealt with many, if not all, of the issues one might face when implementing user
`
`interfaces for instant messaging systems. In fact, we helped to build Microsoft’s
`
`famous MSN product, which adopted various user interface elements we created.
`
`8.
`
`As a result of this experience, I became heavily involved with User
`
`Interface design on desktop machines, experience that I have continued to develop
`
`by building desktop, mobile, and web-based applications used by millions of people.
`
`9.
`
`After graduating, I joined Microsoft (as was required by Microsoft in
`
`its acquisition of Flash Communications), where I worked on both client and server
`
`technologies related to instant messaging. My work spanned both the Microsoft
`
`Exchange Instant Messaging product that was released in 2000 and MSN Messenger.
`
`10. From 2000 to 2007, I cofounded, led, and sold a business called
`
`photo.net to Namemedia, which is now part of GoDaddy.com. In early 2000,
`
`Photo.net was considered a top 1,000 website and it received large amounts of
`
`traffic. I built the site from its beginning as a single computer installed in a
`
`datacenter to a full rack of computers in that same data center. As a result, I became
`
`very familiar with the careful design and programming one must employ to build
`
`IPR2024-01087
`
` 3 -
`
`Patent No. 10,656,755
`
`Page 5 of 34
`
`

`

`
`
`and maintain systems that enable both engaging user interfaces and experiences at
`
`scale.
`
`11. At photo.net, we prototyped many fundamental Internet community
`
`features, such as photo sharing, social networking, and memberships. The photo.net
`
`system was written on top of Oracle SQL Database and served up many millions of
`
`web pages each day (corresponding to millions of records) with high-volume inserts.
`
`The system also incorporated an ecommerce platform used to bill subscriptions,
`
`track users, etc. I spent seven years running the site, both writing new features driven
`
`by the Oracle SQL database and maintaining tables containing the many millions of
`
`records generated by the site. I also served as the database administrator, which
`
`provided me with extensive operational experience running such a system. I dealt
`
`with everything from database query speedup to the more mundane day-to-day
`
`issues, such as the maintenance of relational databases, including backups. In
`
`addition to the web interface, we also ran a mobile version of the product based on
`
`WAP, which I helped to develop.
`
`12. After developing one of the first photo sharing sites, we further
`
`increased user engagement by designing an innovative photo rating interface. Our
`
`photo rating system became a huge draw for the site. The rating system increased
`
`engagement by four orders of magnitude, ultimately resulting in Sergey Brin,
`
`cofounder of Google, asking photo.net to provide imagery to Google for a screen
`
`IPR2024-01087
`
` 4 -
`
`Patent No. 10,656,755
`
`Page 6 of 34
`
`

`

`
`
`saver application. Joichi Ito, former head of the MIT Media Lab and a famous
`
`internet luminary called it “the best photo sharing site” in 2001.
`
`13.
`
`I also innovated user interface technology at Nexaweb, where I was an
`
`investor and technical advisor. Nexaweb designed User Interfaces that worked well
`
`over the web. At Nexaweb, I spent a great deal of time solving underlying issues
`
`related to implementing a UI framework both in Java and Javascript. We also
`
`innovated a mechanism using server push prior to the AJAX becoming widely
`
`adopted.
`
`14.
`
`In 2004, I started Scalable Display Technologies based on the
`
`technology in my Ph.D. thesis and the related patent (U.S. Patent No. 6,456,339)
`
`entitled “Super-resolution Display.” As part of my Ph.D. thesis, I designed a special
`
`display that allows one to create ultra-high-resolution displays composed of multiple
`
`projectors tiled with a slight overlap. The system uses camera-based feedback to
`
`create an inverse map that drives the system such that a person can only see a
`
`continuous, seamless display. There is no visible bezel or overlap. During this
`
`project, I became familiar with how and why to drive content onto the display. Thus,
`
`I also became familiar with content encoding/decoding, user interfaces for driving
`
`large displays, multimedia content storage, high resolution imagery, networks,
`
`recording, GPUs, storage of content, etc.
`
`IPR2024-01087
`
` 5 -
`
`Patent No. 10,656,755
`
`Page 7 of 34
`
`

`

`
`
`15. At Scalable Display Technologies, I developed the core algorithms and
`
`implemented—with a team—the entire software system, which included the core
`
`user interface, machine vision and embedded components, and a multimedia
`
`playback system. Our first customer was Pixar. Today, the software is primarily
`
`used by the US military and commercial theme parks, such as Disney and Universal.
`
`Scalable is well-known in the industry for having some of the easiest and intuitive
`
`software to use.
`
`16.
`
`In parallel with my work at Scalable, I lectured at MIT on many
`
`subjects, including the Android operating system. In 2008, I worked with a group
`
`of MIT students on the development of mobile applications for Android with Rich
`
`Miner, a cofounder of Android. Since many students created applications involving
`
`Google Maps and localization, I became familiar with these technologies. I also
`
`served as a lecturer and mentor at MIT Play Labs, which was an incubator for
`
`augmented reality (AR) and virtual reality (VR) software for mobile handsets and
`
`headset applications. Notably, in addition to mobile handsets, Android runs on the
`
`Oculus headset. As part of that program, I worked with several startup companies
`
`on mobile applications, including a startup that developed applications for Telegram,
`
`which is a cross-platform, cloud-based instant messaging system.
`
`17. Since 2014, I have worked as an independent consultant for several
`
`companies including NEC, Hitachi, Hi Marley, and Estee Lauder.
`
`IPR2024-01087
`
` 6 -
`
`Patent No. 10,656,755
`
`Page 8 of 34
`
`

`

`
`
`18.
`
`In 2018, I became a senior partner at nCent Labs. In that role, I
`
`consulted on the development of an incentive-market-based platform for block
`
`chains and cryptocurrency. Another part of my work at nCent Labs focused on the
`
`development of SMS messaging applications for the nCent platform. In that part, I
`
`developed a full stack application around the protocol.
`
`19. From 2019 to 2020, I served as a Technical Lead of the Skunkworks at
`
`Hydrow, which is a startup company that develops indoor rowing machines. In this
`
`role, I worked on special projects, including the development of a VR experience
`
`using Magic Leap and Oculus that immersed users in a team-based rowing crew on
`
`a scull using VR cinematography. I leveraged my experience with user interfaces
`
`and Human-Computer Interaction (“HCI”) to develop a system that gives people a
`
`sense of rowing on the water with others, while in reality, they were rowing on a
`
`rowing machine.
`
` We accomplished our goal through a clever use of
`
`cinematography. The result was quite compelling and increased user engagement
`
`with their exercise program.
`
`20.
`
`In 2020, I started a company called Skyline Nav AI Inc. that develops
`
`technology that uses a visual location (i.e., the skyline) to geo-locate the place a
`
`picture was taken. At Skyline, I built and now maintain a full user experience
`
`centered around a workflow for people to snap a picture and use the image’s skyline
`
`and downloaded 3D mode to localize a person. I have written applications for both
`
`IPR2024-01087
`
` 7 -
`
`Patent No. 10,656,755
`
`Page 9 of 34
`
`

`

`
`
`Android and iOS that communicate with a backend server running the machine
`
`vision algorithms.
`
`21. Over the past decade, I have served as a technical consultant and expert
`
`witness on patent litigation matters. In the course of this work, I have provided
`
`consulting services to a wide variety of technology companies, including
`
`BritishTelecom, Apple,
`
`IBM, Philips, Shopify, Zillow, Polaris Powered
`
`Technologies, Amazon, Salesforce, Hitachi, Slack, Harris Teeter, and others.
`
`22. During the course of my career, I was the inventor of the subject matter
`
`for approximately 15 U.S. Patents. I have also received additional patents,
`
`including: U.S. Patent No. 8,817,111, entitled “System and method of calibrating a
`
`display system free of variation in system input resolution,” U.S. Patent No.
`
`8,994,757, “System and method for providing improved display quality by display
`
`adjustment and image processing using optical feedback,” U.S. Patent No.
`
`9,215,455, “System and method of calibrating a display system free of variation in
`
`system input resolution,” U.S. Patent No. 9,369,683, “System and method for
`
`calibrating a display system using manual and semi-manual techniques,” and U.S.
`
`Patent No. 9,497,447, “System and method for color and intensity calibrating of a
`
`display system for practical usage.”
`
`23.
`
`I am also a co-inventor of patented technology related to instant
`
`messaging, including U.S. Patent No. 5,943,478, which describes associated
`
`IPR2024-01087
`
` 8 -
`
`Patent No. 10,656,755
`
`Page 10 of 34
`
`

`

`
`
`technology that I developed related to pop-up, two-way messaging over the Internet.
`
`While at Microsoft, I was an inventor on several patents, including: U.S. Patent No.
`
`6,260,148 relating to methods and systems for message forwarding and property
`
`notifications using electronic subscriptions; and U.S. Patent Nos. 6,415,318 and
`
`6,604,133 relating to inter-enterprise messaging systems using bridgehead servers.
`
`Aspects of these patents relate specifically to messaging and notification technology
`
`in telecommunications systems.
`
`24.
`
`I serve on the advisory boards of several technology companies,
`
`including Paneve, which develops general purpose ASICs coupled with compiler
`
`technology; Nexaweb, which develops real-time web application frameworks using
`
`HTTPS; Antix Labs, which develops compiler technology for a universal gaming
`
`platform; and Permabit, which develops content-addressable storage.
`
`25.
`
`In addition to my above experience, I have published papers on subjects
`
`relating to computing systems, computer network communications, databases, and
`
`other subjects within the realm of electrical and computer engineering. I have also
`
`received several awards for my contributions as an inventor and entrepreneur,
`
`including the Global Indus. Technovator Award 2009 and Laureate of 2009
`
`Computer World Honors Program. My publication and award history are listed in
`
`more detail on my CV.
`
`IPR2024-01087
`
` 9 -
`
`Patent No. 10,656,755
`
`Page 11 of 34
`
`

`

`
`
`III. COMPENSATION
`26. My work on this matter is being billed at my customary rate of $700 an
`
`hour. Also, I am being reimbursed for reasonable expenses I incur in relation to my
`
`services. I have no pecuniary interest in the outcome of this proceeding. I
`
`understand I will be paid regardless of the outcome of any proceeding in which my
`
`work is used.
`
`IV. LEGAL CONSIDERATION
`I am not an attorney. My understanding of the law is based on
`27.
`
`information provided by counsel for Smith Interface Technologies, LLC.
`
`A. Obviousness
`I understand that a claimed invention is obvious and, therefore, not
`28.
`
`patentable if the subject matter claimed would have been obvious to a person of
`
`ordinary skill in the art at the time that the invention of the ’755 Patent (a
`
`“POSITA”), which I have been asked to treat as August 5, 2011.
`
`29.
`
`I understand that a claim can be obvious in view of a single prior art
`
`reference (e.g., via modification of that prior art reference) or multiple prior art
`
`references (e.g., via a combination of two or more prior art references), if such a
`
`modification or combination was within the skill of a POSITA. I understand that
`
`there must be some articulated reasoning with some rational underpinning to support
`
`a conclusion of obviousness. I also understand that to establish a finding of
`
`obviousness one must show that one of ordinary skill in the art would have had a
`IPR2024-01087
` 10 -
`Patent No. 10,656,755
`
`Page 12 of 34
`
`

`

`
`
`motivation to combine the prior art references to produce the claimed invention and
`
`a reasonable expectation that the combination would be successful.
`
`30.
`
` I further understand that exemplary rationales that may support a
`
`conclusion of obviousness include: (1) simply arranging old elements in a way in
`
`which each element performs the same function it was known to perform, and the
`
`arrangement yields expected results, (2) merely substituting one element for another
`
`known element in the field, if the substitution yields no more than a predictable
`
`result, (3) combining elements in a way that was “obvious to try” because of a design
`
`need or market pressure, where there was a finite number of identified, predictable
`
`solutions, (4) that design incentives or other market forces in a field would have
`
`prompted variations in a work that were predictable to a person of ordinary skill in
`
`the art, and (5) that there was some teaching, suggestion, or motivation in the prior
`
`art that would have led a POSITA to modify or combine prior art references to arrive
`
`at the claimed invention.
`
`B. Claim Interpretation
`I understand that a claim term is interpreted according to its ordinary
`31.
`
`and customary meaning as a POSITA would have understood the term in light of the
`
`surrounding claim language, other claims, the specification, and the patent’s
`
`prosecution history, which are referred to as intrinsic evidence. I also understand
`
`that prior art references cited in the patent’s prosecution history are considered
`
`IPR2024-01087
`
` 11 -
`
`Patent No. 10,656,755
`
`Page 13 of 34
`
`

`

`
`
`intrinsic evidence. I further understand that evidence outside the patent and its
`
`prosecution history (e.g., dictionaries and technical articles), may inform the context
`
`in which a POSITA would have understood the claims of a patent. I understand this
`
`ordinary and customary meaning applies absent unique circumstances, such as
`
`where a patent clearly expresses an intent to set forth a special meaning for a term.
`
`V. Overview of Task & Basis for Opinions
`I have been asked to review the ’755 Patent. I have been asked to
`32.
`
`provide opinions related to certain issues from the perspective of a person of ordinary
`
`skill, having knowledge of the relevant art, as of August 5, 2011, and—except where
`
`otherwise noted—the opinions stated in this declaration are from that perspective.
`
`The qualifications and abilities of such a person are described in paragraph 38 below.
`
`33. More particularly, I have been informed that this proceeding is
`
`currently at a preliminary stage during which the Patent Trial and Appeal Board (the
`
`Board) will decide whether the Petition establishes a reasonable likelihood of at least
`
`one claim of the ’755 Patent being invalid. I also understand that the Board may
`
`review the Petition under a stricter “compelling merits” standard that requires the
`
`Petition to make a stronger showing of invalidity; however, my analysis is not
`
`performed using this higher standard. At this preliminary stage, I have been asked
`
`to focus my analysis on the Petition’s proposed invalidity theories for the ’755
`
`Patent’s independent claims 1 and 23.
`
`IPR2024-01087
`
` 12 -
`
`Patent No. 10,656,755
`
`Page 14 of 34
`
`

`

`
`
`34. My opinions are based on my education, training, and experience as
`
`well as items that I reviewed to prepare my opinions, including the Petition, the
`
`documents listed in the Exhibit List included with the Petition, and any other items
`
`that I reference in my below analysis.
`
`35.
`
`I understand that other issues may arise that require further explanation,
`
`and I will provide that explanation if appropriate. As a result, I respectfully reserve
`
`the right to update and supplement this declaration and the information and opinions
`
`provided herein.
`
`VI. Level of Skill in the Relevant Art
`I understand that a patent must be analyzed from the perspective of a
`36.
`
`person of ordinary skill in the art (“POSITA”) as of the time of invention of the
`
`patent, which is typically considered to be the patent’s earliest filing date. While I
`
`do not necessarily agree with every aspect of Dr. Cockburn’s explanation of the level
`
`of ordinary skill in the art, I do not currently believe the aspects of that explanation
`
`I would change or clarify would impact the analysis set forth in this declaration. For
`
`purposes of this declaration, my opinions are therefore provided from the perspective
`
`of the POSITA described by Dr. Cockburn’s declaration (EX1003) as of August 5,
`
`2011, which is produced below:
`
`“at least a bachelor’s degree in computer science, computer
`
`engineering, electrical engineering, human-computer interaction, or a related
`
`IPR2024-01087
`
` 13 -
`
`Patent No. 10,656,755
`
`Page 15 of 34
`
`

`

`
`
`field, and would have had at least two years of relevant work experience in
`
`the design and development of graphical user interfaces (GUIs), human-
`
`computer interfaces, or equivalents thereof. Less work experience may be
`
`compensated by a higher level of education and vice versa.” EX1003, ¶26.
`
`VII. OVERVIEW OF THE ’755 PATENT
`37. The ’755 Patent “generally relate[s] to touch screens.” EX1001, 1:37-
`
`38.
`
`38. Specifically, the ’755 Patent focuses on improving user interfaces for
`
`increasingly complex touchscreen devices. EX1001, Abstract.
`
`39. For example, the claims include, responsive to the magnitude of a
`
`gesture exceeding a threshold, dynamically changing the appearance of a
`
`background—without changing the appearance of foreground objects—displaying
`
`in sequence multiple images based at least in part on the magnitude of the gesture.
`
`EX1001, Cl. 1, 23.
`
`40.
`
`“Gesture magnitude refers to the magnitude of the associated gesture
`
`dynamics. For example, in one embodiment, the gesture magnitude of a swipe
`
`gesture may include the length of the swipe and/or the contact pressure exerted
`
`during the swipe.” EX1001, 67:1–5.
`
`IPR2024-01087
`
` 14 -
`
`Patent No. 10,656,755
`
`Page 16 of 34
`
`

`

`
`
`41.
`
`“[T]he operations associated with a gesture may be carried out as a
`
`function of the associated gesture dynamics,” which “are aspects of a gesture that
`
`may vary without changing the identity of the gesture.” EX1001, 65:18–22.
`
`42. Examples of such aspects include “contact point velocity, contact point
`
`acceleration, contact pressure velocity, contact pressure acceleration, time to
`
`complete gesture, and/or any other aspect which may be associated with a gesture.”
`
`EX1001, 65:22–27.
`
`43.
`
` “[A] selection function may determine one or more presentation
`
`properties of a selection,” “vary one or more presentation properties as a function of
`
`contact pressure,” or “determine one or more fundamental properties of a selection.”
`
`EX1001, 29:30–59.
`
`44.
`
`“[P]ossible presentation properties may include, but are not limited to,
`
`shading, color, contrast, brightness, line style, animation routine, transparency,
`
`pattern, border style, area style, and/or any other visual property that may describe
`
`an object on a display.” EX1001, 29:31–36.
`
`45. For example, the selection (e.g., background of the first user interface)
`
`can dynamically change as a function of user input based on that input exceeding a
`
`threshold (e.g., pressure exerted on the device exceeds a threshold). EX1001,
`
`Abstract, 29:43–46.
`
`IPR2024-01087
`
` 15 -
`
`Patent No. 10,656,755
`
`Page 17 of 34
`
`

`

`46. Figure 22 of the ’755 Patent depicts such as a configuration, in which
`
`the user can “dynamically chang[e] an appearance of a background [blurring] based
`
`at least in part on a magnitude of a contact [pressure].” EX1001, Figure 22.
`
`
`
`
`
`Independent Claims 1 and 23
`A.
`47. The ’755 Patent has two independent Claims 1 and 23.
`
`48. Claim 1 is directed to an “electronic device.” EX1001, 101:10.
`
`49. Claim 23 is directed to a “method.” EX1001, 108:51.
`
`50. Claims 1 and 23 are substantially similar.
`
`51. Claims 1 and 23 each recite “a display, a touch-sensitive surface, one
`
`or more processors, and memory.” Limitations [1.a], [1.b], [1.c], [1.d], [23.a],
`
`[23.b], [23.c], [23.d].
`
`IPR2024-01087
`
` 16 -
`
`Patent No. 10,656,755
`
`Page 18 of 34
`
`

`

`
`
`52. Claim 1 recites “one or more programs, wherein the one or more
`
`programs are stored in the memory and configured to be executed by the one or more
`
`processors, the one or more programs including instructions for.” Limitation [1.e].
`
`53. Claims 1 and 23 then each recite “displaying a first user interface on
`
`the display, wherein the first user interface includes: a background with an
`
`appearance, and one or more foreground objects.” Limitations [1.f], [1.g], [1.h],
`
`[23.e], [23.f], [23.g].
`
`54. Claims 1 and 23 then each recite “while” or “with” “displaying the first
`
`user interface on the display, detecting a first input by a first contact on the touch-
`
`sensitive surface at a location in the first user interface that corresponds to the
`
`background of the first user interface.” Limitations [1.i], [23.h].
`
`55. Claims 1 and 23 then each recite “in response to detecting the first input
`
`by the first contact, in accordance with a determination that the first contact has a
`
`magnitude that is above a threshold, dynamically changing the appearance of the
`
`background of the first user interface without changing an appearance of the one or
`
`more foreground objects in the first user interface.” Limitations [1.j], [23.i].
`
`56.
`
` Claims 1 and 23 then each recite “wherein the dynamic change in the
`
`appearance of the background of the first user interface is based at least in part on
`
`the magnitude of the first contact.” Limitations [1.k], [23.j].
`
`IPR2024-01087
`
` 17 -
`
`Patent No. 10,656,755
`
`Page 19 of 34
`
`

`

`
`
`57. Claims 1 and 23 then each recite “wherein the dynamic change in the
`
`appearance of the background of the first user interface includes displaying in
`
`sequence at least some of a plurality of images based at least in part on the magnitude
`
`of the first contact.” Limitations [1.l], [23.k].
`
`VIII. OVERVIEW OF THE PRIOR ART
`A. U.S. Patent Application Publication No.
`(“Hackborn”)
`58. U.S. Patent Application Publication No. 2011/0119610 is titled “Live
`
`2011/0119610
`
`Wallpaper” and lists as its inventors Dianne K. Hackborn, Michael A. Cleron,
`
`Romain P. Guy, and Joseph M. Onorato. I will refer to this reference as “Hackborn.”
`
`59. Hackborn “relates to components of computer operating systems,
`
`including background desktop components commonly referred to as computer
`
`wallpaper.” EX1004, [0002].
`
`60. Hackborn is directed at systems and methods for “animating” a
`
`traditional wallpaper on the home screen of a mobile computer device to produce a
`
`“live wallpaper.” See, e.g., EX1004, [0009] (“method further includes producing an
`
`animated moving display of the graphical wallpaper representation”).
`
`61. Hackborn accomplishes this by making the traditional wallpaper “an
`
`object that includes executable code” which is executed in response to input and
`
`“cause[s] the wallpaper 104 to change in appearance.” EX1004, [0026].
`
`IPR2024-01087
`
` 18 -
`
`Patent No. 10,656,755
`
`Page 20 of 34
`
`

`

`62. Hackborn’s Figure 1 shows a “general home screen layout,” that
`
`includes “wallpaper 104” and “icons 108.” See, e.g., EX1004, [0026], [0030].
`
`
`
`
`
`63. Hackborn explains that “the wallpaper shows grass 105” and “may
`
`ordinarily look like a traditional wallpaper in the form of a static image.” EX1004,
`
`[0026]. However, this “wallpaper may be active, and in this example may respond
`
`to inputs from a user of the computing device.” Id. (emphasis added).
`
`64.
`
`“In the example shown in FIG. 1, a user is swiping her finger across a
`
`touch-sensitive display screen of the computing device” and “[i]n response, the
`
`blades of grass 105 displayed in the area of the screen swiped by the user are caused
`
`to bend or sway as if contacted by the user's finger during the swipe.” Id., [0028]
`
`(emphasis added).
`
`IPR2024-01087
`
` 19 -
`
`Patent No. 10,656,755
`
`Page 21 of 34
`
`

`

`
`
`65. Hackborn also offers details of the “touch-sensitive display screen,” in
`
`that “[a] touch interface 706 may take a familiar form and may interpret input by a
`
`user on a touchscreen” by “coordinat[ing] with the display controller 704 to identify
`
`what item or items on a display a user is selecting by touching at a particular
`
`location.” EX1004, [0054].
`
`66. Thus, Hackborn discloses a display controller determining what a user
`
`selects—e.g., wallpaper 104 vs. icon 108—based on location of a user input rather
`
`than the type of user input or gesture.
`
`67. Nor does Hackborn limit the wallpaper-moving user inputs to swipes;
`
`instead, Hackborn explicitly refers to a user swiping as an “example.” EX1004,
`
`[0028].
`
`68. As an alternative to a wallpaper always reacting to user inputs,
`
`Hackborn explains that “[o]ther examples of live wallpaper may not be immediately
`
`responsive to user input on the screen of the device” and may instead “become
`
`responsive to direct user input only after a predetermined user input is received by
`
`the device.” EX1004, [0028].
`
`69.
`
`“Thus, in a first mode, user input on a touchscreen may be interpreted
`
`as being directed to icons displayed on a layer over the wallpaper, and in another
`
`mode (which can be switched to using the techniques just described), user input may
`
`be interpreted as being directed to items displayed on the wallpaper.” Id.
`
`IPR2024-01087
`
` 20 -
`
`Patent No. 10,656,755
`
`Page 22 of 34
`
`

`

`
`
`70. Here too, Hackborn does not disclose or suggest that the type of input
`
`impacts whether a user input is interpreted as applying to the wallpaper or an icon;
`
`rather, in one mode, all user inputs are directed to the wallpaper and, in another
`
`mode, all user inputs are directed to icons.
`
`71. Hackborn also notes that “[e]xamples of predetermined user inputs
`
`that may be received by the device to [switch from the icon mode to the wallpaper
`
`mode, i.e.,] cause the live wallpaper to become responsive to user input can include
`
`agitating or shaking the device, selecting an icon, button (not shown in FIG. 1), or
`
`area of the display screen (e.g., a corner of the screen) reserved for activating the
`
`live wallpaper, making a particular movement on or across the display screen,
`
`tapping on the display screen a predetermined number of times within a
`
`predetermined time period, and others.” EX1004, [0028].
`
`72. Hackborn’s only discussion of a tap input is therefore in the context of
`
`a device in the first mode (in which all inputs are to the icon display layer; not the
`
`wallpa