as) United States
`a2) Patent Application Publication 10) Pub. No.: US 2010/0103118 Al
`(43) Pub. Date: Apr. 29, 2010
`
`Townsendet al.
`
`US 20100103118A1
`
`(54) MULTI-TOUCH OBJECT INERTIA
`SIMULATION
`
`(75)
`
`Inventors:
`
`Reed L. Townsend, Kirkland, WA
`(US); Xiao Tu, Sammamish, WA
`(US); Bryan D. Scott, Bothell, WA
`(US); Todd A. Torset, Woodinville,
`WA (US); Kenneth W. Sykes,
`Oakton, VA (US); Samir S.
`Pradhan,Bellevue, WA (US);
`Jennifer A. Teed, Redmond, WA
`(US)
`
`Correspondence Address:
`MICROSOFT CORPORATION
`ONE MICROSOFT WAY
`
`REDMOND, WA 98052 (US)
`
`(73) Assignee:
`
`MICROSOFT CORPORATION,
`Redmond, WA (US)
`
`(21) Appl. No.:
`
`12/258,439
`
`(22)
`
`Filed:
`
`Oct. 26, 2008
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`(2006.01)
`GO6F 3/041
`(52) US. CMe coeccccccssssssssssssssssvseseesessessssasisssssenes 345/173
`
`(57)
`
`ABSTRACT
`
`Theinertia system provides a commonplatform and applica-
`tion-programming interface (API) for applications to extend
`the input received from various multi-touch hardware devices
`to simulate real-world behavior of application objects. To
`movenaturally, application objects should exhibit physical
`characteristics such as elasticity and deceleration. When a
`user lifts all contacts from an object, the inertia system pro-
`vides additional manipulation events to the application so that
`the application can handle the events as if the user wasstill
`moving the object with touch. The inertia system generates
`the events based ona simulation ofthe behaviorofthe objects.
`If the user moves an object into another object, the inertia
`system simulates the boundary characteristics of the objects.
`Thus, the inertia system provides more realistic movement
`for application objects manipulated using multi-touch hard-
`ware and the API provides a consistent feel to manipulations
`across applications.
`
`210
`
`220
`
`240
`Input
`230
`Changes
`250
`
`eet
`Inputs from
`
`
`
`
`
`Manipulation
`
`
`
`Application/Control
`Hardware
`
`
`System
`Interface
`
` Manipulation
`Events
`
`260
`
`
`
`
` 270
`
`Inertia Inputs
`and Driving Timer
`
`Inertia System
`
`Inertia Events
`290
`
`APPLE-1006
`
`1
`
`APPLE-1006
`
`

`

`Patent Application Publication
`
`Apr. 29, 2010 Sheet 1 of 7
`
`US 2010/0103118 Al
`
`100
`
`Manipulation System
`110
`
`Interface
`
`Manipulation
`Processor(s)
`
`Input
`Transformation
`
`Simulation
`
`Hardware
`Interface
`
`Application
`
`Figure I
`
`2
`
`

`

`Patent Application Publication
`
`Apr. 29, 2010 Sheet 2 of 7
`
`US 2010/0103118 Al
`
`210
`
`240
`
`Input
`Changes
`230
`
`
`Inputs from
`
`ee
`Manipulation
`Application/Control Po System
`Hardware
`
`
`Interface
`
`
`
`Manipulation
`
`
`Events
`
`260
`
`
`270
`
`Inertia Inputs
`
`and Driving Timer
`
`
`
`280
`Inertia System
`
`
`
`
`Inertia Events
`290
`
`Figure 2
`
`3
`
`

`

`Patent Application Publication
`
`Apr. 29, 2010 Sheet 3 of 7
`
`US 2010/0103118 Al
`
`310
`
`Figure3
`
`4
`
`

`

`Patent Application Publication
`
`Apr. 29, 2010 Sheet 4 of 7
`
`US 2010/0103118 Al
`
`
`
`Application
`Input Loop
`
`
`
`410
`
`Receive Low-Level Touch
`Input
`
`420
`
`Identify Touched Application
`Object
`
`430
`
`Send Touch Input/Object
`ID to Manipulation API
`
`440
`
`Receive Manipulation Event
`
`450
`
`Handle Manipulation Event
`
`460
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`WaitforNextTouchInput :
` Next Input
`
`
`Received?
`
`Figure 4
`
`5
`
`5
`
`

`

`Patent Application Publication
`
`Apr. 29, 2010 Sheet 5 of 7
`
`US 2010/0103118 Al
`
`
`
`
` 510
`
`Create Processor
`
`
` Add Contact to
`Manipulation Processor
`
`
`Manipulation API
`
`Receive Touch Input/
`Application Object ID
`
`Identify Manipulation
`Processor
`
`Processor Exists?
`
`Contact Added?
`
`530
`
`Remove Contact from
`Manipulation Processor
`
`
`
`Figure 5
`
`6
`
`

`

`Patent Application Publication
`
`Apr. 29, 2010 Sheet 6 of 7
`
`US 2010/0103118 Al
`
`Application
`Inertia Loop
`
`6 10
`
`Determine User Released
`Object
`
`Initialize Inertia System
`
`620
`
`630
`
`: Call Inertia Processing :
`
`640
`
`650
`
`660
`
`
` 680
`
`Receive Inertia Event
`
`Handle Inertia Event
`
`670
`
`Wait for Timer
`
`Complete?
`
`Y
`
`Figure 6
`
`7
`
`

`

`Patent Application Publication
`
`Apr. 29, 2010 Sheet 7 of 7
`
`US 2010/0103118 Al
`
`Inertia Processing
`
`710
`
`Receive Initial Parameters
`
`720
`
`730
`
`740
`
`Initialize Simulation
`
`Receive Process Call
`
`Simulate Movement
`
`
`
`760
`Movement
`Y
`Set Complete
`Complete?
`
`
`Fire Inertia Event
`
`Figure 7
`
`©Oo
`
`8
`
`

`

`US 2010/0103118 Al
`
`Apr. 29, 2010
`
`MULTI-TOUCH OBJECT INERTIA
`SIMULATION
`
`BACKGROUND
`
`[0001] A tablet PC, or pen computer, is a notebook or
`slate-shaped mobile computer, equipped with a touch screen
`or graphics tablet/screen hybrid technology that allows the
`user to operate the computer with a stylus, digital pen, or
`fingertip instead of a keyboard or mouse. Tablet PCs offer a
`more natural form of input, as sketching and handwriting are
`a much more familiar form of input than a keyboard and
`mouse, especially for people who are new to computers.
`Tablet PCs can also be more accessible because those whoare
`
`physically unable to type can utilize the additional features of
`a tablet PC to be able to interact with the electronic world.
`
`may expect to be able to “toss”a file from one side of the
`desktop to another by flicking his/her finger. This type of
`movementis not supported by current multi-touch applica-
`tions, which would expect the userto drag his/her finger from
`one side of the screen all the way to the other. Even if an
`application provides support for this type ofmovement, other
`applications could not benefit from it and thus application
`authors would have to repeat the work ofthe first application
`author to offer the same functionality in their applications.
`
`SUMMARY
`
`[0005] The inertia system provides a commonplatform and
`application-programminginterface (API) for applications to
`extend the input received from various multi-touch hardware
`devices to simulate real-world behavior of objects. The
`manipulations received by the application only describe the
`movement of an object based on the movement of contacts
`with the multi-touch hardware. However, to move naturally,
`objects should also exhibit physical characteristics such as
`elasticity and deceleration. When a userlifts all contacts from
`an object, the inertia system provides additional manipulation
`events to the application so that the application can handle the
`events as if the user wasstill moving the object with touch.
`However, the inertia system actually generates the events
`based ona simulation ofthe behaviorofthe objects. Ifthe user
`moves an object into another object, the inertia system sends
`manipulation events based on the boundary characteristics of
`the objects. Thus, the inertia system provides morerealistic
`movement for application objects that a user manipulates
`using multi-touch hardware and the API provides a consistent
`feel to manipulations across applications.
`[0006] This Summary is provided to introduce a selection
`of concepts in a simplified form that are further described
`below in the Detailed Description. This Summary is not
`intended to identify key features or essential features of the
`claimed subject matter, nor is it intended to be used to limit
`the scope of the claimed subject matter.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0002] Multi-touch (or multitouch) denotes a set of inter-
`action techniquesthat allow computerusers to control graphi-
`cal applications using multiple fingers or input devices(e.g.,
`stylus). Multi-touch implementations usually include touch
`hardware(e.g., a screen, table, wall, and so on) and software
`that recognizes multiple simultaneous touch points. Multi-
`touch stands in contrast to traditional touch screens (e.g.,
`computer touchpad, ATM, shopping kiosk) that only recog-
`nize one touch point at a time. Multi-touch hardware can
`sense touches using heat, finger pressure, high capture rate
`cameras, infrared light, optic capture, tuned electromagnetic
`induction, ultrasonic receivers,
`transducer microphones,
`laser rangefinders, shadow capture, and other mechanisms.
`Manyapplications for multi-touch interfaces exist and appli-
`cation designers and users are proposing even more. Some
`uses are individualistic (e.g., Microsoft Surface, Apple
`iPhone, HTC Diamond). As a new input method, multi-touch
`offers the potential for new user experience paradigms.
`[0003] An application cannot use multi-touch hardware
`without an interface for the application software to receive
`information from the multi-touch hardware. Unfortunately,
`each multi-touch hardware device includes its own propri-
`etary interface and application authors must have specific
`knowledgeof a hardware device to write software that works
`with the device. For example, a multi-touch hardware pro-
`FIG. 1 is a block diagram that illustrates compo-
`[0007]
`vider may provide a kernel-mode driver and a user-mode
`nents of the inertia system, in one embodiment.
`application interface through which user-mode software
`[0008]
`FIG. 2 is adata flow diagram thatillustrates atypical
`applications can communicate with the multi-touch hardware
`operating environmentof the inertia system and the flow of
`to receive touch information. An application author writes
`data between components, in one embodiment.
`software that communicates with the user-mode application
`[0009]
`FIG. 3 is a display diagram thatillustrates an appli-
`interface, but the application author’s software works only
`cation object manipulated by user touch, in one embodiment.
`with that multi-touch hardware. A computer user with a dif-
`[0010]
`FIG. 4 is a flow diagram that illustrates the input
`ferent multi-touch hardware device cannot use the applica-
`loop processing of a multi-touch application using theinertia
`tion author’s software unless the application author produces
`system to handle manipulation events, in one embodiment.
`a different version ofthe software that operates correctly with
`
`[0011] FIG. 5isaflow diagram thatillustrates the process-
`the computer user’s device. This produces a very limited
`ing of the inertia system when the system receives touch
`potential market for application authors, reduces the incen-
`input, in one embodiment.
`tive to write applications supporting multi-touch interactions,
`[0012]
`FIG.6is a flow diagram thatillustrates the process-
`and keeps the cost of the most popular devices high for which
`ing of a multi-touch application using the inertia system to
`the greatest numberof applicationsis available.
`process inertia events, in one embodiment.
`[0004] Another problem is thedifficulty for applications to
`[0013]
`FIG. 7 is a flow diagram thatillustrates the process-
`determine a user’s intentions based on touch input received
`ing of the simulation component of the inertia processing
`from multi-touch hardware. Touch input may be received as a
`system, in one embodiment.
`list of coordinates where the hardware senses touch input at
`any given time. Each application has to include software to
`interpret the coordinates and determine the user’s intention.
`In addition,the user’s intention may extend beyondthe actual
`touch input received. The user may expect virtual objects to
`behave howtheydo in the physical world. For example, a user
`
`DETAILED DESCRIPTION
`
`[0014] The inertia system provides a commonplatform and
`APIfor applications to extend the input received from various
`multi-touch hardware devices to simulate real-world behav-
`
`9
`
`

`

`US 2010/0103118 Al
`
`Apr. 29, 2010
`
`ior ofobjects. For example, real world objects do not typically
`stop moving when a user stops pushing them, but rather
`exhibit some inertia and keep moving until friction slows
`them finally to a stop. In some embodiments, the touch input
`first goes through a process to interpret the movementof one
`or more contacts as manipulations. Manipulations map more
`directly to user intentions than do individual touch inputs and
`add support for basic transformationofobjects using multiple
`touch contacts. An application can use manipulationsto sup-
`port rotating, resizing, and translating multiple objects (e.g.,
`photos) at
`the same time. The manipulations may be
`described as two-dimensional (2D) affine transforms that
`contain rotation, scale (e.g., zoom), and translation (e.g., pan)
`information.
`
`information. For
`work together to provide touch input
`example, the operating system may provide a commondriver
`model for multi-touch hardware manufacturers to provide
`touch information for their particular hardware. The operat-
`ing system maytranslate touch information received through
`this model
`into window messages (e.g.. WM_TOUCH
`described herein) and pass these messages to the application.
`Thus, the hardware interface 110 may involve the coordina-
`tion of the hardware, a hardware driver, and an operating
`system layer. Theresult is a series of messagesto the inertia
`system that identify a particular contact (e.g., touch of a
`finger), and the coordinates of the contact over time. For
`example, the operating system may provide a message when
`a new contact is set down on the multi-touch hardware, a
`message each time the contact moves, and a message when
`[0015] Each touch of the multi-touch hardwareis called a
`the contactis lifted away from the multi-touch hardware.
`contact. For example, when a usersets his/her finger on the
`multi-touch hardware, moveshis/her finger around, andlifts
`[0019] One or more manipulation processors 120 use the
`his/her finger, that series of events is a single contact. For
`input transformation component 130 to interpret movement
`example, if the user moves two contacts closer together or
`of each contact associated with a particular application
`further apart, the system may determinethat theuseris scal-
`object. The manipulation processor 120 may determinethat a
`ing (e.g., zooming into or out from) an object. As another
`useris using multiple contacts to perform a single action. For
`example, if the user moves multiple contacts in a circular
`example, a user could touchaphoto with all five fingers ofone
`hand andtwist his/her hand to indicate an intention to rotate
`motion, then the system may interpret the movement as a
`rotation of an object. Each application can define objects that
`the photo. The manipulation processor 120 receives five sepa-
`are relevantdifferently, so it is up to the application to attach
`rate contacts (one for each finger) and the change in coordi-
`nates of each contact as the user rotates his/her hand. The
`an instance ofthe system (called a manipulation processor) to
`each object that a user can manipulate using touch input
`within the application. For example, a photo browsing appli-
`cation may attach a manipulation processorto each displayed
`photo, so that the user can movethe photos around,scale the
`photos, rotate the photos, and so forth.
`[0016] The manipulations handled by the application only
`describe the movementofan object based on the movementof
`contacts. However, to move naturally, objects should also
`exhibit physical characteristics such as elasticity and decel-
`eration. When a userlifts all contacts from an object, the
`inertia system provides additional manipulation events to the
`application so that the application can handle the events as if
`the user wasstill moving the object with touch. However, the
`inertia system actually generates the events based on a simu-
`lation ofthe behavior of the objects. For example, if the user
`lifted the contacts while the object had a velocity in a particu-
`lar direction, then inertia system continues sending events
`that indicate that the object is moving in that direction, slow-
`ing downovertimeas the object decelerates. Ifthe user moves
`an object into another object, such as the edge of the screen,
`the inertia system sends manipulation events based on the
`boundary characteristics of the objects. For example, if an
`application author defines two objects as being elastic, then
`the two objects may bounce off each other when a user moves
`the objects into each other. Thus, the inertia system provides
`morerealistic movement for application objects that a user
`manipulates using multi-touch hardware and the API pro-
`vides a consistent feel to manipulations across applications.
`[0017]
`FIG. 1 is a block diagram that illustrates compo-
`nents of the inertia system, in one embodiment. The inertia
`system 100 includes a hardware interface 110, one or more
`manipulation processors 120, an input transformation com-
`ponent 130, a simulation component 140, and an application
`interface 150. Each of these components is described in fur-
`ther detail herein.
`
`manipulation processor 120 determines that each contact is
`grabbing the same object and performing the samerotation.
`The system will inform the application that the user rotated
`the object, but the application can ignore whether the user
`used two, five, or any particular numberoffingers or other
`contacts to perform the rotation. This greatly simplifies the
`authoring of the application because the application author
`can handle those types of manipulations that are relevant to
`the application and leave it to the inertia system to interpret
`the meaning of each low-level touch input received from the
`multi-touch hardware.
`
`[0020] The manipulation processor 120 uses the input
`transformation component 130 to make determinations about
`the meaning of received movements ofvarious contacts, both
`alone and in concert. For example,if a user is manipulating a
`photo with two fingers, which creates two corresponding
`input contacts, then the manipulation processor 120 uses the
`input transformation component 130 to determine the mean-
`ing of relative movements between the two contacts. If the
`two contacts move apart, then the input transformation com-
`ponent 130 may determine that the user is scaling the object to
`change the object’s size. If the two contacts rotate, then the
`input transformation component 130 may determinethat the
`useris rotating the object. If the two contacts both slide in a
`particular direction, then the input transformation component
`130 may determine the user is panning the object to a new
`location. Although each type of movementis discussed sepa-
`rately, note that a user can makeall three types of movements
`at the sametime, and the input transformation processor can
`report the overall
`transformation to the application. For
`example, a user can rotate, scale, and pan an object all in one
`motion.
`
`[0021] The simulation component 140 simulates the con-
`tinued movementof an application objectafter the user stops
`touching the object based on initialization parameters and
`constraints defined for the object. An application mayinitial-
`ize the simulation component 140 with the final state of the
`manipulation processor 120 associated with the object. The
`
`10
`
`[0018] The hardware interface 110 communicates with the
`hardware to receive touch contacts and movements. The hard-
`
`ware interface 110 may include several subcomponents that
`
`10
`
`

`

`US 2010/0103118 Al
`
`Apr. 29, 2010
`
`application may also define various characteristics of the
`object, such as how the object’s boundaries should behave.
`The simulation component 140 uses techniques based on
`physics to simulate the behavior of the object for a period
`after the user releases the object. For example, the simulation
`component 140 may continuetofire notifications to the appli-
`cation in the same form as the manipulation events received
`by the application while the user was moving the object. The
`application can then focus on reacting to the movementof the
`object rather than being concerned with whatactions (user or
`physical) caused the object to move. Those ofordinary skillin
`the art will recognize numerous well-known techniques for
`simulating the equivalent physical behaviorofvirtual objects
`in software.
`
`[0022] The application interface 150 communicates with
`the application to receive information and provide manipula-
`tion transformsto the application. The application interface
`150 receives initialization information from the application.
`The initialization information may specify which types of
`transforms the application object supports for a particular
`object and associated manipulation processor as well as ini-
`tialization data for the simulation component 140 when the
`user is no longer movingthe object. For example, someappli-
`cation objects may support scaling but not rotation. The ini-
`tialization information mayalso specify a pivot point of the
`object. The inertia system provides manipulation transforms
`to the application through the application interface. For
`example, when the inertia system receives low-level touch
`input that the system interprets as a recognized transform
`(e.g., arotation), the system fires an eventto notify the appli-
`cation about the manipulation. The application processes the
`manipulation transform to modify the object based on the
`transform. For example,ifthe user rotated the object, then the
`application maystore the new orientation of the object to use
`the next time the application displays the object. As another
`example, if the object continued to rotate after the user
`released it based on calculations ofthe simulation component
`140, then the application maystore the new orientation of the
`object.
`[0023] Thecomputing device on which the system is imple-
`mented mayinclude a central processing unit, memory, input
`devices (e.g., keyboard and pointing devices), output devices
`(e.g., display devices), and storage devices(e.g., disk drives).
`The memory and storage devices are computer-readable
`media that may be encoded with computer-executable
`instructions that implement the system, which means a com-
`puter-readable medium that containsthe instructions. In addi-
`tion, the data structures and message structures maybe stored
`or transmitted via a data transmission medium, such as a
`signal on a communication link. Various communication
`links may be used, suchas the Internet, a local area network,
`a wide area network, a point-to-point dial-up connection, a
`cell phone network, and so on.
`[0024] Embodiments ofthe system may be implemented in
`various operating environments that include personal com-
`puters, server computers, handheld or laptop devices, multi-
`processor systems, microprocessor-based systems, program-
`mable consumerelectronics, digital cameras, network PCs,
`minicomputers, mainframe computers, distributed comput-
`ing environments that include any of the above systems or
`devices, and so on. The computer systems maybe cell phones,
`personal digital assistants, smart phones, personal computers,
`programmable consumerelectronics, digital cameras, and so
`on.
`
`[0025] The system maybe described in the general context
`of computer-executable instructions, such as program mod-
`ules, executed by one or more computers or other devices.
`Generally, program modules include routines, programs,
`objects, components, data structures, and so on that perform
`particular tasks or implementparticular abstract data types.
`Typically, the functionality of the program modules may be
`combinedor distributed as desired in various embodiments.
`
`FIG. 2 isadata flow diagram thatillustrates atypical
`[0026]
`operating environmentof the inertia system and the flow of
`data between components, in one embodiment. A multi-touch
`hardware device produces inputs 210 through a hardware
`interface. For example, the hardware maysendthe inputs 210
`to an operating system through a software driver provided by
`the hardware manufacturer. The hardware interface provides
`input events 220 to an application 230. For example, an appli-
`cation may inform the operating system that the application
`230 supports multi-touch user input and register to receive
`messagesrelated to multi-touch user input. The application
`230 receives low-level
`touch input
`information as input
`changes 240 and forwardsthe input changes 240 to a manipu-
`lation system 250. For example, the input changes 240 may
`describe each movement of one or more touch contacts with
`
`the hardware using a set of coordinates that indicate each
`contact’s current position and other movement characteris-
`tics. The manipulation system 250 interprets the input
`changes 240 and notifies the application 230 of one or more
`manipulation events 260 that indicate higher-level manipula-
`tions that the user is performing on a displayed object. For
`example, if the movementof the contacts indicates that the
`user intends to rotate the object, the manipulation events 260
`indicate a degree of rotation.
`[0027] Whenthe user is done movingthe object (e.g., when
`the application receives a notification that each contact touch-
`ing an object has been removed from the touch hardware), the
`application 230 sends initialization information 270 to the
`inertia system 280. The inertia system 280 determines a next
`position of the object and providesinertia events 290 similar
`to the manipulation events 260 that the manipulation system
`250 provided when the user was moving the object. The
`application 230 also provides a driving timerto periodically
`call the inertia system 280 to provide the next position of the
`object through inertia events 290. The application 230 pro-
`cesses the inertia events in a way similar to manipulation
`events.
`
`[0028] Although the diagram illustratesthat the application
`first receives touch input and passes the touch input to the
`manipulation system andinertia, in some embodiments, these
`systemsreceive touch input directly from the hardware inter-
`face,
`interpret the touch input, and provides interpreted
`manipulation events to the application. Likewise, the appli-
`cation may not know that a separate inertia system 280 pro-
`vides inertia events after a user stops moving an object with
`touch,but rather may receive events from oneinterface during
`the time the user is moving the object and afterwards when the
`object is moving based on inertia. This represents an alterna-
`tive architecture that provides similar resultant functionality
`but gives the application less control over the processing of
`the input. For example, the application may not be able to
`define individual application objects to which the system
`attaches individual manipulation processors. The RTS plug-
`in described herein is one example of this alternative archi-
`tecture for the system.
`
`11
`
`11
`
`

`

`US 2010/0103118 Al
`
`Apr. 29, 2010
`
`FIG. 3 isa display diagram that illustrates an appli-
`[0029]
`cation object manipulated by user touch, in one embodiment.
`An application may simultaneously display and receive touch
`input for many such objects. For example, an operating sys-
`tem shell application may display one or more document
`objects stored on the user’s computer desktop. In the display
`310, the document object 320 represents a document on the
`user’s desktop that the user wants to drag to the recycle bin
`330 using touch. The user performsa flicking motion of the
`documentobject 320 that results in system processinga first
`manipulation location 340, second manipulation location
`350, and third manipulation location 360. The application
`receives the first manipulation location 340 when the user
`initially touches the document object 320 with one or more
`fingers (i.e., contacts). The application receives the second
`manipulation location 350 as the user slides his/her fingers
`across the screen. The application receives the third manipu-
`lation location when the user lifts his/her fingers from the
`screen. The arrows 365 represent the vectors of the document
`object’s 320 movement.
`[0030] Without inertia, the document object 320 would
`stopat the third manipulation location 360, whichis likely not
`whatthe user intends. The inertia system provides additional
`manipulation locations to the application as if the user was
`still touching and moving the documentobject 320 based on
`the documentobject’s 320 velocity when the userreleases the
`document object 320. The application receives the first iner-
`tia-based manipulation location 370 when the application
`initializes the inertia system and calls the inertia system’s
`processing functionforthefirst time. The application receives
`the second inertia-based manipulation location 380 as the
`application continuesto call the inertia system’s processing
`function. Because the final manipulation location 380 of the
`documentobject 320 is over the recycle bin 330, the applica-
`tion processes the contact between the two objects (e.g., by
`placing the documentobject 320 in the recycle bin 330). In the
`example illustrated, even though the inertia system deceler-
`ates the movementof the documentobject 320, the document
`object 320 is still able to move a fair distance across the
`display 310 based on the high initial velocity of the user’s
`movementof the document object 320 at the start.
`[0031]
`FIG. 4 is a flow diagram that illustrates the input
`loop processing of a multi-touch application using the inertia
`system to handle manipulation events, in one embodiment. In
`block 410, the application receives low-level touch input. For
`example, an operating system or instance ofthe inertia system
`receives touch contact information from multi-touch hard-
`ware and forwards the touch contact information to the appli-
`cation. In block 420, the application identifies the object to
`whichthe input applies. For example, the application may hit
`test the coordinates of the received input by comparing the
`coordinates with the coordinates of each application object
`displayed by the application. If the touch input is within the
`boundaries of a displayed application object, then the appli-
`cation determinesthat the touch input appliesto that object. In
`block 430, the application sends the received touch input and
`the information about the identified application object to a
`manipulationAPIfor invoking the inertia system (see FIG.5).
`For example, the application may create a numeric identifier
`for each application object and pass the numeric identifier to
`the inertia system each time touch input correspondsto that
`object.
`In block 440, the application receives a manipula-
`[0032]
`tion event from the inertia system that describes one or more
`
`manipulations of the identified application object. For
`example, the application may receive an event describing a
`2D affine transform ofthe application object. Note that block
`440 is illustrated serially after block 430 for simplicity of
`illustration. In practice, the application may receive many
`touch input events before the inertia system notifies the appli-
`cation with a manipulation event. There is not necessarily a
`one-to-one mapping of touch input events to manipulation
`events. Because manipulation events represent a higher-level
`interpretation of low-level touch inputs, multiple touch inputs
`may make up a single manipulation event. In block 450, the
`application handles the received manipulation event. For
`example, ifthe recetved manipulation eventis a rotation, then
`the application mayrotate the application object on the screen
`and store the application objects new location for use when
`the application object is displayed again. The inertia system
`frees the application from performingsteps specific to a par-
`ticular multi-touch hardware device or even from knowing
`which hardware device is providing the multi-touch input. In
`addition, the inertia system frees the application from pro-
`cessing individual contact movementand allowsthe applica-
`tion to focus on processing transforms at the application
`object level.
`[0033]
`Inblock 460, the application waits for the next touch
`input. For example, the application may call an operating
`system provided message API, such as GetMessage on
`Microsoft Windows that waits for the next message to be
`delivered to the application’s message queue. In decision
`block 470, if the application receives the next touch input,
`then the application loops to block 410 to process the input,
`else the application loops to block 460 to continue waiting for
`further input. When the application closes, the application
`exits the input loop (not shown).
`[0034]
`FIG. 5 isa flow diagram thatillustrates the process-
`ing of the inertia system when the system receives touch
`input, in one embodiment. In block 505, the system receives
`touch input along with information identifying an application
`object with which the touch input is associated. For example,
`the touch input may include coordinates or other location
`information of one or more touch contacts, and the applica-
`tion object information may include an identifier that the
`application assignedto a particular displayed object that the
`touch input is over on the multi-touch hardware. In block 510,
`the system identifies a manipulation processor associated
`with the application object. In decision block 520, if the
`system hasnot previously associated a manipulation proces-
`sor with the application object, then the system continues at
`block 530, else the system continues at block 540. In block
`530, the system creates a manipulation processor and associ-
`ates it with the application object, then continuesat block 540.
`[0035]
`In de