as) United States
`a2) Patent Application Publication 10) Pub. No.: US 2013/0212541 Al
` Dolencetal. (43) Pub. Date: Aug. 15, 2013
`
`
`
`US 20130212541A1
`
`(54) METHOD, A DEVICE AND A SYSTEM FOR
`RECEIVING USER INPUT
`
`(52) U.S. CI.
`CPC voeeeccecccesseessccnsrseesnees GO6F 3/04883 (2013.01)
`USPC woes cseeeceeceeseesesenesseseeeeeceenenes 715/863
`
`(75)
`
`Inventors: Andre Dolenc, Espoo (FI); Erkki
`Riekkola, Vantaa (FI)
`
`(73) Assignee: NOKIA CORPORATION,Espoo (FI)
`
`(21) Appl. No.:
`
`13/701,367
`
`(22)
`
`PCTFiled:
`
`Jun. 1, 2010
`
`(86) PCT No.:
`§ 371 (c)(1),
`(2), (4) Date:
`
`PCT/FT10/50445
`
`Apr. 25, 2013
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`GO6F 3/0488
`
`(2006.01)
`
`(57)
`
`ABSTRACT
`
`The invention relates to a method, a device and system for
`receiving user input. User interface events are first formed
`from low-level events generated by a user interface input
`device such as a touch screen. The user interface events are
`modified by forming information on a modifier 5 for the user
`interface events such as time and coordinate information. The
`events and their modifiers are sent to a gesture recognition
`engine, where gesture information is formed from the user
`interface events and their modifiers. The gesture information
`is then used as user input to the apparatus. In other words, the
`gestures may not be 10 formed directly from the low-level
`events of the input device. Instead, user interface events are
`formed from the low-level events, and gestures are then rec-
`ognized from these user interface events.
`
`660
`
`:
`
`:
`
`S Applications
`
`
`
`
`650
`Process Ulevents with modifiers, and
`Gesture
`LL
`Recognizers|delivergesture events to applications
`
`Gesture
`Engine
`
`640
`LL
`
`630
`LO
`
`Low-Level
`Events
`
`620
`
`610
`XY
`
`Rules on how gestures can take and lose
`control of events
`
`Basic Ul Events:
`
`Touch (down), Release, Move, Hold
`
`,
`;
`Driver events after processing
`
`Driver events: Up, Down, Drag
`
`1
`
`APPLE 1013
`
`APPLE 1013
`
`1
`
`

`

`Patent Application Publication
`
`Aug. 15, 2013 Sheet 1 of 12
`
`US 2013/0212541 Al
`
`110
`
`120
`_/
`
`130
`
`140
`
`150
`_/
`
`160
`_/
`
`
`
`Receive user
`interface event
`
` | F
`
`orm modifier
`information
`
`|
`
`
`
`Form gesture
`information
`
`Use gesture
`information as
`
`
` user input
`
`Fig. 1
`
`
`
`
`
`Receive low-
`level event
`
` ! F
`
`orm user
`interface event
`using low-level
`event
`
`!
`
`2
`
`

`

`Patent Application Publication
`
`258
`
`US 2013/0212541 Al
`
`Aug. 15,2013 Sheet 2 of 12
`
`240 25
`
`3
`
`

`

`Patent Application Publication
`
`Aug. 15,2013 Sheet 3 of 12
`
`US 2013/0212541 Al
`
`Evenis Functions
`
`Touch down
`Basic interaction element. Default behaviour: Indicates which
`object has been touched(visible, haptic, audio feedback)
`Basicinteraction element: By default performs the default action
`forthe object
`Basic inleracltion element By defaull makes louched object or
`Move
`whole canvasfollow the movement
`Touch down + Touch release
`Touch down: 1) Default
`2) While canvasor contentis moving: Stops ongoing movement.
`Touchrelease: 1) Default
`2) After movements and stopping of he movementno actionis
`performed.
`Touch down: Default
`Hold: Gives indication (visible, haptic, audio) that something is
`appearing and after the defined timeout
`1) opens item specific menu for touched object;
`2) activates editing modein (text) viewers and brings cursor
`visible into the touchedposition.
`Double tap
`Two consecutive touch down +
`Zoom toggle (zoom in/zoom out)
`release sequensesinto the same
`place within a set time limit
`
`
`
`- 805
`
`_-307
`
`
`
`the snap pointor end of content.
`
`_-301
`
`303
`
`Touch release
`
`Touch down + Hold
`
`
`
`
`
`Events
`
`Touch down + Move
`
`Functions
`
`Touch down: Default
`Move (before the holding time for long press is completed):
`1) Default: The content, control handle or whole canvasfollows movement
`2) Speed scrolling: The speed of the scrolling is controlled by movement
`3) Organise mode:The object selected with touch downfollows movement.
`The possible drop location is indicated by moving objects accordingly or by
`some otherindication.
`Stap: The movementis stopped.
`
`(any
`direction)
`
`
`Touch releaseafter
`Touch release:
`dragging
`1) Default: No default action is performed for the touched object after the
`whole contenthas been moved by dragging. Cancels the action when dragged
`outside of the allowed content area before drop.
`2) Speed scrolling: Scrolling stops
`3) Organise mode:Places the dragged object into its indicated location.
`Touch down: Default
`
`Touch down + Move +
`
`Touchrelease
`
`Move: The contentor whole canvasfollows the finger movements.
`Release: The contentcontinuesits movementwith the direction and speed
`that it had at the momentof touch release until stopped manually or reaches
`
`Fig. 3b
`
`4
`
`

`

`Patent Application Publication
`
`Aug. 15,2013 Sheet 4 of 12
`
`US 2013/0212541 Al
`
`410Com
`
`0
`
`poe
`
`Fig. 4a
`
`100
`
`Release
`
`a N 470
`
`450
`\
`
`~ 480
`
`Fig. 4b
`
`5
`
`

`

`Patent Application Publication
`
`Aug. 15,2013 Sheet 5 of 12
`
`US 2013/0212541 Al
`
`511
`
`512
`
`516
` Fig. 5c
`
`513
`
`514
`
`515
`
`Fig. 5a
`
`6
`
`

`

`Patent Application Publication
`
`Aug. 15,2013 Sheet 6 of 12
`
`US 2013/0212541 Al
`
`Applications
`
`Process Ul events with modifiers, and
`Gesture
`LL
`Recognizers|delivergesture events to applications
`
`Rules on how gestures can take and lose
`control of events
`
`Basic Ul Events:
`
`Touch (down), Release, Move, Hold
`
`Driver events after processing
`
`Driver events: Up, Down, Drag
`
`Fig. 6
`
` 640
`
`
`\ Gesture
`Engine
`
`630
`LO
`
`620
`
`Low-Level
`
`Events
`
`610
`YO
`
`7
`
`

`

`Patent Application Publication
`
`Aug. 15, 2013 Sheet 7 of 12
`
`US 2013/0212541 Al
`
`GN
`730
`
`uolreoddy
`
`
`
`
`
`
`
`(Orsss
`(Necesses
`
`
`Flick Stop:[3|Flick Stop }
`Flick Stop
`
`>OD
`Panning
`730
`Tap
`pi
`Tap
`Tap
`SC
`
`=o=
`
`Panning |720
`
`
`
`LOT
`
`/
`
`o3
`
`) A
`
`
`
`
`
`Flick
`
`Initial state:
`Flick Stop is
`disabled.
`
`723
`
`Touch Ul event
`will not
`activate any
`recognizer
`
`Slow enough
`Moveactivates
`Panning
`
`
`715
`Flick Stop
`
`
`uoneojddy
` Tap
`
`Faster Moveis ignored
`by Panning, reaching
`Flick. Flick forces
`Panning to exit.
`
`Appis notified
`that Panning
`ended, and Flick
`starts.
`
`Fig. 7b
`
`ppis informed, Move is
`consumed, and does not
`reach Flick.
`
`
`
`asencenrentencened
`Tap
`Panning
`
`Flick
`
`Flick Stop
`notifies app,
`and disables
`itself.
`
`f 116
`
`i
`Flick Stop
`l
`Tap
`= 8||
`iS 3
`c|}
`Panning
`
`
`Flick enables
`Flick Stop.
`
`
`
`
`
`aB
`
`
`
`8
`
`

`

`Patent Application Publication
`
`Aug. 15,2013 Sheet 8 of 12
`
`US 2013/0212541 Al
`
`812,1¢ Driver UP eventor Release Ul event (context should
`
`makeit obvious which one is meant)
`Driver DOWN eventor Touch UI event (ditto)
`
`
`
`814
`Driver DRAG eventor Move UI event, in any direction
`—
`(ditto)
`815
`Hold Ul event
`816| po|Anydriveror Ul event
`
`Fig. 8a
`
`Leo od99 59>
`AA A
`A
`
`Touch time ee
`
`
` 820
`
`Hold time
`
`Y
`Hold Ul
`
`0
`
`t
`
`even
`
`a
`
`Fig. 8b
`
`9
`
`

`

`Patent Application Publication
`
`Aug. 15, 2013 Sheet 9 of 12
`
`US 2013/0212541 Al
`
`
`
`970
`
`Send
`gesture(s) to 7
`
`application
`
`
`
` 980 Use gestures |~
`
`for input
`
`
`
`
`
`910
`
`920
`
`930
`
`940
`
`945
`
`Receiving
`hardware
`events
`
`Filter data
`
`: l
`
`Receive low- __
`evel driver
`data
`
`Receive user _
`interface
`events
`
`t
`
`Receive
`modifiers
`
`
`
`948
`
`|-~
`
`952
`
`
`
`
`
`v
`
`962
`
`Recognize
`gesture #2
`
`Fig. 9
`
`Form hold
`events
`
` Vv
`Recognize
`start of
`gesture #2
`
` Vv
`Recognize
`start of
`Gesture #1
`
`
`
`v
`
`Recognize
`gesture #1
`
`
`
`10
`
`

`

`Patent Application Publication
`
`Aug. 15,2013 Sheet 10 of 12
`
`US 2013/0212541 Al
`
`!!
`
`!!!!!!!!!!!!! '!
`
`I!
`I!!
`!!
`
`!!I!!!!!!I!!I
`
`:-InTouchTime
`
`‘InHoldTime_U
`
`‘Dispatch
`
`‘InTouchArea
`
`Touch Timer >0
`
`Consume event
`
`Touch Timer =0
`
`
`4
`:
`
`
`= 1.
`Initialize timers
`r 2,
`
` Hold Timer >0
`
`Produce TOUCH |TouchArea>0O
`NA.
`
`t 1
`
`Consume event
`
`Produce
`RELEASE
`
`Resistive UP and__»
`Suppress Timer > 0?
`
`Leee ee ee ee ee eee ee ee ee ee eee ee eee ee eee ee eee 1
`
`DRAG
`
`1.
`
`Produce
`
`MOVE
`
`Matches
`HOLD
`
`criteria?
`
`\
`
`Does not
`
`match HOLD — ‘Dispatch
`
`criteria?
`
`Hold Timer
`
`Initialize
`
`.
`
`‘-InHoldTime_U
`
`Fig. 10b
`
`11
`
`1 Capacitive UP, or
`Resistive UP and
`i
`-0?
`DRAG,UP Suppress Timer=0?
`
`
`
`1I
`
`Initialize timer
`
`11
`
`

`

`Patent Application Publication
`
`Aug. 15,2013 Sheet 11 of 12
`
`US 2013/0212541 Al
`
`- DRAGinside Area
`
`
`—> |F1.
`Consume event |} ‘InTouchTime
`
`MUP
`
`Produce TOUCH
`
`|
`
`-DRAG outside Area
`-Capacitive UP
`
`-Resistive UP
`al
`—_,
`
`.
`
` -TOUCH timeout
`——>
`.
`Clearall timers
`HOLD timer = 0?
`
`Clear TOUCH timer
`
`Produce TOUCH ee TOUCHtimeout and
`
`HOLD timer > 0?
`
`:InHoldTime_ U
`—
`
`:InTouchArea
`
`Resistive UP and
`HOLD timer = 0?
`
`Fig. 10c
`
`- DRAG inside Area
`
`— #1.
`Consume event
`:
`MPR RRRRRR
`
`
`
`>
`
`‘-InTouchArea
`
`-DRAG outside Area
`-UP event
`
`|e]
`
`Fig. 10d
`
`DOWN
`
`Suppress Timeout
`
`—_
`
`
`Clear Suppress Timer —
`
`.
`Clear SuppressTimer — “INIT
`
`Rename as DRAG
`
`.
`
`Produce RELEASE
`
`Fig. 10e
`
`12
`
`12
`
`

`

`Patent Application Publication
`
`Aug. 15,2013 Sheet 12 of 12
`
`US 2013/0212541 Al
`
`DOWN — :InHoldTime_D
`
`
`Area —>FL. Consume event | -InHoldTime_U
`
`
`
`
`
`
`DRAG inside HOLD
`
`DRAG outside HOLD
`
`Area, or
`
`Capacitive UP
`
`—_
`
`1.
`
`Clear HOLD timer
`
`|
`
`j
`
`‘Dispatch
`
`
`
`Resistive UP ——F1. Consumeevent — ‘InHoldTime_D
`
`Init Suppress
`
`.
`
` Timer
`Restart Timer
`
`HOLD timeout ————————
`
`.
`
`Produce HOLD
`
`|
`
`:INHoldTime_U
`
`Fig. 10f
`
`UP
`—*||:InHoldTime_U
`
`Timeouts
`
`Produce RELEASE
`
`“INIT
`
`Clear timers
`
`
`risenounaree?HOLD
`
`[insideHOLDarea?_|->HOLDarea? ‘InHoldTime_U
`
` Consume event
`Clear Suppress
`
`
`
`area? Produce MOVEfs.Producemove ‘Dispatch
`
`
`
`Clear HOLD timer
`
`Fig. 10g
`
`13
`
`13
`
`

`

`US 2013/0212541 Al
`
`Aug. 15, 2013
`
`METHOD, A DEVICE AND A SYSTEM FOR
`RECEIVING USER INPUT
`
`BACKGROUND
`
`[0001] Advances in computer technology have madeit pos-
`sible to manufacture devices that are powerful in terms of
`computing speed andyet easily movable or even pocket-sized
`like the contemporary mobile communication devices and
`multimedia devices. There are also ever more advanced fea-
`
`tures and software applications in familiar homeappliances,
`vehicles for personal transportation and even houses. These
`advanced devices and software applications require input
`methods and devices that are capable enoughfor controlling
`them. Perhaps for this reason, touch input in forms of touch
`screens and touch pads has recently become more popular.
`Currently, such devices are able to replace more conventional
`input means like the mouse and the keyboard. However,
`implementing the input needs of the most advanced software
`applications and user input systems may require more than
`just a replacement of the conventional input means.
`[0002] There is, therefore, a need for solutions that improve
`the usability and versatility of user input means such as touch
`screens and touch pads.
`
`SUMMARY
`
`[0003] Now there has been invented an improved method
`and technical equipment implementing the method, by which
`the above problems maybeatleast alleviated. Various aspects
`of the invention include a method, an apparatus, a server, a
`client and a computer readable medium comprising a com-
`puter program stored therein, which are characterized by
`what is stated in the independent claims. Various embodi-
`ments of the invention are disclosed in the dependentclaims.
`[0004]
`In one example embodiment, user interface events
`(higher-level events) are first formed from low-level events
`generated by a user interface input device such as a touch
`screen. The user interface events may be modified by forming
`information ona modifier for the userinterface events such as
`time and coordinate information. The user interface events
`and their modifiers are sent to a gesture recognition engine,
`where gesture information is formed from the user interface
`events and possibly their modifiers. The gesture information
`is then used as user input to the apparatus. In other words,
`according to one example embodimentthe gestures may not
`be formed directly from the low-level events of the input
`device. Instead, higher-level events i.e. user interface events
`are formed from the low-level events, and gestures are then
`recognized from these user interface events.
`[0005] According to a first aspect, there is provided a
`methodfor receiving user input, comprising receiving a low-
`level event from a user interface input device, forming a user
`interface event using said low-level event, forming informa-
`tion on a modifier for said user interface event, forming ges-
`ture information from saiduser interface event and said modi-
`fier, and using said gesture information as user input to an
`apparatus.
`[0006] According to an embodiment, the method further
`comprises forwarding said user interface event and said
`modifier to a gesture recognizer, and forming said gesture
`information by said gesture recognizer. According to an
`embodiment, the method further comprises receiving a plu-
`rality of user interface events from a user interface input
`device, forwarding said user interface events to a plurality of
`
`gesture recognizers, and formingat least two gestures by said
`gesture recognizers. According to an embodiment, the user
`interface eventis one of the group of touch, release, move and
`hold. According to an embodiment, the method further com-
`prises forming said modifier from at least one of the group of
`time information, area information, direction information,
`speed information,and pressure information. According to an
`embodiment, the method further comprises forming a hold
`user interface event in responseto a touch input or key press
`input being held in place for a predeterminedtime, and using
`said hold event in forming said gesture information. Accord-
`ing to an embodiment, the method further comprises receiv-
`ing at least two distinct user interface events from a multi-
`touch touch input device, and using said at least two distinct
`user interface events for forming a multi-touch gesture.
`According to an embodiment, the user interface input device
`comprisesat least one of the group ofa touch screen, a touch
`pad, a pen, a mouse,a haptic input device, a data glove anda
`data suit. According to an embodiment, the user interface
`event is one of the group of touch down,release, hold and
`move.
`
`[0007] According to a second aspect, there is provided an
`apparatus comprising at least one processor, memory includ-
`ing computer program code, the memory and the computer
`program code configured to, with the at least one processor,
`cause the apparatus to receive a low-level event from a user
`interface input module, form a user interface event using said
`low-level event, form information on a modifier for said user
`interface event, form gesture information from said userinter-
`face event and said modifier, and use said gesture information
`as user input to an apparatus.
`
`[0008] According to an embodiment, the apparatus further
`comprises computer program code configured to cause the
`apparatus to forwardsaid user interface event and said modi-
`fier to a gesture recognizer, and form said gesture information
`by said gesture recognizer. According to an embodiment, the
`apparatus further comprises computer program code config-
`ured to cause the apparatus to receive a plurality of user
`interface events from a user interface input device, forward
`said user interface events to a plurality of gesture recognizers,
`and form at least two gestures by said gesture recognizers.
`According to an embodiment, the user interface event is one
`ofthe group oftouch, release, move and hold. According to an
`embodiment, the apparatus further comprises computer pro-
`gram code configured to cause the apparatus to form said
`modifier from at least one of the group of time information,
`area information, direction information, speed information,
`and pressure information. According to an embodiment, the
`apparatus further comprises computer program code config-
`ured to cause the apparatus to form a hold user interface event
`in response to a touch input or key press input being held in
`place for a predetermined time, and use said hold event in
`forming said gesture information. According to an embodi-
`ment, the apparatus further comprises computer program
`code configured to cause the apparatus to receive at least two
`distinct user interface events from a multi-touch touch input
`device, and usesaid at least two distinct user interface events
`for forming a multi-touch gesture. According to an embodi-
`ment, the user interface module comprises at least one of the
`group of a touch screen, a touch pad, a pen, a mouse,a haptic
`input device, a data glove and a data suit. According to an
`embodiment, the apparatus is one of a computer, portable
`communication device, a home appliance, an entertainment
`
`14
`
`14
`
`

`

`US 2013/0212541 Al
`
`Aug. 15, 2013
`
`device such as a television, a transportation device such as a
`car, ship or an aircraft, or an intelligent building.
`[0009] According to a third aspect, there is provided a sys-
`tem comprising at least one processor, memory including
`computer program code, the memory and the computer pro-
`gram code configured to, with the at least one processor,
`cause the system to receive a low-level event from a user
`interface input module, forming a user interface event using
`said low-level event, form information on a modifier for said
`user interface event, form gesture information from said user
`interface event and said modifier, and use said gesture infor-
`mation as user input to an apparatus. According to an embodi-
`ment, the system comprisesat least two apparatuses arranged
`in communication connection to each other, wherein a first
`apparatus of said at least two apparatuses is arranged to
`receive said low-level event and a second apparatusofsaid at
`least two apparatusesis arranged to form said gesture infor-
`mation in response to receiving a user interface event from
`said first apparatus.
`[0010] According to a fourth aspect, there is provided an
`apparatus comprising, processing means, memory means,
`and meansfor receiving a low-level event from a userinter-
`face input means, means for forming a user interface event
`using said low-level event, means for forming information on
`a modifier for said user interface event, means for forming
`gesture information from said user interface event and said
`modifier, and meansfor using said gesture information as user
`input to an apparatus.
`[0011] Accordingto a fifth aspect, there is provided a com-
`puter program product stored on a computer readable
`medium and executable in a data processing device, the com-
`puter program product comprising a computer program code
`section for receiving a low-level event from a user interface
`input device, forming a user interface event using said low-
`level event, a computer program code section for forming
`information on a modifier for said user interface event, a
`computer program codesection for forming gesture informa-
`tion from said user interface event and said modifier, and a
`computer program codesection for using said gesture infor-
`mationas user input to an apparatus. According to an embodi-
`ment, the computer program product is an operating system.
`
`DESCRIPTION OF THE DRAWINGS
`
`Inthe following, various embodiments ofthe inven-
`[0012]
`tion will be described in more detail with reference to the
`appendeddrawings, in which
`[0013]
`FIG. 1 shows a methodfor gesture based user input
`according to an example embodiment;
`[0014]
`FIG. 2 shows devices and a system arranged to
`receive gesture based user input according to an example
`embodiment;
`[0015]
`FIGS. 3a and 34 show different example gestures
`composed of touch user interface events;
`[0016]
`FIG. 4a showsa state diagram of a low-level input
`system according to an example embodiment;
`[0017]
`FIG. 46 showsa state diagram of a user interface
`event system generating user interface events and comprising
`a hold state according to an example embodiment;
`[0018]
`FIGS. 5a, 55 and 5c show examples of hardware
`touch signals such as micro-drag signals during a hold user
`interface event;
`[0019]
`FIG. 6 shows a block diagram oflevels of abstrac-
`tion of a userinterface system and a computer program prod-
`uct according to an example embodiment;
`
`FIG. 7a shows a diagram of a gesture recognition
`[0020]
`engine according to an example embodiment;
`[0021]
`FIG. 76 shows a gesture recognilion engine in
`operation according to an example embodiment;
`[0022]
`FIGS. 8a and 8b show generation of a hold user
`interface event according to an example embodiment;
`[0023]
`FIG. 9 shows a methodfor gesture based user input
`according to an example embodiment; and
`[0024]
`FIGS. 10a-10g show state and event diagrams for
`producing user interface events according to an example
`embodiment.
`
`DESCRIPTION OF THE EXAMPLE
`EMBODIMENTS
`
`Inthe following, several embodimentsof the inven-
`[0025]
`tion will be described in the context of a touch user interface
`and methods and devices for the same. It is to be noted,
`however, that the invention is not limited to touch user inter-
`face. In fact, the different embodiments have applications
`widely in any environment where improvementsofuser inter-
`face operations are required. For example, devices with a
`large touch screen such as e-booksand digital newspapers or
`personal computers and multimedia devices such as tablets
`and tables may benefit from the use of the invention. Like-
`wise, user interface systems such as navigation interfaces of
`various vehicles, ships and aircraft may benefit from the
`invention. Computers, portable communication devices,
`home appliances, entertainment devices such as televisions,
`and intelligent buildings may also benefit from the use of the
`different embodiments. The devices employing the different
`embodiments may comprise a touch screen, a touch pad, a
`pen, a mouse,a haptic input device, a data glove or a datasuit.
`Also, three-dimensional input systems e.g. based on haptics
`mayuse the invention.
`[0026]
`FIG. 1 shows a methodfor gesture based user input
`according to an example embodiment. At stage 110, a low-
`level event is received. The low-level events may be generated
`by the operating system of the computer as a response to a
`person using an input device such as a touch screen or a
`mouse. The user interface events may also be generated
`directly by specific user input hardware, or by the operating
`system as a response to hardware events.
`[0027] At stage 120, at least one user interface event is
`formed or generated. The user interface events may be gen-
`erated from the low-level events e.g. by averaging, combin-
`ing, thresholding, by using timer windowsorbyusingfilter-
`ing, or by any other means. For example, two low-level events
`in sequence maybeinterpreted as a user interface event. User
`interface events may also be generated programmatically for
`example from otheruser interface events or as a response to a
`trigger in the program. The user interface events may be
`generated locally by using user input hardware or remotely
`e.g. so that the low-level events are received from a remote
`computer acting as a terminal device.
`[0028] At stage 130, at least one user interface event is
`received. There may be a plurality of user interface events
`received, and user interface events may be combined
`together, split and grouped together and/or used as such as
`individual user interface events. The user interface events
`
`may be received from the same device e.g. the operating
`system, or the user interface events may be received from
`another device e.g. over a wired or wireless communication
`connection. Such another device may be a computeracting as
`
`15
`
`15
`
`

`

`US 2013/0212541 Al
`
`Aug. 15, 2013
`
`a terminal device to a service, or an input device connected to
`a computer, such as a touch pad or touch screen.
`[0029] Atstage 140, modifier informationfor the userinter-
`face event
`is formed. The modifier information may be
`formed by the operating system from the hardware events
`and/or signals or other low-level events and data, or it may be
`formed by the hardware directly. The modifier information
`may be formedat the same time withthe user interface event,
`or it may be formed before or after the user interface event.
`The modifier information may be formed by using a plurality
`of lower-level events or other events. The modifier informa-
`tion may be commonto a numberofuser interface events or
`it may be different for different user interface events. The
`modifier information may comprise position information
`such as a point orarea ontheuserinterface that was touched
`or clicked, e.g. in the form of 2-dimensionalor 3-dimensional
`coordinates. The modifier information may comprise direc-
`tion information e.g. on the direction of movement, drag or
`change of the point of touch or click, and the modifier may
`also comprise information on speed of this movement or
`change. The modifier information may comprise pressure
`data e.g. from a touch screen, and it may comprise informa-
`tion on the area that was touched, e.g. so that it can be
`identified whether the touch was made bya finger or by a
`pointing device. The modifier information may comprise
`proximity data e.g. as an indication of how close a pointer
`device or a finger is from a touch input device. The modifier
`information may comprise timing data e.g. the time a touch
`lasted, or the time between consecutive clicks or touches, or
`clock event information or other timerelated data.
`
`[0030] At stage 150, gesture information is formed from at
`least one user interface event andthe respective modifier data.
`The gesture information may be formed by combining a
`numberof user interface events. The user interface event or
`
`events and the respective modifier data are analyzed by a
`gesture recognizer that outputs a gesture signal whenever a
`predetermined gesture is recognized. The gesture recognizer
`may be a slate machine, or it may be based on pattern recog-
`nition ofother kind, or it may be a program module. A gesture
`recognizer may be implementedto recognize a single gesture
`or it may be implemented to recognize multiple gestures.
`There may be one or more gesture recognizers operating
`simultaneously,in a chain or partly simultaneously andpartly
`in chain. The gesture may be, for example, a touch gesture
`such as a combination of touch/tap, move/drag and/or hold
`events, and it may require a certain timing (e.g. speed of
`double-tap) or range or speed of movementin order to be
`recognized. The gesture mayalso berelative in nature,thatis,
`it may not require any absolute timingsor ranges or speeds,
`but may dependontherelative timings, ranges and speeds of
`the parts of the gesture.
`[0031] At stage 160, the gesture information is used as user
`input. For example, a menu option maybe triggered when a
`gesture is detected, or a change in the modeor behaviorof the
`program maybeactuated. The user input may be received by
`one or more programsorby the operating system, or by both.
`The behaviorafter receiving the gesture maybe specific to the
`receiving program. The receiving of the gesture by the pro-
`gram maystart even before the gesture has been completed so
`that the program can preparefor action orstart the action as a
`response to the gesture even before the gesture has been
`completed. At the same time, one or more gestures may be
`formed and used by the programs and/or the operating sys-
`tem, and the control of the programs and/or the operating
`
`system may happen in a multi-gesture manner. The forming
`of the gestures may take place simultaneously or it may take
`place in a chain sothatfirst, one or more gestures are recog-
`nized, and after that other gestures are recognized. The ges-
`tures may comprise single-touch or multi-touch gestures, that
`is, they may comprise a single point of touchorclick, or they
`may comprise multiple points of touch or click. The gestures
`may besingle gestures or multi-gestures. In multi-gestures,
`two or more essentially simultaneous or sequential gestures
`are used as user input. In multi-gestures, the underlying ges-
`tures maybe single-touch or multi-touch gestures.
`[0032]
`FIG. 2 shows devices and a system arranged to
`receive gesture based user input according to an example
`embodiment. The different devices may be connected via a
`fixed network 210 suchasthe Internetor a local area network;
`or a mobile communication network 220 such as the Global
`
`System for Mobile communications (GSM) network, 3rd
`Generation (3G) network, 3.5th Generation (3.5G) network,
`4th Generation (4G) network, Wireless Local Area Network
`(WLAN), Bluetooth®, or other contemporary and future net-
`works. Different networks are connected to each other by
`means of a communication interface 280. The networks com-
`prise network elements such as routers and switches to handle
`data (not shown), and communication interfaces such as the
`base stations 230 and 231 in orderfor providing access for the
`different devices to the network, and the basestations 230,
`231 are themselves connected to the mobile network 220 via
`a fixed connection 276 or a wireless connection 277.
`
`[0033] There may be a numberof servers connected to the
`network, and in the example of FIG. 2a are shown a server
`240 for offering a network service requiring user input and
`connectedto the fixed network 210, a server 241 for process-
`ing user input received from another device in the network
`and connectedto the fixed network 210, and a server 242 for
`offering a network service requiring user input and for pro-
`cessing user input received from another device and con-
`nected to the mobile network 220. Some ofthe above devices,
`for example the computers 240, 241, 242 may be such that
`they make up the Internet with the communication elements
`residing in the fixed network 210.
`[0034] There are also a numberof end-user devices such as
`mobile phones and smart phones 251, Internet access devices
`(Internet tablets) 250 and personal computers 260 of various
`sizes and formats. These devices 250, 251 and 260 can also be
`made of multiple parts. The various devices may be con-
`nected to the networks 210 and 220 via communication con-
`nections such as a fixed connection 270, 271, 272 and 280 to
`the internet, a wireless connection 273 to the internet 210, a
`fixed connection 275 to the mobile network 220, and a wire-
`less connection 278, 279 and 282 to the mobile network 220.
`The connections 271-282 are implemented by means of com-
`munication interfacesat the respective ends of the communi-
`cation connection.
`
`FIG. 26 shows devices for receiving user input
`[0035]
`according to an example embodiment. As shownin FIG.28,
`the server 240 contains memory 245, one or more processors
`246, 247, and computer program code 248 residing in the
`memory 245 for implementing, for example, gesture recog-
`nition. The different servers 241, 242, 290 may contain at
`least these same elements for employing functionality rel-
`evant to each server. Similarly, the end-user device 251 con-
`tains memory 252, at least one processor 253 and 256, and
`computer program code 254 residing in the memory 252 for
`implementing, for example, gesture recognition. The end-
`
`16
`
`16
`
`

`

`US 2013/0212541 Al
`
`Aug. 15, 2013
`
`user device mayalso have at least one camera 255 for taking
`pictures. The end-user device may also contain one, two or
`more microphones 257 and 258 for capturing sound. The
`different end-user devices 250, 260 may contain at least these
`same elements for employing functionality relevant to each
`device. Some end-user devices may be equipped witha digital
`camera enabling taking digital pictures, and one or more
`microphones enabling audio recording during, before, or
`after taking a picture.
`[0036]
`It needs to be understood that different embodi-
`ments allow different parts to be carried out in different ele-
`ments. For example, receiving the low-level events, forming
`the user interface events, receiving the user interface events,
`forming the modifier information and recognizing gestures
`maybe carried outentirely in one user device like 250, 251 or
`260, or receiving the low-level events, forming the user inter-
`face events, receiving the user interface events, forming the
`modifier
`information and recognizing gestures may be
`entirely carried out in one server device 240, 241, 242 or 290,
`or receiving the low-level events, forming the user interface
`events, receiving the user interface events, forming the modi-
`fier information and recognizing gestures may be carried out
`across multiple user devices 250, 251, 260 or across multiple
`network devices 240, 241, 242, 290, or across user devices
`250, 251, 260 and network devices 240, 241, 242, 290. For
`example, low-level events may be received in one device, the
`user interface events and the modifier information may be
`formed in another device and the gesture recognition may be
`carried out in a third device. As another example, the low-
`level events may be received in one device, and formed into
`user interface events together with the modifier information,
`and the user interface events and the modifier information
`may be used ina seconddeviceto form the gestures and using
`the gestures as input. Receiving the low-level events, forming
`th