a2, United States Patent
`US 6,323,846 B1
`(10) Patent No.:
`Nov. 27, 2001
`(45) Date of Patent:
`Westermanetal.
`
`US006323846B1
`
`(54) METHOD AND APPARATUS FOR
`INTEGRATING MANUAL INPUT
`
`Sarah A. Douglas and Anant Kartik Mithal, The Ergonomics
`of Computer Pointing Devices (1997).
`
`(75)
`
`Inventors: Wayne Westerman, Wellington, MO
`(US); John G. Elias, Townsend, DE
`(US)
`
`(73) Assignee: University of Delaware, Newark, DE
`(US)
`
`(*) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`US.C. 154(b) by 0 days.
`
`(List continued on next page.)
`
`Primary Examiner—Richard Hjerpe
`Assistant Examiner—Ronald Laneau
`(74) Attorney, Agent, or Firm—Connolly Bove Lodge &
`Hutz LLP
`
`(57)
`
`ABSTRACT
`
`(21) Appl. No.: 09/236,513
`
`(22)
`
`Filed:
`
`Jan. 25, 1999
`
`(60)
`
`Related U.S. Application Data
`Provisional application No. 60/072,509, filed on Jan. 26,
`1998.
`
`Apparatus and methods are disclosed for simultaneously
`tracking multiple finger and palm contacts as hands
`approach,
`touch, and slide across a proximity-sensing,
`compliant, and flexible multi-touch surface. The surface
`consists of compressible cushion, dielectric, electrode, and
`circuitry layers. A simple proximity transduction circuit is
`placed under each electrode to maximize signal-to-noise
`ratio and to reduce wiring complexity. Such distributed
`CSL)
`TMC0 eeecccceeeecccssssssnsnesesceesnsnnnseeeceesnnnees G09G 5/00
`transduction circuitry is economical for large surfaces when
`
`(52)US.C1.ceeeecsesesssessnessnssssesssenseenseenscenseeneesses 345/173
`implemented with thin-film transistor techniques. Scanning
`(58) Field of Search 0... 345/173, 174,
`and signal offset removal on an electrode array produces
`345/178, 184, 156, 158
`low-noise proximity images. Segmentation processing of
`each proximity image constructs a group of electrodes
`corresponding to each distinguishable contact and extracts
`shape, position and surface proximity features for each
`group. Groups in successive images which correspondto the
`same hand contact are linked by a persistent path tracker
`whichalso detects individual contact touchdownandliftoff.
`Combinatorial optimization modules associate each con-
`tact’s path with a particular fingertip, thumb, or palm of
`either hand on the basis of biomechanical constraints and
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`4,734,685
`4,746,770
`4,968,877
`5,189,403
`5,305,017 *
`5,376,948
`
`3/1988 Watanabe.
`5/1988 McAvinney.
`11/1990 McAvinneyetal. .
`2/1993 Franz et al.
`.
`4/1994 Gerpheide wi cceeeenn 345/174
`12/1994 Roberts .
`
`(List continued on next page.)
`OTHER PUBLICATIONS
`
`Subutai Ahmad, “A Usable Real-Time 3D Hand Tracker,”
`Proceedings of the 28” Asilomar Conference on Signals,
`Systems, and Computers—Part 2 (of 2), vol. 2 (Oct. 1994).
`
`contact features. Classification of intuitive hand configura-
`tions and motions enables unprecedented integration of
`typing, resting, pointing, scrolling, 3D manipulation, and
`handwriting into a versatile, ergonomic computer input
`device.
`
`118 Claims, 45 Drawing Sheets
`
`
`ELECTRODE
`SCANNING
`ys
`HARDWARE
`
`j
`PROXIMITY IMAGE
`CALIBRATION AND / 8
`FORMATION
`
`]
`CONTACT
`TRACKING AND
`IDENTIFICATION
`
`vy
`
`
`
`
`
`
`
`
`
`
`
`HAND
`
`FINGER
`EC 12TYPING
`
`
`MOTION
`
`tj SYNCHRONIZATION|
`PENGRIP
`RECOGNIZER
`DETECTOR
`COMPONENT
`
`
`
`
`DETECTOR
`
`
`EXTRACTION
`
`
`
`CHORD MOTION
`RECOGNIZER
`
` p?
`
` 24
`HOST
`HOST
`£
`
`
`
`DISPLAY }—— COMPUTER }#—>
`COMMUNICATION YY
`INTERFACE
`
`SYSTEM
`
`
`
`20
`
`APPLE-1021
`
`1
`
`APPLE-1021
`
`

`

`US 6,323,846 B1
`
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`5,463,388
`5,495,077
`5,530,455
`5,543,590
`5,543,591 *
`5,563,632
`5,565,658
`5,666,113
`5,675,361
`5,689,253
`5,767,457
`5,790,107
`5,821,690
`5,821,930
`5,825,352
`5,880,411
`
`10/1995 Boie etal. .
`2/1996 Milleretal. .
`.
`6/1996 Gillick et al.
`.
`8/1996 Gillespie et al.
`8/1996 Gillespie et al. we 345/173
`10/1996 Roberts .
`10/1996 Gerpheide .
`9/1997 Logan .
`10/1997 Santilli .
`11/1997 Hargreavesetal. .
`6/1998 Gerpheideet al.
`.
`8/1998 Kasseret al.
`.
`10/1998 Martensetal. .
`10/1998 Hansen .
`.
`10/1998 Bissetet al.
`.
`3/1999 Gillespie et al.
`OTHER PUBLICATIONS
`
`Donald L. Fisher et al., “Repetitive Motion Disorders: The
`Design of Optimal Rate—-Rest Profiles,” Human Factors,
`35(2):283-304 (Jun. 1993).
`Masaaki Fukumoto and Yoshinobu Tonomura, “Body
`Coupled Fingering: Wireless Wearable Keyboard,” CHI 97,
`pp. 147-154 (Mar. 1997).
`
`Frederick S. Hillier and Gerald J. Lieberman, Introduction to
`Operations Research (1986).
`
`Robert J. K. Jacob et al., “Integrality and Separability of
`Input Devices,” ACM Transactions on Computer-Human
`Interaction, 1:3-26 (Mar. 1994).
`
`Seonkyoo Lee, “A Fast Multiple-Touch-Sensitive Input
`Device,” Master’s Thesis, University of Toronto (1984).
`
`Francis K.H. Quek, “Unencumbered Gestural Interaction,”
`IEEE Multimedia, 3:36-47 (Winter 1996).
`
`Robert G. Radwin, “Activation Force and Travel Effects on
`Overexertion in Repetitive Key Tapping,” Human Factors,
`39(1):130-140 (Mar. 1997).
`
`Dean Rubine and Paul McAvinney, “Programmable Finger—
`Tracking Instrument Controllers,” Computer Music Journal,
`vol. 14, No. 1 (Spring 1990).
`
`Shumin Zhai et al., “Dual Stream Input for Pointing and
`Scrolling,” Proceedings of CHI °97 Extended Abstracts
`(1997).
`
`* cited by examiner
`
`2
`
`

`

`U.S. Patent
`
`Nov.27, 2001
`
`Sheet 1 of 45
`
`US 6,323,846 B1
`
`ELECTRODE
`SCANNING
`HARDWARE
`
`6
`
`CALIBRATION AND
`PROXIMITY IMAGE
`
` CONTACT
`
`
`
`TRACKING AND
`IDENTIFICATION
`
`8
`
`10
`
`12
`
`
` HAND
`FINGER
`
`
`TYPING
`SYNCHRONIZATION
`MOTION
`PENGRIP
`
`
`RECOGNIZER
`bercoron
`COMPONENT
`
`
`EXTRACTION
`
`
`
`FORMATION
`
`DETECTOR CHORD MOTION
` 22
`
`
`RECOGNIZER
`
`24
`
`DISPLAY
`
`HOST
`COMPUTER
`SYSTEM
`
`20
`
`HOST
`COMMUNICATION
`INTERFACE
`
`FIG. 1
`
`3
`
`

`

`U.S. Patent
`
`Nov.27, 2001
`
`Sheet 2 of 45
`
`US 6,323,846 B1
`
`
`
`4
`
`

`

`U.S. Patent
`
`Nov.27, 2001
`
`Sheet 3 of 45
`
`US 6,323,846 B1
`
`
`
`FIG. 3A
`
`
`
`5
`
`

`

`U.S. Patent
`
`Nov.27, 2001
`
`Sheet 4 of 45
`
`US 6,323,846 B1
`
`
`
`TIME
`
`FIG. 4B
`
`6
`
`

`

`U.S. Patent
`
`Nov.27, 2001
`
`US 6,323,846 B1
`
`Sheet 5 of 45
`
`FIG. 5B
`
`7
`
`

`

`U.S. Patent
`
`Sheet 6 of 45
`
`US 6,323,846 B1
`
`N
`+
`
`h
`+
`
`Nov.27, 2001
`
`aa
`
`ro;
`
`ty
`WLP
`
`M
`AL
`
`h
`NS
`aL) Ss
`
`8
`
`

`

`U.S. Patent
`
`Nov.27, 2001
`
`Sheet 7 of 45
`
`US 6,323,846 B1
`
`
`
`FIG. 7A
`
`
`
`FIG. 7B
`
`9
`
`

`

`U.S. Patent
`
`Nov.27, 2001
`
`Sheet 8 of 45
`
`US 6,323,846 B1
`
`44h75
`
`45hStaPatstatatsHs,ee He
`PNAS
`ehccCL
`HhgoEnEhch
`ihSHatiatsttaar(srte
`rdchdchEhEhLL Lh
`naearatisetaatiatts
`Pohchct)
`ehehchcheeLL
`
`FIG. 8
`
`10
`
`

`

`U.S. Patent
`
`78
`
`Nov.27, 2001
`
`78
`
`Sheet 9 of 45
`
`US 6,323,846 B1
`
`11
`
`11
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 10 of 45
`
`US 6,323,846 B1
`
`
`
`12
`
`12
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 11 of 45
`
`US 6,323,846 B1
`
`DrEaarrrna
`
`0eT
`
` 85
`
`13
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 12 of 45
`
`US 6,323,846 B1
`
`
`
`14
`
`14
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 13 of 45
`
`
`
`15
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 14 of 45
`
`US 6,323,846 B1
`
`POSITIONONSURFACE(YAXIScm)
`VERTICAL
`
`—a1cooOrhf
`
`oO
`
`0
`
`2
`
`16
`14
`12
`10
`8
`6
`4
`HORIZONTAL POSITION ON SURFACE (X AXIS cm)
`
`18
`
`FIG. 14
`
`16
`
`

`

`U.S. Patent
`
`Nov.27, 2001
`
`Sheet 15 of 45
`
`US 6,323,846 B1
`
`a a
`
`—,PO
`
`— oO
`
`
`
`VERTICALPOSITIONONSURFACE(YAXIScm)
`
`00
`
` ifj
`
`
`HORIZONTAL POSITION ON SURFACE xAXIS‘cm)
`
`4
`
`6
`
`8
`
`10
`
`12
`
`FIG. 15
`
`17
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 16 of 45
`
`US 6,323,846 B1
`
`
` CURRENT
`
`PROXIMITY
`IMAGE
`
`240
`
`IMAGE
`SEGMENTATION
`
`
`
`
`
`PATHS FROM
`PARAMETERIZED
`
`
`
`LECTRODE GROUP
`
`CONTACT PATH
`TRACKING
`
`
`PREVIOUS IMAGES
`
`
`
`
`NEW PATHS &
`
`
`UPDATED PATH
`
`PARAMETERS
`
`
`
`FINGER & PALM
`IDENTIFICATION
`
`247
`
`
`HAND
`IDENTIFICATION
`
`
`
`IDENTIFIED
`CONTACT PATHS
`
`HAND POSITION
`ESTIMATION
`
`
`
`
`
`
`
`ESTIMATED HAND &
`FINGER OFFSETS
`
`FIG. 16
`
`18
`
`18
`
`

`

`U.S. Patent
`
`Nov.27, 2001
`
`Sheet 17 of 45
`
`US 6,323,846 B1
`
`GET HAND'S
`IDENTIFIED PATHS
`
`250
`
`
`
`“COMPUTE OFFSETS BETWEEN
`EACH FINGER'S MEASURED AND
`
`DEFAULT POSITIONS
`
`254
`
`255
`
`COMPUTE AVERAGE OF
`OFFSETS WEIGHTED BY
`CONTACT PROXIMITY
`
`ADJUST FILTER POLE TO
`CURRENT IDENTIFICATION
`CONFIDENCE
`
`256
`
`257
`
`COMPUTE WEIGHTED
`AVERAGE OF HAND
`CONTACT VELOCITIES
`
`
`
`AUTOREGRESSIVELY UPDATE HAND
`OFFSET ESTIMATES FROM MEASURED
`OFFSETS AND VELOCITIES
`
`2
`
`88
`
`UPDATE FINGER
`OFFSET ESTIMATES
`
`259
`
`CONVERT ESTIMATED OFFSETS
`TO ABSOLUTE POSITIONS
`
`260
`
`19
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 18 of 45
`
`US 6,323,846 B1
`
`240
`
` CURRENT
`PROXIMITY
`IMAGE
`
`
`
`DIFFUSE
`CURRENT IMAGE
`
`
`SMOOTHED
`PROXIMITY
`
`IMAGE
`
`
`SEARCH FOR
`
`SIGNIFICANT
`
`LOCAL MAXIMA
`
`
`
`LOCAL
`
`MAXIMUM
`PIXELS
`
`267
`
`
`
`FLATTENED
`FINGERTIP
`
`FEEDBACK
`
`252
`
`
`
`ESTIMATED HAND
`
`POSITION OFFSETS
`
`DEFINE SEGMENTATION
`STRICTNESS REGIONS
`
`266
`
`
`
`CONSTRUCT ELECTRODE
`GROUPS AROUND LOCAL
`
`MAXIMUM PIXELS
`270
`GROUPS
`
`COMBINE
`OVERLAPPING
`GROUPS
`
`FIT ELLIPSES TO
`COMBINED
`
`272
`
`
`
`
`PARAMETERIZED
`
`ELECTRODE GROUPS
`FIG. 18
`
`242
`
`20
`
`20
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 19 of 45
`
`US 6,323,846 B1
`
`
`
`FIG. 19
`
`21
`
`21
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 20 of 45
`
`US 6,323,846 B1
`
`15
`
`-20
`
`-15
`
`-10
`
`-5
`
`10
`VERTICALSURFACEPOSITION(cm)
`
`
`
`
`
`
`
`
`-20
`
`-15
`
`10
`5
`0
`-5
`-10
`HORIZONTAL SURFACE POSITION (cm)
`FIG. 20C
`
`15
`
`20
`
`22
`
`22
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 21 of 45
`
`US 6,323,846 B1
`
`290
`
`
`RAW PROXIMITY >
`BACKGROUND?
`
`
`
`REGION?
`
`
`292
`
`SEARCHING
` SEGMENTATION
`
`
`HORIZONTAL?
`
`
`300
`
`GET NEXT ELECTRODEIN
`
`DIRECTION OF SEARCH
`
`(A)
`
`N
`
`REACHED BACKGROUND
`LEVEL EDGE
`
`304
`
`306
`
`N
`
`A
`
`N
`
`310
`
`294
`
`
`SEARCHING
`HORIZONTAL?
`
`312 |
`
`296
`
`
`
`REACHED EDGE
`BETWEEN
`FINGERTIP AND
`
`THUMB OR PALM
`
`
`
`
`
`REACHED EDGE
`
`BETWEEN FINGERS
`
`END
`
`
`
`
`DIST. TO LOCAL MAX
`HORIZ.
`
`
`OR DIAGONAL
`MINIMUM ?
`
`
`314
`
`298
`
`TALL
`HORIZONTAL
`MINIMUM ?
`
`
`
`
`
`300
`
`REACHED EDGE
`BETWEEN PALM
`HEELS
`
`END
`
`FIG. 21
`
`23
`
`23
`
`

`

`U.S. Patent
`
`Nov.27, 2001
`
`Sheet 22 of 45
`
`US 6,323,846 B1
`
`START
`
`PREDICT CURRENT POSITIONS
`OF EXISTING PATHS
`
`320
`
`FOR EACH GROUP
`FIND CLOSEST PATH
`
`322
`
`
`
`
`FOR EACH PATH, FIND
`CLOSEST GROUP WITHIN
`
`TRACKING RADIUS
`
`
`$24
`
`
`
`FORM GROUP-PATHPAIRSIF
` GROUP & ACTIVE PATH ARE
`
`CLOSEST TO ONE ANOTHER
`
`
`ATTEMPT TO PAIR REMAINING
`GROUPS WITH RECENTLY
`
`DEACTIVATED PATHS
`
`326
`
`334
`
`ALLOCATE NEW PATHS FOR ANY
`REMAINING UNPAIRED GROUPS
`
`336
`
`DEACTIVATE ANY REMAINING
`UNPAIRED PATHS
`
`344
`
`UPDATE PATH
`PARAMETERS
`
`346
`
`FIG. 22
`
`24
`
`24
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 23 of 45
`
`US 6,323,846 B1
`
`DEFINE IDENTITY
`ATTRACTORS AT DEFAULT
`CONTACT POSITIONS
`
`350
`
`352
`
`TRANSLATE ATTRACTOR
`TEMPLATE BY ESTIMATED
`HAND OFFSET
`
`COMPUTE MATRIX OF
`DISTANCES FROM EACH PATH
`TO EACH ATTRACTOR
`
`354
`FEATURES OF EACH PATH
`
`
`
`COMPUTE ATTRACTOR
`WEIGHTING FACTORS FROM
`
`396
`
`
`FIND ASSIGNMENT BETWEEN
`PATHS AND ATTRACTORS
`WHICH MINIMIZES SUM OF
`
`WEIGHTED DISTANCES
`
`358
`
` 360
`
`
`
`# FINGER
`
`HAND
`
`ATTRACTORS
`
`
`
`ASSIGNMENTS
`ASSIGNED > 1
`
`
`
`OR<5?
`
`362
`
`ENTATIVE 2
`
`
`UPDATE FINGER
`COUNTS AND
`SUBSETS
`
`VERIFY THUMB
`ASSIGNMENT
`
`368
`
`END
`
`FIG, 23
`
`25
`
`25
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 24 of 45
`
`US 6,323,846 B1
`
`(cm)
`
`VERTICALSURFACEPOSITION
`
`
`0
`
`2
`
`4
`
`14
`12
`10
`8
`6
`HORIZONTAL SURFACE POSITION (cm)
`
`16
`
`18
`
`20
`
`FIG, 24
`
`26
`
`26
`
`

`

`U.S. Patent
`
`Nov.27, 2001
`
`Sheet 25 of 45
`
`US 6,323,846 B1
`
`150
`100
`50
`CONTACT ORIENTATION (degrees)
`FIG. 25A
`
`5
`4
`3
`2
`1
`CONTACT SIZE (NORMALIZED TOTAL PROXIMITY)
`FIG. 25B
`.
`
`6
`
`5
`4
`3
`2
`1
`TOTAL PROXIMITY DIVIDED BY ECCENTRICITY
`FIG. 25C
`
`0
`0
`
`0
`
`10
`
`RIGHT
`
`ORIENTATION
`
`FACTOR
`
`Nw
`
`e
`sO
`2 =
`ze
`
`ac
`T1K®)
`tw E=
`rg
`= LeLu
`oN 4
`YQ
`0
`
`S
`SE 5 1
`Ite
`<tqg
`ao x <x
`ul We
`
`0
`
`0
`
`6
`4
`2
`DISTANCE TO NEAREST NEIGHBOR CONTACT(cm)
`FIG. 25D
`
`8
`
`27
`
`27
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 26 of 45
`
`US 6,323,846 B1
`
`COMPUTE INTER-PATH
`THUMB FACTORS
`
`
`
`
`
`
`
`
`& ORIENTATION FACTORS
`OF INNERMOST AND NEXT
`INNERMOST CONTACT
`
`
`
`
`
`402
`
`
`
`400 COMBINE WITH THUMB SIZE},
`
`
`
`
`
`INNERMOS
`Combined_thumb_fact >
`
`
`ASSIGNED TO
`is_thumb_thresh?
`
`THUMB?
`
`412
`
` SHIFT INNERMOST
`
`
`
`PATH TO THUMB
`
` Combined_thumb_fact
`
`
`ATTRACTOR
` not_thumb_thresh?
`
`
`
`INNERMOS
`
`
`ASSIGNED TO
`
`THUMB?
`
` EXISTING
`
`
`ASSIGNMENTS
` SHIFT INNERMOST
`OK
`
`
`PATHS AWAY FROM
`THUMB ATTRACTOR
`
`
`
`FIG. 26
`
`28
`
`28
`
`

`

`U.S. Patent
`
`Nov.27, 2001
`
`Sheet 27 of 45
`
`US 6,323,846 B1
`
`GET ALL PATHS ASSIGNED
`TO THE GIVEN HAND
`
`430
`
`COMPUTE DISTANCES
`FROM EACH PATH TO
`
`FIND SHORTEST RING
`CONNECTING ALL PATHS
`AND PASSING THROUGH
`
`OTHER PATHS
`EACH ONCE
`EACH PATH
`
`COMPUTE THUMB & PALM
`WEIGHTING FACTORS FOR
`
`432
`
`434
`
`436
`
`PICK INNERMOST|,—
`PATH IN RING
`
`438
`
`442
`
`440
`
`
`
` INNERMOST
`INNERMOS
`
`PATH A PALM
`PATH THUMB ?
`
`
`HEEL?
`
`
`PATHS ABOVETHIS
`
`
`PATHS AT THIS
`PATHS AT THIS
`VERTICAL LEVEL ARE
`VERTICAL LEVEL
`VERTICAL LEVEL
`FINGERTIPS, BELOW
`
`
`
`ARE PALMS
`
`ARE FINGERTIPS
`ARE PALMS
`
`
`
`
`
`444
`
`END
`
`FIG. 27
`
`29
`
`29
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 28 of 45
`
`US 6,323,846 B1
`
`
`
`& OUTER HAND
`
`
`
`PENGRIP
`N
`
`
`
`PARTS TOUCHING CEND)DETECTED ?
`
`PREVIOUSLY
`
`
`
`
`
`
`GET ESTIMATED FINGER&
`PALM POSITIONS FOR
`
`
`LIFTED HAND PARTS
` GET MEASURED POSITIONS &
`
`
`
`SIZES OF TOUCHING FINGERS
`
`
`& PALM HEELS
`488
`
`
`487
`
`COMPUTE KNUCKLE FACTOR
`
`FOR OUTER FINGERS
`
`489
`
`=
`
`COMPUTEINDEX JUTTING
`FACTOR FOR INNER FINGERS
`
`COMBINE FACTORS & FILTER
`WITH OLD COMBINED FACTORS
`
`490
`
`494
`
`495
`
`
`
`SEND PARAMETERS OF
`INNER FINGER PATHS TO
`
`
` HANDWRITING RECOGNIZER
`
`493
`TOUCHING ?
` INNER FINGERS
`
`
`
`SEND STYLUS LIFT SIGNAL TO
`
`HANDWRITING RECOGNIZER &
`
`KNUCKLE/PALM MOTION TO CURSOR
`
`FIG. 28
`
`END
`
`494
`
`30
`
`30
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 29 of 45
`
`US 6,323,846 B1
`
`START
`
`450
`
`
`
`CONTACT
`Y
`
`
`PROXIMITIES
`
`TABILIZED 2
`
`
`N
`
`PATH EXTENSION
`
`452
`
`RETAIN PREVIOUS
`IDENTIFICATIONS VIA
`
`DEFINE & TRANSLATE LEFT &
`RIGHT ATTRACTOR TEMPLATES
`
`453
`
`END
`
`456
`
`PICK FIRST
`CONTOUR
`
`GENERATE PARTITIONING
`CONTOURS
`
`454
`
`TENTATIVELY DIVIDE HAND
`IDENTITIES ACROSS CONTOUR
`
`TENTATIVELY ASSIGN FINGER
`IDENTITIES WITHIN EACH HAND
`
`458
`
`460
`
`
`
`EVALUATE BIOMECHANICAL
`COHERENCE OF PARTITION
`
`462
`
`
`
`
`
`Cost
`
`LOWESTSO— ASLOWESTCOST
`
`
`
`PICK NEXT
`CONTOUR
`
`LAST
`CONTOUR
`2
`
`464
`,
`
`466
`
`RECORD PARTITION
`
`Yo
`
`-4
`
`73
`
`474
`
`PARTITION
`
`CHOOSE
`LOWEST
`COST HAND
`
`ASSIGN FINAL
`CONTACTIDENTITIES
`WITHIN EACH HAND
`
`END
`
`FIG. 29
`
`31
`
`31
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 30 of 45
`
`US 6,323,846 B1
`
`
`
`
`
`
`
`VERTICALSURFACEPOSITION(cm)
`
`~20
`
`—15
`
`-10
`
`-5
`
`0
`FIG. 30A
`
`5
`
`10
`
`15
`
`20
`
`-20
`
`~15
`
`-10
`
`-5
`
`0
`FIG. 30B
`
`5
`
`10
`
`15
`
`20
`
`
`
`
`
`-20
`
`-15
`
`10
`5
`0
`-5
`-10
`HORIZONTAL SURFACE POSITION (cm)
`FIG. 30C
`
`15
`
`20
`
`32
`
`32
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 31 of 45
`
`US 6,323,846 B1
`
`CLUTCHING
`
`
`
`DIRECTIONFACTOR oO_— Oo-
`
`50
`:
`0
`HORIZONTAL VELOCITY OF RIGHT HAND CLUSTER(mm/s)
`FIG. 31A
`
`—_—
`
`HANDEDNESS
`
`FACTOR
`
`2
`0
`-2
`-4
`-6
`-8
`-10
`VERTICAL POSITION OF OUTERMOST FINGER
`RELATIVE TO NEXT OUTERMOST(cm)
`FIG. 31B
`
`0
`-12
`
`1
`
`6
`W or
`ro
`OG
`Og
`> LL
`
`| =
`
`<
`QO
`
`0
`20
`15
`10
`5
`0
`HORIZONTAL SEPARATION BETWEEN PALM CONTACTS(cm)
`FIG. 31C
`
`33
`
`33
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 32 of 45
`
`US 6,323,846 B1
`
`-150
`
`100
`50
`0
`-50
`-100
`ANGLE BETWEEN INNERMOST AND
`NEXT INNERMOST FINGER CONTACTS(degrees)
`FIG. 32
`
`150
`
`
`
`RIGHTINNER
`
`
`
`ANGLEFACTOR
`
`
`
`SEPARATIONFACTOR
`
`INTER-HAND
`
`Seeeel
`
`--4
`
`-2
`
`6
`4
`2
`0
`ESTIMATED HORIZONTAL
`SEPARATION BETWEEN THUMBS(cm)
`FIG. 33
`
`8
`
`10
`
`12
`
`0-
`
`6
`
`34
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 33 of 45
`
`US 6,323,846 B1
`
`GET HAND'S CURRENT
`PATH PARAMETERS &
`ID'S
`
`500
`
`502
`
`504
`
`SUPPRESSIVE
`FINGER VELOCITY
`FILTERING
`
`MEASURE HAND'S
`POLAR VELOCITY
`COMPONENTS
`
`506
`
`MEASURE HAND'S
`TRANSLATIONAL
`VELOCITY COMPONENTS
`
`
`
`MEASURE HAND'S
`
`
`DIFFERENTIAL TILT
`PRESSURE
`
`COMPONENTS
`
`508
`
`DOWNSCALE
`WEAKER
`COMPONENTS
`
`510
`FASTEST COMPONENT
`
`DEAD-ZONE FILTER ALL
`COMPONENTS
`BY FRACTION OF
`
`512
`
`FIG. 34
`
`35
`
`35
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 34 of 45
`
`US 6,323,846 B1
`
`207
`1
`
`206
`G
`
`FIG. 35
`
`36
`
`36
`
`

`

`U.S. Patent
`
`Nov.27, 2001
`
`Sheet 35 of 45
`
`US 6,323,846 B1
`
`524
`
`N
`
`SET RADIAL AND
`ANGULAR VELOCITY
`
`522
`
`TO ZERO
`
` AT LEAST
`2 FINGERS
`
`DOWN?
`END
`528
`
`GET CURRENT AND PREVIOUS
`POSITIONS OF INNERMOST AND
`OUTERMOST TOUCHING FINGERS
`
`526
`
`COMPUTE RADIAL VELOCITY FROM
`CHANGE IN SEPARATION BETWEEN
`INNERMOST AND OUTERMOST
`
`
`
`COMPUTE ROTATIONAL VELOCITY
`
`
`FROM SEPARATION AND CHANGE IN
`ANGLE BETWEEN INNERMOST AND
`OUTERMOST
`
`
`930
`
`531
`
`COMBINE WITH ROTATION AND
`SCALING ABOUT A FIXED POINT
`BETWEEN THUMB AND OTHER FINGERS
`
`534
`532
`
`CHECK FOR
`
`
`AVERAGE
`aoe
`cae
`
`
`
`
`
`DROPPING ?
`A
`
`
`DECELERATION
`
`
`
`END
`
`FIG. 36
`
`37
`
`37
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 36 of 45
`
`US 6,323,846 B1
`
`
`
`
`
`
`ACCEL RATIO =
`CURRENT SPEED/
`PAST AVERAGE
`
`
`
`
`
`
`ACCEL RATIO
`
`> THRESH NEAR
`ONE ?
`
`
`
`540
`
`INIT TRANSLATION
`WEIGHTINGS TO FINGER
`
`544
`
`DECREASE TRANSLATION
`WEIGHTING OF
`RELATIVELY SLOW
`
`FINGERS
`546
`
`PROXIMITIES
`
`
`INCREASE
`
`DECREASE TRANSLATION
`WEIGHTING OF CENTRAL
`FINGERS AS POLAR
`COMPONENT SPEEDS
`
`
`
`
`
`DIRECTION CLOSE TO
`PAST AVERAGE 2?
`
`
`
`
`
`COMPUTE TRANSLATION
` SET
`
`
`
`VELOCITY AS WEIGHTED
`
`TRANSLATION
`AVERAGE OF FINGER
`DECELERATION
`
`VELOCITIES
`
`FLAG
`
`
`
`CLEAR
` PROXIMITY
`
`
`TRANSLATION
`DROPPING ?
`DECELERATION
`
`
`FLAG
`
`548
`
`556
`
`
`
`UPDATE MOVING
`
`
`WINDOW AVERAGE OF
`
`
`TRANSLATION
`
`VELOCITIES
`
`END
`
`560
`
`FIG, 37
`
`38
`
`38
`
`

`

`U.S. Patent
`
`Nov.27, 2001
`
`Sheet 37 of 45
`
`US 6,323,846 B1
`
`END
`
`564
`
`SET TILT & ROLL
`COMPONENTS
`TO ZERO
`
`
`
`
`
`STORE CURRENT PATH
`PROXIMITIES AS
`REFERENCE PROXIMITIES
`
`
`COMPUTE UNWEIGHTED
`AVERAGE OF PATH POSITIONS
`
`570
`
`
`
`COMPUTE RATIOS OF CURRENT
`PROXIMITY TO REFERENCE
`PROXIMITY FOR EACH PATH
`
`572
`
`SET RATIOS LESS
`THAN ONE TO ONE
`
`574
`
`
`
`
`
`COMPUTE AVERAGE OF
`PATH POSITIONS
`WEIGHTED BY PROXIMITY
`RATIOS
`
`576
`
`COMPUTE TILT & ROLL
`COMPONENTS FROM
`DIFFERENCE VECTOR BETWEEN
`WEIGHTED AND UNWEIGHTED
`
`AVERAGES
`
`7
`
`078
`
`END.
`
`FIG. 38
`
`39
`
`39
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 38 of 45
`
`US 6,323,846 B1
`
`
`
` GET HAND'S CURRENT
`PATH PARAMETERS & ID'S
`
`DELETE
`
`ASSOCIATED
`
`
`KEYPRESS
`SEARCH FOR FINGER
`
`
`
`QUEUE
`SUBSETS PRESSED OR
`
`
`ELEMENTS
`RELEASED
`
`
`SIMULTANEOUSLY
`
`
`
`
`
`
`
`MARKER
`PENDING
`?
`
`
`ANY
`
`# FINGER
`
`PRESSES
`
`RELEASES
`
`SYNCED
`YNCED > 22
`
`?
`
`
`
`
`
`
`CLEAR
`SYNC
`MARKER
`
`DELETE
`ASSOCIATED KEY
`QUEUE
`ELEMENTS
`
`
`
`
`PAUSE SENDING OF
`
`ASSOCIATED
`KEYPRESS QUEUE
`
`ELEMS
`
`
`
`OUCHIN
`
`
`OR HALTED TOO
`
`
`LONG ?
`
`FIG. 39A
`
`40
`
`40
`
`

`

`
`
`
`
`
`SIGNIFICANT
`
`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 39 of 45
`
`US 6,323,846 B1
`
`
`
`DELETE ASSOCIATED
`KEYPRESS QUEUE
`ELEMENTS
`
`620
`
`ATERAL MOTION 7?
`
`
`
`
`
`LOOKUP CHORD
`FROM SYNCED
`FINGER ID'S
`
`630
`
`
`
`
`EVENTS ?
`
`RESTING CHORD:
`NO EVENTS
`GENERATED
`
`APPEND CHORD
`TAP EVENTS TO
`COMM QUEUE
`
`
`
`FIG. 39B
`
`41
`
`41
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 40 of 45
`
`US 6,323,846 B1
`
`
`
`GET HAND'S
`EXTRACTED MOTION &
`
`
`
`
`
` # FINGERS
`
`SYNCED
`SUBSET OR
`ALL FINGERS
`
`
`
`660
`
`DISABLE KEY &
`
`658
`SELECT SLIDE CHORD
`FROM SYNCED SUBSET
`OR COMBINATION OF
`FINGERS TOUCHING
`
`IDENTIFIED PATHS
`
`TOUCHING > 2 ?
`THIS HAND
`CHORD TAPS FOR
`
`
`
`
` N
`
`SLIDE MODE
`
`
`
`
`666
`
`
`
`668
`
`LEAVE
`CHORD
`
`DECELERATION
`FLAG SET ?
`
`
`667
`
`SET CURRENT VELOCITY
`COMPONENTSTO PRE-
`LIFTOFF AVERAGE
`
`673
`
`
`
`
`NEW
`
`
`
`SUBSET
`HOLE SUBSE
`
`PRESSED IN
`SLIDING ?
`
`
`
`SYNC ?
`
`
` SELECT NEW SLIDE
`
`CHORD FROM NEW
`
`
`SYNCED SUBSET
`
`
`674
`
`FIG. 40A
`
`42
`
`42
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 41 of 45
`
`US 6,323,846 B1
`
`PICK FIRST SLICE
`DEFINED FOR SELECTED
`
`675
`
`GAIN FUNCTION TO
`MOTION COMPONENTS
`
`694
`
`CHORD
`
`
`
` PICK NEXT SLICE
` APPLY SLICE'S VELOCITY
`
`
`
`PROJECT
`
`
`LAST SLICE
`
`
`VELOCITY COMPONENTS
`
`
`FOR CHORD ?
`INTO SLICE'S SPEED AND
`
`DIRECTION RANGE
`
`
`
`
`INTEGRATE PROJECTED
`VELOCITY COMPONENTS
`
`
`690
`OVER TIME
`
`
`
`
`DISABLE FURTHER
`
`EVENTS FROM ONE-
`
`SHOT SLICE
`
`
`
`# UNITS OF
`
`
`
`LOOKUP SLICE'S
`
`
`KEY/MOUSE/3D RESET OTHER|_gg
`
`
`
`EVENTS
`SLICES
`
`
`INTEGRATORS
`
`
`
`
`
`
`APPEND EVENTS W/ REMOVE INTEGER #|_96
`
`# MOTION UNITS
`
`
`MOTION UNITS FROM
`
`TO COMM QUEUE
`INTEGRATORS
`
`
`
`
`
`FIG. 40B
`
`43
`
`43
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 42 of 45
`
`US 6,323,846 B1
`
`
`
`RETRIEVE KEY LAYOUT
`REGIONS AND SYMBOLS
`
`700
`
`GET CURRENTIDENTIFIED
`PATHS FOR BOTH HANDS
`
`702
`
`
`
`
`
`706
`
`
`
`
`TRANSLATE HAND'S
`FINGERS
`
`KEY REGIONS BY
`
`
`ON SAME HAND
`
`MEASURED HAND
`
`PRESSED IN
`
`OFFSETS
`
`
`SYNC?
`
`
`
`FINGERS
`
`
`NOTHING
`
`
`PARTIALLY
`TOUCHING
`
`
`
`CLOSED ?
`
`SURFACE FOR
`AWHILE?
`
`
`
`
`
`
`RESET KEY
`ADJUST REGIONS IN
`
`
`LAYOUT
`
`EACH FINGER'S
`
`OFFSETS TO
`
`
`COLUMN BY FINGER
`
`ZERO
`
`OFFSETS
`
`
`
` SYMBOLS
`
`
`
`
`
`
`UPDATE DISPLAYED
`POSITIONS OF KEY
`
`PROCESS
`FINGER TAPS
`ON MORPHED KEY
`LAYOUT
`
`718
`
`FIG. 41
`
`44
`
`44
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 43 of 45
`
`US 6,323,846 B1
`
`
`
`GET ANY PATH RECENTLY
`CREATED BY HAND PART
`TOUCHDOWN
`
`
`
`
`750
`
`752
`
`PATH
`
`
`PROXIMITY
`
`JUST CROSSED
`
`
`
`KEYPRESS
` IDENTIFIED
`
`WRITING ?
`
`
`
`DEBOUNCE
`HAND SLIDING/
`OK ?
`
`757
`
`760
`FIND CLOSEST
`KEY REGION
`
`
`
`
`CREATE KEYPRESS
`APPEND KEYPRESS
`
`QUEUE ELEMENT
`
`QUEUE ELEMENT TO
`
`
`CONTAINING PATHID,
`TAIL OF FIFO
`
`
`
`CLOSESTKEY &
`KEYPRESS QUEUE
`
`
`
`PRESS TIMESTAMP
`
`
`FIG. 42
`
`45
`
`45
`
`

`

`U.S. Patent
`
`Nov. 27, 2001
`
`Sheet 44 of 45
`
`US 6,323,846 B1
`
`
` DELETE CURRENT
`
`ELEMENT FROM
`KEYPRESS QUEUE
`
`
`
`
`
`
`
`PATH INA
`
`PATH STILL
`FINGER SLID
`
`
`SYNCHRONIZED
`
`
`
`IDENTIFIED AS
`
`
`
`TOO FAR ?
`SUBSET ?
`
`FINGER ?.
`
`
`
` TIME
`
`
`SINCE PRESS<
`
`JAP TIMEOUT
`
` APPEND PRECEDING
`
`
`
`
`MODIFIERS &
`ELEMENT'S KEY
`REGION SYMBOL TO
`HOST COMM QUEUE
`
`
`
`
`
`SKIP TO NEXT
`
`KEY REGION
`
`ELEMENTIN
`
`A MODIFIER ?
`
`
`
`
`
`
`
`
`MOST
`FINGERS
` IMPULSIVE
`
`TOUCHING
`?
`
`
`?
`
`FIG. 43A
`
`46
`
`46
`
`

`

`U.S. Patent
`
`Nov.27, 2001
`
`Sheet 45 of 45
`
`US 6,323,846 B1
`
`
`
`
`TYPEMATIC
`
`
`
`STARTED FOR
`FINGER PRESS>
`ELEMENT?
`
`TAP TIMEOUT
`
`
`?
`
`
`
`
`ANOTHER
`HAND'S
`
`
`ASYNCHRONOUS
`
`OTHER FINGERS
`
`TAP ?
`LIFTED > .5s ?
`
`
`
`
`
`
`
`COMPUTE REPEAT
`INTERVAL FROM
`
`
`CURRENT
`
` INITIALIZE
`
`FINGER PROXIMITY
`
`
`TYPEMATIC
`
`MODE FOR
`
`ELEMENT
` TIME
`
`
`
`SINCE LAST
`
`
`SEND > REPEAT
`
`INTERVAL
`?
`
`UPDATE LAST TYPEMATIC
`SEND TIMESTAMP
`
`FIG. 43B
`
`47
`
`
`
`
`APPEND PRECEDING
`MODIFIERS & ELEMENT'S
`KEY REGION SYMBOL TO
`HOST COMM QUEUE
`
`
`
`
`
`47
`
`

`

`US 6,323,846 B1
`
`1
`METHOD AND APPARATUS FOR
`INTEGRATING MANUAL INPUT
`
`The present application is based upon provisional appli-
`cation Serial No. 60/072,509, filed Jan. 26, 1998.
`BACKGROUND OF THE INVENTION
`A. Field of the Invention
`
`The present invention relates generally to methods and
`apparatus for data input, and, more particularly, to a method
`and apparatus for integrating manual input.
`B. Description of the Related Art
`Many methods for manual input of data and commandsto
`computers are in use today, but each is mostefficient and
`easy to use for particular types of data input. For example,
`drawing tablets with pens or pucks excel at drafting,
`sketching, and quick command gestures. Handwriting with
`a stylus is convenient for filling out forms which require
`signatures, special symbols, or small amounts of text, but
`handwriting is slow compared to typing and voice input for
`long documents. Mice, finger-sticks and touchpads excel at
`cursor pointing and graphical object manipulations such as
`drag and drop. Rollers, thumbwheels and trackballs excel at
`panning and scrolling. The diversity of tasks that many
`computer users encounterin a single day call for all of these
`techniques, but few users will pay for a multitude of input
`devices, and the separate devices are often incompatible in
`a usability and an ergonomic sense. For instance, drawing
`tablets are a must for graphics professionals, but switching
`between drawing and typing is inconvenient because the pen
`must be put down or held awkwardly between the fingers
`while typing. Thus, there is a long-felt need in the art for a
`manual input device which is cheap yet offers convenient
`integration of common manual input techniques.
`Speech recognition is an exciting, new technology which
`promises to relieve some of the input burden on user hands.
`However, voice is not appropriate for inputting all types of
`data either. Currently, voice input is best-suited for dictation
`of long text documents. Until natural language recognition
`matures sufficiently that very high level voice commands
`can be understood by the computer, voice will have little
`advantage over keyboard hot-keys and mouse menus for
`command and control. Furthermore, precise pointing,
`drawing, and manipulation of graphical objects is difficult
`with voice commands, no matter how well speech is under-
`stood. Thus,
`there will always be a need in the art for
`multi-function manual
`input devices which supplement
`voice input.
`input device which combines the
`A generic manual
`typing, pointing, scrolling, and handwriting capabilities of
`the standard input device collection must have ergonomic,
`economic, and productivity advantages which outweigh the
`unavoidable sacrifices of abandoning device specialization.
`The generic device must tightly integrate yet clearly distin-
`guish the different types of input. It should therefore appear
`modeless to the user in the sense that the user should not
`
`need to provide explicit mode switch signals such as
`buttonpresses, arm relocations, or stylus pickups before
`switching from one input activity to another. Epidemiologi-
`cal studies suggest that repetition and force multiply in
`causing repetitive strain injuries. Awkward postures, device
`activation force, wasted motion, and repetition should be
`minimized to improve ergonomics. Furthermore, the work-
`load should be spread evenly over all available muscle
`groups to avoid repetitive strain.
`Repetition can be minimized by allocating to several
`graphical manipulation channels those tasks which require
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`complex mouse pointer motion sequences. Commongraphi-
`cal user interface operations such as finding and manipulat-
`ing a scroll bar or slider control are muchlessefficient than
`specialized finger motions which cause scrolling directly,
`without
`the step of repositioning the cursor over an
`on-screen control. Preferably the graphical manipulation
`channels should be distributed amongst many finger and
`hand motion combinations to spread the workload. Touch-
`pads and mice with auxilliary scrolling controls such as the
`Cirque® Smartcat touchpad with edge scrolling, the IBM®
`ScrollPoint™ mouse with embeddedpointing stick, and the
`Roller Mouse described in U.S. Pat. No. 5,530,455 to Gillick
`et al. represent small improvements in thisarea, butstill do
`not provide enough direct manipulation channels to elimi-
`nate many often-used cursor motion sequences.
`Furthermore, as S. Zhai et al. found in “Dual Stream Input
`for Pointing and Scrolling,” Proceedings of CHI
`’°97
`Extended Abstracts (1997), manipulation of more than two
`degrees of freedom at a time is very difficult with these
`devices, preventing simultaneous panning, zooming and
`rotating.
`Another common method for reducing excess motion and
`repetition is to automatically continue pointing or scrolling
`movementsignals once the user has stopped movingorlifts
`the finger. Related art methods can be distinguished by the
`conditions under which such motion continuation is enabled.
`
`In U.S. Pat. No. 4,734,685, Watanabe continues image
`panning whenthe distance and velocity of pointing device
`movement exceed thresholds. Automatic panning is stopped
`by moving the pointing device back in the opposite
`direction, so stopping requires additional precise move-
`ments. In U.S. Pat. No. 5,543,591 to Gillespie et al., motion
`continuation occurs when the finger enters an edge border
`region around a small touchpad. Continued motion speed is
`fixed and the direction correspondsto the direction from the
`center of the touchpadto the finger at the edge. Continuation
`mode ends whenthe finger leaves the border region or lifts
`off the pad. Disadvantageously, users sometimes pause at the
`edge of the pad without
`intending for cursor motion to
`continue, and the unexpected motion continuation becomes
`annoying. U.S. Pat. No. 5,327,161 to Logan et al. describes
`motion continuation when the finger enters a border area as
`well, but in an alternative trackball emulation mode, motion
`continuation can be a function solely oflateral finger veloc-
`ity and direction at liftoff. Motion continuation decays due
`to a friction factor or can be stopped by a subsequent
`touchdown on the surface. Disadvantageously, touch veloc-
`ity at liftoff is not a reliable indicator of the user’s desire for
`motion continuation since when approaching a large target
`on a display at high speeds the user may not stop the pointer
`completely before liftoff. Thus it would be an advancein the
`art to provide a motion continuation method which does not
`become activated unexpectedly when the user
`really
`intended to stop pointer movementata target but happensto
`be on a border or happens to be movingat significant speed
`during liftoff.
`Many attempts have been made to embed pointing
`devices in a keyboard so the hands don’t have to leave
`typing position to access the pointing device. These include
`the integrated pointing key described in U.S. Pat. No.
`5,189,403 to Franz et al.,
`the integrated pointing stick
`disclosed by J. Rutledge and T. Selker in “Force-to-Motion
`Functions for Pointing,” Human-Computer Interaction-
`INTERACT’90, pp. 701-06 (1990), and the position sensing
`keys described in U.S. Pat. No. 5,675,361 to Santilli.
`Nevertheless, the limited movement range and resolution of
`these devices leads to poorer pointing speed and accuracy
`
`48
`
`48
`
`

`

`US 6,323,846 B1
`
`3
`than a mouse, and they add mechanical complexity to
`keyboard construction. Thusthere exists a need in the art for
`pointing methods with higher resolution, larger movement
`range, and more degrees of freedom yet which are easily
`accessible from typing hand positions.
`Touch screens and touchpads often distinguish pointing
`motions from emulated button clicks or keypresses by
`assuming very little lateral fingertip motion will occur
`during taps on the touch surface which are intended as
`clicks. Inherent in these methods is the assumption that
`tapping will usually be straight down from the suspended
`finger position, minimizing those components of finger
`motion tangential to the surface. This is a valid assumption
`if the surface is not finely divided into distinct key areas or
`if the user does a slow, “hunt and peck” visual search for
`each key before striking. For example,
`in U.S. Pat. No.
`5,543,591 to Gillespie et al. a touchpad sendsall lateral
`motions to the host computer as cursor movements.
`However,if the finger is lifted soon enough after touchdown
`to count as a tap and if the accumulated lateral motions are
`not excessive, any sent motions are undone and a mouse
`button click is sent instead. This method only works for
`mouse commands such as pointing which can safely be
`undone,not for dragging or other manipulations. In U'S. Pat.
`No. 5,666,113 to Logan, taps with less than about “16"lateral
`motion activate keys on a small keypad while lateral motion
`in excess of Yio" activates cursor control mode. In both
`
`patents cursor mode is invoked by default when a finger
`stays on the surface a long time.
`However, fast touch typing on a surface divided into a
`large array of key regions tends to produce more tangential
`motions along the surface than related art filtering tech-
`niques can tolerate. Such an array contains keys in multiple
`rows and columns which may not be directly under the
`fingers, so the user must reach with the hand or flex or
`extend fingers to touch many of the key regions. Quick
`reaching and extending imparts significant
`lateral finger
`motion while the finger is in the air which maystill be
`present when the finger contacts the surface. Glancing taps
`with as much as 4"lateral motion measured at the surface
`
`can easily result. Attempting to filter or suppress this much
`motion would make the cursor seem sluggish and unrespon-
`sive. Furthermore, it may be desirable to enter a typematic
`or automatic key repeat mode instead of pointing mode
`when the finger is held in one place on the surface. Any
`lateral shifting by the fingertip during a prolonged finger
`press would also be picked up as cursor jitter without heavy
`filtering. Thus, there is a need in the art for a method to
`distinguish keying from pointing on the same surface via
`more robust hand configuration cues than lateral motion of
`a single finger.
`An ergonomic typing system should require minimal key
`tapping force, easily distinguish finger taps from resting
`hands, and cushion the fingers from the jarring force of
`surface impact. Mechanical and membrane keyboards rely
`on the spring force in the keyswitches to prevent activation
`when the hands are resting on the keys. This causes an
`irreconcilable tradeoff between the ergonomic desires to
`reduce the fatigue from key activating force and to relax the
`full weight of the hands onto the keys during rest periods.
`Force minimization on touch surfaces is possible with
`capacitive or active optical sensing, which do not rely on
`finger pressure, rather than