Hand tracking

Hands are input sources, not a separate API

A tracked hand does not arrive through some parallel channel. It is an XRInputSource like any other, and if the hand-tracking feature was granted it carries an extra property: inputSource.hand, an XRHand. Everything you already wrote against inputSources, handedness and the select event keeps working — a pinch fires select the same way a trigger pull does.

This matters more than it sounds. The portable way to handle hands is usually to not handle them specially at all: let select drive your interactions, and reach for joint poses only when you actually need the skeleton. Applications that branch on "is this a hand" at the top of their input code end up maintaining two interaction systems that drift apart.

Hand tracking is an optional feature, requested by name. Put it in optionalFeatures rather than requiredFeatures unless your app is genuinely useless without it — a required feature that the device cannot grant fails the whole session request, and most headsets let the user turn hand tracking off in system settings.

Requesting hands and reading a joint

Joint poses are only available inside a frame callback, and only against a reference space. There is no way to ask "where is the index tip right now" outside the loop, because outside the loop the question has no well-defined answer.

const session = await navigator.xr.requestSession('immersive-vr', {
  // Optional, not required: the user can disable hand tracking in system
  // settings, and a required feature that cannot be granted kills the session.
  optionalFeatures: ['hand-tracking'],
});

function onFrame(time, frame) {
  for (const source of session.inputSources) {
    // No hand property means this input is a controller, or the feature
    // was not granted. Both are normal.
    if (!source.hand) continue;

    const indexTip = source.hand.get('index-finger-tip');
    const thumbTip = source.hand.get('thumb-tip');
    if (!indexTip || !thumbTip) continue;

    // getJointPose can return null on ANY frame: occlusion, the hand leaving
    // the tracking volume, or the runtime simply losing confidence.
    const a = frame.getJointPose(indexTip, referenceSpace);
    const b = frame.getJointPose(thumbTip, referenceSpace);
    if (!a || !b) continue;

    const dx = a.transform.position.x - b.transform.position.x;
    const dy = a.transform.position.y - b.transform.position.y;
    const dz = a.transform.position.z - b.transform.position.z;
    const distance = Math.hypot(dx, dy, dz);

    // a.radius is the runtime's estimate of the joint's thickness in metres.
    // This is a preview threshold, not a calibrated gesture detector.
    const threshold = (a.radius + b.radius) * 1.4;
    setPinchPreview(source, distance < threshold);
  }
}

This fragment updates a preview on every valid frame; setPinchPreview is an application UI callback, not a click handler. Use the stateful example below for actions. Note the two separate checks. get() only returns undefined for a key that is not a joint name, since an XRHand always holds all 25. getJointPose returning null is the tracking signal, and the specification makes it all-or-nothing per hand: when a hand is partly hidden the runtime must either emulate the hidden joints or report null for every joint. You will not get a hand with a missing thumb. You will get a whole hand in which some fingertips are guesses, which is worth remembering before trusting a pinch that was detected while the other hand was in the way.

The 25 joints, and how they are named

Joint names are strings from a fixed vocabulary. There are 25 per hand: the wrist, four joints for the thumb, and five for each of the other four fingers.

wrist

The single root joint. Everything else is conceptually downstream of it, though the API gives you each pose independently rather than as a hierarchy.

thumb-metacarpal … thumb-tip

Four joints: metacarpal, phalanx-proximal, phalanx-distal, tip. The thumb is the exception — it has no intermediate phalanx, so loops written for five joints per finger break on it.

{index,middle,ring,pinky}-finger-metacarpal … -tip

Five joints each: metacarpal, phalanx-proximal, phalanx-intermediate, phalanx-distal, tip. The metacarpal sits inside the palm, so it is useful for orientation and almost never for contact.

XRJointPose.radius

A per-joint thickness estimate in metres, and the one piece of the API most people ignore. It can help scale a threshold, but it may be emulated and does not remove the need to test different users and devices.

Why pinch detection needs state

A threshold produces a boolean on each frame. An action needs transitions: start, held, release and cancellation when tracking is lost. Noise near one threshold can alternate the boolean repeatedly while the user tries to hold still.

Use a smaller distance to start than to release. If a pose disappears, choose an explicit cancellation policy; a short grace period can bridge a brief loss, but a stale held state must not last indefinitely. Joint radii can help choose a scale, but the runtime may emulate them. They do not establish physical fingertip contact or guarantee that one threshold works for every hand.

Use select for ordinary selection. A custom detector is useful when the interaction requires a gesture the runtime does not expose. The runnable trace below shows the event and timeout logic separately from the pose-reading fragment.

What the demo above is doing

The skeleton is built to the real joint layout — count them and you will find 25, with the thumb correctly one joint shorter than the fingers. The thumb and index curl toward each other while the other three hold a relaxed rest pose, which is roughly what a real pinch looks like to a tracker.

The distance driving the highlight is measured between the two fingertip joints in world space every frame, not baked into the animation. That is why moving the threshold changes the result rather than just changing a label: at 5 mm almost nothing counts as a pinch, and at 6 cm the gesture is "detected" while the fingers are still visibly apart — which is exactly the failure mode a too-generous threshold produces on real hardware.

Turning on the joint radius draws each joint's thickness estimate as a translucent sphere. Their intersection shows contact in this geometric model; it does not establish physical contact on a tracked hand.

Check the hand input you actually receive

A browser name alone does not establish that joints are available in this session. Check each stage and keep controller or other primary-action input usable.

API behaviour follows the W3C Hand Input Module linked below. This is a verification procedure, not a device compatibility matrix or hardware test report.

Mistakes that cost the most time

These share a shape: they all assume the hand is always there and always the same size.

Requiring the hand-tracking feature

Putting it in requiredFeatures means the session request rejects on any device where the user has hand tracking switched off. Ask for it optionally and degrade.

Assuming getJointPose is non-null

It returns null whenever tracking is lost — a hand behind the other hand, at the edge of the camera view, or moving fast. This happens constantly, not rarely. Because the rule is per hand rather than per joint, testing one joint tells you whether the whole hand is tracked this frame.

Calling getJointPose fifty times a frame

Two hands are 50 joints, and each call allocates a pose object. XRFrame.fillPoses writes every transform into a Float32Array you own, and fillJointRadii does the same for radii; both return false when the data is not valid. Use them once you draw a full skeleton, and keep getJointPose for the two joints a pinch needs.

A hardcoded pinch distance

Hand size and pose estimates vary. Joint radii can inform a threshold, but validate the resulting gesture with multiple users and allow adjustment.

Looping five joints across every finger

The thumb has four. Code that indexes a fixed five-element array per finger reads past the end and either throws or silently uses the wrong joint.

Reimplementing select as pinch

The runtime already fires select for a pinch, with its own tuning and hysteresis. A hand-rolled version is worse and does not work with controllers.

A pinch that fires once, survives jitter, and cancels

A distance check inside every frame callback can fire dozens of actions during one held gesture. Store a state per input source, emit an event only on a transition, and use separate distances to enter and leave the hold. The following sample uses 18 mm to start and 26 mm to release.

Those distances and the 100 ms tracking timeout are teaching parameters. They are not calibrated for a particular hand or headset. The input is distance in millimetres and monotonically increasing time in milliseconds. Convert a distance calculated from WebXR positions by multiplying by 1000.

Run the supplied trace in a console. Distances oscillate between 17 and 19 mm, one pose disappears briefly, and a later hold loses tracking for longer. A cancel event lets a drawing tool discard an unfinished stroke without treating tracking loss as a deliberate release. Ordinary button selection should still use the runtime’s select events.

function createPinch() {
  let held = false;
  let lastValid = null;
  return function update(distanceMm, timeMs) {
    if (!Number.isFinite(distanceMm) || distanceMm < 0) {
      if (held && timeMs - lastValid >= 100) {
        held = false;
        return 'cancel';
      }
      return null;
    }
    // Also expire a hold when callbacks stopped during tracking loss.
    if (held && lastValid !== null && timeMs - lastValid >= 100) {
      held = false;
      lastValid = timeMs;
      return 'cancel';
    }
    lastValid = timeMs;
    if (!held && distanceMm <= 18) {
      held = true;
      return 'start';
    }
    if (held && distanceMm >= 26) {
      held = false;
      return 'release';
    }
    return null;
  };
}

const update = createPinch();
const samples = [
  [0, 30], [16, 17], [32, 19], [48, 17],
  [64, null], [80, 20], [96, 27],
  [112, 17], [224, null],
];
for (const [time, distance] of samples) {
  const event = update(distance, time);
  if (event) console.log(time, event);
}
// 16 start; 96 release; 112 start; 224 cancel

Expected events: 16 start, 96 release, 112 start, 224 cancel. Call createPinch separately for each input source; clear its state when that source disappears or the session ends.

Read the trace before changing the thresholds

Try replacing the 19 mm sample with 27 mm: the hold should release and the next 17 mm sample should start again. Then remove the intermediate missing sample and return after a long gap. The timeout must still cancel the old hold. Keep separate tests for source removal and session visibility changes, where frame callbacks may stop entirely.

Synthetic distances and timestamps. The trace checks state transitions; it does not establish tracking accuracy or a universal gesture threshold.

Further reading

The joint name vocabulary is worth having open the first time you write this code — the names are long and a typo silently returns undefined rather than throwing.

• W3C — WebXR Hand Input Module — The normative joint list, XRHand, and the radius semantics.
• MDN — XRHand — Practical reference for get() and the joint name strings.
• MDN — XRFrame.getJointPose() — What the returned pose contains, and when it is null.
• VR controllers — The input source model that hands plug into, and why select is the portable action.
• WebXR sessions — Requesting optional features, and the reference space joint poses resolve against.