> ## Documentation Index
> Fetch the complete documentation index at: https://tbd-6fc993ce-hypeship-add-rate-limiting-docs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Computer Controls

> Control the computer's mouse, keyboard, and screen

Use OS-level controls to move and click the mouse, type and press keys, scroll, drag, and capture screenshots from a running browser session. Both `moveMouse` and `dragMouse` use human-like [Bézier curves](https://en.wikipedia.org/wiki/B%C3%A9zier_curve) by default.

## Click the mouse

Simulate mouse clicks at specific coordinates. You can select the button, click type (down, up, click), number of clicks, and optional modifier keys to hold.

<CodeGroup>
  ```typescript Typescript/Javascript theme={null}
  import Kernel from '@onkernel/sdk';

  const kernel = new Kernel();
  const kernelBrowser = await kernel.browsers.create();

  // Basic left click at (100, 200)
  await kernel.browsers.computer.clickMouse(kernelBrowser.session_id, {
    x: 100,
    y: 200,
  });

  // Double right-click while holding Shift
  await kernel.browsers.computer.clickMouse(kernelBrowser.session_id, {
    x: 100,
    y: 200,
    button: 'right',
    click_type: 'click',
    num_clicks: 2,
    hold_keys: ['Shift'],
  });
  ```

  ```python Python theme={null}
  from kernel import Kernel

  kernel = Kernel()
  kernel_browser = kernel.browsers.create()

  # Basic left click at (100, 200)
  kernel.browsers.computer.click_mouse(
      id=kernel_browser.session_id,
      x=100,
      y=200,
  )

  # Double right-click while holding Shift
  kernel.browsers.computer.click_mouse(
      id=kernel_browser.session_id,
      x=100,
      y=200,
      button="right",
      click_type="click",
      num_clicks=2,
      hold_keys=["Shift"],
  )
  ```

  ```bash CLI theme={null}
  # Click the mouse at coordinates (100, 200)
  kernel browsers computer click-mouse <session id> --x 100 --y 200

  # Double-click the right mouse button
  kernel browsers computer click-mouse <session id> --x 100 --y 200 --num-clicks 2 --button right
  ```
</CodeGroup>

## Move the mouse

Move the cursor to specific screen coordinates. By default, the cursor follows a human-like Bezier curve path instead of teleporting instantly. You can control this with `smooth` and `duration_ms`.

| Parameter     | Type    | Default | Description                                                                                                |
| ------------- | ------- | ------- | ---------------------------------------------------------------------------------------------------------- |
| `x`           | integer | —       | X coordinate to move the cursor to                                                                         |
| `y`           | integer | —       | Y coordinate to move the cursor to                                                                         |
| `smooth`      | boolean | `true`  | Use human-like Bezier curve path instead of instant teleport                                               |
| `duration_ms` | integer | auto    | Target duration in milliseconds for smooth movement (50–5000). Omit for automatic timing based on distance |
| `hold_keys`   | array   | —       | Modifier keys to hold during the move                                                                      |

<CodeGroup>
  ```typescript Typescript/Javascript theme={null}
  import Kernel from '@onkernel/sdk';

  const kernel = new Kernel();
  const kernelBrowser = await kernel.browsers.create();

  // Human-like smooth movement (default)
  await kernel.browsers.computer.moveMouse(kernelBrowser.session_id, {
    x: 500,
    y: 300,
  });

  // Smooth movement with custom duration
  await kernel.browsers.computer.moveMouse(kernelBrowser.session_id, {
    x: 800,
    y: 600,
    smooth: true,
    duration_ms: 1500,
  });

  // Instant teleport (disable smooth)
  await kernel.browsers.computer.moveMouse(kernelBrowser.session_id, {
    x: 100,
    y: 200,
    smooth: false,
  });
  ```

  ```python Python theme={null}
  from kernel import Kernel

  kernel = Kernel()
  kernel_browser = kernel.browsers.create()

  # Human-like smooth movement (default)
  kernel.browsers.computer.move_mouse(
      id=kernel_browser.session_id,
      x=500,
      y=300,
  )

  # Smooth movement with custom duration
  kernel.browsers.computer.move_mouse(
      id=kernel_browser.session_id,
      x=800,
      y=600,
      smooth=True,
      duration_ms=1500,
  )

  # Instant teleport (disable smooth)
  kernel.browsers.computer.move_mouse(
      id=kernel_browser.session_id,
      x=100,
      y=200,
      smooth=False,
  )
  ```

  ```bash CLI theme={null}
  # Smooth movement (default)
  kernel browsers computer move-mouse <session id> --x 500 --y 300

  # Instant teleport
  kernel browsers computer move-mouse <session id> --x 500 --y 300 --smooth=false
  ```
</CodeGroup>

### Smooth vs instant movement

<Frame caption="Blue = smooth Bezier path, Red = instant teleport">
  <img src="https://mintcdn.com/tbd-6fc993ce-hypeship-add-rate-limiting-docs/DSgZ1Y4zFjJ_wkvA/images/smooth-mouse-demo.gif?s=27f28b9f7568cccfd2ef55419078c45f" width="1280" height="720" data-path="images/smooth-mouse-demo.gif" />
</Frame>

Try it yourself — open [live view](/browsers/live-view) and paste this (requires CLI v0.15.4+ — run `kernel upgrade` to update):

```bash theme={null}
ID=$(kernel browsers create -o json | jq -r '.session_id')
echo "Open live view: https://dashboard.onkernel.com/browsers/$ID"
HTML=$(curl -sL "https://gist.githubusercontent.com/ulziibay-kernel/f6d8063ba223a81293cfe1fde06d8624/raw/cursor-trail-demo.html")
kernel browsers playwright execute $ID "await page.setContent($(echo "$HTML" | jq -Rs .))"
sleep 3

kernel browsers playwright execute $ID "await page.evaluate(() => document.dispatchEvent(new CustomEvent('kernel-mode', {detail:'instant'})))"
for coords in "200 200" "900 150" "1700 250" "1700 700" "960 800" "200 750"; do
  kernel browsers computer move-mouse $ID --x ${coords% *} --y ${coords#* } --smooth=false
  sleep 0.5
done

kernel browsers playwright execute $ID "await page.evaluate(() => document.dispatchEvent(new CustomEvent('kernel-mode', {detail:'smooth'})))"
for coords in "200 200" "900 150" "1700 250" "1700 700" "960 800" "200 750"; do
  kernel browsers computer move-mouse $ID --x ${coords% *} --y ${coords#* } --duration-ms 1200
  sleep 0.3
done
```

## Take screenshots

Capture a full-screen PNG or a specific region.

<CodeGroup>
  ```typescript Typescript/Javascript theme={null}
  import fs from 'fs';
  import { Buffer } from 'buffer';
  import Kernel from '@onkernel/sdk';

  const kernel = new Kernel();
  const kernelBrowser = await kernel.browsers.create();

  // Full screenshot
  {
    const response = await kernel.browsers.computer.captureScreenshot(kernelBrowser.session_id);
    const blob = await response.blob();
    const buffer = Buffer.from(await blob.arrayBuffer());
    fs.writeFileSync('screenshot.png', buffer);
  }

  // Region screenshot
  {
    const response = await kernel.browsers.computer.captureScreenshot(kernelBrowser.session_id, {
      region: { x: 0, y: 0, width: 800, height: 600 },
    });
    const blob = await response.blob();
    const buffer = Buffer.from(await blob.arrayBuffer());
    fs.writeFileSync('region.png', buffer);
  }
  ```

  ```python Python theme={null}
  from kernel import Kernel

  kernel = Kernel()
  kernel_browser = kernel.browsers.create()

  # Full screenshot
  with open('screenshot.png', 'wb') as f:
      image_data = kernel.browsers.computer.capture_screenshot(id=kernel_browser.session_id)
      f.write(image_data.read())

  # Region screenshot
  with open('region.png', 'wb') as f:
      image_data = kernel.browsers.computer.capture_screenshot(
          id=kernel_browser.session_id,
          region={"x": 0, "y": 0, "width": 800, "height": 600},
      )
      f.write(image_data.read())
  ```

  ```bash CLI theme={null}
  # Take a full screenshot
  kernel browsers computer screenshot <session id> --to screenshot.png

  # Take a screenshot of a specific region
  kernel browsers computer screenshot <session id> --to region.png --x 0 --y 0 --width 800 --height 600
  ```
</CodeGroup>

## Type text

Type literal text, optionally with a delay in milliseconds between keystrokes.

<CodeGroup>
  ```typescript Typescript/Javascript theme={null}
  import Kernel from '@onkernel/sdk';

  const kernel = new Kernel();
  const kernelBrowser = await kernel.browsers.create();

  await kernel.browsers.computer.typeText(kernelBrowser.session_id, {
    text: 'Hello, World!',
  });

  await kernel.browsers.computer.typeText(kernelBrowser.session_id, {
    text: 'Slow typing...',
    delay: 100,
  });
  ```

  ```python Python theme={null}
  from kernel import Kernel

  kernel = Kernel()
  kernel_browser = kernel.browsers.create()

  kernel.browsers.computer.type_text(
      id=kernel_browser.session_id,
      text="Hello, World!",
  )

  kernel.browsers.computer.type_text(
      id=kernel_browser.session_id,
      text="Slow typing...",
      delay=100,
  )
  ```

  ```bash CLI theme={null}
  # Type text in the browser
  kernel browsers computer type <session id> --text "Hello, World!"

  # Type text with a 100ms delay between keystrokes
  kernel browsers computer type <session id> --text "Slow typing..." --delay 100
  ```
</CodeGroup>

## Press keys

Press one or more key symbols (including combinations like "Ctrl+t" or "Ctrl+Shift+Tab"). Optionally hold modifiers and/or set a duration to hold keys down.

<CodeGroup>
  ```typescript Typescript/Javascript theme={null}
  import Kernel from '@onkernel/sdk';

  const kernel = new Kernel();
  const kernelBrowser = await kernel.browsers.create();

  // Tap a key combination
  await kernel.browsers.computer.pressKey(kernelBrowser.session_id, {
    keys: ['Ctrl+t'],
  });

  // Hold keys for 250ms while also holding Alt
  await kernel.browsers.computer.pressKey(kernelBrowser.session_id, {
    keys: ['Ctrl+Shift+Tab'],
    duration: 250,
    hold_keys: ['Alt'],
  });
  ```

  ```python Python theme={null}
  from kernel import Kernel

  kernel = Kernel()
  kernel_browser = kernel.browsers.create()

  # Tap a key combination
  kernel.browsers.computer.press_key(
      id=kernel_browser.session_id,
      keys=["Ctrl+t"],
  )

  # Hold keys for 250ms while also holding Alt
  kernel.browsers.computer.press_key(
      id=kernel_browser.session_id,
      keys=["Ctrl+Shift+Tab"],
      duration=250,
      hold_keys=["Alt"],
  )
  ```

  ```bash CLI theme={null}
  # Press one or more keys (repeatable --key)
  kernel browsers computer press-key <session id> --key Ctrl+t

  # Hold for a duration and add optional modifiers
  kernel browsers computer press-key <session id> --key Ctrl+Shift+Tab --duration 250 --hold-key Alt
  ```
</CodeGroup>

## Scroll

Scroll the mouse wheel at a specific position. Positive `delta_y` scrolls down; negative scrolls up. Positive `delta_x` scrolls right; negative scrolls left.

<CodeGroup>
  ```typescript Typescript/Javascript theme={null}
  import Kernel from '@onkernel/sdk';

  const kernel = new Kernel();
  const kernelBrowser = await kernel.browsers.create();

  await kernel.browsers.computer.scroll(kernelBrowser.session_id, {
    x: 300,
    y: 400,
    delta_x: 0,
    delta_y: 120,
  });
  ```

  ```python Python theme={null}
  from kernel import Kernel

  kernel = Kernel()
  kernel_browser = kernel.browsers.create()

  kernel.browsers.computer.scroll(
      id=kernel_browser.session_id,
      x=300,
      y=400,
      delta_x=0,
      delta_y=120,
  )
  ```

  ```bash CLI theme={null}
  # Scroll at a position
  kernel browsers computer scroll <session id> --x 300 --y 400 --delta-y 120
  ```
</CodeGroup>

## Drag the mouse

Drag by pressing a button, moving along a path of points, then releasing. By default, drag movement uses human-like Bezier curves between waypoints. Set `smooth: false` to use linear interpolation with `steps_per_segment` and `step_delay_ms` instead.

| Parameter           | Type    | Default | Description                                                                                                                             |
| ------------------- | ------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `path`              | array   | —       | Ordered list of `[x, y]` coordinate pairs to move through (minimum 2 points)                                                            |
| `button`            | string  | `left`  | Mouse button: `left`, `middle`, or `right`                                                                                              |
| `smooth`            | boolean | `true`  | Use human-like Bezier curves between waypoints. When `true`, `steps_per_segment` and `step_delay_ms` are ignored                        |
| `duration_ms`       | integer | auto    | Target duration in milliseconds for the entire drag when `smooth=true` (50–10000). Omit for automatic timing based on total path length |
| `delay`             | integer | `0`     | Delay in milliseconds between button down and starting to move                                                                          |
| `steps_per_segment` | integer | `10`    | Number of interpolation steps per path segment (only when `smooth=false`)                                                               |
| `step_delay_ms`     | integer | `50`    | Delay in milliseconds between steps (only when `smooth=false`)                                                                          |
| `hold_keys`         | array   | —       | Modifier keys to hold during the drag                                                                                                   |

<CodeGroup>
  ```typescript Typescript/Javascript theme={null}
  import Kernel from '@onkernel/sdk';

  const kernel = new Kernel();
  const kernelBrowser = await kernel.browsers.create();

  // Human-like smooth drag (default)
  await kernel.browsers.computer.dragMouse(kernelBrowser.session_id, {
    path: [
      [100, 200],
      [400, 350],
      [700, 200],
    ],
  });

  // Smooth drag with custom duration
  await kernel.browsers.computer.dragMouse(kernelBrowser.session_id, {
    path: [
      [100, 200],
      [400, 350],
      [700, 200],
    ],
    smooth: true,
    duration_ms: 2000,
  });

  // Linear interpolation drag (legacy behavior)
  await kernel.browsers.computer.dragMouse(kernelBrowser.session_id, {
    path: [
      [100, 200],
      [400, 350],
      [700, 200],
    ],
    smooth: false,
    steps_per_segment: 10,
    step_delay_ms: 50,
  });
  ```

  ```python Python theme={null}
  from kernel import Kernel

  kernel = Kernel()
  kernel_browser = kernel.browsers.create()

  # Human-like smooth drag (default)
  kernel.browsers.computer.drag_mouse(
      id=kernel_browser.session_id,
      path=[[100, 200], [400, 350], [700, 200]],
  )

  # Smooth drag with custom duration
  kernel.browsers.computer.drag_mouse(
      id=kernel_browser.session_id,
      path=[[100, 200], [400, 350], [700, 200]],
      smooth=True,
      duration_ms=2000,
  )

  # Linear interpolation drag (legacy behavior)
  kernel.browsers.computer.drag_mouse(
      id=kernel_browser.session_id,
      path=[[100, 200], [400, 350], [700, 200]],
      smooth=False,
      steps_per_segment=10,
      step_delay_ms=50,
  )
  ```

  ```bash CLI theme={null}
  # Smooth drag (default)
  kernel browsers computer drag-mouse <session id> \
    --point 100,200 \
    --point 400,350 \
    --point 700,200

  # Linear interpolation drag
  kernel browsers computer drag-mouse <session id> \
    --point 100,200 \
    --point 400,350 \
    --point 700,200 \
    --smooth=false \
    --steps-per-segment 10 \
    --step-delay-ms 50
  ```
</CodeGroup>

### Smooth vs linear drag

<Frame caption="Blue = smooth Bezier curves, Red = linear interpolation">
  <img src="https://mintcdn.com/tbd-6fc993ce-hypeship-add-rate-limiting-docs/DSgZ1Y4zFjJ_wkvA/images/smooth-drag-demo.gif?s=e19772de2babbef83ab4f04668745955" width="960" height="540" data-path="images/smooth-drag-demo.gif" />
</Frame>