---
title: "Building ಲಿಪಿ: Three-Tier Speech Recognition in the Browser"
description: "On-device first, cloud fallback, Whisper polish. A Kannada voice editor that works everywhere, built with zero frameworks."
url: "https://h3manth.com/scribe/lipi-architecture/"
canonical_url: "https://h3manth.com/scribe/lipi-architecture/"
markdown_url: "https://h3manth.com/scribe/blog/lipi-architecture.md"
publishDate: "2026-07-13T06:00:00.000Z"
author: "Hemanth HM"
category: "ai"
tags: ["ai","speech","whisper","kannada","web-speech-api","transformers-js","on-device"]
word_count: 687
reading_time_minutes: 4
x_markdown_tokens: 1304
content_signal: "ai-train=yes, search=yes, ai-input=yes"
---

# Building ಲಿಪಿ: Three-Tier Speech Recognition in the Browser

> On-device first, cloud fallback, Whisper polish. A Kannada voice editor that works everywhere, built with zero frameworks.

I wanted to type in Kannada without switching keyboards. Speaking felt more natural than typing, but every speech-to-text tool either required a server, needed an app install, or didn't support Kannada well.

So I built [ಲಿಪಿ (Lipi)](https://h3manth.com/ai/lipi/), a voice editor that runs entirely in the browser. No backend. No framework. Five files.

## The problem with one engine

Web Speech API streams text in real time. Words appear as you speak. But it's cloud-dependent and not available on Firefox.

Whisper (via Transformers.js) runs locally. Accurate, private, offline. But it can't stream. You record first, transcribe after.

Neither alone is enough. So I used both.

## Three tiers

```
User clicks record
├── Tier 1: On-device Speech API (processLocally)
│   └── Chrome 130+ with language pack installed
├── Tier 2: Cloud Speech API (webkitSpeechRecognition)
│   └── Chrome/Edge, cloud-based, real-time streaming
└── Tier 3: Whisper ONNX chunks (Transformers.js worker)
    └── Firefox, Safari, or if Speech API fails
```

The app detects what's available and picks the best tier automatically.

## Tier 1: On-device speech

Chrome recently added `processLocally` to the Web Speech API. Here's how Lipi uses it:

```javascript
if ('processLocally' in recognition) {
  const availability = await SpeechRecognition.available({
    langs: ['kn-IN'],
    processLocally: true
  });

  if (availability === 'available') {
    recognition.processLocally = true;
  } else if (availability === 'downloadable') {
    await SpeechRecognition.install({ langs: ['kn-IN'], processLocally: true });
    recognition.processLocally = true;
  }
}
```

Real-time streaming like cloud, but no data leaves the device. Best of both worlds.

## Tier 2: Cloud Speech API

The familiar `webkitSpeechRecognition`. Lipi configures it for Kannada with interim results:

```javascript
recognition = new webkitSpeechRecognition();
recognition.lang = 'kn-IN';
recognition.continuous = true;
recognition.interimResults = true;
```

Interim text appears as a dimmed span that gets replaced when the final result arrives. It feels like the words are streaming out of your mouth onto the screen.

## Tier 3: Whisper fallback

For browsers without Speech API, the app falls back to Whisper running in a Web Worker via Transformers.js:

```javascript
// Record raw PCM at 16kHz
scriptProc.onaudioprocess = (e) => {
  pcmChunks.push(new Float32Array(e.inputBuffer.getChannelData(0)));
};

// Every 5 seconds, flush to Whisper
chunkTimer = setInterval(() => flushChunk(false), 5000);
```

Not as smooth as streaming, but it works everywhere. The model downloads once and caches in the browser's Cache API.

## The Polish button

Here's where it gets interesting. While the Web Speech API streams text live, the app silently records raw PCM audio in parallel. After you stop, a "Polish" button appears.

Click it, and Whisper re-transcribes your audio from the raw recording. The result replaces only the recorded text. Typed text before and after is untouched.

```
"typed text" [start-mark] {speech text → replaced by Whisper} [end-mark] "more typed text"
```

Whisper gets a Kannada initial prompt to condition its decoder:

```javascript
opts.initial_prompt = 'ಕನ್ನಡ ಭಾಷೆಯಲ್ಲಿ ಸ್ಪಷ್ಟವಾಗಿ ಬರೆಯಿರಿ. ಸರಿಯಾದ ವಿರಾಮ ಚಿಹ್ನೆಗಳನ್ನು ಬಳಸಿ.';
```

This biases the decoder toward clean Kannada script with proper punctuation.

## Phonetic typing

Don't want to talk? Type `namaskara` and it becomes ನಮಸ್ಕಾರ. The transliteration engine intercepts keystrokes, buffers Latin characters, and converts on space.

On mobile, virtual keyboards don't fire `keydown` reliably. Lipi uses `beforeinput` as a fallback:

```javascript
ed.addEventListener('beforeinput', e => {
  if (e.inputType === 'insertText' && /^[a-zA-Z]$/.test(e.data)) {
    e.preventDefault();
    buf += e.data;
    showBuf();
  }
});
```

Desktop uses `keydown` (fires first, blocks `beforeinput`). Mobile falls through to `beforeinput`. Both work.

## The stack

Five files. No build step. No node_modules.

```
index.html    # landing + editor (single page, hash routing)
index.css     # design system with oklch colors
app.js        # speech, recording, transliteration, UI
worker.js     # Whisper inference in a Web Worker
transliterate.js # phonetic-to-Kannada mapping
```

The Whisper model preloads in the background via `requestIdleCallback` when the editor opens. By the time you need it, it's ready.

## Trade-offs

- **Web Speech API accuracy** varies by accent. The Polish button exists precisely for this.
- **Whisper Small** is ~240MB. It caches after first download, but that first load takes time.
- **On-device Speech API** is Chrome 130+ only. The fallback chain handles everything else.

That's it. No server. No API keys. Just the browser doing what browsers should do.

[Try it](https://h3manth.com/ai/lipi/)