Taking AI Agents to the Race Track: Lessons from the High-Velocity Field Test

What an experience. Taking AI agents from notebooks to race tracks was one of the most incredible collaborative efforts I’ve been part of. GDE++ indeed. 🤓

This is just the beginning, and I’m really looking forward to more experiments like this.

The Setup

Picture this: a laptop strapped into a race car, GPS devices wired up, real-time telemetry rendering on a UI, and Gemini Nano running inference-all while the car is doing 150mph around the track.

That was our field test.

We talked to the CEO of the race track. We got expert pro driver assessments to improve our agents. We debugged code between sessions while mechanics prepped the car. This wasn’t a hackathon in a conference room, this was AI meeting reality.

The Split-Brain Architecture

Real-time racing coaching needs two very different capabilities:

Instant reactions - “BRAKE!” can’t wait 500ms
Strategic advice - “You’re lifting too early compared to the pro”

We built a dual-path system:

┌────────────────────────────────────────────────────────────┐
│              10Hz GPS TELEMETRY STREAM                     │
│   (lat, lon, speed, heading, timestamp) @ 100ms intervals  │
└─────────────────────────┬──────────────────────────────────┘
                          │
              ┌───────────┴───────────┐
              ▼                       ▼
┌──────────────────────┐    ┌─────────────────────────┐
│    GEMINI NANO       │    │    GEMINI FLASH 3.0     │
│    Chrome Browser    │    │    Cloud REST API       │
├──────────────────────┤    ├─────────────────────────┤
│ Inference: <50ms     │    │ Inference: 500-1500ms   │
│ Context: 1KB         │    │ Context: 32KB           │
│ Output: 1 token      │    │ Output: Full analysis   │
└──────────────────────┘    └─────────────────────────┘

Nano: The Reflexes

Gemini Nano runs entirely in Chrome via the window.LanguageModel API. No network round-trip-the model lives on-device.

const getInstantFeedback = async (telemetry: TelemetryData): Promise<HotAction> => {
  const session = await window.LanguageModel.create({
    systemPrompt: `Racing copilot. Respond with ONE action word.
      
      DECISION MATRIX:
      | Condition                        | Action      |
      |----------------------------------|-------------|
      | brakePos > 50 AND longG < -0.8   | THRESHOLD   |
      | brakePos > 10 AND latG > 0.4     | TRAIL_BRAKE |
      | latG > 1.0 AND throttle < 20     | COMMIT      |
      | latG reducing AND throttle < 50  | THROTTLE    |
      | throttle > 80 AND latG < 0.3     | PUSH        |
      | throttle < 10 AND brakePos < 10  | NO_COAST    |`
  });

  const result = await session.prompt(
    `Telemetry: ${JSON.stringify(telemetry)}`,
    { 
      responseConstraint: {
        type: "object",
        properties: {
          action: {
            type: "string",
            enum: ["THRESHOLD", "TRAIL_BRAKE", "COMMIT", "THROTTLE", "PUSH", "NO_COAST", "MAINTAIN"]
          }
        }
      }
    }
  );

  session.destroy();
  return JSON.parse(result);
};

The responseConstraint schema forces valid output-no parsing failures, no hallucinations, guaranteed enum values.

Flash: The Strategist

For lap analysis and technique comparison, we use Gemini Flash via REST:

const BASE_URL = "https://generativelanguage.googleapis.com/v1beta/models";

const getCoachingAdvice = async (
  currentTelemetry: TelemetryData,
  referenceLap: Lap,
  coach: CoachPersona
): Promise<ColdAdvice> => {
  
  const response = await fetch(
    `${BASE_URL}/gemini-flash-3.0:generateContent?key=${API_KEY}`,
    {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        contents: [{ 
          parts: [{ text: buildPrompt(currentTelemetry, referenceLap, coach) }] 
        }],
        generationConfig: { 
          responseMimeType: "application/json",
          maxOutputTokens: 500 
        }
      })
    }
  );

  const result = await response.json();
  return parseAdvice(result.candidates[0].content.parts[0].text);
};

GPS-Derived Virtual Sensors

Professional telemetry requires OBD-II hardware to read brake pressure and throttle position. We made it work with GPS alone by deriving these values from physics:

interface TelemetryPoint {
  lat: number;
  lng: number;
  speed: number;      // GPS-derived m/s
  gLat: number;       // Lateral G-force (calculated)
  gLong: number;      // Longitudinal G-force (calculated)
  brake: number;      // Virtual sensor: 0-100
  throttle: number;   // Virtual sensor: 0-100
  time: number;
}

const calculateVirtualSensors = (
  current: GPSPoint, 
  previous: GPSPoint, 
  dt: number
): TelemetryPoint => {
  // Longitudinal G from velocity derivative
  const longG = (current.speed - previous.speed) / dt / 9.81;
  
  // Lateral G from heading change rate and speed
  const headingDelta = normalizeAngle(current.heading - previous.heading);
  const angularVelocity = headingDelta / dt;
  const latG = (current.speed * angularVelocity) / 9.81;

  // Virtual sensors inferred from G-forces
  const isBraking = longG < -0.5;  // Significant deceleration
  const isThrottle = longG > 0.1;   // Positive acceleration

  return {
    lat: current.lat,
    lng: current.lng,
    speed: current.speed * 2.237,  // Convert to mph
    gLat: latG,
    gLong: longG,
    brake: isBraking ? Math.min(100, Math.abs(longG) * 120) : 0,
    throttle: isThrottle ? Math.min(100, longG * 200) : 0,
    time: current.timestamp
  };
};

The CAN Bus Discovery

One of the biggest takeaways: AI can now interface with hardware effortlessly.

And then there was CAN bus (Controller Area Network)-the protocol that connects everything in a modern vehicle. Every sensor, every ECU, every control module talks over this two-wire bus.

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Engine    │     │   Brakes    │     │  Dashboard  │
│    ECU      │     │    ABS      │     │   Cluster   │
└──────┬──────┘     └──────┬──────┘     └──────┬──────┘
       │                   │                   │
       └───────────────────┴───────────────────┘
                     CAN Bus (500kbps)
                           │
                    ┌──────┴──────┐
                    │  OBD-II     │
                    │  Adapter    │
                    └──────┬──────┘
                           │
                    ┌──────┴──────┐
                    │   Laptop    │
                    │  + AI Agent │
                    └─────────────┘

With a $20 OBD-II adapter, you can read:

Real brake pressure (not GPS-derived)
Actual throttle position
Engine RPM, coolant temp, fuel level
Wheel speeds (individual!)
Steering angle

The next iteration will tap directly into CAN for ground-truth telemetry.

Predictive Geofencing

Best coaching comes before you need it. We defined geofences around every corner using Haversine distance:

const GEOFENCE_RADIUS = 200; // meters

const getFeedForwardAdvice = (
  currentPos: { lat: number; lon: number },
  corners: Corner[]
): string | null => {
  
  for (const corner of corners) {
    const distance = haversineDistance(
      currentPos.lat, currentPos.lon,
      corner.lat, corner.lon
    );

    // Trigger in 150-200m approach band
    if (distance <= GEOFENCE_RADIUS && distance > (GEOFENCE_RADIUS - 50)) {
      return `APPROACHING ${corner.name}: ${corner.advice}`;
    }
  }
  return null;
};

const haversineDistance = (lat1: number, lon1: number, lat2: number, lon2: number): number => {
  const R = 6371e3; // Earth radius in meters
  const φ1 = lat1 * Math.PI / 180;
  const φ2 = lat2 * Math.PI / 180;
  const Δφ = (lat2 - lat1) * Math.PI / 180;
  const Δλ = (lon2 - lon1) * Math.PI / 180;

  const a = Math.sin(Δφ/2) * Math.sin(Δφ/2) +
            Math.cos(φ1) * Math.cos(φ2) *
            Math.sin(Δλ/2) * Math.sin(Δλ/2);
            
  return R * 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a));
};

Low-Latency Audio

A 50ms AI decision is worthless if audio synthesis takes 200ms. We pre-cache all command audio:

class CoachAudioService {
  private audioContext: AudioContext;
  private buffers = new Map<string, AudioBuffer>();
  private minRepeatInterval = 1500; // Prevent spam
  
  async initialize(): Promise<void> {
    this.audioContext = new AudioContext();
    
    const commands = [
      'THRESHOLD', 'TRAIL_BRAKE', 'COMMIT', 'THROTTLE',
      'PUSH', 'NO_COAST', 'GOOD', 'SEND_IT'
    ];
    
    await Promise.all(
      commands.map(async cmd => {
        const response = await fetch(`/audio/${cmd}.mp3`);
        const buffer = await this.audioContext.decodeAudioData(
          await response.arrayBuffer()
        );
        this.buffers.set(cmd, buffer);
      })
    );
  }

  play(action: string): void {
    const buffer = this.buffers.get(action);
    if (!buffer) return;

    const source = this.audioContext.createBufferSource();
    source.buffer = buffer;
    source.connect(this.audioContext.destination);
    source.start(0);  // Instant - no network, no TTS synthesis
  }
}

Result: decision-to-audio in under 10ms.

Coach Personas

Different drivers respond differently. Tony needs encouragement. Rachel wants physics. Garmin wants data.

type CoachPersona = 'TONY' | 'RACHEL' | 'AJ' | 'GARMIN' | 'SUPER_AJ';

const COACH_PROMPTS: Record<CoachPersona, string> = {
  TONY: `Encouraging, feel-based. "Commit!", "Good hustle!", "Send it!"`,
  
  RACHEL: `Physics teacher. Focus on weight transfer, platform balance.
           "Smooth brake release", "Settle the rear"`,
           
  AJ: `Direct commands. Connect vehicle state to driver input.
       "Lat G stable-hammer throttle", "Rotation complete-unwind"`,
       
  GARMIN: `Data robot. Delta times only. "Brake 0.2s later", "-0.3s sector"`,
  
  SUPER_AJ: `Adaptive. Switch persona based on error type:
             Safety → Imperative ("STABILIZE!")
             Technique → Physics ("Smooth release")
             Confidence → Encouragement ("Commit now!")
             Optimization → Delta ("Brake later")`
};

Heuristic Fallback

Not everyone runs Chrome Canary with Nano flags. So there’s always a rule-based fallback:

const getHeuristicAction = (data: TelemetryData): HotAction => {
  const { speedMph, throttle, brakePos, latG, longG } = data;

  if (brakePos > 30 && longG < -0.2) return { action: "THRESHOLD", color: "#ef4444" };
  if (brakePos > 5 && Math.abs(latG) > 0.1) return { action: "TRAIL_BRAKE", color: "#ef4444" };
  if (Math.abs(latG) > 0.3 && throttle < 40) return { action: "COMMIT", color: "#3b82f6" };
  if (Math.abs(latG) > 0.15 && throttle < 60) return { action: "THROTTLE", color: "#f97316" };
  if (throttle > 60 && Math.abs(latG) < 0.1) return { action: "PUSH", color: "#f97316" };
  if (throttle < 15 && brakePos < 5 && speedMph > 50) return { action: "NO_COAST", color: "#eab308" };
  
  return { action: "MAINTAIN", color: "#a855f7" };
};

Tuned over hundreds of simulated laps. Not as smart as Nano, but works everywhere.

Designing for the Real World

Racing pushes hardware to the limit-and that made it the perfect stress test:

High G-forces taught us to secure every connection
Thermal extremes forced us to optimize for efficiency
Remote locations validated our offline-first edge architecture
Split-second timing proved the value of pre-cached responses

Every constraint became a design feature. No connectivity? Edge inference. Synthesis latency? Pre-cached audio. No Nano support? Heuristic fallback. The environment forced good architecture.

What’s Next

This was a fantastic AI field test, and there’s definitely more to come.

The pattern we validated-edge AI for reflexes, cloud AI for strategy, human expertise as ground truth-applies beyond racing:

Sports coaching
Surgical guidance
Head up display feedback
Agent with persona of the coach we meet in the room

Any domain with real-time feedback and verified human expertise to learn from.

Thank You

Huge thanks to Ajeet Mirwani for organizing this and giving us the thrill of being in a race car. Taking agents to the track was unforgettable.

And to the entire GDE team who collaborated across domains-AI, hardware, UX, racing-to make this happen. What an incredible effort.

This is just the beginning. 🏎️

Built as part of the GDE High-Velocity AI Field Test.

#ai