OmniAI::Google

LICENSE RubyGems GitHub Yard CircleCI

A Google implementation of the OmniAI APIs.

Installation

gem install omniai-google

Usage

Client

A client is setup as follows if ENV['GOOGLE_API_KEY'] exists:

client = OmniAI::Google::Client.new

A client may also be passed the following options:

  • api_key (required - default is ENV['GOOGLE_API_KEY'])
  • credentials (optional)
  • host (optional)
  • version (optional - options are v1 or v1beta)

Configuration

Vertex AI and Google AI offer different options for interacting w/ Google's AI APIs. Checkout the Vertex AI and Google AI differences to determine which option best fits your requirements.

Configuration w/ Google AI

If using Gemini simply provide an api_key:

OmniAI::Google.configure do |config|
  config.api_key = 'sk-...' # defaults is `ENV['GOOGLE_API_KEY']`
end

Configuration w/ Vertex AI

If using Vertex supply the credentials, host, location_id and project_id:

OmniAI::Google.configure do |config|
  config.credentials = File.open("./credentials.json") # default is `ENV['GOOGLE_CREDENTIALS_PATH']` / `ENV['GOOGLE_CREDENTIALS_JSON']`
  config.host = 'https://us-east4-aiplatform.googleapis.com' # default is `ENV['GOOGLE_HOST']`
  config.location_id = 'us-east4' # defaults is `ENV['GOOGLE_LOCATION_ID']`
  config.project_id = '...' # defaults is `ENV['GOOGLE_PROJECT_ID']`
end

Note for Transcription: When using transcription features, ensure your service account has the necessary permissions for Google Cloud Speech-to-Text API and Google Cloud Storage (for automatic file uploads). See the GCS Setup section below for detailed configuration.

Credentials may be configured using:

  1. A File / String / Pathname.
  2. Assigning ENV['GOOGLE_CREDENTIALS_PATH'] as the path to the credentials.json.
  3. Assigning ENV['GOOGLE_CREDENTIALS_JSON'] to the contents of credentials.json.

Chat

A chat completion is generated by passing in a simple text prompt:

completion = client.chat('Tell me a joke!')
completion.text # 'Why did the chicken cross the road? To get to the other side.'

A chat completion may also be generated by using the prompt builder:

completion = client.chat do |prompt|
  prompt.system('Your are an expert in geography.')
  prompt.user('What is the capital of Canada?')
end
completion.text # 'The capital of Canada is Ottawa.'

Model

model takes an optional string (default is gemini-3.8-flash):

completion = client.chat('How fast is a cheetah?', model: OmniAI::Google::Chat::Model::GEMINI_FLASH)
completion.text # 'A cheetah can reach speeds over 100 km/h.'

Google API Reference model

Temperature

temperature takes an optional float between 0.0 and 2.0:

completion = client.chat('Pick a number between 1 and 5', temperature: 2.0)
completion.text # '3'

Google API Reference temperature

Stream

stream takes an optional a proc to stream responses in real-time chunks instead of waiting for a complete response:

stream = proc do |chunk|
  print(chunk.text) # 'Better', 'three', 'hours', ...
end
client.chat('Be poetic.', stream:)

Max Output Tokens

max_tokens: caps the response, mapping to Gemini's generationConfig.maxOutputTokens:

client.chat("Summarize this page.", model: "gemini-3.7-flash", max_tokens: 8_000)

It can also be set globally, and a per-call value wins:

OmniAI::Google.configure do |config|
  config.chat_options[:max_tokens] = 8_000
end

The key must be a symbol. chat_options["max_tokens"] is neither read nor withheld from the payload, so it reaches Gemini as an unknown top-level field and the request fails with a 400.

The value is passed through unchanged — no floor is imposed, so the number you ask for is the number that reaches the wire. When the cap is hit, response.finish_reason.reason is :length.

Size it as thinking headroom plus expected answer. Unlike Anthropic's answer-only ceiling, Gemini spends this budget on thinking before emitting an answer. A cap sized to the expected answer alone gets consumed by thinking on any request the model reasons about:

max_tokens: 200  ->  finish_reason :length, 196 output tokens, 115 of them thinking, 81 characters of answer

Extended Thinking

Gemini models support extended thinking, which shows the model's reasoning process.

# Enable thinking with model defaults
response = client.chat("What is 25 * 25?", model: OmniAI::Google::Chat::Model::GEMINI_PRO, thinking: true)
Thinking Effort

Control how much the model thinks using the effort option ("low", "medium", or "high"):

# High effort (more reasoning tokens)
response = client.chat("Solve this step by step.", model: OmniAI::Google::Chat::Model::GEMINI_PRO, thinking: { effort: "high" })

# Medium effort
response = client.chat("Explain briefly.", model: OmniAI::Google::Chat::Model::GEMINI_PRO, thinking: { effort: "medium" })

# Low effort (fewer reasoning tokens)
response = client.chat("What is 1+1?", model: OmniAI::Google::Chat::Model::GEMINI_PRO, thinking: { effort: "low" })
Accessing Thinking Content
response.choices.first.message.content.each do |part|
  case part
  when OmniAI::Chat::Thinking
    puts "Thinking: #{part.thinking}"
  when OmniAI::Chat::Text
    puts "Response: #{part.text}"
  end
end
Streaming with Thinking
client.chat("What are the prime factors of 1234567?", model: OmniAI::Google::Chat::Model::GEMINI_PRO, thinking: true, stream: $stdout)

Google API Reference thinking

Upload

An upload is especially useful when processing audio / image / video / text files. To use:

CAT_URL = 'https://images.unsplash.com/photo-1472491235688-bdc81a63246e?fm=jpg'
DOG_URL = 'https://images.unsplash.com/photo-1517849845537-4d257902454a?fm=jpg'

begin
  cat_upload = client.upload(CAT_URL)
  dog_upload = client.upload(DOG_URL)

  completion = client.chat(stream: $stdout) do |prompt|
    prompt.user do |message|
      message.text 'What are these photos of?'
      message.url(cat_upload.uri, cat_upload.mime_type)
      message.url(dog_upload.uri, dog_upload.mime_type)
    end
  end
ensure
  cat_upload.delete!
  dog_upload.delete!
end

Google API Reference stream

Transcribe

Audio files can be transcribed using Google's Speech-to-Text API. The implementation automatically handles both synchronous and asynchronous recognition based on file size and model type.

Basic Usage

# Transcribe a local audio file
result = client.transcribe("path/to/audio.mp3")
result.text # "Hello, this is the transcribed text..."

# Transcribe with specific model
result = client.transcribe("path/to/audio.mp3", model: "latest_long")
result.text # "Hello, this is the transcribed text..."

Multi-Language Detection

The transcription automatically detects multiple languages when no specific language is provided:

# Auto-detect English and Spanish
result = client.transcribe("bilingual_audio.mp3", model: "latest_long")
result.text # "Hello, how are you? Hola, ¿cómo estás?"

# Specify expected languages explicitly
result = client.transcribe("audio.mp3", language: ["en-US", "es-US"], model: "latest_long")

Detailed Transcription with Timestamps

Use VERBOSE_JSON format to get detailed timing information, confidence scores, and language detection per segment:

result = client.transcribe("audio.mp3", 
  model: "latest_long", 
  format: OmniAI::Transcribe::Format::VERBOSE_JSON
)

# Access the full transcript
result.text # "Complete transcribed text..."

# Access detailed segment information
result.segments.each do |segment|
  puts "Segment #{segment[:segment_id]}: #{segment[:text]}"
  puts "Language: #{segment[:language_code]}"
  puts "Confidence: #{segment[:confidence]}"
  puts "End time: #{segment[:end_time]}"
  
  # Word-level timing (if available)
  segment[:words].each do |word|
    puts "  #{word[:word]} (#{word[:start_time]} - #{word[:end_time]})"
  end
end

# Total audio duration
puts "Total duration: #{result.total_duration}"

Models

The transcription supports various models optimized for different use cases:

# For short audio (< 60 seconds)
client.transcribe("short_audio.mp3", model: OmniAI::Google::Transcribe::Model::LATEST_SHORT)

# For long-form audio (> 60 seconds) - automatically uses async processing
client.transcribe("long_audio.mp3", model: OmniAI::Google::Transcribe::Model::LATEST_LONG)

# For phone/telephony audio
client.transcribe("phone_call.mp3", model: OmniAI::Google::Transcribe::Model::TELEPHONY_LONG)

# For medical conversations
client.transcribe("medical_interview.mp3", model: OmniAI::Google::Transcribe::Model::MEDICAL_CONVERSATION)

# Latest generation multilingual model (recommended)
client.transcribe("audio.mp3", model: OmniAI::Google::Transcribe::Model::CHIRP_3)

# Other available models
client.transcribe("audio.mp3", model: OmniAI::Google::Transcribe::Model::CHIRP_2) # Enhanced model
client.transcribe("audio.mp3", model: OmniAI::Google::Transcribe::Model::CHIRP)   # Universal model

Available Model Constants:

  • OmniAI::Google::Transcribe::Model::CHIRP_3 - Latest-generation multilingual ASR model (recommended)
  • OmniAI::Google::Transcribe::Model::CHIRP_2 - Enhanced universal model
  • OmniAI::Google::Transcribe::Model::CHIRP - Universal model
  • OmniAI::Google::Transcribe::Model::LATEST_SHORT - Optimized for audio < 60 seconds
  • OmniAI::Google::Transcribe::Model::LATEST_LONG - Optimized for long-form audio
  • OmniAI::Google::Transcribe::Model::TELEPHONY - For phone/telephony audio
  • OmniAI::Google::Transcribe::Model::TELEPHONY_SHORT - For short phone calls
  • OmniAI::Google::Transcribe::Model::TELEPHONY_LONG - For long phone calls
  • OmniAI::Google::Transcribe::Model::MEDICAL_CONVERSATION - For medical conversations
  • OmniAI::Google::Transcribe::Model::MEDICAL_DICTATION - For medical dictation

Region note: CHIRP_3 is only served from the us and eu multi-region endpoints (not global, and not zonal regions like us-east4). The provider maps the configured location_id to its multi-region parent — any us* region resolves to us, any eu/europe* region resolves to eu — and defaults to us when nothing is configured. This means a Vertex AI client configured with a zonal location_id (e.g. us-east4) for Gemini will still route CHIRP_3 correctly. CHIRP_2 is always routed to us-central1.

Supported Formats

  • Input: MP3, WAV, FLAC, and other common audio formats
  • GCS URIs: Direct transcription from Google Cloud Storage
  • File uploads: Automatic upload to GCS for files > 10MB or long-form models

Advanced Features

Automatic Processing Selection:

  • Files < 60 seconds: Uses synchronous recognition
  • Files > 60 seconds or long-form models: Uses asynchronous batch recognition
  • Large files: Automatically uploaded to Google Cloud Storage

GCS Integration:

  • Automatic file upload and cleanup
  • Support for existing GCS URIs
  • Configurable bucket names

Error Handling:

  • Automatic retry logic for temporary failures
  • Clear error messages for common issues
  • Graceful handling of network timeouts

Google Speech-to-Text API Reference

Gemini Transcribe

gemini-3.5-transcribe is a Gemini model served by Vertex generateContent, not Speech-to-Text, so it needs Vertex credentials. Pass it as the model; the Chirp models above are unaffected.

transcription = client.transcribe("interview.wav", model: OmniAI::Google::Transcribe::Model::GEMINI_3_5_TRANSCRIBE)
transcription.text

# Speaker labels per utterance (experimental, up to 8 speakers)
transcription = client.transcribe("interview.wav", model: "gemini-3.5-transcribe", diarization: true, format: "verbose_json")
transcription.segments # => [{ "id" => 0, "text" => "...", "speaker" => "spk:0" }, ...]

# Language hints and custom vocabulary (up to 1,000 terms); the language is auto-detected when omitted
client.transcribe("interview.wav", model: "gemini-3.5-transcribe", language: "es-US", vocabulary: %w[ibuprofeno])

# Word times (experimental, off by default; see the caveat below)
client.transcribe("interview.wav", model: "gemini-3.5-transcribe", word_timestamps: true, format: "verbose_json")
  • Location: served from global and the us / eu multi-regions, never a single region. A zonal location_id maps to its multi-region (us by default, eu for eu* / europe*); global is kept.
  • Input: wav, mp3, flac, ogg, webm, m4a or aiff (detected from the bytes), or a gs:// URI. Files over 10 MB are uploaded to the <project_id>-speech-audio bucket and deleted afterwards.
  • Length: Google caps a file at 15 minutes when diarization or timestamps are on, so split longer recordings.
  • Word timestamps can drop speech. In live tests on a two-speaker, 8-second clip (2026-10-15), word_timestamps: true returned only the first word of the answer, and with diarization: true as well it dropped the whole first utterance. Google labels both features experimental and warns that word timestamps degrade accuracy. diarization: alone returned the complete transcript but no times.

GCS Setup for Transcription

For transcription to work properly with automatic file uploads, you need to set up Google Cloud Storage and configure the appropriate permissions.

1. Create a GCS Bucket

You must create a bucket named {project_id}-speech-audio manually before using transcription features:

# Using gcloud CLI
gsutil mb gs://your-project-id-speech-audio

# Or create via Google Cloud Console
# Navigate to Cloud Storage > Browser > Create Bucket
2. Service Account Permissions

Your service account needs the following IAM roles for transcription to work:

Required Roles:

  • Cloud Speech Editor - Grants access to edit resources in Speech-to-Text
  • Storage Bucket Viewer - Grants permission to view buckets and their metadata, excluding IAM policies
  • Storage Object Admin - Grants full control over objects, including listing, creating, viewing, and deleting objects

To assign roles via gcloud CLI:

# Replace YOUR_SERVICE_ACCOUNT_EMAIL and YOUR_PROJECT_ID with actual values
SERVICE_ACCOUNT="your-service-account@your-project-id.iam.gserviceaccount.com"
PROJECT_ID="your-project-id"

# Grant Speech-to-Text permissions
gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:$SERVICE_ACCOUNT" \
    --role="roles/speech.editor"

# Grant Storage permissions
gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:$SERVICE_ACCOUNT" \
    --role="roles/storage.objectAdmin"

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:$SERVICE_ACCOUNT" \
    --role="roles/storage.legacyBucketReader"

Or via Google Cloud Console:

  1. Go to IAM & Admin > IAM
  2. Find your service account
  3. Click "Edit Principal"
  4. Add the required roles listed above
3. Enable Required APIs

Ensure the following APIs are enabled in your Google Cloud Project:

# Enable Speech-to-Text API
gcloud services enable speech.googleapis.com

# Enable Cloud Storage API  
gcloud services enable storage.googleapis.com
4. Bucket Configuration (Optional)

You can customize the bucket name by configuring it in your application:

# Custom bucket name in your transcription calls
# The bucket must exist and your service account must have access
client.transcribe("audio.mp3", bucket_name: "my-custom-audio-bucket")

Important Notes:

  • The default bucket name follows the pattern: {project_id}-speech-audio
  • You must create the bucket manually before using transcription features
  • Choose an appropriate region for your bucket based on your location and compliance requirements
  • Audio files are automatically deleted after successful transcription
  • If transcription fails, temporary files may remain and should be cleaned up manually

Live

A Gemini Live session (Vertex BidiGenerateContent over a WebSocket): stream microphone audio in, get speech, transcripts and tool calls back. Vertex credentials only.

client = OmniAI::Google::Client.new(
  project_id: "...", credentials: "...",
  location_id: "us", host: "https://aiplatform.us.rep.googleapis.com" # gemini-3.8-live: us, eu or us-central1 (not global)
)

session = client.live(
  model: OmniAI::Google::Live::Model::GEMINI_3_8_LIVE,
  system: "You are an intake assistant...",
  tools: [RecordAnswer.new],       # OmniAI::Tool; run automatically, results sent back
  tool_behavior: :blocking,        # default; or :non_blocking, or { "flag" => :non_blocking }
  voice: "Kore",                   # optional
  vad: { silence_ms: 800 },        # optional
  compression: { trigger_tokens: 10_000, target_tokens: 5_000 } # optional (or true): see Limits
) do |event|
  case event
  when OmniAI::Google::Live::Event::Audio then speaker.write(event.data) # PCM16, 24 kHz, mono
  when OmniAI::Google::Live::Event::Transcript then caption(event.role, event.text) # :user / :model
  when OmniAI::Google::Live::Event::Interrupted then speaker.flush
  when OmniAI::Google::Live::Event::GoAway then reconnect_within(event.time_left)
  when OmniAI::Google::Live::Event::Usage then track(event.usage, event.data) # data has per-modality counts
  when OmniAI::Google::Live::Event::Closed then reconnect_or_finish(event.code, event.error)
  end
end

session.send_audio(pcm16_16khz) # from any thread
session.send_text("We are at question 12; answers so far: ...") # e.g. to re-seed after reconnecting
session.close
  • Threads: each session has one socket and one reader thread. Events are yielded, and tools run, on that thread; send_* may be called from any thread. A send that blocks longer than write_timeout: (default 5 s) closes the session and raises Live::TimeoutError, as does a connect (TCP + TLS) slower than 10 s. An idle connection is pinged every 15 s; with no answer in 10 s it is dropped, so a silently dead path still ends the session. The session always ends with exactly one Event::Closed, including when the socket drops or your handler raises (event.error).
  • Tools: a raising tool answers the model with its error class only, and the session carries on.
  • Limits (Google): a connection lasts about 10 minutes, and an audio-only session 15 minutes without compression:. Measured live at us (2026-10-15): GoAway arrived at 9 min (time_left 30 s, not the 60 s Google documents) and the server closed with 1000 at 9.5 min. Reconnect when GoAway arrives, not on a timer: the documented numbers are off.
  • Cost: every turn re-bills the whole context. compression: bounds it: in a live test with a 10,000-token trigger and 5,000-token target, promptTokenCount dropped from about 8,500 to 3,400 whenever the next turn would pass the trigger (a sawtooth instead of linear growth; 51% fewer prompt tokens over 12 turns). The oldest turns are dropped, so keep anything the model must remember in your own state and re-send it when needed. The gem does not reconnect: open a new session and re-seed it (resumption: takes Google's resumption handles if you prefer that).
  • Privacy: the gem never logs or raises audio, transcripts or tool arguments. Live::Error#message carries codes and ids only; Google's close reason is on #reason, and Event::Closed#inspect leaves it out. A server frame that fails to parse closes the session with a Live::Error naming only the exception class.

Embed

Text can be converted into a vector embedding for similarity comparison usage via:

response = client.embed('The quick brown fox jumps over a lazy dog.')
response.embedding # [0.0, ...]