Class: OmniAI::Google::Live

Inherits:
Object
  • Object
show all
Defined in:
lib/omniai/google/live.rb,
lib/omniai/google/live/event.rb,
lib/omniai/google/live/connection.rb

Overview

A Gemini Live API session on Vertex (BidiGenerateContent over a WebSocket): stream microphone audio in, receive speech, transcripts and tool calls back.

Usage:

session = client.live(model: "gemini-3.8-live", system: "...", tools: [RecordAnswer.new]) do |event|
case event
when OmniAI::Google::Live::Event::Audio then play(event.data)
when OmniAI::Google::Live::Event::Closed then reconnect
end
end
session.send_audio(pcm16_16khz)
session.close

Each session holds one socket and one reader thread. Events are yielded, and tools run, on that thread; the send_* methods may be called from any thread. The gem never logs or raises audio, transcripts or tool arguments.

Defined Under Namespace

Modules: Event, Model Classes: ClosedError, Connection, Error, TimeoutError

Constant Summary collapse

DEFAULT_MODEL = 
Model::GEMINI_3_8_LIVE
INPUT_MIME_TYPE = 
"audio/pcm;rate=16000"
PATH = 
"/ws/google.cloud.aiplatform.v1.LlmBidiService/BidiGenerateContent"
SETUP_TIMEOUT =

seconds

15
WRITE_TIMEOUT =

seconds

5
BEHAVIORS = 
{ blocking: "BLOCKING", non_blocking: "NON_BLOCKING" }.freeze
ACTIVITY = 
{ "ACTIVITY_START" => :start, "ACTIVITY_END" => :end }.freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(client:, model: DEFAULT_MODEL, system: nil, tools: nil, tool_behavior: :blocking, voice: nil, transcription: true, vad: nil, compression: false, resumption: nil, write_timeout: WRITE_TIMEOUT) {|event| ... } ⇒ Live

Returns a new instance of Live.

Parameters:

  • client (OmniAI::Google::Client) —

    must use Vertex credentials, a project and a location (e.g. "us")

  • model (String) (defaults to: DEFAULT_MODEL)
  • system (String, nil) (defaults to: nil) —

    the system instruction

  • tools (Array<OmniAI::Tool>, nil) (defaults to: nil) —

    run automatically; each result is sent back as a tool response

  • tool_behavior (Symbol, Hash) (defaults to: :blocking) —

    :blocking (default) or :non_blocking, or a Hash of tool name to either

  • voice (String, nil) (defaults to: nil) —

    a prebuilt voice name, e.g. "Kore"

  • transcription (Boolean) (defaults to: true) —

    transcribe both the user's and the model's speech (default true)

  • vad (Hash, nil) (defaults to: nil) —

    automatic activity detection: disabled:, start_sensitivity: / end_sensitivity: (:high or :low), prefix_padding_ms:, silence_ms:

  • compression (Boolean, Hash) (defaults to: false) —

    sliding-window context compression, which lifts the 15-minute session cap and bounds the context re-billed every turn. true uses Google's defaults (trigger at 80% of the context window); or pass trigger_tokens: / target_tokens:

  • resumption (Boolean, String, nil) (defaults to: nil) —

    true to receive resumption handles, or a handle to resume

  • write_timeout (Numeric) (defaults to: WRITE_TIMEOUT) —

    seconds a send may block before the session is closed

Yield Parameters:



77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
 
# File 'lib/omniai/google/live.rb', line 77

def initialize(client:, model: DEFAULT_MODEL, system: nil, tools: nil, tool_behavior: :blocking, voice: nil,
  transcription: true, vad: nil, compression: false, resumption: nil, write_timeout: WRITE_TIMEOUT, &handler)
  @client = client
  @model = model
  @system = system
  @tools = tools || []
  @tool_behavior = tool_behavior
  @voice = voice
  @transcription = transcription
  @vad = vad
  @compression = compression
  @resumption = resumption
  @write_timeout = write_timeout
  @handler = handler
  @state = :new
end

Instance Attribute Details

#session_id ⇒ String? (readonly)

Returns the server's session id, once open.

Returns:

  • (String, nil) —

    the server's session id, once open



59
60
61
 
# File 'lib/omniai/google/live.rb', line 59

def session_id
  @session_id
end

Class Method Details

.open ⇒ Live

Opens a session and waits for the server's setupComplete.

Returns:

Raises:

  • (Error) —

    if the server rejects the setup or it times out



54
55
56
 
# File 'lib/omniai/google/live.rb', line 54

def self.open(...)
  new(...).tap(&:open)
end

Instance Method Details

#close ⇒ Object 



141
142
143
 
# File 'lib/omniai/google/live.rb', line 141

def close
  @connection&.close
end

#end_audio ⇒ Object

Tells the server the microphone paused, so it flushes any buffered audio.



130
131
132
 
# File 'lib/omniai/google/live.rb', line 130

def end_audio
  deliver(realtimeInput: { audioStreamEnd: true })
end

#open ⇒ Live

Returns:

Raises:



97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
 
# File 'lib/omniai/google/live.rb', line 97

def open
  raise ArgumentError, "#{self.class.name} requires Vertex credentials" unless @client.credentials?

  @ready = Queue.new
  @connection = Connection.new(
    url:,
    headers: { "Authorization" => @client.send(:auth) }, # a fresh token per connection
    write_timeout: @write_timeout,
    on_open: method(:on_open),
    on_message: method(:on_message),
    on_close: method(:on_close)
  )
  log("LIVE connect #{host} #{@model}")
  begin
    @connection.connect
  rescue Connection::ConnectTimeout => e
    raise TimeoutError, "#{self.class.name} #{e.message} (model=#{@model})"
  end
  await_setup!
  self
end

#open? ⇒ Boolean

Returns:

  • (Boolean) 


120
121
122
 
# File 'lib/omniai/google/live.rb', line 120

def open?
  @state == :open && @connection.open?
end

#send_audio(audio) ⇒ Object

Parameters:

  • audio (String) —

    raw PCM16 audio, 16 kHz, mono, little-endian



125
126
127
 
# File 'lib/omniai/google/live.rb', line 125

def send_audio(audio)
  deliver(realtimeInput: { audio: { mimeType: INPUT_MIME_TYPE, data: Base64.strict_encode64(audio) } })
end

#send_text(text) ⇒ Object

Sends a complete user turn of text, e.g. to re-seed context after reconnecting.

Parameters:

  • text (String) 


137
138
139
 
# File 'lib/omniai/google/live.rb', line 137

def send_text(text)
  deliver(clientContent: { turns: [{ role: "user", parts: [{ text: }] }], turnComplete: true })
end