Source URL: https://simonwillison.net/2024/Dec/17/openai-webrtc/#atom-everything
Source: Simon Willison’s Weblog
Title: OpenAI WebRTC Audio demo
Feedly Summary: OpenAI WebRTC Audio demo
OpenAI announced a bunch of API features today, including a brand new WebRTC API for setting up a two-way audio conversation with their models.
They tweeted this opaque code example:
async function createRealtimeSession(inStream, outEl, token) {
const pc = new RTCPeerConnection();
pc.ontrack = e => outEl.srcObject = e.streams[0];
pc.addTrack(inStream.getTracks()[0]);
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);
const headers = { Authorization: Bearer ${token}, ‘Content-Type’: ‘application/sdp’ };
const opts = { method: ‘POST’, body: offer.sdp, headers };
const resp = await fetch(‘https://api.openai.com/v1/realtime’, opts);
await pc.setRemoteDescription({ type: ‘answer’, sdp: await resp.text() });
return pc;
}
So I pasted that into Claude and had it build me this interactive demo for trying out the new API.
My demo uses an OpenAI key directly, but the most interesting aspect of the new WebRTC mechanism is its support for ephemeral tokens.
This solves a major problem with their previous realtime API: in order to connect to their endpoint you need to provide an API key, but that meant making that key visible to anyone who uses your application. The only secure way to handle this was to roll a full server-side proxy for their WebSocket API, just so you could hide your API key in your own server.
Ephemeral tokens solve that by letting you make a server-side call to request an ephemeral token which will only allow a connection to be initiated to their WebRTC endpoint for the next 60 seconds. The user’s browser then starts the connection, which will last for up to 30 minutes.
Tags: claude, audio, openai, ai, llms, ai-assisted-programming, tools, generative-ai, api, security
AI Summary and Description: Yes
Summary: The text discusses OpenAI’s newly announced WebRTC API, highlighting its ability to facilitate two-way audio conversations while addressing security concerns around API key visibility through the use of ephemeral tokens. This development showcases how emerging technologies can enhance real-time communication capabilities while minimizing security risks for developers.
Detailed Description:
The provided text presents significant developments regarding OpenAI’s new WebRTC API, which enables two-way audio conversations with their models. Here are the major points derived from the content:
– **OpenAI’s WebRTC API**: The launch of this API is a notable enhancement to OpenAI’s capabilities in real-time communication.
– **Code Example**: The API allows developers to set up audio streaming efficiently, with a focus on usability and functionality.
– **Security Improvement with Ephemeral Tokens**:
– **Previous Security Concerns**: Earlier versions required developers to expose their API keys to end-users, introducing substantial security risks.
– **Ephemeral Tokens Solution**: The introduction of ephemeral tokens allows a secure server-side request to generate a temporary token, granting limited-time access to the API (valid for only 60 seconds).
– This significantly reduces the risk because the token does not persist beyond the necessity of the connection setup.
– **Connection Durability**: Once initiated, the connection can last up to 30 minutes, enabling users to engage in prolonged interactions without compromising security.
– **Implications for Developers**: This API can be beneficial for developers aiming to integrate audio conversations securely in their applications without implementing complex proxy solutions.
In summary, the text highlights a crucial evolution in API security, showcasing how modern technology can facilitate advanced functionalities like real-time communications while addressing significant security concerns, which is particularly relevant for professionals in AI, software development, and cloud computing.