OpenAI API integration

What production integration adds to a working demo

A demo sends a prompt and prints the reply. A product has to deal with timeouts, malformed output, a customer pasting a 200-page file, a burst of traffic at month end and a finance team asking why the bill doubled. Each of those becomes a line of the integration.

• Model selection per task. A small, fast model is often enough for routing and tagging. A larger one is reserved for steps where quality shows to the user.
• Structured outputs. We define a JSON schema, so the response is parsed data and not a paragraph you try to scrape.
• Tool calling. The model requests a function you control, your code runs it, and the result goes back. You decide what the model may touch.
• Retries and backoff. Rate limits and transient errors are normal. Requests are retried with delays, and safe-to-repeat actions are marked so they are not duplicated.
• Spend controls. Per-user and per-day limits, and an alert when usage moves outside its normal range.
• Logging. Prompt version, model, tokens used, latency and outcome for each call, with personal data masked where possible.

Worked example: ticket triage with a confidence threshold

Suppose an online business gets support emails and wants them tagged and routed. This invented flow shows how the pieces fit.

1. ReceiveA new email triggers your backend. We strip signatures and quoted replies so the model sees only the new message.
2. ClassifyThe API is called with a schema: category (billing, shipping, technical, other), urgency (1 to 3), language, and a confidence value from 0 to 1.
3. ValidateCode checks that the category is in the allowed list. An unknown value is rejected, not accepted.
4. RouteConfidence above 0.85 goes straight to the right queue. Anything below goes to a human triage inbox with the model's guess attached.
5. LearnHumans' corrections are stored. Every month we replay them against the prompt to see whether it improved.

The threshold is not magic. We choose it by scoring a few hundred real emails and picking the point where wrong auto-routes are rarer than your tolerance.

Choosing between models and providers

We test two or three models on your own examples before deciding, because public benchmarks rarely match your task. We also keep the model call behind one internal function, so changing provider is a configuration change and a retest rather than a rewrite. If you are weighing vendors, read OpenAI API vs Claude API and our page on Claude API integration.

For a broader picture of connecting AI to your existing systems, see AI integration services. For apps built around a model, see LLM application development.

Data handling and privacy

Before any customer data goes to an API, we write down what is sent, where it is processed, how long the provider keeps it and what your privacy notice says. Business API terms differ from consumer chat products, and some accounts offer options such as zero data retention or regional processing. Availability changes, so we check the current terms for your account rather than assume.

• Send the minimum. Names and email addresses are removed or replaced with tokens when the task does not need them.
• Keep API keys on the server, never in browser code, and rotate them on a schedule.
• For EU and UK users, record the processing in your records of processing and check transfer terms. For California users, check what your notice says about service providers.
• Do not let user-supplied text decide which tools run. Validate in code.

Gotchas

• Model versions change. Pin a specific model version where the platform allows it, and retest before moving to a newer one.
• Token counts surprise people. Long conversation history is resent on each call. We trim or summarize.
• Output limits. Very long documents need splitting, and splitting needs care to avoid losing context.
• Rate limits depend on account tier and grow with usage history. Plan launches with enough headroom.
• Streaming is nice for chat but complicates validation. We stream only where the user benefits.

Timeline

The quote states the scope and the test set used to accept the work.