Research preview

Making a request

One SID-2 episode begins with a question and ends with ranked IDs. The question reaches SID-2 along an arrow labeled + tools. SID-2 sends search and read tool calls to your search backend and episode cache, which return snippets or chunks and metadata. Parallel arrows in both directions represent multiple calls across turns. The report comes from SID-2, not directly from the backend. A dashed box highlights Making a request: the question sent to SID-2.

Access SID-2 through the OpenAI Responses API. SID-2 does not support chat completions or stateful serving. Set store=False (store: false in TypeScript); you own the conversation and resend all of it on every call in the episode.

SID-2 lives behind the first-party SID API at https://api.sid-1.com/v1, authenticated with a SID_API_KEY as Authorization: Bearer sid-sk-.... Send requests to POST /v1/responses. Create a key on the API keys page. You may need to contact SID to enable SID-2 access for your account.

import os
from openai import OpenAI

client = OpenAI(base_url="https://api.sid-1.com/v1", api_key=os.environ["SID_API_KEY"])
MODEL = "sid-2"
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.sid-1.com/v1",
  apiKey: process.env.SID_API_KEY,
});
const MODEL = "sid-2";
import (
	"os"

	"github.com/openai/openai-go/v3"
	"github.com/openai/openai-go/v3/option"
)

client := openai.NewClient(
	option.WithBaseURL("https://api.sid-1.com/v1"),
	option.WithAPIKey(os.Getenv("SID_API_KEY")),
)
const model = "sid-2"

The model ID is sid-2.

SID-2 is also available through Baseten, authenticated with a BASETEN_API_KEY. Send requests to POST /v1/responses.

import os
from openai import OpenAI

client = OpenAI(base_url="https://inference.baseten.co/v1", api_key=os.environ["BASETEN_API_KEY"])
MODEL = "sid/sid-2-preview"
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://inference.baseten.co/v1",
  apiKey: process.env.BASETEN_API_KEY,
});
const MODEL = "sid/sid-2-preview";
import (
	"os"

	"github.com/openai/openai-go/v3"
	"github.com/openai/openai-go/v3/option"
)

client := openai.NewClient(
	option.WithBaseURL("https://inference.baseten.co/v1"),
	option.WithAPIKey(os.Getenv("BASETEN_API_KEY")),
)
const model = "sid/sid-2-preview"

SID-2 is also available as an AWS Marketplace model package that you deploy to an Amazon SageMaker real-time endpoint. A SageMaker endpoint serves a single route, POST /invocations, and authenticates with AWS SigV4 request signing instead of a bearer token. The responses-sagemaker package provides an httpx transport that handles both, so the OpenAI client and client.responses.create work unchanged. See the repository for installation.

Set the region with AWS_DEFAULT_REGION or your AWS profile. boto3 ignores AWS_REGION, so setting only that signs requests for the wrong region. Your credentials need sagemaker:InvokeEndpoint on the endpoint.

import boto3
import httpx
from openai import OpenAI
from responses_sagemaker import SageMakerResponsesTransport

session = boto3.Session()  # credentials and region, as boto3 resolves them
client = OpenAI(
    base_url=f"https://runtime.sagemaker.{session.region_name}.amazonaws.com/endpoints/<endpoint-name>",
    api_key="unused",  # SageMaker authenticates with SigV4; the SDK requires a key
    http_client=httpx.Client(
        transport=SageMakerResponsesTransport(credentials=session.get_credentials(), region=session.region_name)
    ),
)
MODEL = "sid/sid-2-preview"

The API accepts tool descriptions and reasoning effort in the standard OpenAI API parameters. Additionally, the API accepts two new parameters, "tool_examples" and "reward", passed in the "extra_body" of the Python API request. In TypeScript, put the sid object directly in the request body and pass headers as the second argument to client.responses.create. In Go, set sid and the tools on the request body with option.WithJSONSet, and the header with option.WithHeader.

Create a new session ID for every episode and pass it in the x-session-affinity header. This information is used for request routing to provide a higher KV cache hit rate, ensuring lower latency and cost.

Build one request object and send it unchanged on every turn. Only the input should grow:

# One dict, sent unchanged on every request.
request = dict(
    model=MODEL,
    max_output_tokens=4096,
    store=False,
    include=["reasoning.encrypted_content"],
    instructions=INSTRUCTIONS,
    tools=TOOLS,
    reasoning={"effort": REASONING_EFFORT},
    extra_body={"sid": {"tool_examples": TOOL_EXAMPLES, "reward": REWARD}},
    extra_headers={"x-session-affinity": session_id},
)

history = [{"role": "user", "content": QUESTION}]
response = client.responses.create(input=history, **request)
import { randomUUID } from "node:crypto";
import type { ResponseCreateParamsNonStreaming, ResponseInput } from "openai/resources/responses/responses";

const sessionId = randomUUID();
const request: Omit<ResponseCreateParamsNonStreaming, "input"> & { sid: object } = {
  model: MODEL,
  max_output_tokens: 4096,
  store: false,
  include: ["reasoning.encrypted_content"],
  instructions: INSTRUCTIONS,
  tools: TOOLS,
  reasoning: { effort: REASONING_EFFORT },
  sid: { tool_examples: TOOL_EXAMPLES, reward: REWARD },
};
const options = { headers: { "x-session-affinity": sessionId } };
const history: ResponseInput = [{ role: "user", content: QUESTION }];
let response = await client.responses.create({ ...request, input: history }, options);
import (
	"crypto/rand"

	"github.com/openai/openai-go/v3"
	"github.com/openai/openai-go/v3/option"
	"github.com/openai/openai-go/v3/responses"
	"github.com/openai/openai-go/v3/shared"
)

// One request, sent unchanged on every turn.
request := responses.ResponseNewParams{
	Model:           model,
	MaxOutputTokens: openai.Int(4096),
	Store:           openai.Bool(false),
	Include:         []responses.ResponseIncludable{responses.ResponseIncludableReasoningEncryptedContent},
	Instructions:    openai.String(instructions),
	Reasoning:       shared.ReasoningParam{Effort: reasoningEffort},
}
// The tools and the sid extension are plain JSON, set on the request body.
options := []option.RequestOption{
	option.WithJSONSet("tools", tools),
	option.WithJSONSet("sid", map[string]any{"tool_examples": toolExamples, "reward": reward}),
	option.WithHeader("x-session-affinity", rand.Text()), // a new session ID for every episode
}

history := responses.ResponseInputParam{
	responses.ResponseInputItemParamOfMessage(question, responses.EasyInputMessageRoleUser),
}
request.Input = responses.ResponseNewParamsInputUnion{OfInputItemList: history}
resp, err := client.Responses.New(ctx, request, options...)
if err != nil {
	log.Fatal(err)
}

The prompt (instructions, tools, reasoning, and the sid extra body) is fixed for the life of a conversation. It takes effect on the first request and applies to every turn after it.

What comes back

A standard Responses object. The parts you act on:

{
  "id": "resp_...",
  "status": "completed",
  "output": [
    {
      "type": "reasoning",
      "id": "rs_...",
      "content": [],
      "encrypted_content": "ttrs1_..."
    },
    {
      "type": "message",
      "id": "msg_...",
      "role": "assistant",
      "content": [{"type": "output_text", "text": "...", "annotations": []}]
    },
    {
      "type": "function_call",
      "id": "fc_...",
      "call_id": "call_...",
      "name": "search",
      "arguments": "{\"query\": \"...\", \"limit\": 5}"
    }
  ],
  "usage": {
    "input_tokens": 0,
    "input_tokens_details": {"cached_tokens": 0},
    "output_tokens": 0,
    "output_tokens_details": {"reasoning_tokens": 0},
    "total_tokens": 0
  },
  "...": "the remaining standard Responses fields"
}
  • output: the turn's items, in the order the model produced them: a reasoning item, then a message if the model wrote prose, then one function_call per tool call. A turn that only calls tools has no message.
  • encrypted_content: the model's thinking, encrypted. content is always empty; the thinking is never returned in plaintext. Hand the item back unchanged on the next request and the model continues where it left off.
  • arguments: a JSON string, not an object. Parse it before use.
  • status: "completed", or "incomplete" when the turn hit max_output_tokens. An incomplete turn may have been cut off mid-tool-call, so end the episode rather than acting on it.
  • usage.output_tokens_details.reasoning_tokens: how many of the output tokens were thinking. The thinking itself is encrypted, so this is the only measure of it you get. usage.input_tokens_details.cached_tokens reports prefix-cache hits: because the conversation grows by appending, later turns in an episode reuse most of their prompt.

Continuing the conversation

The loop makes one Responses API call per turn, resending the whole conversation as input each time (store=False, so nothing is kept server-side). After a non-terminal turn, append three kinds of items to the history:

  1. The model's output items, verbatim. The reasoning items carry the encrypted_content that lets the model continue its thinking. Model output items cannot be edited, dropped or reordered after they have been produced.
  2. One function_call_output item per tool call, carrying the rendered observation as an output string. Serialize structured results yourself.
  3. A user message telling SID-2 how many turns remain.
history += [item.model_dump(mode="json", exclude_none=True) for item in response.output]
history += [
    {"type": "function_call_output", "call_id": call.call_id, "output": observation}
    for call, observation in zip(calls, observations)
]
history.append({"role": "user", "content": f"You have {remaining} turns remaining out of {MAX_TURNS} total."})

response = client.responses.create(input=history, **request)
history.push(...response.output as ResponseInput);
history.push(...calls.map((call, index) => ({
  type: "function_call_output" as const,
  call_id: call.call_id,
  output: observations[index],
})));
history.push({ role: "user", content: `You have ${remaining} turns remaining out of ${MAX_TURNS} total.` });

response = await client.responses.create({ ...request, input: history }, options);
var calls []responses.ResponseFunctionToolCall
for _, item := range resp.Output {
	// Verbatim, including the encrypted reasoning.
	history = append(history, param.Override[responses.ResponseInputItemUnionParam](json.RawMessage(item.RawJSON())))
	if item.Type == "function_call" {
		calls = append(calls, item.AsFunctionCall())
	}
}
// ... run the calls and render their observations ...
for i, call := range calls {
	output := responses.ResponseInputItemParamOfFunctionCallOutput(observations[i])
	output.OfFunctionCallOutput.CallID = openai.String(call.CallID)
	history = append(history, output)
}
history = append(history, responses.ResponseInputItemParamOfMessage(
	fmt.Sprintf("You have %d turns remaining out of %d total.", remaining, maxTurns),
	responses.EasyInputMessageRoleUser,
))

request.Input = responses.ResponseNewParamsInputUnion{OfInputItemList: history}
resp, err = client.Responses.New(ctx, request, options...)

Send the observations in the order the model made the calls: they are matched to calls by position, not by call_id. If you run tools in parallel, collect the results back into call order before appending them.

Reward

SID-2 is trained to vary its search and reporting strategy to either be biased toward precision or recall based on the "reward" parameter. The following values are accepted:

  • "f1": favor precision, reporting a narrower set of documents
  • "ndcg": favor recall, reporting a broader set of documents
  • "recall@<n>": replace <n> with the number of documents to instruct the model to report exactly this many documents, for <n> = 1, 2, 5, 10, 20, 50

SID-2 is trained to always rank the most relevant documents first, independent of reward.

Tool examples

SID-2 benefits from few-shot demonstrations of tools, especially tools with high complexity. Give your tool-call examples captions that explain how to use them:

TOOL_EXAMPLES = [
    {
        "caption": "Use the semantic search tool with a long, descriptive query:",
        "name": "search",
        "arguments": {
            "query": "what makes SID's approach to retrieval different from standard search pipelines?",
            "limit": 5,
        },
    },
    {
        "caption": "Use the keyword search tool to find documents containing specific words:",
        "name": "text_search",
        "arguments": {"query": "SID AI retrieval posttraining", "limit": 5},
    },
    ...
]
const TOOL_EXAMPLES = [
  {
    caption: "Use the semantic search tool with a long, descriptive query:",
    name: "search",
    arguments: {
      query: "what makes SID's approach to retrieval different from standard search pipelines?",
      limit: 5,
    },
  },
  {
    caption: "Use the keyword search tool to find documents containing specific words:",
    name: "text_search",
    arguments: { query: "SID AI retrieval posttraining", limit: 5 },
  },
  // Add an example for each remaining tool.
];
var toolExamples = []map[string]any{
	{
		"caption": "Use the semantic search tool with a long, descriptive query:",
		"name":    "search",
		"arguments": map[string]any{
			"query": "what makes SID's approach to retrieval different from standard search pipelines?",
			"limit": 5,
		},
	},
	{
		"caption":   "Use the keyword search tool to find documents containing specific words:",
		"name":      "text_search",
		"arguments": map[string]any{"query": "SID AI retrieval posttraining", "limit": 5},
	},
	// Add an example for each remaining tool.
}

Provide at least one example for every tool you give the model.

Rate limits

The first-party SID API uses requests per second (RPS). Every response carries the current budget:

x-ratelimit-limit: 5
x-ratelimit-remaining: 4
x-ratelimit-scope: model:sid-2
retry-after: 1                     # on a 429 only

x-ratelimit-limit is requests per second (RPS) and is set per key, so read it rather than hard-coding a number; the default is single-digit. A 429 returns {"error": {"type": "rate_limit_error", ...}}; honor retry-after.

Treat the advertised limit as an upper bound rather than a set-point. Under a burst, 429 and 200 responses have been observed advertising different limits, so a client that simply copies the last header it saw will oscillate. Back off on a 429 and probe back up on success.

Because the ceiling is low, pace your requests when you run many episodes at once, rather than discovering the limit by collecting 429s.

Baseten uses requests per minute (RPM) and tokens per minute (TPM). Its response headers are:

HeaderValue
x-ratelimit-limit-requestsMaximum requests per minute.
x-ratelimit-remaining-requestsRequests currently available.
x-ratelimit-limit-tokensMaximum tokens per minute.
x-ratelimit-remaining-tokensTokens currently available.

TPM includes input and output tokens, with cached and uncached input tokens counted equally. Check the headers on successful responses and 429 errors for your current limits. Pace requests against both budgets and use exponential backoff after a 429.

See Baseten's rate limits for details.

SageMaker endpoints have no per-endpoint rate limit. The ceiling and latency-throughput tradeoff is the capacity you provisioned: the instance type and instance count behind the endpoint. For an RTX Pro 6000, we find that 8 concurrent requests is a good starting point.

Content