SID-1-Tabular

Research preview

SID-1-Tabular is in research preview. The integration on this page can change between releases.

SID-1-Tabular is a variant of SID-1 trained to retrieve over tabular data and to combine search with complex filters over the columns.

Unlike SID-1, which ends a run by reporting a ranked list of document IDs, SID-1-Tabular ends a run by reporting a search: a query plus the filters it settled on. The results of that search can then be displayed to a user or passed to an LLM as a database view. This is the right model for reasoning over large tables, such as company or people data.

The client, the loop, and the result-handling code are the same as in the SID-1 quickstart. This page covers only the differences. Read the SID-1 docs first for everything the two share.

What's different

	SID-1	SID-1-Tabular
Terminal tool	`report_helpful_ids`	`report_search` / `report_text_search`
What the model submits as the answer	a model-picked, ranked list of IDs	a final search query, with filters
Search results rendered as	XML `<doc>` blocks	a markdown table

Everything else (semantic search, lexical text_search, read, the loop, the turn budget, capping turns) works as it does for SID-1.

Point the client at SID-1-tabular

Create the client the same way as the quickstart, then pass sid-1-tabular as the model.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.sid-1.com/v1",
    api_key=os.environ["SID_API_KEY"],
)

response = client.chat.completions.create(model="sid-1-tabular", messages=messages, tools=tools)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.sid-1.com/v1",
  apiKey: process.env.SID_API_KEY,
});

const response = await client.chat.completions.create({ model: "sid-1-tabular", messages, tools });

System prompt

The structure is the same as the SID-1 recommended prompt. The report tools change, and the tool list documents your filters. The prompt below is the adapted version, using the example companies filters.

SYSTEM_PROMPT = """
You are an expert research assistant that is given a question and must use the provided search tools to find all companies needed to answer the question.

Steps:
1. Reflect on what information is needed to answer the question and use the search tools to find companies from the company database. Each company has an id.
2. Repeat step 1 until all companies necessary and sufficient to answer the question have been found. Take as many turns and searches as needed – you can make multiple searches per turn! Most questions will require multiple turns. Most questions require at least 5-8 search requests. Many will need more.
3. Use the report_search tool to report a search that gives relevant companies.

The interaction ends once report_search is called. You will be scored based on whether you have found all the companies and whether you reported them in the correct order (NDCG).

Use medium reasoning effort.

Tools:
- search: performs a semantic search with the query. Arguments and filters:
  - query (string, optional): the query to search for
  - limit (int, optional, default 5): the number of results to return
  - primary_industry (list of strings, optional)
  - primary_industry_exclude (list of strings, optional)
  - size (list of strings, optional)
  - size_exclude (list of strings, optional)
  - type (list of strings, optional)
  - type_exclude (list of strings, optional)
  - location (list of strings, optional)
  - location_exclude (list of strings, optional)
  - country (list of strings, optional)
  - country_exclude (list of strings, optional)
  - annual_revenue_min (int, optional)
  - annual_revenue_max (int, optional)
  - funding_raised_min (int, optional)
  - funding_raised_max (int, optional)
  - associated_members (list of strings, optional)
  - associated_members_exclude (list of strings, optional)
  - followers_min (int, optional)
  - followers_max (int, optional)
  - products_and_services (list of strings, optional)
  - products_and_services_exclude (list of strings, optional)
  - extra_cols (list of strings, optional): extra columns to display in the output possible values: ["description", "primary_industry", "size", "type", "location", "country", "url", "annual_revenue", "funding_raised", "associated_members", "followers", "products_and_services"]
- text_search: performs a lexical (full-text) search with the query. Has the same arguments and filters as search.
- read: display the full details of a company. Arguments:
  - id (required): the id of the company as a string
- report_search: report a search. Has the *exact* same arguments as the search tool.
- report_text_search: report a text search. Has the *exact* same arguments as the text_search tool.

To use a tool, enclose it within <tool_call> tags with a Python dictionary containing "name" and "arguments". For example:

<tool_call>
{"name": "search", "arguments": {"query": "machine learning", "limit": 3, "followers_min": 1000, "size": ["101-200", "201-5000"], "extra_cols": ["description"]}}
</tool_call>
"""

const SYSTEM_PROMPT = `
You are an expert research assistant that is given a question and must use the provided search tools to find all companies needed to answer the question.

Steps:
1. Reflect on what information is needed to answer the question and use the search tools to find companies from the company database. Each company has an id.
2. Repeat step 1 until all companies necessary and sufficient to answer the question have been found. Take as many turns and searches as needed – you can make multiple searches per turn! Most questions will require multiple turns. Most questions require at least 5-8 search requests. Many will need more.
3. Use the report_search tool to report a search that gives relevant companies.

The interaction ends once report_search is called. You will be scored based on whether you have found all the companies and whether you reported them in the correct order (NDCG).

Use medium reasoning effort.

Tools:
- search: performs a semantic search with the query. Arguments and filters:
  - query (string, optional): the query to search for
  - limit (int, optional, default 5): the number of results to return
  - primary_industry (list of strings, optional)
  - primary_industry_exclude (list of strings, optional)
  - size (list of strings, optional)
  - size_exclude (list of strings, optional)
  - type (list of strings, optional)
  - type_exclude (list of strings, optional)
  - location (list of strings, optional)
  - location_exclude (list of strings, optional)
  - country (list of strings, optional)
  - country_exclude (list of strings, optional)
  - annual_revenue_min (int, optional)
  - annual_revenue_max (int, optional)
  - funding_raised_min (int, optional)
  - funding_raised_max (int, optional)
  - associated_members (list of strings, optional)
  - associated_members_exclude (list of strings, optional)
  - followers_min (int, optional)
  - followers_max (int, optional)
  - products_and_services (list of strings, optional)
  - products_and_services_exclude (list of strings, optional)
  - extra_cols (list of strings, optional): extra columns to display in the output possible values: ["description", "primary_industry", "size", "type", "location", "country", "url", "annual_revenue", "funding_raised", "associated_members", "followers", "products_and_services"]
- text_search: performs a lexical (full-text) search with the query. Has the same arguments and filters as search.
- read: display the full details of a company. Arguments:
  - id (required): the id of the company as a string
- report_search: report a search. Has the *exact* same arguments as the search tool.
- report_text_search: report a text search. Has the *exact* same arguments as the text_search tool.

To use a tool, enclose it within <tool_call> tags with a Python dictionary containing "name" and "arguments". For example:

<tool_call>
{"name": "search", "arguments": {"query": "machine learning", "limit": 3, "followers_min": 1000, "size": ["101-200", "201-5000"], "extra_cols": ["description"]}}
</tool_call>
`;

Adapt the opening corpus description and the filter list to your columns, the same way as the SID-1 system prompt.

Reasoning effort

SID-1-Tabular supports three reasoning-effort levels: low, medium, and high. Higher effort spends more tokens reasoning before each turn, which helps on more complex queries and filters, at the cost of more latency and tokens per turn. Lower effort is faster and cheaper on simpler lookups.

Set the level with a line in the system prompt, as the recommended prompt above does:

Use medium reasoning effort.

The recommended prompt uses medium. Swap in low or high to trade speed against thoroughness for your workload.

Tell the model what you care about

The last line of the recommended prompt tells the model what matters in the reported search. The recommended prompt asks for NDCG, which rewards putting the most relevant companies first:

You will be scored based on whether you have found all the companies and whether you reported them in the correct order (NDCG).

Swap that line to match what you care about:

NDCG (ranking order, the default): whether you have found all the companies and whether you reported them in the correct order (NDCG)
F1 (coverage and precision): whether you have found all the companies and how precise the search results are (F1 score)
Both: both finding all the relevant companies (F1 score) and ranking them correctly (NDCG)

Define the tools

Pass a single placeholder stub. As with SID-1, the API tools field only switches on tool-call parsing, and its contents are ignored; SID-1-Tabular reads every tool, filter, and example from the system prompt, not from this field.

tools = [{"type": "function", "function": {"name": "dummy"}}]

const tools: OpenAI.Chat.Completions.ChatCompletionTool[] = [
  { type: "function", function: { name: "dummy" } },
];

The tools field is just a switch

Adding description or parameters to the stub does nothing; the model does not read them. Put the tool names, descriptions, and filters in the system prompt. This is the same rule as the quickstart.

Filters

Filters are optional arguments on search and text_search, alongside query and limit. SID-1-Tabular combines them with the query to narrow results, and the search it reports at the end carries the filters it chose. Declare each filter as a tool argument in the system prompt and apply it in your backend.

The filters below cover the shapes SID-1-Tabular is trained to use. The names are examples for a companies corpus; use your own columns.

Family	Shape	Example
Exact match	one value the column must equal	`status="active"`, `country="United States"`
Any-of match	an array; a row matches if it has any value	`tags=["fintech", "payments"]`
Exclude	an array of values to remove from results	`tags_exclude=["crypto"]`
Numeric range	inclusive `_min` / `_max` pair	`year_founded_min=2020`, `team_size_max=50`
Column selection	`extra_cols`, extra columns to return	`extra_cols=["website", "description"]`

A reported search then combines a query with filters:

{
  "name": "report_search",
  "arguments": {
    "query": "fintech companies for small-business lending",
    "tags": ["fintech"],
    "year_founded_min": 2020,
    "limit": 10
  }
}

Extra columns

extra_cols keeps the results table compact by default while letting SID-1-Tabular pull more detail only when it needs it. Return a small default set of columns on every search (a name, a status, a short description) and offer the rest through extra_cols. When the model passes extra_cols, add those columns to that search's table; leave them off otherwise. This holds down context on routine searches without putting every field into every row.

Compact does not mean inaccessible. The model can always reach any field it needs through one of two paths:

extra_cols on a search or text_search adds the requested columns to that result's table.
read returns the full record for a single id, with every field, regardless of extra_cols.

List the allowed values in the extra_cols line of the system prompt so the model knows which columns it can request, as the recommended prompt does.

Format results as a table

SID-1-Tabular reads search results as a markdown table, one row per record, unlike SID-1, which uses XML <doc> blocks. Use your record's columns as the table header. id is required and must be stable for the whole run (see ID rules), because read and the report tools resolve records by it; the other columns are yours.

NO_RESULTS = "No records matched this query. Try different keywords, a broader query, or looser filters."


def format_docs_table(docs: list[dict]) -> str:
    if not docs:
        return NO_RESULTS
    columns = list(docs[0].keys())  # e.g. ["id", "name", "batch", "status"]
    header = "| " + " | ".join(columns) + " |"
    divider = "| " + " | ".join("---" for _ in columns) + " |"
    rows = ["| " + " | ".join(str(doc.get(col, "")) for col in columns) + " |" for doc in docs]
    return "\n".join([header, divider, *rows])

const NO_RESULTS = "No records matched this query. Try different keywords, a broader query, or looser filters.";

function formatDocsTable(docs: Record<string, unknown>[]): string {
  if (docs.length === 0) return NO_RESULTS;
  const columns = Object.keys(docs[0]); // e.g. ["id", "name", "batch", "status"]
  const header = `| ${columns.join(" | ")} |`;
  const divider = `| ${columns.map(() => "---").join(" | ")} |`;
  const rows = docs.map((doc) => `| ${columns.map((col) => String(doc[col] ?? "")).join(" | ")} |`);
  return [header, divider, ...rows].join("\n");
}

A tool message then carries a table like this:

| id | name | batch | status |
| --- | --- | --- | --- |
| co_18f2 | Acme Pay | W21 | active |
| co_9d3a | Globex Capital | S19 | acquired |

Keep columns focused: every column is read on every turn, so include only fields that help the model judge relevance. Append tool results verbatim, the same as SID-1; do not summarize, rewrite, or re-rank rows.

Implement the report tools

report_search and report_text_search take the same arguments as search and text_search, filters included. Implement each by running the corresponding search with those arguments and returning the result IDs in backend order. Those IDs are the answer, so the loop stops once either is called.

This is the only change to the executor from the loop page. Where SID-1 reads the ids argument off report_helpful_ids, SID-1-Tabular runs the reported search and takes the IDs the backend returns.

import json


def execute_tool_call(tc) -> tuple[str, list[str] | None]:
    name = tc.function.name
    args = json.loads(tc.function.arguments)

    if name == "read":
        return format_docs_table([read(args["id"])]), None

    # search, text_search, report_search, report_text_search share the same arguments:
    # a query, an optional limit, and any number of column filters.
    query = args.pop("query", "")
    limit = args.pop("limit", 5)
    filters = args  # e.g. {"status": "active", "year_founded_min": 2020, "tags": ["fintech"]}

    backend = text_search if name in ("text_search", "report_text_search") else search
    docs = backend(query, limit, **filters)

    if name in ("report_search", "report_text_search"):
        ids = [doc["id"] for doc in docs]  # backend order is the answer order
        return json.dumps(ids), ids

    return format_docs_table(docs), None

type ToolResult = { text: string; ids: string[] | null };

async function executeToolCall(
  tc: OpenAI.Chat.Completions.ChatCompletionMessageToolCall,
): Promise<ToolResult> {
  const name = tc.function.name;
  const args = JSON.parse(tc.function.arguments);

  if (name === "read") {
    return { text: formatDocsTable([await read(args.id)]), ids: null };
  }

  // search, text_search, report_search, report_text_search share the same arguments:
  // a query, an optional limit, and any number of column filters.
  const { query = "", limit = 5, ...filters } = args;
  const backend =
    name === "text_search" || name === "report_text_search" ? textSearch : search;
  const docs = await backend(query, limit, filters);

  if (name === "report_search" || name === "report_text_search") {
    const ids = docs.map((doc) => doc.id as string); // backend order is the answer order
    return { text: JSON.stringify(ids), ids };
  }

  return { text: formatDocsTable(docs), ids: null };
}

Your search and text_search accept the filters as extra arguments and apply them in your backend. See running the tools on your backend for the search functions and customize the tools for declaring filters. The loop is unchanged: when ids comes back non-null, that turn called a report tool, so you record the IDs and stop.

Content