Batch inference

Batch is the big hammer on sference. Reach for it when volume, not latency, is the constraint: send an array of inference rows and let the scheduler place work on spot-friendly European GPUs, completing within the 24h completion window ("24h" is the only supported value). For user-facing inference, use realtime sync endpoints (/v1/chat/completions, /v1/messages, blocking /v1/responses); for discounted sync work that can tolerate queuing, use service_tier: "flex" on /v1/chat/completions or /v1/responses.

Typical control loop

1. POST /v1/batches: send an inline requests[] array (each row has optional custom_id and a body object).
2. GET /v1/batches/{batch_id}: poll status, token totals, and request_count while work drains.
3. GET /v1/batches/{batch_id}/results or GET /v1/batches/{batch_id}/results.jsonl: fetch per-row outcomes once the batch is terminal (completed, failed, or cancelled).

Retries should be idempotent: set a stable custom_id per row when you need correlation across replays.

One model per batch

Every row in requests[] must use the same body.model. Mixing models in one batch returns HTTP 400 before anything is enqueued:

requests[1] (custom_id="row-b"): all batch rows must use the same model; expected "Qwen/Qwen3.6-35B-A3B", got "other-model"

The model must also be available in your platform catalog (platform_status=available). Unknown or unavailable ids fail at create with Model "…" is not available for inference. Split multi-model workloads into separate batch jobs (or use Responses & streams with per-request models).

Request shape

{
  "window": "24h",
  "requests": [
    {
      "custom_id": "row-a",
      "body": {
        "model": "Qwen/Qwen3.6-35B-A3B",
        "messages": [{ "role": "user", "content": "Hello" }]
      }
    },
    {
      "custom_id": "row-b",
      "body": {
        "model": "Qwen/Qwen3.6-35B-A3B",
        "messages": [{ "role": "user", "content": "Summarize this." }]
      }
    }
  ]
}

Base URL: https://api.sference.com/v1 (Bearer API key).

Row body shapes (normalize → validate)

At create time the API normalizes each row body to internal chat-completions format, validates it, then persists. Invalid rows return HTTP 400 with requests[i] and optional custom_id. Nothing is enqueued, so you never get late worker failures like body.messages must be a list.

Chat completions

{
  "model": "Qwen/Qwen3.6-35B-A3B",
  "messages": [{ "role": "user", "content": "Summarize this." }],
  "temperature": 0.2,
  "max_tokens": 512
}

messages must be a non-empty array. Optional fields match POST /v1/chat/completions (tools, tool_choice, …).

Responses API (normalized at create)

Use the same fields as POST /v1/responses. The API converts input → messages, max_output_tokens → max_tokens, and similar mappings before enqueue:

{
  "model": "Qwen/Qwen3.6-35B-A3B",
  "input": [{ "role": "user", "content": "Summarize this." }],
  "instructions": "Reply in one sentence.",
  "max_output_tokens": 512
}

String shorthand for input is supported. After create, stored rows always contain messages; workers never see raw Responses shape.

Rejected at create

JSONL via CLI / SDK

For file-based workflows, use sference batch submit or client.submit_batch(input_file=...). Each JSONL line is one batch row. Use the same model on every line (or pass --model / model= for content-only lines only).

OpenAI-style envelope: the SDK sends only custom_id + inner body; method / url are ignored:

{"custom_id":"a","method":"POST","url":"/v1/chat/completions","body":{"model":"…","messages":[{"role":"user","content":"hi"}]}}
{"custom_id":"b","method":"POST","url":"/v1/responses","body":{"model":"…","input":[{"role":"user","content":"hi"}]}}

Content-only: requires global model= on submit:

{"content":"Classify this log line."}

See CLI and the OSS CLI README for subcommands and sference batch stream.

Not OpenAI’s file-upload Batch API

sference batches differ from OpenAI Batch:

• No POST /v1/files or batch objects referencing uploaded JSONL file IDs.
• Create uses inline requests[], not a separate file upload step.
• Result rows use result_json / error_json, not OpenAI’s batch result envelope.

For OpenAI Responses workloads at scale, use Responses-shaped body rows in a batch (above) or POST /v1/responses with background: true per request.

CLI quick example

export SFERENCE_API_KEY='sk_...'
sference auth login --api-key "$SFERENCE_API_KEY"
sference batch submit --input-file ./workload.jsonl --model Qwen/Qwen3.6-35B-A3B --window 24h
sference batch wait --batch-id <batch_id> --timeout 86400
sference batch download-results --batch-id <batch_id> --out ./results.jsonl

Python SDK quick example

from sference_sdk import SferenceClient

client = SferenceClient(api_key="sk_...")
batch = client.submit_batch(
    input_file="./workload.jsonl",
    model="Qwen/Qwen3.6-35B-A3B",
    window="24h",
)
done = client.wait_for_completion(batch.id, poll_interval=5.0, timeout=86_400.0)
client.download_results_jsonl(done.id, out="./results.jsonl")

Pass an explicit timeout to wait_for_completion; the default is 30 seconds.

When not to use batches

• Interactive assistants with typing indicators → realtime Responses & streams.
• Latency-tolerant sync work (unattended agents, retries OK) → service_tier: "flex" on the sync endpoints is simpler than a batch job; see Processing modes.

Cross-links