The Batch API is designed for workloads where you have a large number of requests to process but do not need the results immediately. Common use cases include bulk document classification, dataset annotation, offline evaluation runs, nightly summarisation jobs, and any pipeline where throughput matters more than latency. Because batch requests are processed asynchronously and can be scheduled during off-peak periods, they are significantly cheaper than synchronous calls — especially when combined with the :batch routing suffix.
Overview
Batch processing follows an upload-submit-poll pattern:
- Prepare a JSONL file where each line is a self-contained request object.
- Upload the file to get a
file_id.
- Submit a batch job referencing the
file_id.
- Poll the batch status until it reaches a terminal state.
- Download the output file using the
output_file_id from the completed batch.
All requests in a batch run against the same endpoint. Results are written to an output JSONL file with each line containing the original custom_id alongside the response, making it straightforward to correlate outputs with inputs.
Step-by-Step Walkthrough
Batch Status Values
After creation, a batch moves through the following states:
A batch in in_progress state can be cancelled at any time. Results for requests completed before cancellation are still available in the output file.
Failed individual requests within an otherwise successful batch do not change the batch status to "failed". Check each line of the output JSONL for per-request error fields alongside the overall batch status.
Include :batch in your model ID inside the request bodies — for example "model": "mistral-small:batch" — to route requests through providers that offer the deepest discounts for asynchronous workloads and achieve maximum cost savings.
