How to Run Programmatic Inference with Hugging Face SDKs

Programmatic inference on Hugging Face usually starts with InferenceClient from the huggingface hub Python package or the Hugging Face inference JavaScript package. For chat, you can also use the OpenAI compatible endpoint at https://router.huggingface.co/v1. Get oriented at /explore/huggingface.

This guide focuses on SDK patterns you can paste into apps and scripts. Chat specific routing detail is expanded in /blog/how-to-use-hugging-face-inference-providers-for-chat-completion. Browser first discovery stays in /blog/how-to-browse-and-try-models-on-the-hugging-face-hub.

SDK work pays off when the same call must run in CI, in a backend service, and in a notebook without surprising differences. Widgets are for discovery. Clients are for contracts. Write the contract once: model id, suffix policy, max tokens, and error handling. Then reuse it everywhere instead of pasting slightly different snippets into five repos.

When to use SDKs instead of widgets

Use SDKs when you need automation, CI checks, streaming UIs, batch jobs, or org billing hooks. Stay in widgets or Spaces while you are still choosing a model. Move to dedicated Inference Endpoints when you need always on capacity separate from serverless Providers.

You need a fine grained token with Inference Providers permission for Providers API calls. Monthly credits then pay as you go apply. Figures can change; read https://huggingface.co/docs/inference-providers/en/pricing. Hub PRO seat prices belong on huggingface.co/pro or huggingface.co/pricing, not mixed into inference invoices casually.

If your app already speaks the OpenAI chat format, pointing the base URL at the Hugging Face router can reduce glue code. Still verify that your features beyond basic chat, such as tools or multimodal parts, are supported for the chosen model and provider before you promise them in a product UI.

How the clients fit together

InferenceClient wraps Providers tasks including chat completions. The JS package mirrors common flows for browser and Node apps. Routing suffixes for fastest, cheapest, and preferred change provider selection. Named providers appear when listed for a model. Bill to options support org billing on Team or Enterprise setups when required.

Keep a thin adapter layer in your codebase that owns token loading, default model config, and logging redaction. Feature code should call your adapter, not scatter InferenceClient construction across handlers. That structure makes it easier to swap fastest routing for cheapest routing in bulk jobs without hunting string literals.

Step by step SDK workflow

1. Install clients and set your HF token

Install the official packages. Load your HF token from the environment. Fail fast if the variable is missing so local mistakes are obvious.

# Python
pip install -U huggingface_hub
# JavaScript
npm install @huggingface/inference
export HF_TOKEN="hf_***"

2. Prove chat with a tiny script

Start with chat completions and a short max tokens limit. Confirm the model id you already validated on the Hub.

from huggingface_hub import InferenceClient
import os

assert os.environ.get("HF_TOKEN"), "HF_TOKEN missing"
client = InferenceClient(api_key=os.environ["HF_TOKEN"])
r = client.chat.completions.create(
    model="org/chat-model:fastest",
    messages=[{"role": "user", "content": "Reply with the word pong only."}],
    max_tokens=8,
)
print(r.choices[0].message.content)

3. Add streaming for interactive apps

Stream when users watch tokens appear. Keep non stream mode for batch scoring where full responses are easier to validate.

import { InferenceClient } from "@huggingface/inference";

const client = new InferenceClient(process.env.HF_TOKEN);
const stream = client.chatCompletionStream({
  model: "org/chat-model",
  messages: [{ role: "user", content: "Write three bullet release risks." }],
  max_tokens: 200,
});
for await (const chunk of stream) {
  const delta = chunk.choices?.[0]?.delta?.content;
  if (delta) process.stdout.write(delta);
}

4. Use task helpers beyond chat when Providers support them

Clients expose task helpers for embeddings, speech, and other modalities when providers host them. Do not force the chat router into non chat jobs. Confirm task support on the model page.

from huggingface_hub import InferenceClient
import os

client = InferenceClient(api_key=os.environ["HF_TOKEN"])
# Example shape: feature extraction / embeddings when supported
vecs = client.feature_extraction(
    "How do I reset my password?",
    model="org/embedding-model",
)
print(type(vecs), getattr(vecs, "shape", len(vecs)))

5. Compare routing suffixes in code

Parameterize the suffix. Log latency and output length. Pick defaults from evidence, not habit.

from huggingface_hub import InferenceClient
import os, time

client = InferenceClient(api_key=os.environ["HF_TOKEN"])
base = "org/chat-model"
prompt = [{"role": "user", "content": "Summarize CI flakiness in 4 bullets."}]
for suffix in [":fastest", ":cheapest", ":preferred"]:
    t0 = time.time()
    r = client.chat.completions.create(
        model=base + suffix,
        messages=prompt,
        max_tokens=200,
    )
    print(suffix, round(time.time() - t0, 2), "s", len(r.choices[0].message.content or ""))

6. Add guardrails for production scripts

Cap tokens, retries, and timeouts. Separate staging keys from production. Document whether calls use personal or org billing.

Production guardrails
[ ] HF_TOKEN from secret store only
[ ] max_tokens capped per route
[ ] timeout and retry policy set
[ ] :cheapest for offline bulk where quality allows
[ ] bill_to / X-HF-Bill-To configured if org requires it
[ ] Alert on elevated 4xx/5xx rates
[ ] Do not log raw prompts that contain PII

Copyable SDK examples for common jobs

Keep these as starting points. Swap model ids after Hub validation.

curl control against the chat router

curl https://router.huggingface.co/v1/chat/completions \
  -H "Authorization: Bearer $HF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "org/chat-model:cheapest",
    "messages": [{"role":"user","content":"One sentence definition of eval harness."}],
    "max_tokens": 64
  }'

OpenAI client compatibility smoke test

from openai import OpenAI
import os

client = OpenAI(base_url="https://router.huggingface.co/v1", api_key=os.environ["HF_TOKEN"])
print(client.chat.completions.create(
    model="org/chat-model",
    messages=[{"role": "user", "content": "Say ready"}],
    max_tokens=8,
).choices[0].message.content)

Batch chat over a CSV of prompts

import csv, os
from huggingface_hub import InferenceClient

client = InferenceClient(api_key=os.environ["HF_TOKEN"])
with open("prompts.csv") as f, open("out.csv", "w", newline="") as g:
    reader = csv.DictReader(f)
    writer = csv.DictWriter(g, fieldnames=["id", "answer"])
    writer.writeheader()
    for row in reader:
        r = client.chat.completions.create(
            model="org/chat-model:cheapest",
            messages=[{"role": "user", "content": row["prompt"]}],
            max_tokens=128,
        )
        writer.writerow({"id": row["id"], "answer": r.choices[0].message.content})

Text to image style call note

# Pseudocode: use the task API when a provider hosts text-to-image
# Do NOT send this to router.huggingface.co/v1 chat endpoint
from huggingface_hub import InferenceClient
import os
client = InferenceClient(api_key=os.environ["HF_TOKEN"])
image = client.text_to_image(
    "watercolor sketch of a mountain hut at dawn",
    model="org/text-to-image-model",
)
image.save("hut.png")

Error handling wrapper

from huggingface_hub import InferenceClient
import os, time

client = InferenceClient(api_key=os.environ["HF_TOKEN"])

def chat_once(messages, model="org/chat-model:fastest", tries=3):
    last = None
    for i in range(tries):
        try:
            return client.chat.completions.create(
                model=model, messages=messages, max_tokens=200
            )
        except Exception as e:
            last = e
            time.sleep(1.5 * (i + 1))
    raise RuntimeError(f"chat failed after retries: {last}")

Org billing header experiment

curl https://router.huggingface.co/v1/chat/completions \
  -H "Authorization: Bearer $HF_TOKEN" \
  -H "X-HF-Bill-To: your-org-name" \
  -H "Content-Type: application/json" \
  -d '{"model":"org/chat-model","messages":[{"role":"user","content":"ping"}],"max_tokens":8}'
# Confirm header requirements in current Team/Enterprise docs

CI golden prompt job

# ci_infer_smoke.sh
set -euo pipefail
test -n "${HF_TOKEN}"
python - <<'PY'
from huggingface_hub import InferenceClient
import os
c = InferenceClient(api_key=os.environ["HF_TOKEN"])
r = c.chat.completions.create(
  model="org/chat-model:fastest",
  messages=[{"role":"user","content":"Reply with OK"}],
  max_tokens=4,
)
assert "OK" in (r.choices[0].message.content or "").upper()
print("smoke passed")
PY

Config file for model defaults

# inference_defaults.toml
model_id = "org/chat-model"
route_suffix = ":fastest"
max_tokens = 256
temperature = 0.2
# bulk_jobs use :cheapest after quality signoff

Weak vs strong integration plan

Weak: paste a snippet from a blog and ship
Strong: validate on Hub widget, pin model id + suffix in config,
add smoke CI, cap tokens, separate Hub seats from inference credits,
document Endpoints migration trigger (SLA, concurrency)

Tips and verification

Keep SDK versions pinned. Re run smoke tests after model or provider changes. When demos live on Spaces, align the same model ids using /blog/how-to-run-and-explore-hugging-face-spaces-demos. When you publish weights, remember /blog/how-to-upload-and-share-a-model-on-hugging-face does not auto enable Providers.

Add a weekly credit check to the same calendar reminder as certificate renewals. Small monthly credits disappear quietly during load tests. Pair the check with a glance at error rates so you notice provider trouble before users file tickets. Keep the pricing docs link in the runbook because credit numbers can change.

• Fail fast when the HF token environment variable is missing
• Keep chat on router.huggingface.co/v1 and other tasks on the right helpers
• Measure fastest versus cheapest routing before freezing defaults
• Cap tokens and redact PII in logs
• Separate serverless Providers from dedicated Inference Endpoints in architecture docs

Common mistakes

• Calling Providers APIs without Inference Providers permission on the token
• Sending text to image or other non chat tasks to the chat only router
• Hard coding suffixes without latency or cost measurements
• Logging secrets or raw customer prompts
• Assuming Hub PRO includes unlimited inference
• Skipping retries and timeouts in batch jobs
• Confusing Spaces demos with production SDK contracts

Related Hugging Face articles: /blog/how-to-use-hugging-face-inference-providers-for-chat-completion, /blog/how-to-browse-and-try-models-on-the-hugging-face-hub, and /blog/how-to-upload-and-share-a-model-on-hugging-face. Return to /explore/huggingface for the Explore overview.