How to Use Hugging Face Inference Providers for Chat Completion

Inference Providers give serverless multi provider inference on Hugging Face. For chat, you can use InferenceClient in Python or JavaScript, or the OpenAI compatible chat endpoint at https://router.huggingface.co/v1. Start with the product map at /explore/huggingface.

This guide focuses on chat completion only. You will create a fine grained token, pick a model id, choose routing suffixes such as fastest or cheapest, stream answers, and verify credits. Browser first tries belong in /blog/how-to-browse-and-try-models-on-the-hugging-face-hub. Broader SDK patterns sit in /blog/how-to-run-programmatic-inference-with-hugging-face-sdks.

When to use Inference Providers for chat

Use Providers when you want managed chat without standing up GPUs yourself. It fits prototypes, internal tools, and apps that can tolerate pay as you go usage after monthly credits. Choose dedicated Inference Endpoints instead when you need always on capacity you control. Do not expect the OpenAI compatible router to cover non chat tasks such as text to image.

Hub seats and inference credits are separate. Free, PRO, Team, and Enterprise Hub plans are not the same bill as Providers usage. Official docs describe monthly credits that can change: Free about $0.10, PRO $2.00, and Team or Enterprise $2.00 per seat, then pay as you go. Confirm live numbers at https://huggingface.co/docs/inference-providers/en/pricing.

How chat routing works

You pass a model id and messages. Default routing behaves like the fastest suffix. You can append cheapest or preferred, or a named provider when available. Widgets on model pages appear when a provider hosts the model. Uploading a model does not auto enable Providers widgets.

Step by step chat workflow

1. Create a fine grained token with Inference Providers permission

In Hugging Face settings, create a fine grained token and enable Inference Providers permission. Store it as an HF token environment variable in a secret manager. Never commit tokens to git.

# Shell: export for local experiments only
export HF_TOKEN="hf_***"
# Confirm the token is present without printing it
test -n "$HF_TOKEN" && echo "HF_TOKEN is set"

2. Pick a chat model id you already tried

Prefer a model you validated in a widget or the Inference Playground. Copy the exact org/name string. Add a routing suffix only when you have a reason.

Model id choices
Base: org/chat-model
Fastest (default policy): org/chat-model:fastest
Cheapest: org/chat-model:cheapest
Preferred: org/chat-model:preferred
Named provider (when listed): org/chat-model:providerName
Rule: start with base or :fastest, then A/B suffixes with the same messages

3. Call chat with InferenceClient (Python)

Install the huggingface hub package and use InferenceClient. Keep max tokens and temperature explicit so runs are reproducible across teammates.

from huggingface_hub import InferenceClient
import os

client = InferenceClient(api_key=os.environ["HF_TOKEN"])
resp = client.chat.completions.create(
    model="org/chat-model",
    messages=[
        {"role": "system", "content": "You are a concise product analyst."},
        {"role": "user", "content": "List 5 risks of shipping without evals."},
    ],
    max_tokens=300,
    temperature=0.2,
)
print(resp.choices[0].message.content)

4. Call the OpenAI compatible router with curl

The chat only endpoint is https://router.huggingface.co/v1. Use Bearer auth with your token. This path is helpful when an existing OpenAI style client already exists.

curl https://router.huggingface.co/v1/chat/completions \
  -H "Authorization: Bearer $HF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "org/chat-model",
    "messages": [
      {"role": "user", "content": "Explain LoRA in five short sentences."}
    ],
    "max_tokens": 250
  }'

5. Stream tokens for UI friendly responses

Streaming reduces perceived wait time. Enable stream in the client and append chunks to your UI. Keep the same model id and messages when you compare stream versus non stream quality.

from huggingface_hub import InferenceClient
import os

client = InferenceClient(api_key=os.environ["HF_TOKEN"])
stream = client.chat.completions.create(
    model="org/chat-model:fastest",
    messages=[{"role": "user", "content": "Write a 6 line release note."}],
    max_tokens=200,
    stream=True,
)
for event in stream:
    delta = event.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

6. Bill to an organization when Team or Enterprise requires it

Team and Enterprise setups may need org billing. Prefer the documented bill to option or Bill To header when your org requires it. Confirm the exact field names in current Hugging Face docs before you ship.

# curl shape for org billing header (confirm docs for your plan)
curl https://router.huggingface.co/v1/chat/completions \
  -H "Authorization: Bearer $HF_TOKEN" \
  -H "Content-Type: application/json" \
  -H "X-HF-Bill-To: your-org-name" \
  -d '{"model":"org/chat-model","messages":[{"role":"user","content":"ping"}],"max_tokens":16}'

Copyable chat prompts and client templates

These examples mix prompt design with client settings so chat calls stay testable.

JavaScript InferenceClient chat

import { InferenceClient } from "@huggingface/inference";

const client = new InferenceClient(process.env.HF_TOKEN);
const out = await client.chatCompletion({
  model: "org/chat-model",
  messages: [
    { role: "system", content: "Answer in Markdown bullets only." },
    { role: "user", content: "Pros and cons of RAG vs fine tuning for FAQs." },
  ],
  max_tokens: 350,
});
console.log(out.choices[0].message.content);

Routing suffix A/B script notes

A/B plan
Prompt fixed: "Summarize this policy in 5 bullets: ..."
Run A: model:fastest
Run B: model:cheapest
Run C: model:preferred
Log: latency, cost signal if shown, refusal rate, factual misses
Keep winner as default in config, not hard coded in ten files

Tool calling style message scaffold

System: You may propose a tool call as JSON only when needed.
User: What is the weather in Paris tomorrow?
If tools unsupported by provider: answer with a clear limitation line instead of inventing weather.
Assistant format when tools allowed:
{"tool":"get_weather","args":{"city":"Paris","when":"tomorrow"}}

Structured JSON output request

Return ONLY valid JSON with keys:
risks (array of strings),
severity (low|medium|high),
next_test (string)
User content: Review this release plan: ...
If the provider cannot enforce schema, still ask for JSON and validate in code.

Vision language message shape

# Pseudocode message when provider supports VLMs
messages = [{
  "role": "user",
  "content": [
    {"type": "text", "text": "List defects visible on this PCB photo."},
    {"type": "image_url", "image_url": {"url": "https://example.com/pcb.jpg"}}
  ]
}]
# Confirm provider VLM support before relying on this path

OpenAI SDK pointed at the HF router

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://router.huggingface.co/v1",
    api_key=os.environ["HF_TOKEN"],
)
r = client.chat.completions.create(
    model="org/chat-model:cheapest",
    messages=[{"role": "user", "content": "Give three eval ideas for a chatbot."}],
)
print(r.choices[0].message.content)

Credit guard checklist

Before load tests
[ ] Read https://huggingface.co/docs/inference-providers/en/pricing
[ ] Confirm monthly credit remaining in account billing UI
[ ] Cap max_tokens in staging
[ ] Prefer :cheapest for bulk offline jobs
[ ] Separate Hub PRO seat spend from inference spend in finance notes

Failure triage template

If chat fails
401/403: token missing Inference Providers permission or gated model not accepted
404/model error: wrong model id or provider no longer hosts it
Empty content: check max_tokens and provider refusals
Unexpected task error: you may have hit a non chat route by mistake
Next: retry :fastest, then open model page widget as a control

System prompt for support drafts

System: You draft customer emails. Be polite, short, and specific.
Never invent refund policies. If policy is unknown, ask a clarifying question.
User: Customer says package missing for order {{ORDER_ID}}.

Tips and verification

Keep a golden chat suite in CI with tiny max tokens. Alert when providers start failing a model id. Document whether production uses fastest, cheapest, or a named provider. For uploading models that you later hope to serve, see /blog/how-to-upload-and-share-a-model-on-hugging-face and remember upload does not auto enable Providers widgets.

Write a short runbook for on call teammates: which model id is live, which suffix is default, where credits are checked, and what to do when a provider returns empty content. Include a link to the Hub model page and to the pricing docs so people do not guess. Refresh that runbook after any model swap even if the product UI looks unchanged.

• Use fine grained tokens with Inference Providers permission
• Treat router.huggingface.co/v1 as chat only
• A/B routing suffixes with identical messages
• Validate JSON in code even when you ask for JSON in the prompt
• Re read pricing docs when credits or pay as you go matter

When quality slips, bisect changes in this order: prompt, model id, routing suffix, then client library version. Most regressions come from an unnoticed model revision or a suffix change that looked harmless in staging. Keep one pinned control prompt that must still pass after every deploy.

Common mistakes

• Calling chat with a token that lacks Inference Providers permission
• Expecting the OpenAI compatible router to run text to image or other non chat tasks
• Hard coding a provider suffix everywhere before measuring fastest versus cheapest
• Mixing Hub seat pricing with inference credit balances
• Assuming upload to the Hub enables Providers widgets automatically
• Skipping gated model acceptance until production breaks
• Ignoring bill to or Bill To header needs on Team or Enterprise org billing

Related Hugging Face articles: /blog/how-to-browse-and-try-models-on-the-hugging-face-hub, /blog/how-to-run-programmatic-inference-with-hugging-face-sdks, and /blog/how-to-upload-and-share-a-model-on-hugging-face. Return to /explore/huggingface for the Explore overview.