gradient-inference

# 🦞 Gradient AI — Serverless Inference

> ⚠️ **This is an unofficial community skill**, not maintained by DigitalOcean. Use at your own risk.

> *"Why manage GPUs when the ocean provides?" — ancient lobster proverb*

Use DigitalOcean's [Gradient Serverless Inference](https://docs.digitalocean.com/products/gradient-ai-platform/how-to/use-serverless-inference/) to call large language models without managing infrastructure. The API is **OpenAI-compatible**, so standard SDKs and patterns work — just point at `https://inference.do-ai.run/v1` and swim.

## Authentication

All requests need a **Model Access Key** in the `Authorization: Bearer` header.

```bash
export GRADIENT_API_KEY="your-model-access-key"
```

**Where to get one:** [DigitalOcean Console](https://cloud.digitalocean.com) → Gradient AI → Model Access Keys → Create Key.

📖 *[Full auth docs](https://docs.digitalocean.com/products/gradient-ai-platform/how-to/use-serverless-inference/#create-a-model-access-key)*

---

## Tools

### 🔍 List Available Models

Window-shop for LLMs before you swipe the card.

```bash
python3 gradient_models.py                    # Pretty table
python3 gradient_models.py --json             # Machine-readable
python3 gradient_models.py --filter "llama"   # Search by name
```

Use this before hardcoding model IDs — models are added and deprecated over time.

**Direct API call:**
```bash
curl -s https://inference.do-ai.run/v1/models \
  -H "Authorization: Bearer $GRADIENT_API_KEY" | python3 -m json.tool
```

📖 *[Models reference](https://docs.digitalocean.com/products/gradient-ai-platform/details/models/)*

---

### 💬 Chat Completions

The classic. Send structured messages (system/user/assistant roles), get a response. OpenAI-compatible, so you probably already know how this works.

```bash
python3 gradient_chat.py \
  --model "openai-gpt-oss-120b" \
  --system "You are a helpful assistant." \
  --prompt "Explain serverless inference in one paragraph."

# Different model
python3 gradient_chat.py \
  --model "llama3.3-70b-instruct" \
  --prompt "Write a haiku about cloud computing."
```

**Direct API call:**
```bash
curl -s https://inference.do-ai.run/v1/chat/completions \
  -H "Authorization: Bearer $GRADIENT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai-gpt-oss-120b",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello!"}
    ],
    "temperature": 0.7,
    "max_tokens": 1000
  }'
```

📖 *[Chat Completions docs](https://docs.digitalocean.com/products/gradient-ai-platform/how-to/use-serverless-inference/#chat-completions)*

---

### ⚡ Responses API (Recommended)

DigitalOcean's [recommended endpoint](https://docs.digitalocean.com/products/gradient-ai-platform/how-to/use-serverless-inference/#responses-api) for new integrations. Simpler request format and supports **prompt caching** — a.k.a. "stop paying twice for the same context."

```bash
# Basic usage
python3 gradient_chat.py \
  --model "openai-gpt-oss-120b" \
  --prompt "Summarize this earnings report." \
  --responses-api

# With prompt caching (saves cost on follow-up queries)
python3 gradient_chat.py \
  --model "openai-gpt-oss-120b" \
  --prompt "Now compare it to last quarter." \
  --responses-api --cache
```

**Direct API call:**
```bash
curl -s https://inference.do-ai.run/v1/responses \
  -H "Authorization: Bearer $GRADIENT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai-gpt-oss-120b",
    "input": "Explain prompt caching.",
    "store": true
  }'
```

**When to use which:**
| | Chat Completions | Responses API |
|---|---|---|
| **Request format** | Array of messages with roles | Single `input` string |
| **Prompt caching** | ❌ | ✅ via `store: true` |
| **Multi-step tool use** | Manual | Built-in |
| **Best for** | Structured conversations | Simple queries, cost savings |

📖 *[Responses API docs](https://docs.digitalocean.com/products/gradient-ai-platform/how-to/use-serverless-inference/#responses-api)*

---

### 🖼️ Generate Images

Turn text prompts into images. Because sometimes a chart isn't enough.

```bash
python3 gradient_image.py --prompt "A lobster trading stocks on Wall Street"
python3 gradient_image.py --prompt "Sunset over the NYSE" --output sunset.png
python3 gradient_image.py --prompt "Fintech logo" --json
```

**Direct API call:**
```bash
curl -s https://inference.do-ai.run/v1/images/generations \
  -H "Authorization: Bearer $GRADIENT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "dall-e-3",
    "prompt": "A lobster analyzing candlestick charts",
    "n": 1
  }'
```

📖 *[Image generation docs](https://docs.digitalocean.com/products/gradient-ai-platform/how-to/use-serverless-inference/#image-generation)*

---

## 🧠 Model Selection Guide

Not all models are created equal. Choose wisely, young crustacean:

| Model | Best For | Speed | Quality | Context |
|-------|----------|-------|---------|---------|
| `openai-gpt-oss-120b` | Complex reasoning, analysis, writing | Medium | ★★★★★ | 128K |
| `llama3.3-70b-instruct` | General tasks, instruction following | Fast | ★★★★ | 128K |
| `deepseek-r1-distill-llama-70b` | Math, code, step-by-step reasoning | Slow | ★★★★★ | 128K |
| `qwen3-32b` | Quick triage, short tasks | Fastest | ★★★ | 32K |

> **🦞 Pro tip: Cost-aware routing.** Use a fast model (e.g., `qwen3-32b`) to score or triage, then only escalate to a strong model (e.g., `openai-gpt-oss-120b`) when depth is needed. Enable prompt caching for repeated context.

Always run `python3 gradient_models.py` to check what's currently available — the menu changes.

📖 *[Available models](https://docs.digitalocean.com/products/gradient-ai-platform/details/models/)*

---

### 💰 Model Pricing Lookup

Check what models cost *before* you rack up a bill. Scrapes the official [DigitalOcean pricing page](https://docs.digitalocean.com/products/gradient-ai-platform/details/pricing/) — no API key needed.

```bash
python3 gradient_pricing.py                    # Pretty table
python3 gradient_pricing.py --json             # Machine-readable
python3 gradient_pricing.py --model "llama"    # Filter by model name
python3 gradient_pricing.py --no-cache         # Skip cache, fetch live
```

**How it works:**
- Fetches live pricing from DigitalOcean's docs (public page, no auth)
- Caches results for 24 hours in `/tmp/gradient_pricing_cache.json`
- Falls back to a bundled snapshot if the live fetch fails

> **🦞 Pro tip:** Run `python3 gradient_pricing.py --model "gpt-oss"` before choosing a model to see the cost difference between `gpt-oss-120b` ($0.10/$0.70) and `gpt-oss-20b` ($0.05/$0.45) per 1M tokens.

📖 *[Pricing docs](https://docs.digitalocean.com/products/gradient-ai-platform/details/pricing/)*

---

## CLI Reference

All scripts accept `--json` for machine-readable output.

```
gradient_models.py   [--json] [--filter QUERY]
gradient_chat.py     --prompt TEXT [--model ID] [--system TEXT]
                     [--responses-api] [--cache] [--temperature F]
                     [--max-tokens N] [--json]
gradient_image.py    --prompt TEXT [--model ID] [--output PATH]
                     [--size WxH] [--json]
gradient_pricing.py  [--json] [--model QUERY] [--no-cache]
```

## External Endpoints

| Endpoint | Purpose |
|----------|---------|
| `https://inference.do-ai.run/v1/models` | List available models |
| `https://inference.do-ai.run/v1/chat/completions` | Chat Completions API |
| `https://inference.do-ai.run/v1/responses` | Responses API (recommended) |
| `https://inference.do-ai.run/v1/images/generations` | Image generation |
| `https://docs.digitalocean.com/.../pricing/` | Pricing page (scraped, public) |

## Security & Privacy

- All requests go to `inference.do-ai.run` — DigitalOcean's own endpoint
- Your `GRADIENT_API_KEY` is sent as a Bearer token in the Authorization header
- No other credentials or local data leave the machine
- Model Access Keys are scoped to inference only — they can't manage your DO account
- Prompt caching entries are scoped to your account and automatically expire

## Trust Statement

> By using this skill, prompts and data are sent to DigitalOcean's Gradient Inference API.
> Only install if you trust DigitalOcean with the content you send to their LLMs.

## Important Notes

- Run `python3 gradient_models.py` before assuming a model exists — they rotate
- All scripts exit with code 1 and print errors to stderr on failure