inferencer

inferencer is a lightweight R package for calling hosted and local foundation model inference APIs through a simple and mostly consistent interface. It also ships with a small shell companion for selected hosted providers when you want terminal usage without writing R code.

It currently supports:

The package is intentionally minimal. It focuses on a few common tasks:

  1. listing available models from each provider
  2. sending simple prompt-based inference requests
  3. requesting embeddings
  4. working with image-generation and multimodal model inputs

It also includes Gemini TTS support through query_gemini() plus write_gemini_audio(), OpenRouter and Gemini embedding helpers, Gemini and OpenRouter image-generation helpers, and lower-level multimodal wrappers for non-text inputs. It also includes query_fallback() on the R side for simple ordered fallback across Gemini, OpenRouter, and Groq.

More advanced provider-specific parameters may be added gradually in future versions.

Basic setup

Set your API keys first:

Sys.setenv(GEMINI_API_KEY = "your_key_here")
Sys.setenv(OPENAI_API_KEY = "your_key_here")
Sys.setenv(GROQ_API_KEY = "your_key_here")
Sys.setenv(OPENROUTER_API_KEY = "your_key_here")
Sys.setenv(CEREBRAS_API_KEY = "your_key_here")
Sys.setenv(OLLAMA_API_KEY = "your_key_here")
Sys.setenv(DASHSCOPE_API_KEY = "your_key_here")
Sys.setenv(ZHIPUAI_API_KEY = "your_key_here")
Sys.setenv(DEEPSEEK_API_KEY = "your_key_here")
Sys.setenv(MOONSHOT_API_KEY = "your_key_here")
Sys.setenv(MINIMAX_API_KEY = "your_key_here")

Load the package:

library(inferencer)

Local OpenAI-Compatible Servers

query_openai_compatible() and list_openai_compatible_models() support any local or self-hosted server that implements OpenAI-compatible /v1/models and /v1/chat/completions endpoints, including llama.cpp, vLLM, LM Studio, and LocalAI. Their defaults target an unauthenticated server on http://127.0.0.1:8080; no model runtime, launch command, or model path is bundled with this package.

local_models <- list_openai_compatible_models()

reply <- query_openai_compatible(
  prompt = "Define meticulous.",
  system = "Return JSON only.",
  model = "Qwen3-0.6B",
  response_format = list(
    type = "json_schema",
    json_schema = list(
      name = "vocabulary_entry",
      strict = TRUE,
      schema = list(type = "object")
    )
  ),
  temperature = 0,
  max_tokens = 120
)

Supply api_key for a protected deployment, or override url when the server uses another address or port.

Shell scripts

The package includes optional executable zsh helpers in inst/shell. They are kept inside inferencer because they mirror the R wrappers closely and stay small enough not to justify a separate package.

The shell layer currently includes query helpers, model-listing helpers, and a terminal markdown renderer:

Run scripts from the repo:

inst/shell/query_openrouter "Summarize retrieval-augmented generation."

query_local_llm is a Node.js 18+ script. It targets a local server's OpenAI-compatible chat endpoint at http://127.0.0.1:8080 by default and uses Qwen3-0.6B-Q4_0 unless a model is supplied. It accepts generic request options:

inst/shell/query_local_llm \
  --system "Return a JSON object only." \
  --temperature 0 \
  --max-tokens 120 \
  --timeout 120 \
  --response-format '{"type":"json_object"}' \
  "Define meticulous." \
  "Qwen3-0.6B-Q4_0"

Use LOCAL_LLM_API_URL, LOCAL_LLM_API_KEY, LOCAL_LLM_MODEL, and LOCAL_LLM_TIMEOUT to change its endpoint, optional authentication, model default, or request timeout. Pass --json to print the full server response.

Or add the installed shell directory to PATH:

system.file("shell", package = "inferencer")

export PATH="$(Rscript -e 'cat(system.file(\"shell\", package = \"inferencer\"))'):$PATH"

Shell API keys should live in .zprofile, not .Renviron, but they use the same names as the R wrappers:

export GEMINI_API_KEY="your_key_here"
export GROQ_API_KEY="your_key_here"
export OPENROUTER_API_KEY="your_key_here"
export CEREBRAS_API_KEY="your_key_here"
export OLLAMA_API_KEY="your_key_here"

Example shell usage:

query_openrouter "Summarize the main uses of retrieval-augmented generation."
query_gemini "Write three title ideas for a data engineering memo." "gemini-2.5-flash"
query_ollama "Explain principal component analysis in one paragraph." "gpt-oss:120b"
query_fallback "Draft a concise status update for today's analysis."
query_openrouter --json "Return a short JSON object."
query_openrouter "Write release notes in markdown." | render_markdown_terminal
list_openrouter_free_models
list_openrouter_models --json

Each query_* shell helper takes:

  1. prompt as the first argument
  2. optional model as the second argument

By default, query scripts print response text. With --json, they print the full parsed JSON payload.

query_fallback uses this fixed order with each function's default model:

  1. query_gemini
  2. query_openrouter
  3. query_groq

If all three calls fail, it exits with a non-zero status.

List available models

OpenAI

openai_models <- list_openai_models()
head(openai_models)

Gemini

gemini_models <- list_gemini_models()
head(gemini_models)

Parsed JSON list:

gemini_json <- list_gemini_models(json_list = TRUE)

Groq

groq_models <- list_groq_models()
head(groq_models)

OpenRouter

openrouter_models <- list_openrouter_models()
head(openrouter_models)

Parsed JSON list:

openrouter_json <- list_openrouter_models(json_list = TRUE)

Refresh benchmark results directly from OpenRouter's benchmarks endpoint:

or_benchmarks <- list_openrouter_benchmarks(
  source = "artificial-analysis",
  task_type = "coding"
)
head(or_benchmarks)

extract_openrouter_benchmarks() remains available to inspect benchmark fields already present in a saved models response, but it is not the recommended interface for refreshing benchmark data:

or_benchmarks <- extract_openrouter_benchmarks(openrouter_json)
head(or_benchmarks)

Filter model categories from the general catalog:

openrouter_embedding_models <- list_openrouter_embedding_models()
openrouter_image_models <- list_openrouter_image_models()
openrouter_audio_models <- list_openrouter_audio_models()
openrouter_multimodal_models <- list_openrouter_multimodal_models()

List video generation models and their supported capabilities:

openrouter_video_models <- list_openrouter_video_models()
head(openrouter_video_models)

Ollama Cloud

ollama_models <- list_ollama_models()
head(ollama_models)

Providers Hosted in Mainland China

qwen_models <- list_qwen_models()
zhipu_models <- list_zhipu_models()
deepseek_models <- list_deepseek_models()
moonshot_models <- list_moonshot_models()
minimax_models <- list_minimax_models()

Qwen and MiniMax accounts are region-specific. The defaults use their international endpoints; set QWEN_MODELS_URL, QWEN_API_URL, MINIMAX_MODELS_URL, and MINIMAX_API_URL when using a different account region, including mainland-China endpoints.

Query models

OpenAI

OpenAI requests use the Responses API:

query_openai("Explain retrieval-augmented generation in plain English.")

Use structured Responses API input or inspect the complete response:

response <- query_openai(
  input = list(
    list(
      role = "user",
      content = list(list(type = "input_text", text = "Summarize this topic."))
    )
  ),
  json_list = TRUE
)

Gemini

query_gemini("Explain what a large language model is in simple terms.")

Specify model and generation settings:

query_gemini(
  prompt = "Write three short taglines for an AI consulting firm.",
  model = "gemini-2.5-flash",
  temperature = 0.8,
  top_p = 0.95
)

Gemini TTS:

audio_b64 <- query_gemini(
  prompt = paste(
    "TTS the following conversation between Joe and Jane:",
    "Joe: Hows it going today Jane?",
    "Jane: Not too bad, how about you?"
  ),
  model = "gemini-2.5-flash-preview-tts",
  response_modalities = "AUDIO",
  speech_config = list(
    multiSpeakerVoiceConfig = list(
      speakerVoiceConfigs = list(
        list(
          speaker = "Joe",
          voiceConfig = list(
            prebuiltVoiceConfig = list(voiceName = "Kore")
          )
        ),
        list(
          speaker = "Jane",
          voiceConfig = list(
            prebuiltVoiceConfig = list(voiceName = "Puck")
          )
        )
      )
    )
  )
)

write_gemini_audio(audio_b64, "out.wav", format = "wav")

Gemini embeddings:

embed_gemini(c("machine learning", "data science"))

Gemini text-to-image:

img_b64 <- generate_image_gemini("A watercolor skyline at sunrise")

Gemini multimodal input:

query_gemini_content(
  parts = list(
    list(text = "Describe this audio clip."),
    list(inlineData = list(mimeType = "audio/mp3", data = "BASE64_AUDIO_HERE"))
  )
)

Groq

query_groq("Summarize the difference between R and Python in 5 bullet points.")

Specify model:

query_groq(
  prompt = "Give me a concise explanation of vector databases.",
  model = "llama-3.3-70b-versatile",
  temperature = 0.2,
  max_tokens = 300
)

Request an SSE response and assemble its text:

query_groq("Explain vector databases briefly.", stream = TRUE)

Groq SSE responses are currently buffered and parsed before the function returns. This does not print tokens incrementally to the terminal.

OpenRouter

query_openrouter("What are the main use cases of retrieval-augmented generation?")

Use a free model:

query_openrouter(
  prompt = "Rewrite this in a more professional tone: our app is pretty good at searching files",
  model = "stepfun/step-3.5-flash:free",
  temperature = 0
)

Use OpenRouter's task-aware Auto Router explicitly when paid routing is acceptable. auto_router_cost_tier selects its cost band, while session_id helps related requests retain the selected model:

query_openrouter(
  prompt = "Summarize the first section of this report.",
  model = "openrouter/auto",
  auto_router_cost_tier = "low",
  session_id = "report-summary-2026-09"
)

The default remains openrouter/free; Auto Router bills at the selected model's rate. Use json_list = TRUE to inspect the response model field.

Request token log probabilities and retain the complete response:

response <- query_openrouter(
  "Classify this review as positive or negative.",
  logprobs = TRUE,
  top_logprobs = 5,
  json_list = TRUE
)

Fallback

query_fallback("Explain retrieval-augmented generation in plain English.")

Providers Hosted in Mainland China

query_qwen("Explain retrieval-augmented generation in plain English.")
query_zhipu("Explain retrieval-augmented generation in plain English.")
query_deepseek("Explain retrieval-augmented generation in plain English.")
query_moonshot("Explain retrieval-augmented generation in plain English.")
query_minimax("Explain retrieval-augmented generation in plain English.")

Use domestic-provider fallback when global provider APIs are unavailable from a given network. Its fixed order is Qwen, Zhipu, DeepSeek, Moonshot, then MiniMax. Each provider attempt has an independent timeout.

query_china_fallback(
  "Explain retrieval-augmented generation in plain English.",
  timeout = 60
)

query_china_free_fallback() never intentionally calls a paid model. At present, it uses only Zhipu's documented permanently free glm-z1-flash; list_china_free_models() exposes this curated policy. New-user promotional quotas from other providers are not included because they expire or are account-specific.

query_china_free_fallback("Explain retrieval-augmented generation in plain English.")

OpenRouter embeddings:

embed_openrouter(c("alpha", "beta"))

OpenRouter text-to-image:

generate_image_openrouter(
  "A minimalist product photo of a mechanical keyboard on oak"
)

OpenRouter multimodal input:

query_openrouter_content(
  content = list(
    list(type = "text", text = "What is in this image?"),
    list(type = "image_url", image_url = list(url = "https://example.com/cat.png"))
  ),
  model = "meta-llama/llama-3.3-70b-instruct:free"
)

Cerebras

query_cerebras("Explain inflation targeting in one paragraph.")

Current public model catalog:

cerebras_models <- list_cerebras_models()

Use list_cerebras_models() to inspect the current public model catalog before pinning a model ID, because Cerebras public models may refresh frequently.

Specify model:

query_cerebras(
  prompt = "Write a short introduction to algorithmic trading.",
  model = "gemma-4-31b"
)

Ollama Cloud

query_ollama("Explain why the sky is blue.")

Specify model:

query_ollama(
  prompt = "Give me a concise explanation of principal component analysis.",
  model = "gpt-oss:120b"
)

Example workflow: compare outputs across providers

prompt <- "Explain retrieval-augmented generation in plain English."

list(
  gemini = query_gemini(prompt),
  groq = query_groq(prompt),
  openrouter = query_openrouter(prompt),
  cerebras = query_cerebras(prompt),
  ollama = query_ollama(prompt)
)

Live provider smoke tests

The normal test suite mocks HTTP requests and never consumes API quota. Release validation can opt into four live behavior checks for OpenAI Responses, OpenRouter log probabilities, buffered Groq SSE assembly, and Cerebras' shared OpenAI-compatible transport:

Rscript tools/smoke-test-live-providers.R

The script requires the corresponding provider keys. Override the default live test models with INFERENCER_OPENROUTER_LOGPROBS_MODEL and INFERENCER_GROQ_STREAM_MODEL when provider availability changes.

Example workflow: inspect free or low-cost model candidates

Cerebras public models as parsed JSON

cb_json <- list_cerebras_models(json_list = TRUE)
names(cb_json)

Gemini models as parsed JSON

gm_json <- list_gemini_models(json_list = TRUE)
names(gm_json)

Gemini model families currently visible in the API

Note: provider support differs by modality and model family. Model IDs and capabilities should still be checked against the live provider model catalogs.

OpenRouter free models

or_models <- list_openrouter_models()
or_models[, pricing.prompt := sapply(pricing, `[[`, "prompt")]
or_models[pricing.prompt == 0]

OpenRouter free model notes