Skip to main content

Quickstart

Run a small Q&A evaluation, then inspect its scores and saved files. Complete Installation first. Choose local GPU inference or a model API below; for tool-using tasks, use the agentic smoke test.

1. Activate your environment​

source .venv/bin/activate

2. Run a benchmark​

Choose the backend you installed. Both examples use run-all to query, grade, and summarize 25 questions per benchmark, with one trial. Use a new output directory for this first run.

Requires a supported Linux GPU host and .[vllm,metrics]. The model download may require HF_TOKEN and publisher access approval; see credentials.

faith run-all \
  --model-paths meta-llama/Llama-3.1-8B-Instruct \
  --benchmarks ctibench-mcqa seceval \
  --datastore-location out/run-001 \
  --model-engine vllm \
  --generation-mode chat_comp \
  --prompt-format chat \
  --num-trials 1 \
  --sample-size 25 \
  --n-shot 0 \
  --stats accuracy

3. Inspect the results​

On a complete run, the console table has an entry for each benchmark with accuracy columns. accuracy_mean is the mean trial accuracy; with one trial, it is that trial's score. --stats accuracy selects these columns; without --stats, the table contains only run metadata.

Under out/run-001, each experiment contains:

  • experiment.json: configuration and execution metadata, saved when the query stage completes.
  • metrics.json: computed metrics from eval.
  • trials/…/benchmark-log.json: the saved questions and model outputs.

Verify that each benchmark has 25 evaluated records and inspect error counts before interpreting the score. See the completeness checks and metric definitions. The storage reference explains the full directory hierarchy.

Run an agentic benchmark​

This smoke test runs one vulnerability-localization task. Install .[openai,docker,metrics], start Docker, and complete the Docker preflight, including runner download access.

The agent executes commands in a local container; the model runs remotely. Set OPENAI_API_KEY through your approved secret mechanism. One task can make multiple billed model requests, so keep the sample size small. The benchmark's limits bound its execution.

Agentic smoke query
# OPENAI_API_KEY must already be set in the runner environment.

faith query \
  --model-paths gpt-4.1-2025-04-14 \
  --model-engine openai \
  --benchmarks vulnerability-localization-a \
  --generation-mode chat_comp \
  --prompt-format chat \
  --num-trials 1 \
  --sample-size 1 \
  --datastore-location out/agentic-smoke

Both vulnerability-localization phases require a read-only workspace. The supplied image supports this policy. Custom deployments must meet the environment requirements.

query writes logs but does not create metrics. Grade and summarize the smoke test separately:

Agentic smoke results
faith eval --experiment-path out/agentic-smoke
faith summarize --experiment-path out/agentic-smoke \
  --stats mean_file_f1 error_rate stopped_reason

Expect one evaluated task, no recorded execution error, and a termination breakdown explaining how the task ended. A low F1 score alone is not a setup failure. Inspect benchmark-log.json for tool output and errors; a successful command exit is not sufficient validation. See the completeness checks.

For other deployments, see serving a local model or using Kubernetes. All agentic runs require the openai engine, chat_comp generation mode, and chat prompt format; see the full requirements.

After the first run​

  • Evaluate more samples: Remove --sample-size for a full benchmark, or increase --num-trials to measure variation across trials. Check expected runtime and API cost first.
  • Export results: See CSV output or BigQuery setup.
  • Use cloud storage: Configure Google Cloud tooling before selecting a gs:// datastore. See the CLI overview for local downloads needed by later phases.
  • Run phases separately: Use query, eval, and summarize independently.

Keeping a long run alive​

Keep a remote run active across SSH disconnections

A full run can take several hours. When benchmarking on a remote machine, use a persistent terminal session so an SSH disconnection does not terminate the run. From your local machine, open or reconnect to a screen session named NAME:

ssh -t user@REMOTE "screen -RdS NAME"

This creates the session if it does not exist or reattaches to an existing session, disconnecting any previous attachment. Session controls:

  • Detach: ctrl-a d
  • Scroll within a session: ctrl-a ESC
  • Force-disconnect a frozen SSH session: type ~ .

API configuration options​

Use a local endpoint or a packaged model configuration

With vllm, FAITH loads model weights in-process. With openai, it connects to a running OpenAI-compatible chat API. To use a local server through the same client, add --engine-kwargs base_url='"http://localhost:8100/v1"'. For tool-using tasks, follow Serve a model for agentic runs.

For the hosted API example, replace --model-paths, --model-engine, and --prompt-format with --model-configs instruct/gpt-4.1-2025-04-14 to use a packaged configuration. This also sets context length and generation parameters instead of leaving them at the CLI defaults. See Model config files for settings and override precedence.

Where to go next​