Skip to main content
FAITH · Cisco Foundation AI

The open harness for benchmarking cybersecurity models.

FAITH — Foundation-AI's Test Hub/Harness — provides a common interface to benchmarks that evaluate the cybersecurity knowledge and capabilities of language models.

QuickstartDocumentationBenchmark catalog
bash — faith
$ faith run-all \
  --model-paths meta-llama/Llama-3.1-8B-Instruct \
  --benchmarks ctibench-mcqa seceval \
  --model-engine vllm \
  --prompt-format chat \
  --sample-size 25 \
  --num-trials 1 \
  --stats accuracy \
  --datastore-location out/

[query]  querying 2 benchmarks · 1 model · logs → out/
[eval]  grading responses and aggregating metrics
[summarize]  reading saved metrics → summary table

Evaluation results

Example

3 models · 2 benchmarks · accuracy.mean

Example values, not measured results or output from the command above.
ModelBenchmarkSamplesAccuracy
Llama-3.1-70B-Instructctibench-mcqa25—
GPT-4.1ctibench-mcqa25—
Foundation-Sec-8Bctibench-mcqa25—
Llama-3.1-70B-Instructseceval25—
GPT-4.1seceval25—
Foundation-Sec-8Bseceval25—

A three-phase pipeline

Each phase is a dedicated subcommand, so you can re-run any step independently — or chain all three with run-all.

01

Query

faith query

Records Q&A responses or agentic trajectories in the datastore. Q&A supports vLLM, OpenAI, OpenRouter, and SageMaker. Agentic execution requires the openai engine.

02

Eval

faith eval

Grades saved records and aggregates metrics within and across trials, writing metrics.json for each experiment.

03

Summarize

faith summarize

Reads saved metrics and experiment metadata for table or CSV output, or BigQuery ingestion.

Q&A and agentic evaluation

Both use the same query, eval, and summarize workflow.

one exchange

Q&A benchmarks

Evaluate model responses to benchmark questions using benchmark-specific grading.

many turns

Agentic benchmarks

Evaluate models as they use tools to complete tasks in a configured execution environment.

What you can do with FAITH

01

Evaluate multiple models on multiple benchmarks

Both --model-paths and --benchmarks take lists, so one run covers every pair.

02

Query local or hosted models

In-process vLLM for Q&A, plus OpenAI, OpenRouter, and AWS SageMaker endpoints. Agentic runs use an OpenAI-compatible tool-calling endpoint.

03

Define a custom benchmark

Define a benchmark.yaml and run it with --custom-benchmarks. Reuse built-in components or implement additional loaders, tools, and scoring functions as needed.

04

Record experiment settings

Completed query stages record resolved model and benchmark configurations and trial seeds. Retain data and deployment revisions separately when needed to reproduce an evaluation.

05

Analyze at scale in BigQuery

Ingest normalized metrics for SQL leaderboards and performance-over-time tracking.

06

Build dashboards

Power Looker Studio leaderboards and comparison charts from pre-built views.

One interface, many model backends

Q&A supports the engines below. Agentic execution requires the openai engine with a compatible tool-calling endpoint.

vLLMLocal

Hugging Face models on supported hardware

OpenAIAPI

GPT-family models

OpenRouterAPI

Claude, Gemini, and more

SageMakerAPI

Models on AWS endpoints

Run an initial evaluation

Install FAITH, configure a supported model and benchmark, and follow the quickstart to generate and evaluate a small set of records.

Install FAITHView on GitHub