class Riffer::Evals::ScenarioResult

Represents the result of evaluating a single scenario.

Contains the input, output, ground truth, and individual evaluator results.

scenario_result = Riffer::Evals::ScenarioResult.new( input: “What is Ruby?”, output: “A programming language.”, ground_truth: “A programming language”, results: [result1, result2] )

scenario_result.scores # => { MyEvaluator => 0.85 }

Attributes

ground_truth [R]

The ground truth used during evaluation.

input [R]

The input that was evaluated.

output [R]

The agent output for this scenario.

results [R]

Individual evaluation results.

Public Class Methods

new (input:, output:, ground_truth:, results:)

Source

# File lib/riffer/evals/scenario_result.rb, line 33
def initialize(input:, output:, ground_truth:, results:)
  @input = input
  @output = output
  @ground_truth = ground_truth
  @results = results
end

Initializes a new scenario result.

: (input: String, output: String, ground_truth: String?, results: Array) -> void

Public Instance Methods

scores ()

Source

# File lib/riffer/evals/scenario_result.rb, line 43
def scores
  results.each_with_object({}) do |result, hash|
    hash[result.evaluator] = result.score
  end
end

Returns scores keyed by evaluator class.

: () -> Hash[singleton(Riffer::Evals::Evaluator), Float]

to_h ()

Source

# File lib/riffer/evals/scenario_result.rb, line 52
def to_h
  {
    input: input,
    output: output,
    ground_truth: ground_truth,
    scores: scores.transform_keys(&:name),
    results: results.map(&:to_h)
  }
end

Returns a hash representation of the scenario result.

: () -> Hash[Symbol, untyped]