All articles

Model Musings

Confidence Intervals in SimPy: Quantifying Uncertainty

Published
Reading time
6 min

A point estimate without a confidence interval is a guess with false precision. Here’s how to do it properly.

What’s a Confidence Interval?

A 95% confidence interval means: if you ran this analysis 100 times with different random seeds, about 95 of those intervals would contain the true mean.

It’s not “95% probability the true value is in this range.” That’s a common misconception.

Basic Calculation

import numpy as np
from scipy import stats

def confidence_interval(data, confidence=0.95):
    """Calculate confidence interval for the mean."""
    n = len(data)
    mean = np.mean(data)
    se = stats.sem(data)  # Standard error
    t_value = stats.t.ppf((1 + confidence) / 2, df=n-1)
    margin = t_value * se

    return {
        'mean': mean,
        'lower': mean - margin,
        'upper': mean + margin,
        'margin': margin,
        'n': n
    }

# Usage
results = [4.2, 5.1, 3.8, 4.5, 5.2, 4.1, 4.8, 3.9, 4.6, 5.0]
ci = confidence_interval(results)
print(f"Mean: {ci['mean']:.2f}")
print(f"95% CI: [{ci['lower']:.2f}, {ci['upper']:.2f}]")

Output:

Mean: 4.52
95% CI: [4.16, 4.88]

From Multiple Replications

The standard workflow: run the model several times with different seeds, take one number from each run and put the interval around their mean. Here the model is a two-server counter at 90% load (the same one the replications guide uses), and each run reports its mean wait. The later sections build on this function and on confidence_interval above.

import random
import simpy

def run_replication(seed, servers=2, run_time=1000):
    """Run one simulation replication, return mean wait time."""
    rng = random.Random(seed)
    env = simpy.Environment()
    counter = simpy.Resource(env, capacity=servers)
    waits = []

    def customer(service_time):
        arrived = env.now
        with counter.request() as req:
            yield req
            waits.append(env.now - arrived)
            yield env.timeout(service_time)

    def arrivals():
        while True:
            yield env.timeout(rng.expovariate(1.0))
            # Service time drawn on arrival, so scenarios see the same customers
            env.process(customer(rng.expovariate(1 / 1.8)))

    env.process(arrivals())
    env.run(until=run_time)
    return float(np.mean(waits))

# Run multiple replications
n_reps = 30
results = [run_replication(42 + i) for i in range(n_reps)]

# Calculate CI
ci = confidence_interval(results)
print(f"Mean wait time: {ci['mean']:.2f} ± {ci['margin']:.2f}")

Output:

Mean wait time: 7.24 ± 1.87

Choosing Sample Size

How many replications for a target half-width?

def required_sample_size(pilot_data, target_half_width, confidence=0.95):
    """Estimate required sample size for target precision."""
    n_pilot = len(pilot_data)
    s = np.std(pilot_data, ddof=1)

    # Initial estimate using normal approximation
    z = stats.norm.ppf((1 + confidence) / 2)
    n_estimate = (z * s / target_half_width) ** 2

    # Refine using t-distribution
    t = stats.t.ppf((1 + confidence) / 2, df=max(1, int(n_estimate) - 1))
    n_required = (t * s / target_half_width) ** 2

    return int(np.ceil(n_required))

# Pilot study with 10 runs
pilot = [run_replication(42 + i) for i in range(10)]
needed = required_sample_size(pilot, target_half_width=0.5)
print(f"Need {needed} replications for half-width of 0.5")

Output:

Need 139 replications for half-width of 0.5

A pilot of ten asks for 139 replications to pin the mean wait down to within half a minute. Treat that as a first estimate rather than a promise: the pilot’s spread is itself an estimate, and for this busy queue the full thirty runs prove more variable than the first ten suggested. Recompute once the main run is in.

Relative Precision

Sometimes you want a percentage precision:

def required_for_relative_precision(pilot_data, target_relative_error, confidence=0.95):
    """Sample size for target relative half-width (e.g., 5% of mean)."""
    cv = np.std(pilot_data, ddof=1) / np.mean(pilot_data)  # Coefficient of variation
    z = stats.norm.ppf((1 + confidence) / 2)

    n_required = (z * cv / target_relative_error) ** 2
    return int(np.ceil(n_required))

# For 5% relative precision
needed = required_for_relative_precision(pilot, 0.05)
print(f"Need {needed} replications for 5% relative precision")

Output:

Need 327 replications for 5% relative precision

Comparing Two Scenarios

Paired comparison with confidence interval:

def compare_scenarios(results_a, results_b, confidence=0.95):
    """
    Paired comparison of two scenarios.

    Returns CI for the difference (A - B).
    """
    differences = np.array(results_a) - np.array(results_b)
    ci = confidence_interval(differences, confidence)

    return {
        'mean_diff': ci['mean'],
        'ci_lower': ci['lower'],
        'ci_upper': ci['upper'],
        'significant': (ci['lower'] > 0) or (ci['upper'] < 0)
    }

# Run both scenarios with same seeds (paired)
results_2_servers = [run_replication(42 + i, servers=2) for i in range(30)]
results_3_servers = [run_replication(42 + i, servers=3) for i in range(30)]

comparison = compare_scenarios(results_2_servers, results_3_servers)
print(f"Difference: {comparison['mean_diff']:.2f}")
print(f"95% CI: [{comparison['ci_lower']:.2f}, {comparison['ci_upper']:.2f}]")
print(f"Significant: {comparison['significant']}")

Output:

Difference: 6.69
95% CI: [4.85, 8.54]
Significant: True

Multiple Metrics

One run can report several numbers at once. Here is the same model returning its mean wait, its utilisation and its throughput per hour:

def run_replication_full(seed, servers=2, run_time=1000):
    """One replication; returns several metrics at once."""
    rng = random.Random(seed)
    env = simpy.Environment()
    counter = simpy.Resource(env, capacity=servers)
    waits, busy, done = [], 0.0, 0

    def customer(service_time):
        nonlocal busy, done
        arrived = env.now
        with counter.request() as req:
            yield req
            waits.append(env.now - arrived)
            busy += min(service_time, run_time - env.now)  # Busy time inside the run
            yield env.timeout(service_time)
            done += 1

    def arrivals():
        while True:
            yield env.timeout(rng.expovariate(1.0))
            env.process(customer(rng.expovariate(1 / 1.8)))

    env.process(arrivals())
    env.run(until=run_time)
    return {
        'wait': float(np.mean(waits)),
        'util': busy / (servers * run_time),
        'throughput': done / run_time * 60
    }

class SimulationResults:
    def __init__(self):
        self.replications = []

    def add_replication(self, metrics):
        """metrics = {'wait': 4.2, 'util': 0.85, 'throughput': 12.5}"""
        self.replications.append(metrics)

    def confidence_intervals(self, confidence=0.95):
        """Calculate CIs for all metrics."""
        results = {}
        metrics = self.replications[0].keys()

        for metric in metrics:
            values = [r[metric] for r in self.replications]
            results[metric] = confidence_interval(values, confidence)

        return results

# Usage
results = SimulationResults()
for i in range(30):
    metrics = run_replication_full(42 + i)
    results.add_replication(metrics)

cis = results.confidence_intervals()
for metric, metric_ci in cis.items():
    print(f"{metric}: {metric_ci['mean']:.2f} [{metric_ci['lower']:.2f}, {metric_ci['upper']:.2f}]")

Output:

wait: 7.24 [5.37, 9.11]
util: 0.90 [0.88, 0.91]
throughput: 59.26 [58.67, 59.85]

Same thirty runs, very different precision. Utilisation and throughput are pinned down to within a percentage point or so, while the mean wait still spans nearly four minutes. Waiting times are the most variable thing a queue produces, so they usually decide how many replications you need.

Visualising Confidence Intervals

import matplotlib.pyplot as plt

def plot_ci_comparison(scenarios):
    """
    scenarios = {
        'Baseline': {'mean': 5.2, 'lower': 4.8, 'upper': 5.6},
        'Improved': {'mean': 3.1, 'lower': 2.9, 'upper': 3.3}
    }
    """
    names = list(scenarios.keys())
    means = [s['mean'] for s in scenarios.values()]
    errors = [[s['mean'] - s['lower'], s['upper'] - s['mean']]
              for s in scenarios.values()]
    errors = np.array(errors).T

    plt.figure(figsize=(8, 5))
    plt.bar(names, means, yerr=errors, capsize=10,
            color='steelblue', alpha=0.7)
    plt.ylabel('Mean Wait Time')
    plt.title('Scenario Comparison with 95% Confidence Intervals')
    plt.tight_layout()
    plt.show()

Bootstrap Confidence Intervals

When normality assumptions don’t hold:

def bootstrap_ci(data, n_bootstrap=10000, confidence=0.95, rng=None):
    """Calculate bootstrap confidence interval."""
    rng = rng or np.random.default_rng()
    bootstrapped_means = []

    for _ in range(n_bootstrap):
        sample = rng.choice(data, size=len(data), replace=True)
        bootstrapped_means.append(np.mean(sample))

    alpha = (1 - confidence) / 2
    lower = np.percentile(bootstrapped_means, alpha * 100)
    upper = np.percentile(bootstrapped_means, (1 - alpha) * 100)

    return {
        'mean': np.mean(data),
        'lower': lower,
        'upper': upper
    }

data = results_2_servers  # The 30 two-server replications from above
t_ci = confidence_interval(data)
boot = bootstrap_ci(data, rng=np.random.default_rng(42))  # Seeded, so it repeats
print(f"t interval:         [{t_ci['lower']:.2f}, {t_ci['upper']:.2f}]")
print(f"Bootstrap interval: [{boot['lower']:.2f}, {boot['upper']:.2f}]")

Output:

t interval:         [5.37, 9.11]
Bootstrap interval: [5.70, 9.22]

The two broadly agree, but the bootstrap interval is lopsided: about 1.5 minutes below the mean and 2.0 above it, where the t interval is symmetric by construction. Two long and unlucky runs drag the distribution of the mean to the right; the bootstrap says so. With thirty replications the difference is modest. The bootstrap cannot conjure information that a handful of runs does not contain, so it is no substitute for running enough of them.

Reporting Results

Good reporting includes:

def format_result(ci, metric_name, decimals=2):
    """Format a confidence interval for reporting."""
    return (
        f"{metric_name}: {ci['mean']:.{decimals}f} "
        f"(95% CI: {ci['lower']:.{decimals}f} to {ci['upper']:.{decimals}f}, "
        f"n={ci['n']})"
    )

print(format_result(confidence_interval(results_2_servers), "Mean wait time"))

Output:

Mean wait time: 7.24 (95% CI: 5.37 to 9.11, n=30)

Common Mistakes

Too few replications:

# Bad: CI is meaninglessly wide
results = [run_replication(i) for i in range(3)]
# Good: CI has reasonable precision
results = [run_replication(i) for i in range(30)]

Ignoring warm-up:

# Bad: includes initialisation bias
mean_wait = simulation_run()
# Good: excludes warm-up period
mean_wait = simulation_run_with_warmup(warmup=100)

Reporting without CI:

# Bad: false precision
print(f"Mean wait: 4.523")
# Good: honest uncertainty
print(f"Mean wait: 4.5 (95% CI: 4.2-4.8)")

Summary

Confidence intervals:

  • Quantify uncertainty in your estimates
  • Require multiple replications
  • Enable meaningful comparisons
  • Should always accompany point estimates

A simulation result without a confidence interval is not entirely trustworthy. Always report your uncertainty.

Next Steps

Ready to build a model of your own?

The free SimPy Handbook covers the foundations; the Simulation Bootcamp covers everything else, from first principles to models a business will act on.