Model Musings
Confidence Intervals in SimPy: Quantifying Uncertainty
- Published
- Reading time
- 6 min
A point estimate without a confidence interval is a guess with false precision. Here’s how to do it properly.
What’s a Confidence Interval?
A 95% confidence interval means: if you ran this analysis 100 times with different random seeds, about 95 of those intervals would contain the true mean.
It’s not “95% probability the true value is in this range.” That’s a common misconception.
Basic Calculation
import numpy as np
from scipy import stats
def confidence_interval(data, confidence=0.95):
"""Calculate confidence interval for the mean."""
n = len(data)
mean = np.mean(data)
se = stats.sem(data) # Standard error
t_value = stats.t.ppf((1 + confidence) / 2, df=n-1)
margin = t_value * se
return {
'mean': mean,
'lower': mean - margin,
'upper': mean + margin,
'margin': margin,
'n': n
}
# Usage
results = [4.2, 5.1, 3.8, 4.5, 5.2, 4.1, 4.8, 3.9, 4.6, 5.0]
ci = confidence_interval(results)
print(f"Mean: {ci['mean']:.2f}")
print(f"95% CI: [{ci['lower']:.2f}, {ci['upper']:.2f}]")
Output:
Mean: 4.52
95% CI: [4.16, 4.88]
From Multiple Replications
The standard workflow: run the model several times with different seeds, take one number from each run and put the interval around their mean. Here the model is a two-server counter at 90% load (the same one the replications guide uses), and each run reports its mean wait. The later sections build on this function and on confidence_interval above.
import random
import simpy
def run_replication(seed, servers=2, run_time=1000):
"""Run one simulation replication, return mean wait time."""
rng = random.Random(seed)
env = simpy.Environment()
counter = simpy.Resource(env, capacity=servers)
waits = []
def customer(service_time):
arrived = env.now
with counter.request() as req:
yield req
waits.append(env.now - arrived)
yield env.timeout(service_time)
def arrivals():
while True:
yield env.timeout(rng.expovariate(1.0))
# Service time drawn on arrival, so scenarios see the same customers
env.process(customer(rng.expovariate(1 / 1.8)))
env.process(arrivals())
env.run(until=run_time)
return float(np.mean(waits))
# Run multiple replications
n_reps = 30
results = [run_replication(42 + i) for i in range(n_reps)]
# Calculate CI
ci = confidence_interval(results)
print(f"Mean wait time: {ci['mean']:.2f} ± {ci['margin']:.2f}")
Output:
Mean wait time: 7.24 ± 1.87
Choosing Sample Size
How many replications for a target half-width?
def required_sample_size(pilot_data, target_half_width, confidence=0.95):
"""Estimate required sample size for target precision."""
n_pilot = len(pilot_data)
s = np.std(pilot_data, ddof=1)
# Initial estimate using normal approximation
z = stats.norm.ppf((1 + confidence) / 2)
n_estimate = (z * s / target_half_width) ** 2
# Refine using t-distribution
t = stats.t.ppf((1 + confidence) / 2, df=max(1, int(n_estimate) - 1))
n_required = (t * s / target_half_width) ** 2
return int(np.ceil(n_required))
# Pilot study with 10 runs
pilot = [run_replication(42 + i) for i in range(10)]
needed = required_sample_size(pilot, target_half_width=0.5)
print(f"Need {needed} replications for half-width of 0.5")
Output:
Need 139 replications for half-width of 0.5
A pilot of ten asks for 139 replications to pin the mean wait down to within half a minute. Treat that as a first estimate rather than a promise: the pilot’s spread is itself an estimate, and for this busy queue the full thirty runs prove more variable than the first ten suggested. Recompute once the main run is in.
Relative Precision
Sometimes you want a percentage precision:
def required_for_relative_precision(pilot_data, target_relative_error, confidence=0.95):
"""Sample size for target relative half-width (e.g., 5% of mean)."""
cv = np.std(pilot_data, ddof=1) / np.mean(pilot_data) # Coefficient of variation
z = stats.norm.ppf((1 + confidence) / 2)
n_required = (z * cv / target_relative_error) ** 2
return int(np.ceil(n_required))
# For 5% relative precision
needed = required_for_relative_precision(pilot, 0.05)
print(f"Need {needed} replications for 5% relative precision")
Output:
Need 327 replications for 5% relative precision
Comparing Two Scenarios
Paired comparison with confidence interval:
def compare_scenarios(results_a, results_b, confidence=0.95):
"""
Paired comparison of two scenarios.
Returns CI for the difference (A - B).
"""
differences = np.array(results_a) - np.array(results_b)
ci = confidence_interval(differences, confidence)
return {
'mean_diff': ci['mean'],
'ci_lower': ci['lower'],
'ci_upper': ci['upper'],
'significant': (ci['lower'] > 0) or (ci['upper'] < 0)
}
# Run both scenarios with same seeds (paired)
results_2_servers = [run_replication(42 + i, servers=2) for i in range(30)]
results_3_servers = [run_replication(42 + i, servers=3) for i in range(30)]
comparison = compare_scenarios(results_2_servers, results_3_servers)
print(f"Difference: {comparison['mean_diff']:.2f}")
print(f"95% CI: [{comparison['ci_lower']:.2f}, {comparison['ci_upper']:.2f}]")
print(f"Significant: {comparison['significant']}")
Output:
Difference: 6.69
95% CI: [4.85, 8.54]
Significant: True
Multiple Metrics
One run can report several numbers at once. Here is the same model returning its mean wait, its utilisation and its throughput per hour:
def run_replication_full(seed, servers=2, run_time=1000):
"""One replication; returns several metrics at once."""
rng = random.Random(seed)
env = simpy.Environment()
counter = simpy.Resource(env, capacity=servers)
waits, busy, done = [], 0.0, 0
def customer(service_time):
nonlocal busy, done
arrived = env.now
with counter.request() as req:
yield req
waits.append(env.now - arrived)
busy += min(service_time, run_time - env.now) # Busy time inside the run
yield env.timeout(service_time)
done += 1
def arrivals():
while True:
yield env.timeout(rng.expovariate(1.0))
env.process(customer(rng.expovariate(1 / 1.8)))
env.process(arrivals())
env.run(until=run_time)
return {
'wait': float(np.mean(waits)),
'util': busy / (servers * run_time),
'throughput': done / run_time * 60
}
class SimulationResults:
def __init__(self):
self.replications = []
def add_replication(self, metrics):
"""metrics = {'wait': 4.2, 'util': 0.85, 'throughput': 12.5}"""
self.replications.append(metrics)
def confidence_intervals(self, confidence=0.95):
"""Calculate CIs for all metrics."""
results = {}
metrics = self.replications[0].keys()
for metric in metrics:
values = [r[metric] for r in self.replications]
results[metric] = confidence_interval(values, confidence)
return results
# Usage
results = SimulationResults()
for i in range(30):
metrics = run_replication_full(42 + i)
results.add_replication(metrics)
cis = results.confidence_intervals()
for metric, metric_ci in cis.items():
print(f"{metric}: {metric_ci['mean']:.2f} [{metric_ci['lower']:.2f}, {metric_ci['upper']:.2f}]")
Output:
wait: 7.24 [5.37, 9.11]
util: 0.90 [0.88, 0.91]
throughput: 59.26 [58.67, 59.85]
Same thirty runs, very different precision. Utilisation and throughput are pinned down to within a percentage point or so, while the mean wait still spans nearly four minutes. Waiting times are the most variable thing a queue produces, so they usually decide how many replications you need.
Visualising Confidence Intervals
import matplotlib.pyplot as plt
def plot_ci_comparison(scenarios):
"""
scenarios = {
'Baseline': {'mean': 5.2, 'lower': 4.8, 'upper': 5.6},
'Improved': {'mean': 3.1, 'lower': 2.9, 'upper': 3.3}
}
"""
names = list(scenarios.keys())
means = [s['mean'] for s in scenarios.values()]
errors = [[s['mean'] - s['lower'], s['upper'] - s['mean']]
for s in scenarios.values()]
errors = np.array(errors).T
plt.figure(figsize=(8, 5))
plt.bar(names, means, yerr=errors, capsize=10,
color='steelblue', alpha=0.7)
plt.ylabel('Mean Wait Time')
plt.title('Scenario Comparison with 95% Confidence Intervals')
plt.tight_layout()
plt.show()
Bootstrap Confidence Intervals
When normality assumptions don’t hold:
def bootstrap_ci(data, n_bootstrap=10000, confidence=0.95, rng=None):
"""Calculate bootstrap confidence interval."""
rng = rng or np.random.default_rng()
bootstrapped_means = []
for _ in range(n_bootstrap):
sample = rng.choice(data, size=len(data), replace=True)
bootstrapped_means.append(np.mean(sample))
alpha = (1 - confidence) / 2
lower = np.percentile(bootstrapped_means, alpha * 100)
upper = np.percentile(bootstrapped_means, (1 - alpha) * 100)
return {
'mean': np.mean(data),
'lower': lower,
'upper': upper
}
data = results_2_servers # The 30 two-server replications from above
t_ci = confidence_interval(data)
boot = bootstrap_ci(data, rng=np.random.default_rng(42)) # Seeded, so it repeats
print(f"t interval: [{t_ci['lower']:.2f}, {t_ci['upper']:.2f}]")
print(f"Bootstrap interval: [{boot['lower']:.2f}, {boot['upper']:.2f}]")
Output:
t interval: [5.37, 9.11]
Bootstrap interval: [5.70, 9.22]
The two broadly agree, but the bootstrap interval is lopsided: about 1.5 minutes below the mean and 2.0 above it, where the t interval is symmetric by construction. Two long and unlucky runs drag the distribution of the mean to the right; the bootstrap says so. With thirty replications the difference is modest. The bootstrap cannot conjure information that a handful of runs does not contain, so it is no substitute for running enough of them.
Reporting Results
Good reporting includes:
def format_result(ci, metric_name, decimals=2):
"""Format a confidence interval for reporting."""
return (
f"{metric_name}: {ci['mean']:.{decimals}f} "
f"(95% CI: {ci['lower']:.{decimals}f} to {ci['upper']:.{decimals}f}, "
f"n={ci['n']})"
)
print(format_result(confidence_interval(results_2_servers), "Mean wait time"))
Output:
Mean wait time: 7.24 (95% CI: 5.37 to 9.11, n=30)
Common Mistakes
Too few replications:
# Bad: CI is meaninglessly wide
results = [run_replication(i) for i in range(3)]
# Good: CI has reasonable precision
results = [run_replication(i) for i in range(30)]
Ignoring warm-up:
# Bad: includes initialisation bias
mean_wait = simulation_run()
# Good: excludes warm-up period
mean_wait = simulation_run_with_warmup(warmup=100)
Reporting without CI:
# Bad: false precision
print(f"Mean wait: 4.523")
# Good: honest uncertainty
print(f"Mean wait: 4.5 (95% CI: 4.2-4.8)")
Summary
Confidence intervals:
- Quantify uncertainty in your estimates
- Require multiple replications
- Enable meaningful comparisons
- Should always accompany point estimates
A simulation result without a confidence interval is not entirely trustworthy. Always report your uncertainty.