Model Musings
Running Multiple Replications in SimPy: Statistical Confidence
- Published
- Reading time
- 6 min
One simulation run proves nothing. Run it again: different results. Which is right? Both. Neither. You need replications.
Why Multiple Replications?
A single simulation run is one sample from a distribution of possible outcomes. You wouldn’t draw conclusions from one data point. Don’t draw them from one simulation.
Multiple replications give you:
- Estimate of the mean
- Estimate of the variance
- Confidence intervals
- Statistical validity
Basic Replication Pattern
A replication is one complete run of the model with its own random numbers. To have something real to replicate, here is a complete model: a counter with two servers, customers arriving about once a minute and taking 1.8 minutes on average to serve, so the servers are busy 90% of the time. Each replication builds a fresh environment and draws from its own random.Random(seed), so nothing leaks from one run into the next. The rest of this post builds on these two functions.
import random
import simpy
import numpy as np
def run_single_replication(seed, run_time=1000, servers=2,
mean_interarrival=1.0, mean_service=1.8):
"""One replication of a multi-server queue; returns the mean wait."""
rng = random.Random(seed) # This replication's own random stream
env = simpy.Environment()
counter = simpy.Resource(env, capacity=servers)
waits = []
def customer(service_time):
arrived = env.now
with counter.request() as req:
yield req
waits.append(env.now - arrived)
yield env.timeout(service_time)
def arrivals():
while True:
yield env.timeout(rng.expovariate(1 / mean_interarrival))
# Draw the service time on arrival, so that every scenario
# sees exactly the same customers
env.process(customer(rng.expovariate(1 / mean_service)))
env.process(arrivals())
env.run(until=run_time)
return np.mean(waits)
def run_replications(n_replications, base_seed=42, **config):
"""Run replications, each with its own seed."""
return [run_single_replication(seed=base_seed + i, **config)
for i in range(n_replications)]
results = run_replications(30)
print("First five:", ", ".join(f"{r:.2f}" for r in results[:5]))
print(f"Range: {min(results):.2f} to {max(results):.2f}")
print(f"Overall mean: {np.mean(results):.2f}")
print(f"Std deviation: {np.std(results, ddof=1):.2f}")
Output:
First five: 4.03, 7.23, 13.06, 7.96, 5.05
Range: 3.33 to 25.13
Overall mean: 7.24
Std deviation: 5.01
Thirty runs of one model, and the mean wait ranges from 3.33 to 25.13 minutes. Queueing theory (the Erlang C formula) puts the long-run average for this system at 7.67 minutes. The replications home in on it together; any one of them alone could have told you almost anything.
How Many Replications?
Rule of thumb: at least 10 to 30.
More precise: enough to achieve the precision you need. Run a small pilot, measure its spread and work out how many replications bring the confidence interval down to the half-width you want:
from scipy import stats
def required_replications(pilot_results, target_half_width, confidence=0.95):
"""Estimate replications needed for target confidence interval width."""
n_pilot = len(pilot_results)
s = np.std(pilot_results, ddof=1)
t_value = stats.t.ppf((1 + confidence) / 2, df=n_pilot - 1)
# Required n to achieve half-width
n_required = (t_value * s / target_half_width) ** 2
return int(np.ceil(n_required))
# Pilot study: pin the mean wait down to within a minute
pilot = run_replications(10)
needed = required_replications(pilot, target_half_width=1.0)
print(f"Need approximately {needed} replications")
Output:
Need approximately 46 replications
Forty-six, says the pilot. Treat that as a first estimate rather than a promise: ten runs are a small sample of the spread itself, and this pilot happened to miss the worst of it. Recompute once the main run is in, and top up if you fall short.
Confidence Intervals
def confidence_interval(data, confidence=0.95):
"""Calculate confidence interval for mean."""
n = len(data)
mean = np.mean(data)
se = stats.sem(data)
t_value = stats.t.ppf((1 + confidence) / 2, df=n-1)
margin = t_value * se
return mean, mean - margin, mean + margin
results = run_replications(30)
mean, lower, upper = confidence_interval(results)
print(f"Mean: {mean:.2f}")
print(f"95% CI: [{lower:.2f}, {upper:.2f}]")
Output:
Mean: 7.24
95% CI: [5.37, 9.11]
Thirty replications put the mean wait between 5.37 and 9.11 minutes: a half-width of nearly two minutes, not the one the pilot promised, because the thirty include a few long, unlucky runs that the first ten did not. Feed these thirty results back into required_replications and it asks for 105. Busy queues are like that; the closer the servers run to full, the more the answer swings from one run to the next.
Parallel Replications
Speed up with parallel processing:
from multiprocessing import Pool
def run_single_replication_wrapper(args):
seed, run_time = args
return run_single_replication(seed, run_time)
def run_parallel_replications(n_replications, base_seed=42, n_workers=4):
"""Run replications in parallel."""
args = [(base_seed + i, 1000) for i in range(n_replications)]
with Pool(n_workers) as pool:
results = pool.map(run_single_replication_wrapper, args)
return results
# Much faster for large replication counts. Keep the guard: Python
# re-imports this file to start the workers on Windows and macOS, and on
# Linux too from Python 3.14.
if __name__ == "__main__":
results = run_parallel_replications(100, n_workers=8)
print(f"{len(results)} replications, mean {np.mean(results):.2f}")
print("Same as the serial run:", results == run_replications(100))
Output:
100 replications, mean 7.65
Same as the serial run: True
Each replication carries its own seed, so the same hundred seeds give the same hundred answers whichever worker happens to run them; the parallel results match the serial ones exactly. One caution if you paste this whole post into a single file: every line outside the guard runs again when the workers start (in each worker, on Windows and macOS), the earlier demonstrations included. Keep the top level of the file to definitions, or give the model a module of its own.
Comparing Scenarios
Run replications for each scenario:
def run_scenario(scenario_config, n_replications=30, base_seed=42):
results = []
for i in range(n_replications):
result = run_single_replication(
seed=base_seed + i,
**scenario_config
)
results.append(result)
return results
# Compare 2 servers vs 3 servers
scenario_a = run_scenario({'servers': 2})
scenario_b = run_scenario({'servers': 3})
print(f"2 servers: {np.mean(scenario_a):.2f} ± {np.std(scenario_a, ddof=1):.2f}")
print(f"3 servers: {np.mean(scenario_b):.2f} ± {np.std(scenario_b, ddof=1):.2f}")
Output:
2 servers: 7.24 ± 5.01
3 servers: 0.55 ± 0.15
The figure after the ± is the standard deviation across replications, not a confidence interval; for that, use confidence_interval from above.
Paired Comparisons
For fair comparison, use same random numbers:
def run_paired_comparison(config_a, config_b, n_replications=30):
"""Run paired replications using common random numbers."""
differences = []
for i in range(n_replications):
seed = 42 + i
result_a = run_single_replication(seed=seed, **config_a)
result_b = run_single_replication(seed=seed, **config_b)
differences.append(result_a - result_b)
return differences
diffs = run_paired_comparison({'servers': 2}, {'servers': 3})
mean_diff, lower, upper = confidence_interval(diffs)
print(f"Mean difference: {mean_diff:.2f} (95% CI {lower:.2f} to {upper:.2f})")
if lower > 0:
print("Scenario A is significantly higher")
elif upper < 0:
print("Scenario B is significantly higher")
else:
print("No significant difference")
Output:
Mean difference: 6.69 (95% CI 4.85 to 8.54)
Scenario A is significantly higher
A third server cuts the mean wait by 6.69 minutes, and the interval sits well clear of zero. Pairing only works if the same random number drives the same thing in both runs. That is why run_single_replication draws each customer’s service time on arrival: drawn when service starts, it would come at a different moment in each scenario and interleave differently with the arrival draws, so the two runs would soon be using different random numbers for the same customers.
Complete Replication Manager
class ReplicationManager:
def __init__(self, simulation_func, base_seed=42):
self.simulation_func = simulation_func
self.base_seed = base_seed
self.results = {}
def run(self, scenario_name, config, n_replications=30):
"""Run replications for a scenario."""
results = []
for i in range(n_replications):
result = self.simulation_func(
seed=self.base_seed + i,
**config
)
results.append(result)
self.results[scenario_name] = results
return results
def summary(self, scenario_name):
"""Get summary statistics for a scenario."""
data = self.results[scenario_name]
mean, lower, upper = confidence_interval(data)
return {
'n': len(data),
'mean': mean,
'std': np.std(data, ddof=1),
'ci_lower': lower,
'ci_upper': upper,
'min': min(data),
'max': max(data)
}
def compare(self, scenario_a, scenario_b):
"""Compare two scenarios."""
a = self.results[scenario_a]
b = self.results[scenario_b]
# Paired t-test
t_stat, p_value = stats.ttest_rel(a, b)
return {
'mean_a': np.mean(a),
'mean_b': np.mean(b),
'difference': np.mean(a) - np.mean(b),
't_statistic': t_stat,
'p_value': p_value,
'significant': p_value < 0.05
}
# Usage
manager = ReplicationManager(run_single_replication)
manager.run('baseline', {'servers': 2}, n_replications=30)
manager.run('improved', {'servers': 3}, n_replications=30)
for name in ('baseline', 'improved'):
s = manager.summary(name)
print(f"{name}: mean {s['mean']:.2f}, 95% CI [{s['ci_lower']:.2f}, {s['ci_upper']:.2f}], n = {s['n']}")
c = manager.compare('baseline', 'improved')
print(f"Difference {c['difference']:.2f}, p = {c['p_value']:.1e}, significant: {c['significant']}")
Output:
baseline: mean 7.24, 95% CI [5.37, 9.11], n = 30
improved: mean 0.55, 95% CI [0.49, 0.60], n = 30
Difference 6.69, p = 3.5e-08, significant: True
Storing Results
Save results for later analysis:
import pandas as pd
def save_results(results, filename):
"""Save replication results."""
df = pd.DataFrame([
{'scenario': scenario, 'replication': i, 'value': value}
for scenario, values in results.items()
for i, value in enumerate(values)
])
df.to_csv(filename, index=False)
def load_results(filename):
"""Load replication results."""
df = pd.read_csv(filename)
results = {}
for scenario in df['scenario'].unique():
results[scenario] = df[df['scenario'] == scenario]['value'].tolist()
return results
save_results(manager.results, 'replications.csv')
print("Round trip intact:", load_results('replications.csv') == manager.results)
Output:
Round trip intact: True
Summary
Multiple replications:
- Run at least 10 to 30 per scenario, and as many as your precision demands
- Use different seeds for each replication
- Calculate confidence intervals
- Use paired comparisons for scenarios
- Parallelise for speed
One run tells you nothing. Many runs tell you everything.