Work out a b test for models instantly with clear inputs, formula shown and shareable results.
Comparing two models on a conversion metric is a two-proportion test. Required sample per arm scales with the variance of the metric and inversely with the square of the effect size, so halving the detectable effect quadruples the sample. That relationship is why tiny model improvements are effectively unmeasurable at modest traffic, and why the minimum detectable effect must be agreed before launch.
Two-proportion sample size
n per arm = (z_alpha sqrt(2 p(1-p)) + z_beta sqrt(p1(1-p1) + p2(1-p2)))^2 / delta^2
Not with a fixed-horizon test; peeking inflates the false-positive rate badly. Use a sequential or Bayesian design if you need to monitor continuously.
Even splits minimise the total sample needed. Uneven splits are for risk management, and they increase the required duration.