# General formulas and definitions¶

## Formulas for A/B testing¶

The main metrics to perform A/B testing are described in [Stu15]. Let us consider two variants $$X_A$$ and $$X_B$$ for testing.

The error probability or probability of $$X_B > X_A$$ is denoted as

$P[X_B > X_A] = \int_{-\infty}^{\infty} \int_{x_A}^{\infty} f(x_A, x_B) \mathop{dx_B} \mathop{dx_A},$

where $$f(x_A, x_B)$$ is the joint probability distribution, under the assumption of independence, i.e. $$f(x_A, x_B) = f(x_A) f(x_B)$$.

The expected loss function given a joint posterior is the expected value of the loss function. The loss function is the expected uplift lost by choosing a given variant. If variant $$X_B$$ is chosen we have

$EL(X_B) = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \max(x_A - x_B, 0) f(x_A, x_B) \mathop{dx_B} \mathop{dx_A}.$

Other metrics also considered are the relative expected loss or uplift and credible intervals. A credible interval is a region which has a specified probability of containing the true value.

## Formulas for Multivariate testing¶

Let us first introduce some properties of the distribution of the maximum of a set of independent random variables with support on the whole real line.

$X_{max} = \max\{X_1, \ldots, X_n\}$

The cumulative distribution function is

$F_{X_{max}}(z) = P\left[\underset{i=1, \ldots, n}\max{X_i} \le z\right] = \prod_{i=1}^n P[X_i \le z] = \prod_{i=1}^n F_{X_i}(z),$

where $$F_{X_i}(z)$$ is the cdf of each random variable $$X_i$$. The probability density functions is obtain after derivation

$f_{X_{max}}(z) = \sum_{i=1}^n f_{X_i}(z) \prod_{j \neq i} F_{X_j}(z).$

where $$f_{X_i}(z)$$ is the pdf of each random variable $$X_i$$.

The probability to beat all is defined as

$P\left[X_i > \underset{j \neq i}\max{X_j}\right] = \int_{-\infty}^{\infty} f(x_i) \prod_{j \neq i} F_{X_j}(x_i) \mathop{dx_i}.$

The expected loss function vs all is defined as

$\mathrm{E}[\max(\underset{j \neq i}\max{X_j} - X_i, 0)]$

Take $$Y = \underset{j \neq i}\max{X_j}$$, then we have

$\begin{split}EL(X_i) &= \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \max(y - x_i, 0) f(y) f(x_i) \mathop{dx_i} \mathop{dy} \\ &= \int_{-\infty}^{\infty} \int_{-\infty}^y y f(y)f(x_i) \mathop{dx_i} \mathop{dy} - \int_{-\infty}^{\infty} \int_{-\infty}^y x_i f(y)f(x_i) \mathop{dx_i} \mathop{dy}\\ &= \int_{-\infty}^{\infty} y f(y) F_{X_i}(y) \mathop{dy} - \int_{-\infty}^{\infty} f(y) F^*_{X_i}(y) \mathop{dy},\end{split}$

where $$F^*_{X_i}(y) = \int_{-\infty}^y x_i f(x_i) \mathop{dx_i}$$.

## References¶

 [Stu15] C. Stucchio. Bayesian A/B Testing at VWO. Visual Web Optimizer, 2015. URL: https://www.chrisstucchio.com/pubs/VWO_SmartStats_technical_whitepaper.pdf.