Bayesian experiment¶

Main class and utilities to run, explore and visualize a Bayesian experiment for A/B and multivariate testing.

Experiment¶

class cprior.experiment.Experiment(name, test, stopping_rule='expected_loss', epsilon=1e-05, min_n_samples=None, max_n_samples=None, verbose=False, **options)¶

Bases: object

Bayesian experiment.

Parameters:

name (str) – Experiment name.
test (object) – A multivariate test instance. This is a class inherit from cprior.cdist.base.BayesMVTest.
stopping_rule (str (default="expected_loss")) – The stopping rule or metric to be used throughout the experiment. Options are “probability”, “probability_vs_all”, “expected_loss” and “expected_loss_vs_all”.
epsilon (float (default=1e-5)) – The epsilon or threshold to be checked throughout the experiment. The experiment will terminate with a winner variant if the metric value reaches epsilon.
min_n_samples (int or None (default=None)) – The minimum number of samples for any variant.
max_n_samples (int or None (default=None)) – The maximum number of samples for any variant.
verbose (int or bool (default=False)) – Controls verbosity of output.
**options – For other keyword-only arguments. For example, nig_metric with options "mu" and "sigma_sq".

variants_¶

The variants in the experiment.

Type:	list

n_variants_¶

The number of variants in the experiment.

Type:	int

n_samples_¶

The total number of samples of all variants throughout the experimentation.

Type:	int

n_updates_¶

The total number of updates throughout the experimentation.

Type:	int

describe()¶: Experiment settings.

load(pickle_path)¶

Load experiment from a given pickle path.

Parameters:	pickle_path (str) – Path of the pickle object.

plot_metric()¶: Plot stopping rule metric over updates/time.

plot_stats()¶: Plot statistics (mean and CI intervals) over updates/time.

run_update(**data)¶

Update one or various Bayesian models with new data.

Parameters:	*data (dict) – Dictionary with key=variant and value=data, e.g. `*{"A": data_A}`.

save(pickle_path)¶

Save this experiment to the given pickle path.

Parameters:	pickle_path (str) – Path of the pickle object.

stats()¶: Experiment main statistics on collected data.

summary()¶

Experiment summary with several decision metrics.

If a winner has been declared, the corresponding row is highlighted in green.

Plotting¶

cprior.experiment.experiment_plot_metric(experiment)¶

Plot stopping rule metric over updates/time.

Parameters:	experiment (object) – A Bayesian experiment instance.

cprior.experiment.experiment_plot_stats(experiment)¶

Plot statistics (mean and CI intervals) over updates/time.

Parameters:	experiment (object) – A Bayesian experiment instance.

Utilities and reporting tools¶

cprior.experiment.experiment_describe(experiment)¶

Experiment settings.

Parameters:	experiment (object) – A Bayesian experiment instance.

cprior.experiment.experiment_stats(experiment)¶

Experiment main statistics on collected data.

Parameters:	experiment (object) – A Bayesian experiment instance.
Returns:	stats
Return type:	pandas.DataFrame

cprior.experiment.experiment_summary(experiment)¶

Experiment summary with several decision metrics.

If a winner has been declared, the corresponding row is highlighted in green.

Parameters:	experiment (object) – A Bayesian experiment instance.
Returns:	stats
Return type:	pandas.DataFrame