Bayesian experiment¶
Main class and utilities to run, explore and visualize a Bayesian experiment for A/B and multivariate testing.
Experiment¶

class
cprior.experiment.
Experiment
(name, test, stopping_rule='expected_loss', epsilon=1e05, min_n_samples=None, max_n_samples=None, verbose=False, **options)¶ Bases:
object
Bayesian experiment.
Parameters:  name (str) – Experiment name.
 test (object) – A multivariate test instance. This is a class inherit from
cprior.cdist.base.BayesMVTest
.  stopping_rule (str (default="expected_loss")) – The stopping rule or metric to be used throughout the experiment. Options are “probability”, “probability_vs_all”, “expected_loss” and “expected_loss_vs_all”.
 epsilon (float (default=1e5)) – The epsilon or threshold to be checked throughout the experiment. The
experiment will terminate with a winner variant if the metric value
reaches
epsilon
.  min_n_samples (int or None (default=None)) – The minimum number of samples for any variant.
 max_n_samples (int or None (default=None)) – The maximum number of samples for any variant.
 verbose (int or bool (default=False)) – Controls verbosity of output.
 **options – For other keywordonly arguments. For example,
nig_metric
with options"mu"
and"sigma_sq"
.

variants_
¶ The variants in the experiment.
Type: list

n_variants_
¶ The number of variants in the experiment.
Type: int

n_samples_
¶ The total number of samples of all variants throughout the experimentation.
Type: int

n_updates_
¶ The total number of updates throughout the experimentation.
Type: int

describe
()¶ Experiment settings.

load
(pickle_path)¶ Load experiment from a given pickle path.
Parameters: pickle_path (str) – Path of the pickle object.

plot_metric
()¶ Plot stopping rule metric over updates/time.

plot_stats
()¶ Plot statistics (mean and CI intervals) over updates/time.

run_update
(**data)¶ Update one or various Bayesian models with new data.
Parameters: **data (dict) – Dictionary with key=variant and value=data, e.g. **{"A": data_A}
.

save
(pickle_path)¶ Save this experiment to the given pickle path.
Parameters: pickle_path (str) – Path of the pickle object.

stats
()¶ Experiment main statistics on collected data.

summary
()¶ Experiment summary with several decision metrics.
If a winner has been declared, the corresponding row is highlighted in green.
Plotting¶

cprior.experiment.
experiment_plot_metric
(experiment)¶ Plot stopping rule metric over updates/time.
Parameters: experiment (object) – A Bayesian experiment instance.

cprior.experiment.
experiment_plot_stats
(experiment)¶ Plot statistics (mean and CI intervals) over updates/time.
Parameters: experiment (object) – A Bayesian experiment instance.
Utilities and reporting tools¶

cprior.experiment.
experiment_describe
(experiment)¶ Experiment settings.
Parameters: experiment (object) – A Bayesian experiment instance.

cprior.experiment.
experiment_stats
(experiment)¶ Experiment main statistics on collected data.
Parameters: experiment (object) – A Bayesian experiment instance. Returns: stats Return type: pandas.DataFrame

cprior.experiment.
experiment_summary
(experiment)¶ Experiment summary with several decision metrics.
If a winner has been declared, the corresponding row is highlighted in green.
Parameters: experiment (object) – A Bayesian experiment instance. Returns: stats Return type: pandas.DataFrame