
Hypothesis

Plan

The hypothesis is formed from what we want to know, understand, study
 Describe
 Prioritize
 Backlog

Use Pattern
 If....., Then....
 SMART

use Experiment description template

Hypothesis. Description
 what and how we want to achieve

Product
 What we do in the product
 what the experience of the test and control groups looks like

Users
 which users and at what point they become a part of the experiment
 Minimum sample size
 use calc
 How
 1
 Statistical Power (1−β)
 Determine the power of the test
 Percent of the time the minimum effect size will be detected, assuming it exists
 The power is usually set at 80% and the confidence level is 95%
 Significance level α
 Percent of the time a difference will be detected, assuming one does NOT exist
 the level of trust
 2
 Determine the minimum effect that we want to notice in the experiment (in %)
 conv2conv1
 conv2 is the expected conversion of the test group
 conv1 is the conversion that we have now
 3
 Solve the equation by substituting the values into the formula and look for X
 X is the required minimum sample size for each group
 x = conv2  conv1 +1.96 * sqrt( (conv1 * (1conv1) / x + (conv2 * (1conv2) /x)

Key metrics
 for evaluating the experiment
 what should change, how to measure it)

Expected effect
 sample size for experiment

An action plan
 depending on the results of the experiment (if X, then we do Y, if Z, then we do Q)

Remebmer
 avoid obvious hypotheses
 facts are not hypotheses, they are current tasks

think about the metric affected by the hypothesis
 The recipe for a good METRIC
 Some notes about metrics for PO
 think about strategy of collection and evaluation of results

Action

Do

What do we do to test the hypothesis?
 think about time

Start the experiment
 measure the results

Data

Check
 what metric will change affect

data collection
 Analyzed changes

Insigth

Act
 % improvement

...and

Belief in effect
 0... 10%

Difficulty in realization
 1..5

It will be interesting
 https://www.iidf.ru/media/articles/lifehacks/haditsikly5layfkhakov/
 http://startupkitchen.blog/carrotquestcasehadicyclesandhowtoincreaseprojectdevelopmentwiththeirhelp/
 HOW WE USE PDCA
 Some notes about AB testing
 Статистика: Задача Метод. A/B Testing. Statistical tests (Ru, En)
 ML продукты. Разбираем основы
 Some notes about Exploratory Data Analysis (Python and Pandas)

New Agileproject. Let's Start!
 https://www.xmind.net/m/k756Qn/

tools

tools
 CLT for means
 Normal Table  z Table  Standard Normal Table  Normal Distribution Table
 Distribution Calculator
 Sample Size Calculator (Evan’s Awesome A/B Tools)

jypyter notebook
 Jupyter Notebook для начинающих: учебник  Еще один блог веб разработчика
 Values of the tdistribution (twotailed)
 Understanding and Interpreting Correlations  an Interactive Visualization
 Diagnostics for simple linear regression