Project

Namespaces

Public Vars

taoensso.touchstone

Simple, Carmine-backed Multi-Armed Bandit (MAB) split-testing. Both more
effective and more convenient than traditional A/B testing. Fire-and-forget!

Redis keys:
  * touchstone:<test-id>:nprospects -> hash, {form-id count}
  * touchstone:<test-id>:scores     -> hash, {form-id score}
  * touchstone:<test-id>:<ts-id>:selection  -> ttl string, form-id
  * touchstone:<test-id>:<ts-id>:committed? -> ttl flag

Ref. http://goo.gl/XPlP6 (UCB1 MAB algo)
     http://en.wikipedia.org/wiki/Multi-armed_bandit
     http://stevehanov.ca/blog/index.php?id=132

*ts-id*

dynamic

Test subject id
view source

ab-select

macro

(ab-select config ts-id test-id & id-form-pairs)
Like `mab-select` but uses simple, A/B-style random selection. Unless you
know all the implications, you probably want `mab-select` instead.
view source

commit!

(commit! {:keys [conn ttl-ms non-uniques?], :or {ttl-ms (* 1000 60 60 2)}, :as config} ts-id test-id value)(commit! config ts-id test-id value & id-value-pairs)
Signals the occurrence of one or more events, each of which may contribute
a specified value (-1 <= value <= 1) to a test score:

    ;; On sign-up button click:
    (commit! {} *ts-id* :my-app.landing.buttons/sign-up 1
                        :my-app.landing/title           0.5)

    ;; On buy button click:
    (commit! {} *ts-id* :my-app/sale-price (if (>= order-item-qty 2) 1 0.8))

There's great flexibility in this to model all kinds of single or multivariate
test->event interactions. Any event can contribute to the score of any test,
positively or negatively, to any extent.

The statistics can get complicated so try keep things simple: resist the urge
to get fancy with the spices.
view source

delete-test

(delete-test {:keys [conn]} test-id)
view source
mab-select
macro
(mab-select config ts-id test-id & id-form-pairs)
Defines a test that selects and evaluates one of the testing forms using the
"UCB1" selection algorithm:
   (mab-select {} *ts-id* :my-test-1
               :my-form-1 "String 1"
               :my-form-2 (do (Thread/sleep 2000) "String 2"))

 Dependent tests can be created through composition:
   (mab-select {} *ts-id* :my-test-1
               :my-form-1 "String 1"
               :my-form-2 (mab-select {} *ts-id* :my-test-1a ...))

 Test forms can be freely added, reordered, or removed for an ongoing test at
 any time, but avoid changing a particular form once id'd.
view source
mab-select-id
macro
(mab-select-id config ts-id test-id & ids)
Like `mab-select` but takes only form ids and uses each id also as its form.

view source
mab-select-ordered
macro
(mab-select-ordered config ts-id test-id & ordered-forms)
Like `mab-select` but takes un-id'd forms and automatically ids them by their
order: :form-1, :form-2, ....
view source
mab-select-permutations
macro
(mab-select-permutations config ts-id test-id take-n & ordered-forms)
Advanced. Defines a positional test with N!/(N-n)! testing forms. Each
testing form will be a vector permutation of the given `ordered-forms`,
automatically id'd for the order of its constituent forms.

Useful for testing the order of the first n forms out of N. The remaining
forms will retain their natural order.
view source
move-test
(move-test {:keys [conn]} old-id new-id)
view source
pr-results
(pr-results {:keys [conn], :as config} test-id)(pr-results config test-id & more)
view source
select-form!
macro
(select-form! config strategy-fn ts-id test-id id-form-pairs)
Implementation detail.

view source
selected-form-id
(selected-form-id {:keys [conn]} ts-id test-id)
Returns subject's currently selected form id for test, or nil. One common
idiom is to use this for creating dependent tests:
    (case (selected-form-id {} *ts-id* :my-test-1)
      :my-form-1 (mab-select {} *ts-id* :my-test-1a ...)
      :my-form-2 (mab-select {} *ts-id* :my-test-1b ...)
      nil)
view source
with-test-subject
macro
(with-test-subject id & body)
Executes body within the context of thread-local test-subject-id binding.

view source