Project

Namespaces

Public Vars

taoensso.carmine.tundra

Semi-automatic archival datastore layer for Carmine.
Use multiple Redis instances (recommended) or Redis databases for local key
namespacing.

Redis keys:
  * carmine:tundra:evictable - set, ks for which `ensure-ks` fail should throw

<urlsafe-str

(<urlsafe-str s)
>urlsafe-str
(>urlsafe-str s)
catcht
macro
(catcht & body)
dirty
(dirty tstore & ks)
Queues given keys for freezing to datastore. Throws an exception if any keys
don't exist. Acts as a Redis command: call within a `wcar` context.

If TundraStore has a :redis-ttl-ms option, **MARKS GIVEN KEYS FOR EXPIRY**!!
** Worker MUST be running AND FUNCTIONING CORRECTLY or DATA WILL BE LOST! **
ensure-ks
(ensure-ks tstore & ks)
BLOCKS to ensure given keys are available in Redis, fetching them from
datastore as necessary. Throws an exception if any previously evicted keys
couldn't be made available. Acts as a Redis command: call within a `wcar`
context.
extend-exists-missing-ks
(extend-exists-missing-ks ttl-ms ks & [only-evictable?])
fetch-keys-delayed
Used to prevent multiple threads from rushing the datastore to get the same
keys, unnecessarily duplicating work.
IDataStore
protocol
Extension point for additional datastores.

members
fetch-keys
(fetch-keys dstore ks)
(fetch-keys dstore ["key" ...]) => [<#{<frozen-val> <ex>}> ...]

put-key
(put-key dstore k v)
(put-key dstore "key" <frozen-val>) => <#{true <ex>}>

IFreezer
protocol
Extension point for compressors, encryptors, etc.

members
freeze
(freeze freezer x)
Returns datastore-ready key val.
(comp put-key freeze): dump-ba -> datastore val.
thaw
(thaw freezer x)
Returns Redis-ready key val.
(comp thaw fetch-key): datastore val -> dump-ba.
ITundraStore
protocol
members
dirty*
(dirty* tstore ks)
ensure-ks*
(ensure-ks* tstore ks)
worker
(worker tstore conn-opts wopts)
Alpha - subject to change.
Returns a threaded message queue worker to routinely freeze Redis keys
marked as dirty to datastore and mark successfully frozen keys as clean.
Logs any errors. THESE ERRORS ARE **IMPORTANT**: an email or other
appropriate notification mechanism is HIGHLY RECOMMENDED. If a worker shuts
down and your keys are set to expire YOU WILL IRREVOCABLY **LOSE DATA**.

Options:
  :nattempts        - Number of times worker will attempt to freeze a key to
                      datastore before failing permanently. >=1.
  :retry-backoff-ms - Amount of time (msecs) to backoff before retrying a
                      failed key freeze. >=0. Can be a (fn [attempt]) -> ms.

  :montior, :eoq-backoff-ms, :nthreads, :throttle-ms, :auto-start
  - Standard `taoensso.carmine.message-queue/worker` opts.
nippy-freezer
Default Nippy Freezer.

tundra-store
(tundra-store datastore & [{:keys [tqname freezer redis-ttl-ms], :or {tqname :default, freezer nippy-freezer}}])
Alpha - subject to change.
Returns a TundraStore with options:
  datastore     - Storage for frozen key data. Default datastores:
                  `taoensso.carmine.tundra.faraday/faraday-datastore`
                  `taoensso.carmine.tundra.s3/s3-datastore`.
  :tqname       - Optional. Worker message queue name.
  :freezer      - Optional. Preps key data to/from datastore. May provide
                  services like compression and encryption, etc. Defaults to
                  Nippy with default options (Snappy compression and no
                  encryption).
  :redis-ttl-ms - Optional! Time after which frozen, inactive keys will be
                  EVICTED FROM REDIS (**DELETED!**). Minimum 10 hours. ONLY
                  use this if you have CONFIRMED that your worker is
                  successfully freezing the necessary keys to your datastore.
                  Otherwise YOU WILL IRREVOCABLY **LOSE DATA**.

See `ensure-ks`, `dirty`, `worker` for TundraStore API.