LMDB Backend#
LMDB Backend utilizes python lmdb package wrapper for LMDB data store.
This backend provides following characteristics:
storage of all event structures
querying by all query data parameters
optimization for latest events querying
optimization for time series querying
fragmentation of time series entries based on event types
in memory caching of event registration with periodic disk persistence
configurable condition rules for filtering event by their content
Storage models#
Storage of events in LMDB Backend is based on models:
key-value storage
time series storage
event id based reference storage
Key-value storage#
Key-value storage is used as storage model for persisting latest events for each event type. Configuration provides parameter defining which events, based on their event type, are stored in key-value store. During backend initialization, content of whole key-value storage is loaded in memory and continuously updated with new event registrations. This enables quick access to event data without need for disk read operations during query executions.
Time series storage#
Time series storage provide storage model where all events are orderly stored based on their event timestamp or event source timestamp. Configuration enables definition of multiple sets defined by event types and time source. For each configured set, single series storage is created. This fragmentation of stored events into independent time series enables fine tuning and optimizations based on expected query types. For general cases, usage of single set of configured event types, which includes all event that should be persisted, is suggested. Events without source timestamp can’t be stored in storage configured for source timestamp storage. Querying of events is based on sequential checking of query conditions which occurred in time segments constrained by query data.
event id based reference storage#
In addition to key-value and time series event storage, additional event references are stored in dedicated event id based reference storage. These references are kept only for events available in key-value or time series storage. Once event is removed (e.g. due to configured limits) and not accessible in other storages, its reference is also removed from reference storage. This storage enables quick and sequential access to events based on their event id.
Event filtering#
During registration and event querying, this backend discards all event occurrences which do not satisfy configured conditions. Conditions are defined and applied based on event type. These conditions provide plausibility of condition composition (by operators all and any) and testing based on event content.
Query planning#
During event querying, simple query planning is done based on following steps:
if query data filters only latest events for each queried event type, then only key-value storage is searched for possible event instances
for all other queries, time series partition is searched, in order as they are defined in configuration, for first series that contains at least one possible event type constrained by query data - only that time series storage partition is used as source of event instances returned by query
if no time series storage could be found, empty list is returned as query result
In case of querying event sequences ordered by event id and based on starting event id, event id based reference storage is used (backends query_flushed method).
Warning
Query which utilizes time series search will retrieve only events from single partition. Thus, time series partitioning should be used in accordance with expected queries as additional optimization technique. It is strongly advised to configure only single timestamp and single source timestamp partition as initial configuration.
Disk persisting#
Registration of new events doesn’t initiate immediate disk writes. All changes
are cached in memory and periodically written to disk as part of single
transaction which includes all storages. This period is defined by
configuration (flush_period). Writing of memory cache is also part of
backend’s standard closing procedure. Single transaction responsible for
writing all memory caches to disk also include cleanup operation which enables
deletion of oldest entries in time series storages. Each time series storage
can have its own configuration defined limit which specifies event persistent
period based on event timestamp or event source timestamp, number of events in
partition or partitions approximated disk usage. event id based reference
storage is updated only during this periodic write transaction. Because of this,
queries utilizing event id reference storage will return only events that are
persisted on disk.
Limiting time series size#
Each time series storage can optionally limit number of stored events. This functionality is associated with disk persisting procedure and is part of same database transaction. Supported limits include:
min_entries
Minimum number of events preserved in database despite of other limits. This property can be used as “low water mark” - number of entries always available.
max_entries
Maximum number of event preserver in database. This property can be used as “high water mark” - number of entries will never exceed this number.
duration
Time in seconds representing maximum duration between current time and time used as time series key. All entries which exceed this duration will be removed.
size
Size in bytes allocated for associated time series. Once time series exceeds this storage size, some of the oldest entries are removed. Number of removed entries is calculated based on average entry size and limiting storage size.
Configuration#
$schema: "https://json-schema.org/draft/2020-12/schema"
$id: "hat-event://backends/lmdb.yaml"
title: LMDB backend
type: object
required:
- db_path
- identifier
- flush_period
- cleanup_period
- conditions
- latest
- timeseries
properties:
db_path:
type: string
identifier:
type:
- "null"
- string
flush_period:
type: number
cleanup_period:
type: number
conditions:
type: array
items:
type: object
required:
- subscriptions
- condition
properties:
subscriptions:
$ref: "hat-event://backends/lmdb.yaml#/$defs/event_types"
condition:
$ref: "hat-event://backends/lmdb.yaml#/$defs/condition"
latest:
type: object
required:
- subscriptions
properties:
subscriptions:
$ref: "hat-event://backends/lmdb.yaml#/$defs/event_types"
timeseries:
type: array
items:
type: object
required:
- order_by
- subscriptions
properties:
order_by:
enum:
- TIMESTAMP
- SOURCE_TIMESTAMP
subscriptions:
$ref: "hat-event://backends/lmdb.yaml#/$defs/event_types"
limit:
$ref: "hat-event://backends/lmdb.yaml#/$defs/limit"
$defs:
event_types:
type: array
items:
type: array
items:
type: string
limit:
type: object
properties:
min_entries:
type: number
description: |
number of entries kept despite of other limits
max_entries:
type: number
description: |
maximum number of entries
duration:
type: number
description: |
limit for the persisted history based on keys
expressed as duration in seconds
size:
type: number
description: |
memory consumption size in bytes that triggers
additional cleanup based on average entry size
condition:
oneOf:
- $ref: "hat-event://backends/lmdb.yaml#/$defs/conditions/all"
- $ref: "hat-event://backends/lmdb.yaml#/$defs/conditions/any"
- $ref: "hat-event://backends/lmdb.yaml#/$defs/conditions/json"
conditions:
all:
type: object
required:
- type
- conditions
properties:
type:
const: all
conditions:
type: array
items:
$ref: "hat-event://backends/lmdb.yaml#/$defs/condition"
any:
type: object
required:
- type
- conditions
properties:
type:
const: any
conditions:
type: array
items:
$ref: "hat-event://backends/lmdb.yaml#/$defs/condition"
json:
type: object
required:
- type
properties:
type:
const: json
data_path:
$ref: "hat-event://backends/lmdb.yaml#/$defs/path"
data_type:
enum:
- "null"
- boolean
- string
- number
- array
- object
data_value: {}
path:
oneOf:
- type: string
- type: integer
- type: array
items:
$ref: "hat-event://backends/lmdb.yaml#/$defs/path"