Memory
The memory module is responsible in allocating and managing chunks of off-heap memory. The allocated memory is managed in a hierarchical model. The root
allocator is managing the memory allocation on a node level, and can have one or more job
allocators. A job
allocator is created for each job (i.e., a Siren Federate search request) and is managing the memory allocation on a job level. A job
can have one or more task
allocators. A task
allocator is created for each task of a job and is managing the memory allocation on a task level. Each allocator specifies a limit for how much off-heap memory it can use.
-
siren.memory.root.limit
- Limit in bytes for the root allocator. Defaults to 750MB.
-
siren.memory.job.limit
- Limit in bytes for the job allocator. Defaults to
siren.memory.root.limit
. -
siren.memory.task.limit
- Limit in bytes for the task allocator. Defaults to
siren.memory.job.limit
.
By default, the job limit is equal to the root limit, and the task limit is equal to the job limit. This enables a simplified configuration for most common scenarios where only the root limit has to be configured. For more advanced scenarios, e.g., with multiple concurrent users, one might want to tune the job and task limits to avoid having a user executing a query that will consume all the available off-heap memory on the root level, leaving no memory for the queries executed by other users.
As a rule of thumb, one should never give more than half of the remaining OS memory to the siren root allocator, in order to leave some memory for the OS cache and cater for Netty’s memory management overhead. For example, if Elasticsearch is configured with a 32GB heap on a machine with 64GB of ram, this leaves 32GB to the OS. The maximum limit that one could set for the root allocator should be 16GB.