Splunk® Enterprise

Managing Indexers and Clusters of Indexers

Acrobat logo Download manual as PDF

Acrobat logo Download topic as PDF

How the indexer stores indexes

As the indexer indexes your data, it creates a number of files. These files contain two types of data:

Together, these files constitute the Splunk Enterprise index. The files reside in sets of directories organized by age. Some directories contain newly indexed data; others contain previously indexed data. The number of such directories can grow quite large, depending on how much data you're indexing.

Why you might care

You might not care, actually. The indexer handles indexed data by default in a way that gracefully ages the data through several stages. After a long period of time, typically several years, the indexer removes old data from your system. You might well be fine with the default scheme it uses.

However, if you're indexing large amounts of data, have specific data retention requirements, or otherwise need to carefully plan your aging policy, you've got to read this topic. Also, to back up your data, it helps to know where to find it. So, read on....

How data ages

Each of the index directories is known as a bucket. To summarize so far:

  • An "index" contains compressed raw data and associated index files.
  • An index resides across many age-designated index directories.
  • An index directory is called a bucket.

A bucket moves through several stages as it ages:

  • hot
  • warm
  • cold
  • frozen
  • thawed

As buckets age, they "roll" from one stage to the next. As data is indexed, it goes into a hot bucket. Hot buckets are both searchable and actively being written to. An index can have several hot buckets open at a time.


Once further conditions are met (for example, the index reaches some maximum number of warm buckets), the indexer begins to roll the warm buckets to cold, based on their age. It always selects the oldest warm bucket to roll to cold. Buckets continue to roll to cold as they age in this manner. After a set period of time, cold buckets roll to frozen, at which point they are either archived or deleted. By editing attributes in indexes.conf, you can specify the bucket aging policy, which determines when a bucket moves from one stage to the next.

If the frozen data has been archived, it can later be thawed. Thawed data is available for searches.

Here are the stages that buckets age through:

Bucket stage Description Searchable?
Warm Data rolled from hot. There are many warm buckets. Data is not actively written to warm buckets. Yes
Frozen Data rolled from cold. The indexer deletes frozen data by default, but you can choose to archive it instead. Archived data can later be thawed. No
Thawed Data restored from an archive. If you archive frozen data, you can later return it to the index by thawing it. Yes

The collection of buckets in a particular stage is sometimes referred to as a database or "db": the "hot db", the "warm db", the "cold db", etc.

What the index directories look like

Each index occupies its own directory under $SPLUNK_HOME/var/lib/splunk. The name of the directory is the same as the index name. Under the index directory are a series of subdirectories that categorize the buckets by stage (hot/warm, cold, or thawed).

The buckets themselves are subdirectories within those directories. The bucket directory names are based on the age of the data.


Bucket stage Default location Notes
Hot $SPLUNK_HOME/var/lib/splunk/defaultdb/db/* There can be multiple hot subdirectories, one for each hot bucket. See "Bucket naming conventions".
Warm $SPLUNK_HOME/var/lib/splunk/defaultdb/db/* There are separate subdirectories for each warm bucket. See "Bucket naming conventions".

The paths for the hot/warm, cold, and thawed directories are configurable, so, for example, you can store cold buckets in a separate location from hot/warm buckets. See "Configure index storage" and "Use multiple partitions for index data".

Important: All index locations must be writable.

Note: In pre-6.0 versions of Splunk Enterprise, replicated copies of cluster buckets always resided in the colddb directory, even if they were hot or warm buckets. Starting with 6.0, hot and warm replicated copies reside in the db directory, the same as for non-replicated copies.

Bucket naming conventions

Bucket names depend on:

  • The stage of the bucket: hot or warm/cold/thawed

Important: Bucket naming conventions are subject to change.

Non-clustered buckets


Clustered buckets





For an introduction to indexer cluster architecture and replicated data streaming, read "Basic indexer cluster architecture".

Bucket names

These are the naming conventions:

Bucket type Hot bucket Warm/cold/thawed bucket





Buckets and Splunk Enterprise administration


  • To learn how to back up your data, read "Back up indexed data." That topic also discusses how to manually roll hot buckets to warm, so that you can then back them up.

In addition, see "indexes.conf" in the Admin Manual.

Last modified on 03 October, 2018
Use the monitoring console to view indexing performance
Configure index storage


Was this documentation topic helpful?



0 out of 1000 Characters