Backup and Recovery of the Elasticsearch Database¶

Elasticsearch Documentation¶

You will find a more detailed description of how to back up and restore the database in the original Elasticsearch documentation: Restore a Snapshot.

The following chapter describes how to back up and restore specific indices in the Elasticsearch database. In addition, we recommend you also back up the Elasticsearch configuration directory on a daily basis at the file system level: Back up Configuration Files.

Backup and snapshot are used interchangeably here.

Outlining the Relationships¶

In the elasticsearch.yml configuration file, set the repository path:
```
path.repo
```
Register a snapshot repository. This is required for defining a snapshot lifecycle policy.
Define a snapshot lifecycle policy.
Create a snapshot.

The snapshots are created according to the schedule of the snapshot lifecycle policy and stored in the linked snapshot repository.
Restore a snapshot, if necessary, via the Kibana user interface.

Snapshot Repositories¶

For storing backups, you can specify one or more repositories in Elasticsearch. These are used for defining snapshot policies.

Literature

For details on the defining of repositories, refer to Register a Snapshot Repository.

Repository Path¶

In the elasticsearch.yml configuration file, you have to define the allowed locations for repositories in advance.

Example - allowed locations

Elasticsearch is allowed to store repositories in the /mount/backups and /mount/long_term_backups directories. You can specify the 2 directories

in separate lines:

path.repo:
    - /mount/backups
    - /mount/long_term_backups

in one single line:

path.repo: ["/mount/backups","/mount/long_term_backups"]

Caution - existing directories with writing access

These directories must already be existing and Elasticsearch must be granted writing access.

Under Linux you can grant wrinting access with the following command:

sudo chown -R elasticsearch:elasticsearch /mount/backups

Registering a Snapshot Repository¶

You can register snapshot repositories

interactively via the Kibana User Interface:

http://localhost:5601/app/management/data/snapshot_restore/repositories

with the appropriate command via the Dev Tools Console in the Kibana User Interface:
```
http://localhost:5601/app/dev_tools#/console
```

Example - registering a repository

The my_repository repository ist to be registered with the path /mount/backups/my_repository_location:

PUT _snapshot/my_repository
{
    "type": "fs",
    "settings": {
    "location": "/mount/backups/my_repository_location"
    }
}

Use the my_repository name of the snapshot repository in the snapshot policies below.

Caution - database failure

In case of a complete failure of the database contents, also the information about the registered repositories is lost. You have to reregister the snapshot directory with the above command first, before you can restore backups.

Verifying a Repository¶

We recommend verifying the connection to a newly registered snapshot repository. You can co this

in the Kibana user interface by clicking the Verify repository button:

http://localhost:5601/app/management/data/snapshot_restore/repositories/my_repository

in the Dev Tools Console with the following command:
```
POST /_snapshot/my_repository/_verify
```

Creating Backups with Snapshot Policies¶

You need snapshot repositories to define a storage location, before you define snapshot policies. You can define any snapshot policy to back up specific data either together or separately. You can create snapshot policies

in the Kibana user interface:

http://localhost:5601/app/management/data/snapshot_restore/policies

in the Dev Tools Console:

http://localhost:5601/app/dev_tools#/console

In Elasticsearch the snapshots created based on a certain policy are incremental.

In the following subchapters, you will find descriptions of two snapshot policies, which you can use to back up the following features:

security, which includes users, roles and role mappings
kibana, which can include different data, e. g. saved searches and dashboards

... for Security Settings¶

In the Kibana Dev Tools Console, you can define a snapshot lifecycle policy to back up users, roles and role mappings.

Example of a backup definition for security settings

Indices belonging to the security feature are to be backed up every night at 1:00. The created snapshots are to be stored in the already registered my_repository snapshot repository. Independent of the allowed maximum age of 30 days, a minimum of 5 and a maximum of 50 snapshots are stored.

PUT _slm/policy/nightly-security-snapshots
{
  "schedule": "0 0 1 * * ?",
  "name": "<nightly-security-snap-{now/d}>",
  "repository": "my_repository",
  "config": {
    "indices": "-.*",
    "include_global_state": false,
    "feature_states": [
      "security"
    ]
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 5,
    "max_count": 50
  }
}

... for Kibana Settings¶

In the Kibana Dev Tools Console, you can define a snapshot lifecycle policy to back up Kibana settings, e. g. searches and dashboards.

Example of a backup definition for Kibana settings

Indices belonging to the kibana feature are to be backed up every night at 0:30. The created snapshots are to be stored in the my_repository snapshot repository. Independent of the allowed maximum age of 30 days, a minimum of 5 and a maximum of 50 snapshots are stored.

PUT _slm/policy/nightly-kibana-snapshots
{
  "schedule": "0 30 0 * * ?",
  "name": "<nightly-kibana-snap-{now/d}>",
  "repository": "my_repository",
  "config": {
    "indices": "-.*",
    "include_global_state": false,
    "feature_states": [
      "kibana"
    ]
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 5,
    "max_count": 50
  }
}

... for SEAL Log, Accounting and Audit Data¶

Standard log messages of a system and accounting and audit data need to be stored for different periods of time.

Below you will find several examples for snapshot lifecycle policies. You can copy them to use them as templates and modify them according to your needs.

Example 1 - hourly backups

This snapshot lifecycle policy takes up to 24 hourly backups, which are stored for one day:

PUT _slm/policy/hourly-audit-snapshots
{
  "name": "<hourly-audit-snap-{now/d}>",
  "schedule": "0 0 * * * ?",
  "repository": "my_repository",
  "config": {
    "indices": "seal-*-audit",
    "include_global_state": false
  },
  "retention": {
    "expire_after": "1d",
    "min_count": 1,
    "max_count": 24
  }
}

Example 2 - daily backups

This snapshot lifecycle policy takes a backup every night at 23:45, which is stored for one month:

PUT _slm/policy/daily-audit-snapshots
{
  "name": "<daily-audit-snap-{now/d}>",
  "schedule": "0 45 23 * * ?",          
  "repository": "my_repository",
  "config": {
    "indices": "seal-*-audit",
    "include_global_state": false
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 1,
    "max_count": 31
  }
}

Example 3 - monthly backups

This snapshot lifecycle policy takes a backup at the first day of every month, which are stored for one year:

PUT _slm/policy/monthly-audit-snapshots
{
  "name": "<monthly-audit-snap-{now/d}>",
  "schedule": "0 56 23 1 * ?",            
  "repository": "my_repository",
  "config": {
    "indices": "seal-*-audit",
    "include_global_state": false
  },
  "retention": {
    "expire_after": "366d",
    "min_count": 1,
    "max_count": 12
  }
}

Hint- long term data storage

We recommend setting up several coordinated snapshot lifecycle policies for data that have to be stored for a longer period and must not get lost.

In the examples above all snapshots are stored in the my_repository directory. You can differentiate between the snapshots of the different Elasticsearch indices by their names.

If, additionally, you wish to backup the snapshot directory itself by means of your operating system, we recommend you use different snapshot repositories.

Literature

For details on backing up snapshot repositories, refer to Backup a repository.

... for Elasticsearch Cluster Configurations¶

Literature

cluster states, refer to Dedicated cluster state snapshots.
configuration files, refer to Back up configuration files.

Restoring Backups¶

Caution - version compatibility

Different Elasticsearch database versions may be incompatible to its previous or following version, refer to Snapshot compatibility.

Register the snapshot repository:

Registering a Snapshot Repository
Select the required snapshot
- in the Kibana user interface:
```
http://localhost:5621/app/management/data/snapshot_restore/snapshots
```
- in the Dev Tools Console:
  Example - listing snapshots
  
  The following command lists the latest 10 snapshots in descending order:
```
GET /_snapshot/my_repository/*?size=10&sort=start_time&order=desc
```

Restoring Security Settings¶

Caution - restoring security settings

If you restore data in an empty Elasticsearch database, you must restore the security settings first.

We strongly recommend you use the security feature to restore your snapshot, as using the .security alias instead may be tricky. If you still want use the .security alias to restore the securtiy settings, consider the following:

The .security alias can only point to one single index. This index depends on the Elasticsearch version you use, e. g. .security-7 for version 7.x, and must not be changed. Restoring the security settings with an index name different from the current index name creates a second alias with the same name. As the software cannot handle this, it locks you out of your Kibana user interface. If this happens, refer to Troubleshooting.

The instruction below describe the restoration of the security feature for a single server Elasticsearch installation.

Literature

For details on restoring data in an Elasticsearch cluster, refer to Restore an entire cluster.

Register the repository, in which the snapshots have been saved:

PUT _snapshot/<my_repository>
{
    "type": "fs",
    "settings": {
    "location": "/mount/backups/<my_repository>_location"
    }
}

Search the latest security snapshot:
```
GET /_snapshot/<my_repository>/*security*?size=1&sort=start_time&order=desc
```
Literature

For details on retrieving information about one or more snapshots, refer to Get snapshot API.

In the response of the command, you will find the name of the snapshot in the snapshot key.
Example of a search response

The search for the lates snapshot provides nightly-security-snap-2022.03.16-ata_2ng7qfifayn_keydna:
```
{
  "snapshots" : [
    {
    "snapshot" : "nightly-security-snap-2022.03.16-ata_2ng7qfifayn_keydna",
    "uuid" : "tO0Btmb3SaOCUdREwJAzSA",
    "repository" : "my_repository",
    ...
    "indices" : [
        ".security-7"
    ],
    ...
    }
  ],
  ...
}
```
Restore the selected security snapshot:
Example of the restoring command

The snapshot provided by the previous search is to be restored:
```
POST /_snapshot/my_repository/nightly-security-snap-2022.03.16-ata_2ng7qfifayn_keydna/_restore
{
  "feature_states": [ "security" ]
}
```
A proper restoration will provide the following result:
```
{
  "accepted" : true
}
```

Restoring Other Snapshots¶

Depending on the objective of restoring the database, you have to do consider different things.

Restoring data exaclty with their old indices, e. g. after a complete loss of data:
- We recommend you do not rename any indices.
- Make sure that no new data arrive during the restoration. You can do this, for example by deactivating the fire wall rules, refer to used ports, or by stopping all Filebeat processes.
Restoring older data to mix them with current data:
- We recommend you rename the indices to be restored.
- You do not necessarily have to close the input channels of the Elasticsearch database.

Literature

For details on restoring any snapshots other than security feature, refer to Restore an index or data stream.

Restoring the Elasticsearch Cluster Configuration¶

Literature

For details on restoring an Elasticsearch cluster, refer to Restore an entire cluster.

Troubleshooting¶

Log on Problems to Kibana After Restoring Security Settings¶

After restoring the security settings you cannot log on to the Kibana user interface.

This problem can be solved by disabling security, restarting the Elasticsearch database and manually removing the incorrectly restored index. If this is done successfully, `xpack.security.enabled` can be switched on again and the access to the following restored data is still protected.

In an editor, open the elasticsearch.yml file:

Linux:
```
/etc/elasticsearch/elasticsearch.yml
```

Windows:

C:\ProgramData\Elastic\Elasticsearch\config\elasticsearch.yml

Deactivate the security settings and save the file:
```
xpack.security.enabled: false
```

Restart Elasticsearch:

Linux:
```
sudo systemctl restart elasticsearch
```
Windows:
```
restart-service elasticsearch
```

In a Browser, open the Kibana Dev Tools Console:
```
http://localhost:5621/app/dev_tools#/console
```
In the Kibana Dev Tools Console, search for all indices with the alias .security:
```
GET _cat/aliases/.security
```
Example of a search result
```
.security .security-7 - - - -
```
Delete all indices found with the previous command:
Example of the deleting command
```
DELETE /.security-7
```
Restore the security feature:

Restoring Security Settings
In the elasticsearch.yml file, reactivate the security and save the file:
```
xpack.security.enabled: true
```

Restart Elasticsearch:

Linux:

sudo systemctl restart elasticsearch

Windows:

restart-service elasticsearch