Upgrade

Upgrade DDN Infinia

This section applies to realm admins only.

If you need to upgrade the firmware for a specific component included in a server, refer to the hardware documentation for your platform.

Key Points

The following are key points about upgrading DDN Infinia:

  • During upgrade, you cannot configure the realm or cluster.
  • During upgrade, you cannot add or remove nodes from the cluster.
  • You can upgrade a stopped cluster.
  • You can stop a cluster if upgrade fails because the cluster is unhealthy.
  • The version activation can be executed only for a running cluster.

Upgrade Methods

DDN Infinia supports both online and offline upgrade methods:

  • Online upgrade — The cluster remains operational throughout the process.
  • Offline upgrade — The cluster is taken offline for the duration of the upgrade.

Online Upgrade

To upgrade with DDN Infinia online, complete the following steps on any node in a cluster:

Procedure

1

Update the cached release metadata from remote server and list the available releases:

$redcli realm upgrade list -u
2

Download the required release bits from remote server to local container image registry:

$redcli realm upgrade download <release_version>

If successful, the command returns a task UUID. Use this task UUID to monitor the progress.

$redcli task show <task_uuid>
Example

This example shows the progress of download task (with a given UUID) monitored till successful.

16:36PM INF Realm upgrade-download task [c75e762e-3422-4dc0-a6ff-e9ed1cc6628e] has been started
2$ redcli task show c75e762e-3422-4dc0-a6ff-e9ed1cc6628e
3
4┌──────────────────────────────────────┬───────────────┬──────────┬──────────┬──────────┬──────────────┬───────┐
5│ UUID                                 │ NAME          │ TYPE     │ DURATION │ STATE    │ MESSAGE      │ ERROR │
6│                                      │               │          │ (sec)    │          │              │       │
7├──────────────────────────────────────┼───────────────┼──────────┼──────────┼──────────┼──────────────┼───────┤
8│ c75e762e-3422-4dc0-a6ff-e9ed1cc6628e │ Realm upgrade │ rupgrade │ 5        │ complete │ Close task   │       │
9│                                      │ task: download│          │          │          │ as completed │       │
10└──────────────────────────────────────┴───────────────┴──────────┴──────────┴──────────┴──────────────┴───────┘
11
12┌───────┬─────────────────────────────┬───────────────────────────────┬──────────┬─────────┬───────┐
13│ STAGE │ NAME                        │ TRANSITION TIME               │ STATE    │ MESSAGE │ ERROR │
14├───────┼─────────────────────────────┼───────────────────────────────┼──────────┼─────────┼───────┤
15│ 0     │ STAGE_INIT                  │ 2023-07-25 18:36:36 +0000 UTC │ complete │         │       │
16├───────┼─────────────────────────────┼───────────────────────────────┼──────────┼─────────┼───────┤
17│ 1     │ STAGE_UPDATE_LOCAL_REGISTRY │ 2023-07-25 18:36:41 +0000 UTC │ complete │         │       │
18├───────┼─────────────────────────────┼───────────────────────────────┼──────────┼─────────┼───────┤
19│ 3     │ STAGE_DONE                  │ 2023-07-25 18:36:41 +0000 UTC │ complete │         │       │
20└───────┴─────────────────────────────┴───────────────────────────────┴──────────┴─────────┴───────┘
3

Apply the required release:

$redcli realm upgrade apply <release_version>

If successful, the command returns a task UUID. Use this task UUID to monitor the progress.

$redcli task show <task_uuid>
Example

This example shows the progress of the upgrade task (with a given UUID) monitored till successful.

14:58PM INF Realm upgrade-apply task [97b1becf-236e-469f-81b6-b7967a6e78a3] has been started
2$ redcli task show 97b1becf-236e-469f-81b6-b7967a6e78a3
3
4┌──────────────────────────────────────┬───────────────┬──────────┬──────────┬──────────┬──────────────┬───────┐
5│ UUID                                 │ NAME          │ TYPE     │ DURATION │ STATE    │ MESSAGE      │ ERROR │
6│                                      │               │          │ (sec)    │          │              │       │
7├──────────────────────────────────────┼───────────────┼──────────┼──────────┼──────────┼──────────────┼───────┤
8│ 97b1becf-236e-469f-81b6-b7967a6e78a3 │ Realm upgrade │ rupgrade │ 80       │ complete │ Close task   │       │
9│                                      │ task: apply   │          │          │          │ as completed │       │
10└──────────────────────────────────────┴───────────────┴──────────┴──────────┴──────────┴──────────────┴───────┘
11
12┌───────┬────────────────────────────────┬───────────────────────────────┬──────────┬─────────┬───────┐
13│ STAGE │ NAME                           │ TRANSITION TIME               │ STATE    │ MESSAGE │ ERROR │
14├───────┼────────────────────────────────┼───────────────────────────────┼──────────┼─────────┼───────┤
15│ 0     │ STAGE_INIT                     │ 2023-07-28 16:58:47 +0000 UTC │ complete │         │       │
16├───────┼────────────────────────────────┼───────────────────────────────┼──────────┼─────────┼───────┤
17│ 1     │ STAGE_DISTRIBUTE_IMAGES        │ 2023-07-28 16:58:52 +0000 UTC │ complete │         │       │
18│       │ _ACROSS_NODES                  │                               │          │         │       │
19├───────┼────────────────────────────────┼───────────────────────────────┼──────────┼─────────┼───────┤
20│ 2     │ STAGE_UPGRADE_SERVICES         │ 2023-07-28 16:58:52 +0000 UTC │ complete │         │       │
21├───────┼────────────────────────────────┼───────────────────────────────┼──────────┼─────────┼───────┤
22│ 3     │ STAGE_UPDATE_REALM_CONFIG      │ 2023-07-28 17:00:07 +0000 UTC │ complete │         │       │
23├───────┼────────────────────────────────┼───────────────────────────────┼──────────┼─────────┼───────┤
24│ 4     │ STAGE_DONE                     │ 2023-07-28 17:00:07 +0000 UTC │ complete │         │       │
25└───────┴────────────────────────────────┴───────────────────────────────┴──────────┴─────────┴───────┘
4

Activate the release:

$redcli realm upgrade activate-version <release_version>

Note that the realm upgrade list command with -a flag shows all available releases on remote server.

Example

This command output shows the compatible and incompatible releases with version details.

1$ redcli realm upgrade list -a
23:48PM INF Realm upgrade list releases
3
4┌────────────────────────────────────────────────────────────────────┐
5│                       REALM UPGRADE RELEASES                       │
6├──────────┬───────────────┬──────────────────────┬──────────────────┤
7│ CHANNEL  │ VERSION       │ DATE                 │ STATE            │
8├──────────┼───────────────┼──────────────────────┼──────────────────┤
9│ unstable │ 1.3.1-beta.1  │ 2024-06-06T17:32:28Z │ installed        │
10├──────────┼───────────────┼──────────────────────┼──────────────────┤
11│ unstable │ 1.3.2-beta.1  │ 2024-06-06T17:32:28Z │ downloaded       │
12├──────────┼───────────────┼──────────────────────┼──────────────────┤
13│ unstable │ 1.3.3-beta.1  │ 2024-06-06T17:32:28Z │ available        │
14├──────────┼───────────────┼──────────────────────┼──────────────────┤
15│ unstable │ 1.2.5-beta.1  │ 2024-06-06T17:32:28Z │ incompatible     │
16└──────────┴───────────────┴──────────────────────┴──────────────────┘

Offline Upgrade

DDN Infinia can be upgraded using an offline process where the cluster is stopped first, upgraded, and then restarted. If you want to do an online upgrade instead, see Online Upgrade.

To do an offline upgrade, complete the following steps on any node in a cluster:

Procedure

1

List the most recent DDN Infinia versions available for download:

$redcli realm upgrade list -u
2

Download the required release bits from the remote server to local container image registry:

$redcli realm upgrade download <release_version>

The command returns a task UUID.

3

Use the task UUID from the prior step to monitor the progress:

$redcli task show <task_uuid>

Once all stages have a STATE of complete, the release bits are fully downloaded.

Example

This example shows the progress of download task (with a given UUID) monitored till successful.

1$ redcli realm upgrade download 1.1.0
26:36PM INF Realm upgrade-download task [c75e762e-3422-4dc0-a6ff-e9ed1cc6628e] has been started
3
4$ redcli task show c75e762e-3422-4dc0-a6ff-e9ed1cc6628e
5┌────────────────────────────────────┬──────────────┬─────────┬──────────┬──────────┬─────────────┬─────────┐
6│                UUID                │     NAME     │ TYPE    │ DURATION │ STATE    │ MESSAGE     │ ERROR   │
7│                                    │              │         │ (sec)    │          │             │         │
8├────────────────────────────────────┼──────────────┼─────────┼──────────┼──────────┼─────────────┼─────────┤
9│c75e762e-3422-4dc0-a6ff-e9ed1cc6628e│Realm upgrade │ rupgrade│    5     │ complete │Close task   │         │
10│                                    │task: download│         │          │          │as completed │         │
11└────────────────────────────────────┴──────────────┴─────────┴──────────┴──────────┴─────────────┴─────────┘
12
13┌────────┬─────────────────────────────┬───────────────────────────────┬───────────┬─────────────┬──────────┐
14│ STAGE  │    NAME                     │    TRANSITION TIME            │  STATE    │  MESSAGE    │ ERROR    │
15├────────┼─────────────────────────────┼───────────────────────────────┼───────────┼─────────────┼──────────┤
16│   0    │ STAGE_INIT                  │ 2023-07-25 18:36:36 +0000 UTC │ complete  │             │          │
17├────────┼─────────────────────────────┼───────────────────────────────┼───────────┼─────────────┼──────────┤
18│   1    │ STAGE_UPDATE_LOCAL_REGISTRY │ 2023-07-25 18:36:41 +0000 UTC │ complete  │             │          │
19├────────┼─────────────────────────────┼───────────────────────────────┼───────────┼─────────────┼──────────┤
20│   3    │ STAGE_DONE                  │ 2023-07-25 18:36:41 +0000 UTC │ complete  │             │          │
21└────────┴─────────────────────────────┴───────────────────────────────┴───────────┴─────────────┴──────────┘
4

Stop the cluster:

$redcli cluster stop <cluster_name>

Confirm that you want to stop it:

1Stopping cluster infinia-cluster. Confirm? [y/N] y

Example output:

18:58AM INF Stopping reds3 daemon...
28:58AM INF Reds3 daemon stopped.
38:59AM INF Cluster <cluster_name> stopping.
48:59AM INF Cluster <cluster_name> stopped.
58:59AM INF 4 out of 5 instances still running
68:59AM INF All instances stopped.
5

Verify that the cluster is fully stopped:

$redcli cluster list

Confirm that all services show a DOWN state.

6

Apply the upgrade, using the version number you downloaded. Use the -a flag to apply the upgrade across the entire realm:

$redcli realm upgrade apply <release_version> -a

The command returns a task UUID.

7

Use the task UUID from the prior step to monitor the upgrade progress. You can use the -w flag to stream progress until completion:

$redcli task show <task_uuid>  -w
Example

This example shows the progress of the upgrade task (with a given UUID) monitored until successful.

1$ redcli realm upgrade apply 1.1.0
24:58PM INF Realm upgrade-apply task [97b1becf-236e-469f-81b6-b7967a6e78a3] has been started
3
4$ redcli task show 97b1becf-236e-469f-81b6-b7967a6e78a3
5┌────────────────────────────────────┬──────────────┬──────────┬──────────┬──────────┬────────────┬─────────┐
6│                UUID                │     NAME     │ TYPE     │ DURATION │ STATE    │ MESSAGE    │ ERROR   │
7│                                    │              │          │ (sec)    │          │            │         │
8├────────────────────────────────────┼──────────────┼──────────┼──────────┼──────────┼────────────┼─────────┤
9│97b1becf-236e-469f-81b6-b7967a6e78a3│Realm upgrade │ rupgrade │ 80       │ complete │Close task  │         │
10│                                    │task: apply   │          │          │          │as completed│         │
11└────────────────────────────────────┴──────────────┴──────────┴──────────┴──────────┴────────────┴─────────┘
12
13┌─────────┬──────────────────────────────┬───────────────────────────────┬──────────┬───────────┬───────────┐
14│  STAGE  │    NAME                      │    TRANSITION TIME            │ STATE    │  MESSAGE  │  ERROR    │
15├─────────┼──────────────────────────────┼───────────────────────────────┼──────────┼───────────┼───────────┤
16│   0     │ STAGE_INIT                   │ 2023-07-28 16:58:47 +0000 UTC │ complete │           │           │
17├─────────┼──────────────────────────────┼───────────────────────────────┼──────────┼───────────┼───────────┤
18│   1     │ STAGE_DISTRIBUTE_IMAGES      │ 2023-07-28 16:58:52 +0000 UTC │ complete │           │           │
19│         │ _ACROSS_NODES                │                               │          │           │           │
20├─────────┼──────────────────────────────┼───────────────────────────────┼──────────┼───────────┼───────────┤
21│   2     │ STAGE_UPGRADE_SERVICES       │ 2023-07-28 16:58:52 +0000 UTC │ complete │           │           │
22├─────────┼──────────────────────────────┼───────────────────────────────┼──────────┼───────────┼───────────┤
23│   3     │ STAGE_UPDATE_REALM_CONFIG    │ 2023-07-28 17:00:07 +0000 UTC │ complete │           │           │
24├─────────┼──────────────────────────────┼───────────────────────────────┼──────────┼───────────┼───────────┤
25│   4     │ STAGE_DONE                   │ 2023-07-28 17:00:07 +0000 UTC │ complete │           │           │
26└─────────┴──────────────────────────────┴───────────────────────────────┴──────────┴───────────┴───────────┘
8

Start the cluster:

$redcli cluster start <cluster_name>

Example output:

19:06AM INF Cluster <cluster_name> is starting.
29:06AM INF Cluster status: boot-replay
39:06AM INF Cluster status: dlm-init
49:06AM INF Cluster status: service-init
59:06AM INF Cluster status: running
69:06AM INF Starting reds3 daemon
79:06AM INF Reds3 daemon started.
9

Activate the upgrade:

$redcli realm upgrade activate-version <release_version>

Example output:

19:13AM INF Autoselected cluster <cluster_name>.
29:13AM INF Realm upgrade version activated.
10

Validate final cluster status to ensure that all nodes and services are healthy and running:

$redcli cluster show

Example output:

17:53PM INF Auto Selecting cluster: c1
27:53PM INF Set limit to 25.
3┌──────┬─────────┬───────────────┬─────────┬────────────┬───────────────────┬───────────┐
4│ NAME │  STATE  │    LEADER     │ FEATURE │   RUNTIME  │ INSTANCES/EVICTED │ DRIVES/EVI│
5│      │         │               │ VERSION │   CONFIG   │                   │           │
6├──────┼─────────┼───────────────┼─────────┼────────────┼───────────────────┼───────────┤
7│   c1 │ running │ <infinia-001> │  2.2.77 │ <your_cfg> │               4/0 │           │
8└──────┴─────────┴───────────────┴─────────┴────────────┴───────────────────┴───────────┘
11

Verify cluster health:

$redcli cluster show -H

All entries in the TABLES section of the output should show Green in the HEALTH column. For example:

17:56PM INF Auto Selecting cluster: c1
27:56PM INF Set limit to 25.
3┌────────────────────────────────────────────────────────────────────────────────┬──────┐
4│                                     CLUSTER                                    │      │
5├─────────┬───────────────┬─────────┬────────────┬───────┬─────┬─────┬─────┬─────┼──────┤
6│  STATE  │    LEADER     │ FEATURE │   RUNTIME  │ EPOCH │ SCE │ CCE │ RDE │ EDE │ Valid│
7│         │               │ VERSION │   CONFIG   │       │     │     │     │     │      │
8├─────────┼───────────────┼─────────┼────────────┼───────┼─────┼─────┼─────┼─────┼──────┤
9│ running │ <infinia-001> │  2.2.77 │ <your_cfg> │    29 │  29 │  29 │  25 │   0 │     4│
10└─────────┴───────────────┴─────────┴────────────┴───────┴─────┴─────┴─────┴─────┴──────┘
11
12┌───────────────────────────────────────────────────────────────────────────────────────┐
13│                                        DRIVE STATUS                                   │
14├──────────┬────────────────┬──────────────┬─────────┬────────┬──────────┬───────┬──────┤
15│ DEGRADED │ DELETE PENDING │ DELETE READY │ EVICTED │ JOINED │ READABLE │ VALID │ WRITE│
16├──────────┼────────────────┼──────────────┼─────────┼────────┼──────────┼───────┼──────┤
17│        0 │              0 │            0 │       0 │     48 │       48 │    48 │      │
18└──────────┴────────────────┴──────────────┴─────────┴────────┴──────────┴───────┴──────┘
19
20┌───────────────────────────────────────────────────────────────────────────────────────┐
21│                                                                    TABLES             │
22├─────────┬───────────┬──────────┬────────┬────────┬─────────┬────────────────────┬─────┤
23│ POOL ID │ POOL NAME │ TABLE ID │ HEALTH │ DRIVES │ EVICTED │ REBALANCING DRIVES │ REBU│
24├─────────┼───────────┼──────────┼────────┼────────┼─────────┼────────────────────┼─────┤
25│       1 │    SYSTEM │        1 │  Green │     48 │       0 │                  0 │     │
26├─────────┼───────────┼──────────┼────────┼────────┼─────────┼────────────────────┼─────┤
27│       1 │    SYSTEM │        2 │  Green │     48 │       0 │                  0 │     │
28├─────────┼───────────┼──────────┼────────┼────────┼─────────┼────────────────────┼─────┤
29│       1 │    SYSTEM │        3 │  Green │     48 │       0 │                  0 │     │
30└─────────┴───────────┴──────────┴────────┴────────┴─────────┴────────────────────┴─────┘
12

Verify the agent/daemon state:

$redcli realm agent-status -A

The output should show that all agents are in a normal state. For example:

17:56PM INF All all agents are in normal state.