Upgrading Reference

This guide explains how to upgrade components of a Teleport cluster in non-standard situations.

To ensure that your Teleport cluster remains up to date with the lowest amount of manual overhead, we recommend signing up for a cloud-hosted Teleport Enterprise account and following Enroll Agents in Automatic Upgrades.

If your infrastructure does not support automatic agent updates, follow this guide to determine the best approach for keeping your Teleport cluster up to date.

Before reading this guide, become familiar with the Upgrading Compatibility Overview guide, which describes the sequence in which to upgrade components in your cluster.

Working with the automatic updater

This section explains how to manage the automatic updater.

On Kubernetes deployments, the updater is a controller that periodically reconciles expected Kubernetes resources with those in the cluster. On a Linux server, the updater is an executable script called teleport-upgrade.

teleport-upgrade commands

The teleport-upgrade tool provides some basic commands to verify and perform an update of the Teleport agent.

teleport-upgrade help
USAGE: /usr/sbin/teleport-upgrade <command>
Tool for automatic upgrades of Teleport agents.
Commands:  run           check for and potentially apply a teleport upgrade.  dry-run       check for new teleport version but do not upgrade.  force         performs an upgrade if an upgrade is available.  version       print the current version of /usr/sbin/teleport-upgrade.  help          show this help text.

The dry-run command can be used to check for an available update without performing an update.

Example output when teleport is already on the latest compatible version.
teleport-upgrade dry-run
[i] no upgrades available (14.3.14 == 14.3.14) [ 582 ]
Example output when an update is available.
teleport-upgrade dry-run
[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ][i] within maintenance window, upgrade will be attempted. [ 596 ]

The run command performs an update if available.

Successful teleport update from 13.4.14 to 14.3.14.
teleport-upgrade run
[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ][i] within maintenance window, upgrade will be attempted. [ 596 ][i] attempting apt install teleport-ent=14.3.14... [ 480 ][...][i] gracefully restarting Teleport (if already running) [ 449 ]
Teleport updates are not attempted when outside the maintenance window.
teleport-upgrade run
[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ][i] upgrade is non-critical and we are outside of maintenance window, not attempting. [ 618 ]

The force command performs an update immediately even when outside the maintenance window.

teleport-upgrade force
[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ][i] attempting apt install teleport-ent=14.3.14... [ 480 ][...][i] gracefully restarting Teleport (if already running) [ 449 ]

Configuring the teleport-upgrade tool

1. Create the upgrade configuration directory:

   sudo mkdir -p /etc/teleport-upgrade.d/

2. If you changed the agent user to run as non-root, create /etc/teleport-upgrade.d/schedule and grant ownership to your Teleport user. Otherwise, you can skip this step:

   sudo touch /etc/teleport-upgrade.d/schedule
   sudo chown your-teleport-user /etc/teleport-upgrade.d/schedule

3. Configure the upgrader to connect to your version server and subscribe to the right release channel:

   echo "teleport.example.com:443/v1/webapi/automaticupgrades/channel/default" | sudo tee /etc/teleport-upgrade.d/endpoint

   Make sure not to include https:// as a prefix to the server address, nor suffix the endpoint with /version.

Choosing a release channel

When configuring the updater, you can select a release channel.

The following channels are available for APT, YUM, and Zypper repos:

Updating the updater

The updater is designed to be minimal and relatively stable, but the updater will receive updates on occasion. Currently, the updater does not have the capability to update itself. Updates to the teleport-ent-updater package must be done manually.

The teleport-ent-updater will be backwards compatible with older versions of Teleport, so you can always update the teleport-ent-updater package to the latest available version.

Version locking

As of Teleport 15.1.10, a version locking mechanism is enabled by the updater. This mechanism locks the version of Teleport and prevents manual updates of the Teleport package. This prevents unintentional updates during routine system maintenance, or an accidental update by a user. The updater still has the capability to update the Teleport package, and all updates of Teleport are expected to be performed by the updater.

The version locking mechanism is implemented using the features of the package managers. In case a user would like to manually update the Teleport version, this can be done with the following commands.

With the apt package manager CLI, the --allow-change-held-packages flag must be provided to bypass the version lock.

apt-get install --allow-change-held-packages "teleport-ent=<target-version>"

With the yum package manager CLI, the --disableexcludes="teleport" flag must be provided to bypass the version lock.

yum install --disablerepo="*" --enablerepo="teleport" --disableexcludes="teleport" "teleport-ent-<target-version>"

With the zypper package manager CLI, the lock must be disabled and then re-enabled after the update.

zypper removelock "teleport-ent"
zypper install --repo="teleport" "teleport-ent-<target-version>"
zypper addlock "teleport-ent"

Automatic update limitations

Automatic updates are not available in all Teleport editions and installation methods. If you cannot use automatic updates, read Manual updates for possible steps.

Automatic updates with Teleport Community Edition

Automatic updates is not currently supported with the community editions of Teleport. Ensure that you are using the Teleport Enterprise edition of the teleport-kube-agent chart. You should see the following when you query your teleport-kube-agent release:

helm -n teleport get values teleport-agent -o json | jq '.enterprise'
true

Automatic updates with direct binary installation

Automatic updates is not currently supported with the direct binary installation method. Automatic updates is only supported with installations via the apt, yum, and zypper package managers.

Automatic updates with GitOps tools

Automatic updates are incompatible with some GitOps tools used for continuous deployment. The teleport-kube-agent Helm chart owns the version of the teleport-agent resource, so when the teleport-agent-updater modifies the image version of the teleport-agent resource, the GitOps tool will detect a drift or a diff in the teleport-agent resource.

ArgoCD deployments

After an automatic update, ArgoCD reports the teleport-agent resource as OutOfSync. As a workaround to this problem use a Diff Customization to ignore the difference in image version. Here is an example deployment using the name teleport-agent and namespace teleport.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: teleport-agent
  namespace: teleport
spec
  ignoreDifferences:
  - group: apps
    kind: StatefulSet
    name: teleport-agent
    namespace: teleport
    jqPathExpressions:
    - .spec.template.spec.containers[] | select(.name == "teleport").image
...

FluxCD deployments

After an automatic update, FluxCD reports a DriftDetected event. As a workaround to this problem modify the drift detection configuration to ignore the difference in image version. Here is an example deployment using the name teleport-agent and namespace teleport.

apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
  name: teleport-agent
  namespace: teleport
spec
  driftDetection:
    mode: enabled
    ignore:
    - paths: ["/spec/template/spec/containers/0/image"]
      target:
        kind: StatefulSet
        name: teleport-agent
        namespace: teleport
...

Manual updates

This section shows you how to upgrade Teleport manually. You can perform manual upgrades on Teleport Auth Service and Proxy Service instances running in self-hosted clusters, as well as all Teleport agents.

Teleport agents

1. Identify the latest compatible Teleport agent version by querying the webapi endpoint of the Teleport Proxy Service:

   curl https://teleport.example.com/webapi/automaticupgrades/channel/stable/cloud/version
   v15.2.1

2. Use the tctl inventory ls command to list connected agents along with their current version. Use the --older-than flag to list agents that are upgradable:

   tctl inventory ls --older-than=v15.2.1
   Server ID                            Hostname       Services       Version Upgrader------------------------------------ -------------- -------------- ------- --------00000000-0000-0000-0000-000000000000 ip-10-1-6-130  Node           v14.4.5 none00000000-0000-0000-0000-000000000001 teleport-proxy Proxy          v15.2.0 none00000000-0000-0000-0000-000000000002 teleport-auth  Auth,Discovery v15.2.0 none...

3. For each agent ID returned by the tctl inventory ls command, copy the ID and run the following tctl command to access the host via tsh:

   HOST=00000000-0000-0000-0000-000000000000
   USER=root
   tsh ssh "${USER?}@${HOST?}"

4. On each Linux server, follow the instructions in the next section to install the new version of the teleport binary.

5. If you have deployed any agents on Kubernetes using the teleport-kube-agent Helm chart, follow the instructions to upgrade the Helm release.

Single Teleport binaries on Linux servers

You can upgrade a single Teleport binary running on a Linux host by running the one-line installation script with a higher version than the current one.

Before upgrading Teleport across a self-hosted cluster, read the Compatibility Overview to ensure you are upgrading components in the correct order.

Complete the following steps on all servers that run the Auth Service and Proxy Service, then on each of your agents:

1. Get the current version:

   teleport version

2. Assign edition to one of the following, depending on your Teleport edition:

   Edition	Value
   Teleport Enterprise (Cloud)	cloud
   Teleport Enterprise (Self-Hosted)	enterprise
   Teleport Community Edition	oss

3. Assign version to the version you want to install.

4. Install the new Teleport version on your Linux server:

   curl https://cdn.teleport.dev/install-v16.4.8.sh | bash -s version edition

   The installation script detects the package manager on your Linux server and uses it to install Teleport binaries. To customize your installation, learn about the Teleport package repositories in the installation guide.

5. Confirm that the version of the teleport binary is the one you expect:

   teleport version

6. Now that you have installed a more recent teleport binary on your Auth Service and Proxy Service instances, restart Teleport on these servers to run the new version.

   Configure your Teleport instance to start automatically when the host boots up by creating a systemd service for it. The instructions depend on how you installed your Teleport instance.

   • Package Manager
   • TAR Archive

   On the host where you will run your Teleport instance, enable and start Teleport:

   sudo systemctl enable teleport
   sudo systemctl start teleport

   On the host where you will run your Teleport instance, create a systemd service configuration for Teleport, enable the Teleport service, and start Teleport:

   sudo teleport install systemd -o /etc/systemd/system/teleport.service
   sudo systemctl enable teleport
   sudo systemctl start teleport

   You can check the status of your Teleport instance with systemctl status teleport and view its logs with journalctl -fu teleport.

Self-hosted Teleport clusters on Kubernetes

The instructions in this section assume that you have configured the teleport-cluster Helm chart with a values file called values.yaml, and that your teleport-cluster release is called teleport-cluster.

1. Shrink the Auth Service pool. You must reduce the number of Auth Service instances to one in order to ensure a consistent cluster state during the upgrade.

   Ensure that your teleport-cluster values file includes the following configuration:

   auth:
     highAvailability:
       replicaCount: 1
   

2. Update the Teleport Helm chart repository so you can install the latest version of the teleport-cluster chart:

   Set up the Teleport Helm repository.

   Allow Helm to install charts that are hosted in the Teleport Helm repository:

   helm repo add teleport https://charts.releases.teleport.dev

   Update the cache of charts from the remote repository so you can upgrade to all available releases:

   helm repo update

3. Upgrade the teleport-cluster Helm release:

   helm upgrade teleport-cluster teleport/teleport-cluster \  --version=16.4.8 \  --values=values.yaml

   The teleport-cluster Helm chart automatically waits for the previous version of the Proxy Service to stop responding to requests before running a new version of the Auth Service.

4. Once you have completed this guide and upgraded the cluster, you can configure your cluster for high availability again.

Teleport agents running on Kubernetes

The instructions in this section assume that you have configured the teleport-kube-agent Helm chart with a values file called values.yaml, and that your teleport-kube-agent release is called teleport-agent.

1. Update the Teleport Helm chart repository so you can install the latest version of the teleport-kube-agent chart:

   Set up the Teleport Helm repository.

   Allow Helm to install charts that are hosted in the Teleport Helm repository:

   helm repo add teleport https://charts.releases.teleport.dev

   Update the cache of charts from the remote repository so you can upgrade to all available releases:

   helm repo update

2. Upgrade the Helm release:

   helm -n teleport upgrade teleport-agent teleport/teleport-kube-agent \  --values=values.yaml \  --version=16.4.8