OSISM 7

Instructions for the upgrade can be found in the upgrade guide.

7.0.0a, 7.0.0b, 7.0.0c, 7.0.0d are pre-releases. Do not use these releases.

7.0.2

Release date: 17. April 2024

  • The Ceph & OpenStack service images have not been rebuilt. No upgrades of Ceph & OpenStack (and associated infrastructure services such as MariaDB or RabbitMQ) are required.

  • Properties for device type vrfs are now supported in the osism.commons.network role.

  • Debian support in osism.commons & osism.services Ansible collection.

    • The roles of the osism.commons collection are now usable with Debian. The roles have been tested with Debian Bookworm.

    • The roles of the osism.services collection are now usable with Debian. The roles have been tested with Debian Bookworm.

  • If the OSISM CLI is executed as root user there is now an informal warning message.

  • Use a single network for ceph frontend & backend in the cookiecutter (https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#network-configuration-reference for details).

  • When synchronising the configuration repository, it is now ensured that you are on the correct branch.

  • Senlin images available again as rolling tag for OpenStack 2023.2.

  • Zun images available again as rolling tag for OpenStack 2023.2 & 2024.1.

  • New FRR configuration template for loadbalancers with an external uplink in the osism.services.frr role.

  • scs:name-v1 and scs:name-v2 extra specs are now set via the OpenStack flavor manager.

  • Task ID is now displayed in the output of the OSISM CLI for prepared tasks.

  • New Makefile target in the OSISM Testbed to fetch the Wireguard configuration file: vpn-wireguard-config.

  • With the play noop it is now possible to run a play with noop. This is useful for testing purposes of the manager service.

  • The stable repository is now used as the default for the Netdata service in the osism.services.netdata role. This avoids package conflicts in future.

7.0.1

Release date: 27. March 2024

  • It’s now possible to also use custom plays in the kolla and in the ceph environments. As for the other environments, custom plays can now be stored there in a e.g. playbook-hello-world.yml file and run by using e.g. osism apply -e kolla hello-world.

  • The version of k3s has been updated to 1.29.2. If Kubernetes is used, upgrade with osism apply kubernetes.

  • There is a new parameter ceph_custom_keys in the copy-ceph-keys play. This makes it possible to copy the keys from custom Ceph pools.

    ceph_custom_keys:
  - src: ceph.client.manila1.keyring
    dest: "{{ configuration_directory }}/environments/kolla/files/overlays/manila/ceph.client.manila1.keyring"

  • There is a new parameter with_keycloak in the cookiecutter. This can now be used to select whether the keycloak integration should be prepared or not.

  • An error in the ceph-pools play has been fixed that prevented the keys from being created for the pools.

  • The ceph-iscsigws play has been removed. The iSCSI gateway is in maintenance as of November 2022. This means that it is no longer in active development and will not be updated to add new features.

  • With ceph_serial it is now possible to define how many hosts Ansible should manage at a single time in the Ceph plays.

    osism apply ceph-mgrs -e ceph_serial=1

  • With ANSIBLE_VERSION it’s now possible to overwrite the use Ansible version when working with the run.sh script inside the manager environment.

  • The osism.commons.known_hosts role has been completely revised.

    • avoid duplicate entries in the destination file

    • avoid comments in the destination file

    • make use of static entries possible

      It’s now possible to add a known_hosts parameter to the host_vars to set static known hosts entries for a specific host. When this parameter is set ssh-keygen will not be used to generate the known hosts entries on the fly.

      known_hosts:
  - ssh-rsa AAAAB3NzaC1y...
  - ecdsa-sha2-nistp256 AAAAE2VjZHN...
  - ssh-ed25519 AAAAC3NzaC1...

    • make use of extra entries possible

      It’s now possible to add a known_hosts_extra parameter to the configuration repository to set extra known hosts entries.

      known_hosts_extra:
  - testbed-node-1.testbed.osism.xyz ssh-rsa AAAAB3Nza...
  - testbed-node-2.testbed.osism.xyz ssh-rsa AAAAB3Nza...

    • The Octavia images have been updated. If Octavia is used, an upgrade must be done with osism apply -a upgrade octavia. We addressed the following issues.

7.0.0

Release date: 20. March 2024

General notes

  • Shortly before the release, gilt made a major release which led to breaking changes. It is therefore important for the moment to install python-gilt < 2 when synchronising the configuration repository against the generics. In the CI and within the container images, we currently use python-gilt == 1.2.3. This is also the version that’s currently installed in the container images and that’s set in the requirements.txt.

Deprecations

  • The role for deploying the Tang service (osism.services.tang) is deprecated in preparation for removal as it is currently not in a usable state. An attempt will be made to make the role usable until the next release. The same applies to Clevis integration via the osism.commons.clevis role.

Removals

  • The role for deploying the OpenLDAP service (osism.services.openldap) has been removed.

  • The manager plays to control the Vault service (seal, unseal, ..) have been removed as these tasks will be realized directly via the OSISM CLI (osism set vault password, ..) in the future.

  • The role for deploying the Bird service (osism.services.bird) has been removed.

New features

  • With the osism manage image octavia command it is possible to rotate the Octavia Amphora image, which is rebuilt daily. Older images are deactivated. The old images must be removed manually after rotating the amphorae instances.

  • With the osism manage image clusterapi command it is possible to import all currently stable Cluster API images (v1.27, v1.28, and v1.29). As soon as new minor or major versions are available, these are also imported. Old and no longer used versions must currently be removed manually.

  • The persistence feature in Octavia can enabled with the new enable_octavia_jobboard parameter. The jobboard in Octavia is an experimental feature. It is not recommended to use it in production.

    enable_octavia_jobboard: "yes"

    This requires an additional database, which is only created when Octavia play is run in bootstrap mode first.

    osism apply -a bootstrap octavia

    The secret octavia_persistence_database_password (see above) must be added to environments/kolla/secrets.yml before.

  • In preparation for the migration to Rook, the Rook operator is deployable on the internal Kubernetes cluster with osism apply rook. The Rook operator is not yet used for the Ceph deployment. For the deployment of Ceph we still use the ceph-ansible project. For the next release a tool called rookify is planned to migrate the Ceph deployment from ceph-ansible to Rook.

  • The roles of the osism.commons collection are now usable with CentOS 9. The roles have been tested with CentOS 9.

  • The roles of the osism.services collection are now usable with CentOS 9. The roles have been tested with CentOS 9.

  • With the openstack-resource-manager it is now possible to clean up orphaned amphora instances of Octavia or volumes that are stuck in the DELETING state.

  • Kubernetes can now be deployed with k3s-ansible on the management plane and the control plane.

  • It is now possible to manage the Ceph pools independently of ceph-osds Play using ceph-pools.

Upgrade notes

  • The switch from classic queue mirroring and durable queues to quorum queues in RabbitMQ has not yet been tested and documented. This is planned for the next release. The switch is not yet recommended.

  • The hosts_interface parameter is now set to internal_interface by default.

  • The Keycloak deployment via Docker Compose, which was previously included as a technical preview, has been completely revised and is now deployed on Kubernetes. No migration from the old deployment via Docker Compose to the new deployment via Kubernetes has been prepared. If you are currently using the Keycloak service, do not upgrade the Keycloak service and contact us in advance.

  • The Keystone role service is required by a number of OpenStack services. The role has been created automatically with new deployments for some time now. It is possible that this role is not yet available on older deployments and must be created once in preparation for the upgrade. You can check whether the role is available in the output of openstack --os-cloud admin role list. If it does not exist, it can be created with openstack --os-cloud admin role create service.

  • The use of ProxySQL for MariaDB is now possible and it is possible to switch to it as part of the upgrade. It is not mandatory and there is no recommendation. The parameter enable_proxysql is added to environments/kolla/configuration.yml for this purpose.

    enable_proxysql: yes

    The secrets listed below (proxysql_admin_password, proxysql_stats_password, mariadb_monitor_password) must also be added or changed.

    When migrating to ProxySQL, it is important to upgrade MariaDB first.

    When migrating to ProxySQL, it is important to perform the loadbalancer upgrade before all OpenStack service upgrades. To make sure that the OpenStack services continue to work after the upgrade when ProxySQL is enabled as part of the upgrade, the ProxySQL service must have been deployed first. The ProxySQL service is deployed with the loadbalancer play.

    It is possible that connectivity with the database may be interrupted for a short time during the migration. It is therefore recommended to carry out extensive tests on the staging environment in advance.

  • The following secrets must be added in environments/kolla/secrets.yml:

    octavia_persistence_database_password:  # generate with: pwgen 32 1
prometheus_bcrypt_salt:                 # generate with: pwgen 22 1 <-- there's a 22
prometheus_grafana_password:            # generate with: pwgen 32 1
prometheus_password:                    # generate with: pwgen 32 1
proxysql_admin_password:                # generate with: pwgen 32 1
proxysql_stats_password:                # generate with: pwgen 32 1

  • The parameter mariadb_monitoring_password in environments/kolla/secrets.yml has to be renamed to mariadb_monitor_password. If the parameter is not present, it is added.

    mariadb_monitor_password:     # generate with: pwgen 32 1

  • The following parameters must be removed from the configuration repository from environments/kolla/configuration.yml:

    ceph_nova_user: nova
ceph_nova_keyring: ceph.client.nova.keyring

  • Parameters for the Netbox service in environments/infrastructure/configuration.yml or secrets.yml must now also be added in environments/manager/configuration.yml or secrets.yml. In an upcoming release, the parameters can be removed from the infrastructure environment.

  • The Ansible callback plugin osism.commons.still_alive is now available to avoid timeouts for long-running tasks. This currently has to be explicitly enabled in the Ansible configuration. This is done in the environments/ansible.cfg file in the configuration repository. The callback plugin is enabled by default in the future. After this change has been made, the update of the manager must be performed. A manager with a version before OSISM 7.0.0 cannot be longer used if this plugin is set in environments/ansible.cfg.

    [defaults]
...
stdout_callback = osism.commons.still_alive

  • In the inventory, the nova_backend parameter must be added to the host vars of compute nodes where local storage is used.

    nova_backend: default

  • The SSL certificate file haproxy.pem is now available in a different location in the haproxy container. Previously it was stored under /etc/haproxy/haproxy.pem. From now on it is stored under /etc/haproxy/certificates/haproxy.pem. If you have customised the configuration for the haproxy service or use overlays for this, adjust the locations of the SSL certificate as required.

  • Due to the upgrade from Fluentd to version 5, some directory names within the container image for Fluentd have changed. If you have worked with overlay files in the Fluentd service, check these in advance. Currently we know that /var/run/td-agent is now available as /var/run/fluentd (check GitHub issue #864 for details). We assume that other directory names have changed similarly.

Known issues

  • If error Couldn't fetch the key client.bootstrap-rbd at /var/lib/ceph/bootstrap-rbd/." occurs when updating Ceph in task create potentially missing keys (rbd and rbd-mirror), create directory /var/lib/ceph/bootstrap-rbd/ on the 1st control node used for Ceph. Use the UID 64045 and the GID 64045. Set 0755 as permissions.

  • There are currently problems when using a custom CA in combination with Octavia. A bugfix for this will be made available with OSISM 7.0.1 (osism/issues#890).

    There is another problem when using Octavia in combination with OVN which leads to a leakage of ports when deleting load balancers (osism/issues#921). A bugfix for this is also expected to be available with OSISM 7.0.1.

  • The manager service is updated via osism update manager. If this command is not yet available, you can use osism-update-manager as an alternative.

    osism: 'update manager' is not an osism command. See 'osism --help'.

Other

  • Refstack 2022.11 results

    ======
Totals
======
Ran: 356 tests in 1221.9879 sec.
 - Passed: 353
 - Skipped: 3
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 715.6658 sec.

References

OpenStack 2023.2 press announcement: https://www.openstack.org/software/openstack-bobcat

OpenStack 2023.2 release notes: https://releases.openstack.org/bobcat/index.html

Release notes for each OpenStack service: