5.0.0

General notes

  • Mitogen is installed in all Ansible containers. The use of the Mitogen strategy plugin for Ansible can be enabled via ansible.cfg in the environments directory

    [defaults]
strategy_plugins = /usr/share/ansible/plugins/mitogen/ansible_mitogen/plugins/strategy
strategy = mitogen_linear

    Unfortunately, the latest usable version of Mitogen (v0.3.3, Jun 12, 2022) is only usable up to Ansible 2.13 and is not yet usable with Ansible 2.14. Therefore, the Mitogen plugin can currently not be used with the osism-ansible container. When a new version of Mitogen is released that is usable with Ansible 2.14, it will be made available in a minor release of OSISM 5. Until then, the Mitogen plugin should not be used.

  • Support for routeros, which could be used to control Mikrotik devices, has been removed. Mikrotik devices were only used by the OSISM project itself in a baremetal testbed. Due to the change of the network hardware used by the OSISM project, only SONiC will be supported as NOS in the future.

  • Since only SONiC will be supported in the future, the following Ansible collections, which were previously available as technical previews, have been removed: arista.cvp, arista.eos, juniper.device.

  • The new Ansible collection osism.sonic, a downstream of dellemc.enterprise_sonic, is used to manage SONiC.

  • If osism is installed via PyPI, Ansible is no longer installed as a dependency. This ensures that the services can also be used with older Ansible versions. If Ansible is to be installed as well, this can be done via an extra requirement using osism[ansible].

  • With enable_auditd it is possible to disable the osism.services.auditd role in the bootstrap and maintenance playbooks.

  • With the dump facts command, it is possible to output the facts of Ansible for specific systems.

    $ osism dump facts mgr003
+--------+---------------------------------------------+----------------------------------------------------------------+
| Host   | Fact                                        | Value                                                          |
+========+=============================================+================================================================+
| mgr003 | _ansible_facts_gathered                     | True                                                           |
+--------+---------------------------------------------+----------------------------------------------------------------+
| mgr003 | ansible_all_ipv4_addresses                  | ['10.20.3.3', '10.15.0.112']                                   |
+--------+---------------------------------------------+----------------------------------------------------------------+
[...]

    $ osism dump facts mgr003 ansible_memory_mb
+--------+-------------------+------------------------------------------------------------+
| Host   | Fact              | Value                                                      |
+========+===================+============================================================+
| mgr003 | ansible_memory_mb | { 'nocache': {'free': 127134, 'used': 1551},               |
|        |                   |   'real': {'free': 122304, 'total': 128685, 'used': 6381}, |
|        |                   |   'swap': { 'cached': 0,                                   |
|        |                   |             'free': 8191,                                  |
|        |                   |             'total': 8191,                                 |
|        |                   |             'used': 0}}                                    |
+--------+-------------------+------------------------------------------------------------+

  • With the log ansible command, it is possible to use the shell of ARA.

    $ osism log ansible
(ara) help

Documented commands (use 'help -v' for verbose/'help <topic>' for details):
===========================================================================
alias  exit  history  quit          run_script  shell
edit   help  macro    run_pyscript  set         shortcuts

Application commands (type help <topic>):
=========================================
complete     host metrics  playbook delete   record delete  result show
expire       host show     playbook list     record list    task delete
help         play delete   playbook metrics  record show    task list
host delete  play list     playbook prune    result delete  task metrics
host list    play show     playbook show     result list    task show

    $ osism log ansible playbook list
+------+-----------+-------------------------------+--------+-----------------+----------------------------+-------+---------+-------+-----------------------------+-----------------+
|   id | status    | controller                    | user   | ansible_version | path                       | tasks | results | hosts | started                     | duration        |
+------+-----------+-------------------------------+--------+-----------------+----------------------------+-------+---------+-------+-----------------------------+-----------------+
| 2322 | completed | osism-ansible.manager_default | dragon | 2.13.7          | /ansible/generic/facts.yml |     3 |      18 |     6 | 2022-12-30T09:19:30.587307Z | 00:00:15.500605 |
| 2321 | completed | osism-ansible.manager_default | dragon | 2.13.7          | /ansible/generic/facts.yml |     3 |      17 |     6 | 2022-12-30T09:09:30.589686Z | 00:00:15.680527 |
| 2320 | completed | osism-ansible.manager_default | dragon | 2.13.7          | /ansible/generic/facts.yml |     3 |      17 |     6 | 2022-12-30T08:59:30.577125Z | 00:00:15.524505 |
| 2319 | completed | osism-ansible.manager_default | dragon | 2.13.7          | /ansible/generic/facts.yml |     3 |      18 |     6 | 2022-12-30T08:49:30.608174Z | 00:00:15.567697 |
+------+-----------+-------------------------------+--------+-----------------+----------------------------+-------+---------+-------+-----------------------------+-----------------+

  • With the log container command, it is possible to get the logs of a container.

    $ osism log container mgr003 fluentd
+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
[...]

  • With the validate command, it is possible to run validators. If you call the validate command without further parameters, all available validators will be displayed.

    The validators used are available in the Ansible collection osism.validations.

    $ osism validate ulimits

PLAY [Run ulimits validator] ***************************************************

TASK [osism.validations.ulimits : Get nofiles limit] ***************************
ok: [mgr003]
[...]

  • With osism validate ceph-config you can now check whether the configuration for Ceph is OK (https://docs.ceph.com/projects/ceph-ansible/en/latest/index.html#configuration-validation).

  • For Ceph, special playbooks were added to validate the deployment status of the OSD, MON and MGR services. The commands to run these are osism validate ceph-osds, osism validate ceph-mons, and osism validate ceph-mgrs.

  • ansible.utils and kubernetes.core Ansible collections are now usable.

  • The container command now allows the use of a prompt

    $ osism container ctl003
ctl003>>> images
REPOSITORY                    TAG       IMAGE ID       CREATED        SIZE
quay.io/osism/fluentd         zed       d30381217d55   21 hours ago   522MB
quay.io/osism/cron            zed       62d87cfb1f53   21 hours ago   254MB
quay.io/osism/kolla-toolbox   zed       5d1afcf2ac05   21 hours ago   826MB
quay.io/osism/rabbitmq        zed       5e36e8a4205a   21 hours ago   308MB
ctl003>>> info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
  scan: Docker Scan (Docker Inc., v0.23.0)
[...]

  • The console command now allows shortcuts to select specific console types.

    $ osism console ctl003  # SSH
dragon@ctl003:~$

$ osism console .ctl003  # Ansible
Welcome to the ansible console. Type help or ? to list commands.

dragon@ctl003 (1)[f:5]$

$ osism console ctl003/rabbitmq  # Container
(rabbitmq)[rabbitmq@ctl003 /]$

$ osism console ctl003/  # Container prompt
ctl003>>>

$ osism console :ctl00[1-3]  # clush
Enter 'quit' to leave this interactive mode
Working with nodes: ctl[001-003]
clush>

  • So far, the facts have been updated every 10 minutes. This is not necessary. Through the standard use of Redis as a cache backend, the facts are kept 24 hours. The facts are now only updated every 12 hours.

  • The use of the OpenStack Image Manager has been integrated. For this purpose, the new command manage was added. With osism manage images, the images on an OpenStack environment can now be managed using the OpenStack Image Manager. Currently, only the image definitions provided by the OpenStack Image Manager can be used. The option to use your own image definitions will be added in the future.

  • If osism apply is executed without a role to be executed, a table with all available roles will now be displayed.

    $ osism apply
2023-01-17 19:19:04.339 | INFO     | osism.commands.apply:take_action:159 - No playbook
given for execution. The playbooks listed in the table can be used.
+------------------------------------------------------------------+----------------+
| Role                                                             | Environment    |
|------------------------------------------------------------------+----------------|
| netdata                                                          | monitoring     |
| remove-netdata                                                   | monitoring     |
| remove-zabbix-agent                                              | monitoring     |
| openstack-health-monitor                                         | monitoring     |
| auditd                                                           | generic        |

  • The osism console command now supports the clustershell and can be used to operate a set group of systems at the same time.

    $ osism console :ctl00[1-3]
Enter 'quit' to leave this interactive mode
Working with nodes: ctl[001-003]
clush>

  • With the role osism.commons.docker_login it is possible to perform a login to a registry independently of osism.services.docker.

  • By switching from ansible to ansible-core, the size of the Ansible manager images has been significantly reduced (appr. 400 MByte / image).

  • With the parameters mariadb_datadir_volume and rabbitmq_datadir_volume it is possible to configure dedicated data volumes for the MariaDB and RabbitmQ services. This allows, for example, dedicated NVMes to be used for these services.

  • Skyline is available as technical preview. Skyline is an OpenStack dashboard optimized by UI and UE, support OpenStack Train+. It has a modern technology stack and ecology, is easier for developers to maintain and operate by users, and has higher concurrency performance.

    To be able tu use the Skyline service add enable_skyline: "yes" to environments/kolla/configuration.yml and add some new secrets to environments/kolla/secrets.yml.

    skyline_database_password:
skyline_keystone_password:
skyline_secret_key:

  • With the playbook ceph-purge-storage-node it is possible to remove all Ceph services on a Ceph storage node and remove all Ceph related data and configuration files.

  • With the playbook ceph-purge-cluster it is possible to remove all Ceph services on all Ceph nodes and remove all Ceph related data and configuration files.

  • With the Ansible role osism.services.osquery osquery can be deployed. osquery uses basic SQL commands to leverage a relational data-model to describe a device. It will be used in OSISM in the future to obtain facts about system states.

  • In the Magnum service, the Cluster API plugin from Vexxhost is available as a technical preview.

  • Use of Refstack can be prepared via the osism.validations.refstack role.

Testbed

  • The testbed has been significantly simplified for new operators and developers and quick start guide has been added.

  • Gnocchi, Ceilometer and Prometheus are deployed by default.

  • Refstack is run on stable-deploy jobs.

  • The documentation was changed to Docusaurus and is now published at docs.scs.community.

  • With deploy-ceph there is a new target to do a standalone Ceph deployment.

  • The CI jobs for the testbed were changed from Ubuntu 20.04 to 22.04 shortly before the release.

  • Terragrunt is now used as a wrapper for Terraform to avoid API timeouts during deployment.

  • Authorization Code flow with PKCE in Keycloak OIDC federation is now used.

  • SCS v2 flavors are used for testing.

  • MariaDB is used as backend for ARA.

  • Keystone port 35357 is no longer used.

  • Various improvements in Neutron with OVN: availability zones, default DNS server.

  • Linting of all Yaml files and Ansible files.

  • Ceph validators are run after a deployment and upgrade.

  • OpenStack Image Manager is used for importing machine images.

  • Speed of CI deployments increased significantly (use of own registry, deactivation of auditd, ..). Pure deployments are currently at under an hour.

  • QuickStart guide has been added.

OpenStack image manager

  • Some of the images (Almalinux, Rockylinux, Ubuntu) defined in etc/images are now automatically updated once a week.

  • OpenStack image manager is now published at PyPI.

  • A container image harbor.services.osism.tech/osism/openstack-image-manager:latest is now available which also contains the default image definitions in /etc/openstack-image-manager.

  • It is possible to add meta-information to individual versions of an image.

    - name: Sample image
  [...]
  meta:
    architecture: x86_64
  versions:
    - version: '1.0'
      [...]
      meta:
        image_build_date: YYYY-MM-DD

  • With the --validate parameter, it is possible to check the images available on an OpenStack environment for conformity with the Sovereign Cloud Stack (SCS) image standard.

  • With the --check parameter, it is possible to check the correctness of local image definitions.

  • With the parameter --filter it is possible to filter the list of images to be processed by their names with regular expressions. E.g. --filter "Ubuntu 2\d.\d\d" can be used to process all Ubuntu images with a version >= 20.04. The --name parameter was removed.

  • The --images parameter can now be passed a single file in addition to a directory with YAML files with included image definitions.

  • All parameters marked as mandatory by the Sovereign Cloud Stack (SCS) image standard are now set or requested accordingly if they are not set.

  • A dedicated crawler service to update the image definitions was sponsored by Plusserver and integration into the code base was started. The rework and further integration will take place in the next release.

  • The documentation was changed to Docusaurus.

Other

  • The Kubernetes CAPI images have been upgraded from Ubuntu 20.04 to Ubuntu 22.04.

  • Garden Linux image is available in version 934.6.

  • In order to avoid having to install the manager and, if necessary, the control nodes manually when setting up a new OSISM environment, an ISO image is now available. osism/node-image automatically installs a new node with Ubuntu 22.04 on a software RAID 1 and prepares everything to be able to start directly.

  • The Netbox plugin OSISM is now published at PyPI.

Deprecations

  • The role osism.services.bird is deprecated. In the future FRRouting (osism.services.frr) will be used.

  • The role osism.services.minikube is deprecated. In the future osism.services.k8s will be used.

  • Heat is deprecated in favor of more generic Infrastructure as Code tools like Terraform as of now and will be removed in the future (exact removal date is not yet known).

  • Swift (currently available as Technical Preview) will be removed in favor of Ceph RGW.

  • Trove (currently available as Technical Preview) will be removed in favor of Kubernetes database operators.

  • Skydive (currently available as Technical Preview) will be removed in the future, the project is not maintained anymore, last commit is 8th Jan 2022 (https://review.opendev.org/c/openstack/kolla/+/869191).

  • The login to a registry with the osism.services.docker role is deprecated in favor of the new osism.commons.docker_login role.

Removals

  • Monasca was deprecated in Kolla. In preparation for the introduction of Monasca in OSISM (which will no longer happen after deprecation), three infrastructure services required only by Monasca have already been built as images: Kafka, Storm, and Zookeeper. These images were removed without prior deprecation as they were only available for the planned Monasca integration and are not yet in use anywhere.

  • The ospurge wrapper script has been removed from the osism.services.openstackclient role. The ospurge project is no longer compatible with the current OpenStack SDK. The command openstack project purge can be used as an alternative.

  • Support for Zabbix was already removed in OSISM 3.0.0. The Ansible collection community.zabbix was still present as a leftover.

  • The docker-compose package is uninstalled by the osism.commons.docker_compose role. The Compose v2 plugin for Docker is now used instead of the old standalone docker-compose CLI. A dummy script has been added to /usr/local/bin which displays a corresponding message when using docker-compose.

  • Due to the transition of our CI to Zuul, the Github action for deploying Devstack has been archived and will not be maintained in the future.

Housekeeping

  • In all docker-compose.yml files the declaration of the version was removed. This is no longer necessary in the latest Compose specification.

Upgrade notes

  • On Ubuntu 20.04, Ansible is only usable up to Ansible 6 (or Ansible Core 2.13). If a manager is still in use on Ubuntu 20.04 and needs to be updated to OSISM 5, either the Ansible version must be explicitly set when using osism-update-manager (ANSIBLE_VERSION=6.7.0 osism-update-manager) or Ansible must be installed via ppa:ansible/ansible.

  • The ARA library has been updated to version 1.6.x in all Ansible containers. If the version of the ARA server on the manager is lower than 1.6.x problems may occur. When updating the Ansible container, the ARA server should also be updated to version 1.6.x accordingly.

  • Kolla-Ansible now only supports OpenSearch instead of ElasticSearch + Kibana. If you are currently deploying ElasticSearch with Kolla Ansible, you should backup the data before starting the upgrade. The contents of the ElasticSearch data volume will be automatically moved to the OpenSearch volume. The ElasticSearch, ElasticSearch Curator and Kibana containers will be removed automatically. The inventory must be updated so that the elasticsearch group is renamed to opensearch, and the kibana group is renamed to opensearch-dashboards. A new secret opensearch_dashboards_password has to be added to the secrets.yml file.

  • The ovn role has been split into ovn-controller and ovn-db roles, therefore users that have ovn_extra_volumes configured need to adapt their config to use ovn_db_extra_volumes or ovn_controller_extra_volumes.

  • The ironic_dnsmasq_dhcp_range and ironic_dnsmasq_default_gateway parameters were replaced with ironic_dnsmasq_dhcp_ranges.

    # Old way
ironic_dnsmasq_dhcp_range: "192.168.112.50,192.168.112.60"
ironic_dnsmasq_default_gateway: "192.168.112.1"

# New way
ironic_dnsmasq_dhcp_ranges:
  - range: "192.168.112.50,192.168.112.60"
    routers: "192.168.112.1"

  • The zuul_tag parameter for the osism.services.zuul role has been renamed to zuul_zuul_tag for consistency with the other container tag variables.

  • The osism.services.hddtemp role has been changed to lm_sensors. This role now removes the hddtemp service on Ubuntu 22.04. In the future, the role will be renamed from hddtemp to lm_sensors.

  • The Celery integration of the manager is now activated by default, because in the future various features will be based on it. If the integration should not to be used, enable_celery: False must be set in environments/manager/configuration.yml.

  • When upgrading from Ubuntu 20.04 to Ubuntu 22.04, it is important to also change the network configuration to Netplan. Details about it are there: https://github.com/osism/issues/issues/495

References

OpenStack Zed press announcement: https://www.openstack.org/software/zed/

OpenStack Zed release notes:

Versions

service

version

aodh

15.0.0

barbican

15.0.1

ceilometer

19.0.1

cinder

21.1.1

cloudkitty

17.0.1

cron

3.0pl1

designate

15.0.1

dnsmasq

2.86

etcd

3.3.27

fluentd

4.4.2

glance

25.1.1

gnocchi

4.5.0

grafana

9.4.3

haproxy

2.4.18

heat

19.0.1

horizon

23.0.1

influxdb

1.8.10

ironic

21.1.1

iscsid

2.1.5

keepalived

2.2.4

keystone

22.0.1

kolla-toolbox

15.1.1

kolla_toolbox

15.1.1

kuryr

10.0.0

libvirt

8.0.0

magnum

15.0.2

manila

15.1.1

mariadb

10.6.12

memcached

1.6.14

mistral

15.0.0

multipathd

0.8.8

neutron

21.1.1

nova

26.1.1

octavia

11.0.1

opensearch

2.3.0

opensearch_dashboards

2.3.0

openvswitch

3.0.1

ovn

22.9.0

placement

8.0.0

prometheus

2.38.0

prometheus_alertmanager

0.24.0

prometheus_blackbox_exporter

0.22.0

prometheus_cadvisor

0.45.0

prometheus_elasticsearch_exporter

1.5.0

prometheus_haproxy_exporter

0.13.0

prometheus_libvirt_exporter

5.0.0

prometheus_memcached_exporter

0.10.0

prometheus_msteams

1.5.1

prometheus_mtail

3.0.0

prometheus_mysqld_exporter

0.14.0

prometheus_node_exporter

1.4.0

prometheus_openstack_exporter

5.0.0

prometheus_ovn_exporter

1.0.4

rabbitmq

3.11.11

redis

6.0.16

senlin

14.0.0

skyline

1.0.0

swift

2.30.1

tgtd

1.0.80

trove

18.0.1