![]() To view the release notes of recent automation controller releases, please visit release notes 4.3. To download and install the latest version, please visit the automation controller installation guide. ![]() If you're interested in detailed information on automation controller, then the automation controller documentation is a must-read. In the next blog, we compare some of the results of the above change in two different clusters of automation controller. With the above change of how callback receiver workers are implemented, the risk of running into OOM issues is reduced and improves the overall performance of automation controller. For more information on making this modification manually, visit the performance tuning guide. Additionally, administrators may set the callback receiver worker manually if they so choose by setting JOB_EVENT_WORKERS property in a custom settings file. For example, if a VM control node has eight CPUs, the installer sets the callback receiver worker to eight.įor Red Hat OpenShift operator based installs, the number of callback receiver workers is set to the CPU limit for the task container if the CPU limit is greater than four. For virtual machine and bare metal installations, the 4.3 installer sets the number of callback receiver workers equal to the number of CPU. This is accomplished by enhancements to the traditional installer and the Red Hat OpenShift operator. Now, in automation controller 4.3, vertically scaling control nodes not only increases capacity to run jobs (which generate events), it proportionally scales the number of callback receiver workers to better handle the output from those jobs and to utilize host resources available to automation controller. While versions of automation controller prior to 4.3 had the option of modifying the JOB_EVENT_WORKERS setting to increase the size of the callback receiver from the default four, it was not a well known administrative setting. As more and more job events were queued up at the redis database (an in-memory database), the underlying control node ran out of memory (OOM) and the redis database processes were killed. During our performance analysis, we noticed that job events were getting queued at the redis database waiting to be processed when a large volume of job events took place that could not be handled by the default four callback receiver workers. Also, if the job templates are run at higher verbosity, that generates even more job events. ![]() Large Ansible Automation Platform clusters generate a huge volume of job events when running at their maximum capacity (max allowed forks), i.e. For customers who vertically scale their control nodes, this could cause performance issues as the callback receiver workers were not scaled based on the capacity of the control node(s). Prior to automation controller 4.3, by default each callback receiver had four workers to process job events regardless of the size of the control node. Each callback receiver has workers that process these job events and saves them in the database. These job events are pushed to the redis database queue and processed by the callback receiver. The following is an example job event: "event": "playbook_on_play_start", The job event data structures contain data from the parameters of the callback plugin hooks plus unique IDs that reference other job events. This enables Ansible to capture the result of a playbook run. ![]() Job events are JSON structures, created when Ansible calls the runner callback plugin hooks. ![]() It receives job events that result from Ansible jobs. This process is I/O and CPU intensive and requires performance considerations.Įvery control node in automation controller has a callback receiver process. This enables reviewing and querying results from across all your infrastructure and automation. The callback receiver is the process in charge of transforming the standard output of Ansible into serialized objects in the automation controller database. We will elaborate on an important change with callback receiver workers in automation controller 4.3 and how it can have an impact on the performance. Automation controller 4.3 is taking that one step further. In the previous blog, we saw that automation controller 4.1 provides significant performance improvements as compared to Red Hat Ansible Tower 3.8. We are excited to announce that the Ansible Automation Platform 2.3 release includes automation controller 4.3. Red Hat Ansible Automation Platform 2 is the next generation automation platform from Red Hat’s trusted enterprise technology experts. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |