Ansible Filtering Lab

As emphasized by the prior lab, the Jinja2 templating engine is utilized throughout the Ansible ecosystem. One of the features provided by Jinja is filtering. Filtering can be accomplished within {{ braces }} in both playbooks and template files and resemble Unix pipes. Jinja itself provides a set of built in filters which are then augmented by those provided by Ansible.

Goals

• Understand how to utilize Ansible filters
• Utilize common Ansible filters
• Create A Custom Filter

Lab Setup

1. Ensure you are in the ciq-basics directory

2. Create a folder for this lab, let’s call it lab03

You are now ready to start the lab.

Understanding Ansible Filters

1. Before we walk through some of the common filters found in Ansible, we should review how a filter is constructed as well as utilized. As mentioned previously, filters resemble unix pipes to signify their action.

Here is a simple playbook that prints out an “Invalid Variable” if a variable has yet to be defined

2. Lets create a new file named filters.yml. For the rest of this exercise, we will just modify the same file over and over instead of creating new files for each task.

---
- name: Print variable if it exists, print default message if not
  hosts: localhost
  gather_facts: false
  connection: local

  tasks:
      - debug:
          msg: "{{ does_not_exist_variable | default('Invalid Variable') }}"

Sample Output

localhost| SUCCESS => {
    "msg": "Invalid Variable"
}

Using the Default Filter with Omit

The default filter becomes especially powerful when combined with Ansible’s omit special value. This allows you to conditionally include or exclude parameters from tasks based on whether variables are defined, creating more flexible and reusable playbooks.

1. Create a playbook called default_omit.yml to demonstrate conditionally omitting task parameters:

- hosts: basics-host
  gather_facts: false
  vars:
    file_owner: "{{ ansible_user }}"
    # file_group is intentionally not defined
    # file_mode is intentionally not defined
  
  tasks:
    - name: Create file with conditional parameters
      file:
        path: /tmp/test_file.txt
        state: touch
        owner: "{{ file_owner | default(omit) }}"
        group: "{{ file_group | default(omit) }}"
        mode: "{{ file_mode | default(omit) }}"
    
    - name: Display file info
      stat:
        path: /tmp/test_file.txt
      register: file_info
    
    - name: Show file ownership
      debug:
        msg: |
          File owner: {{ file_info.stat.pw_name }}
          File group: {{ file_info.stat.gr_name }}
          File mode: {{ file_info.stat.mode }}

2. Notice how parameters are conditionally applied:
   • owner is set because file_owner is defined
   • group and mode are omitted because those variables are undefined
   • When omitted, Ansible uses system defaults
3. Now let’s demonstrate with user management where some parameters are optional:

- name: User management with conditional parameters
  hosts: basics-host
  gather_facts: false
  become: true
  vars:
    username: testuser
    user_shell: /bin/bash
    # user_home is not defined - will use system default
    # user_groups is not defined - no supplementary groups
  
  tasks:
    - name: Create user with conditional parameters
      user:
        name: "{{ username }}"
        shell: "{{ user_shell | default(omit) }}"
        home: "{{ user_home | default(omit) }}"
        groups: "{{ user_groups | default(omit) }}"
        state: present
    
    - name: Display user info
      getent:
        database: passwd
        key: "{{ username }}"
      register: user_info
    
    - name: Show user details
      debug:
        msg: |
          User: {{ username }}
          Shell: {{ user_info.ansible_facts.getent_passwd[username][5] }}
          Home: {{ user_info.ansible_facts.getent_passwd[username][4] }}

Sample Output

TASK [Create file with conditional parameters] *****************************
changed: [localhost]

TASK [Show file ownership] *********************************************
ok: [localhost] => {
    "msg": "File owner: ansible\nFile group: ansible\nFile mode: 644\n"
}

TASK [Create user with conditional parameters] *************************
changed: [localhost]

TASK [Show user details] ********************************************
ok: [localhost] => {
    "msg": "User: testuser\nShell: /bin/bash\nHome: /home/testuser\n"
}

4. You can also use default(omit) with boolean parameters. Create another example:

- name: Service management with conditional parameters
  hosts: basics-host
  gather_facts: false
  become: true
  vars:
    service_name: httpd
    service_enabled: true
    # service_masked is not defined
  
  tasks:
    - name: Manage service with conditional parameters
      service:
        name: "{{ service_name }}"
        enabled: "{{ service_enabled | default(omit) }}"
        masked: "{{ service_masked | default(omit) }}"
        state: started

When service_masked is undefined, the masked parameter is omitted entirely, allowing the service to maintain its current masked state.

Things to try

• Use default(omit) with package installation where version might be optional
• Combine default(omit) with conditionals to create highly flexible role parameters
• Practice using default(omit) with template module parameters like backup and follow

Utilizing Common Filters

In this section, we will create a series of playbooks that demonstrate how to utilize Ansible filters. Playbooks, when utilized, will have a common structure. Either a custom fact will be created using the set_fact module, or a variable provided by Ansible’s fact gathering module will be utilized.

Calculating Checksums

1. A checksum can be calculated against a string using the checksum filter:

- hosts: localhost
  gather_facts: false
  connection: local

  tasks:
      - debug:
          msg: "{{ 'text I want to obtain a checksum' | checksum }}"

Sample Output

localhost | SUCCESS => {
    "msg": "394f552ed03a06108ca7233b0235eba100270820"
}

List Operations

1. Lists can be manipulated using Ansible filters.

First, display the maximum value from a list containing -2, 5 and 10

- hosts: localhost
  gather_facts: false
  connection: local

  tasks:
      - debug:
          msg: "{{ [-2, 5, 10] | max }}"

Sample Output

localhost | SUCCESS => {
    "msg": "10"
}

2. The inverse can be seen using the min filter

- hosts: localhost
  gather_facts: false
  connection: local

  tasks:
      - debug:
          msg: "{{ [-2, 5, 10] | min }}"

Sample Output

localhost | SUCCESS => {
    "msg": "-2"
}

Random Numbers

1. Random numbers can be generated in a variety of ways.

First, print out a random number between 0 and 100

- hosts: localhost
  gather_facts: false
  connection: local

  tasks:
      - debug:
          msg: "{{ 100 | random }}"

Sample Output

localhost | SUCCESS => {
    "msg": "31"
}

2. Or, an even number between 0 and 100 that utilizes the step feature

- hosts: localhost
  gather_facts: false
  connection: local

  tasks:
      - debug:
          msg: "{{ 100 | random(step=2) }}"

Sample Output

localhost | SUCCESS => {
    "msg": "48"
}

Mathematical Operations

1. Various mathematical operations can be performed using Ansible filters.

First, find the square root of 144

- hosts: localhost
  gather_facts: false
  connection: local

  tasks:
      - debug:
          msg: "{{ 144 | root | int }}"

Sample Output

localhost | SUCCESS => {
    "msg": "12"
}

2. In the previous example, we demonstrated how filters can be chained together. The result of the root filter is a decimal value (12.0). Much like piping in Linux, Ansible filters can be combined to take the value of the prior result as their input. The int filter converts a number into an Integer.

3. Now, lets find the absolute value of -169

- hosts: localhost
  gather_facts: false
  connection: local

  tasks:
      - debug:
          msg: "{{ -169 | abs }}"

Sample Output

localhost | SUCCESS => {
    "msg": "169"
}

4. Finally, lets round 13.6

- hosts: localhost
  gather_facts: false
  connection: local

  tasks:
      - debug:
          msg: "{{ 13.6 | round | int }}"

Sample Output

localhost | SUCCESS => {
    "msg": "14"
}

Using the Map Filter

The map filter is one of the most powerful filters in Ansible, allowing you to extract specific attributes from complex data structures or transform lists of objects. It’s particularly useful when working with Ansible facts that contain nested dictionaries and lists.

1. Let’s use the map filter to extract mount points from the system’s mounted filesystems. Create a playbook called map_filter.yml:

- hosts: all
  gather_facts: yes
  tasks:
    - name: Display all mount points as a comma-separated list
      debug:
        msg: "Mount points: {{ ansible_mounts | map(attribute='mount') | join(',') }}"
    
    - name: Display all filesystem types
      debug:
        msg: "Filesystem types: {{ ansible_mounts | map(attribute='fstype') | unique | join(', ') }}"
    
    - name: Display device names with their mount points
      debug:
        msg: "{{ ansible_mounts | map('regex_replace', '^(.*)$', item.device + ' -> ' + item.mount) | list }}"
      vars:
        item: "{{ ansible_mounts[0] }}"
      when: ansible_mounts | length > 0
    
    - name: Show network interface names
      debug:
        msg: "Network interfaces: {{ ansible_interfaces | map('upper') | join(', ') }}"

2. Execute the playbook

Sample Output

TASK [Display all mount points as a comma-separated list] **********************
ok: [node-1] => {
    "msg": "Mount points: /,/boot,/home,/var"
}

TASK [Display all filesystem types] ********************************************
ok: [node-1] => {
    "msg": "Filesystem types: ext4, xfs"
}

TASK [Show network interface names] ********************************************
ok: [node-1] => {
    "msg": "Network interfaces: LO, ETH0, ETH1"
}

The map filter can be used in several ways:

• map(attribute=‘key’) - Extract a specific attribute from objects in a list
• map(‘filter_name’) - Apply another filter to each item in a list
• map(‘regex_replace’, ‘pattern’, ‘replacement’) - Apply regex replacement to each item

3. Let’s create a more complex example that processes user data:

- hosts: localhost
  gather_facts: false
  vars:
    users:
      - name: alice
        email: alice@example.com
        role: admin
      - name: bob
        email: bob@example.com
        role: user
      - name: charlie
        email: charlie@example.com
        role: admin
  tasks:
    - name: Extract just the usernames
      debug:
        msg: "Usernames: {{ users | map(attribute='name') | join(', ') }}"
    
    - name: Get admin email addresses
      debug:
        msg: "Admin emails: {{ users | selectattr('role', 'equalto', 'admin') | map(attribute='email') | join('; ') }}"
    
    - name: Transform usernames to uppercase
      debug:
        msg: "Uppercase usernames: {{ users | map(attribute='name') | map('upper') | join(', ') }}"

Sample Output

TASK [Extract just the usernames] **********************************************
ok: [localhost] => {
    "msg": "Usernames: alice, bob, charlie"
}

TASK [Get admin email addresses] ***********************************************
ok: [localhost] => {
    "msg": "Admin emails: alice@example.com; charlie@example.com"
}

Things to try

• Try using map with other Ansible facts like ansible_all_ipv4_addresses
• Combine map with selectattr to filter and transform data in one pipeline
• Use map with mathematical operations to transform numeric lists

Ternary

1. A common programming construct is a Ternary Operator. This conditional operation allows for an if statement to be shortcut. The operator provides a value if a condition is true or false.

2. Create a playbook called ternary_filter.yml with the following content:

- hosts: all
  gather_facts: yes
  tasks:
    - name: Ternary Example
      debug:
        msg: "{{ (ansible_os_family == 'RedHat') | ternary('Rocky','Not Rocky') }}"

The result of the playbook will display “RedHat/Rocky” if the machine is RedHat/Rocky based. Otherwise it will display “Not RedHat”.

3. Execute the playbook

    ok: [node-1] => {
        "msg": "Rocky"
    }

Things to try

• Practice chaining multiple mathematical filters together
• Use random filters with range() to generate random numbers within specific ranges
  
• Combine checksum filters with conditionals to verify file integrity

Using Lookup Plugins

While not technically filters, lookup plugins are often used alongside filters in Jinja2 expressions and provide powerful ways to retrieve data from external sources during playbook execution. Lookup plugins can read files, query databases, access environment variables, and much more.

Create and run your playbook

1. First, let’s create some sample files to work with. Create a file called servers.txt:

web01
web02
db01
cache01

2. Create another file called config.json:

{
  "database": {
    "host": "localhost",
    "port": 3306,
    "name": "myapp"
  },
  "cache": {
    "host": "redis.local", 
    "port": 6379
  }
}

3. Create a playbook called lookup_examples.yml:

- hosts: localhost
  gather_facts: false
  tasks:
    - name: Read file contents using lookup
      debug:
        msg: "Server list: {{ lookup('file', 'servers.txt').split('\n') | join(', ') }}"
    
    - name: Parse JSON file using lookup
      debug:
        msg: "Database host: {{ (lookup('file', 'config.json') | from_json).database.host }}"
    
    - name: Get environment variable with lookup
      debug:
        msg: "Current user: {{ lookup('env', 'USER') | default('unknown') }}"
    
    - name: Generate random password using lookup
      debug:
        msg: "Random password: {{ lookup('password', '/dev/null length=12 chars=ascii_letters,digits') }}"
    
    - name: Use lookup with loops
      debug:
        msg: "Processing server: {{ item }}"
      loop: "{{ lookup('file', 'servers.txt').split('\n') }}"
      when: item != ""
    
    - name: Combine multiple lookups with filters
      set_fact:
        server_config: |
          {%- set servers = lookup('file', 'servers.txt').split('\n') -%}
          {%- set config = lookup('file', 'config.json') | from_json -%}
          Servers: {{ servers | select | join(', ') }}
          DB Host: {{ config.database.host }}
          DB Port: {{ config.database.port }}
    
    - name: Display combined configuration
      debug:
        msg: "{{ server_config }}"
    
    - name: Use lookup to check if files exist
      debug:
        msg: "Config file exists: {{ lookup('first_found', ['config.json', 'config.yaml'], errors='ignore') != '' }}"

4. Execute the playbook

Sample Output

TASK [Read file contents using lookup] *****************************************
ok: [localhost] => {
    "msg": "Server list: web01, web02, db01, cache01"
}

TASK [Parse JSON file using lookup] ********************************************
ok: [localhost] => {
    "msg": "Database host: localhost"
}

TASK [Get environment variable with lookup] ************************************
ok: [localhost] => {
    "msg": "Current user: ansible"
}

TASK [Generate random password using lookup] ***********************************
ok: [localhost] => {
    "msg": "Random password: aB3kL9mP4xR2"
}

TASK [Use lookup with loops] ************************************************
ok: [localhost] => (item=web01) => {
    "msg": "Processing server: web01"
}
ok: [localhost] => (item=web02) => {
    "msg": "Processing server: web02"
}
ok: [localhost] => (item=db01) => {
    "msg": "Processing server: db01"
}
ok: [localhost] => (item=cache01) => {
    "msg": "Processing server: cache01"
}

Common lookup plugins include:

• file - Read file contents
• env - Access environment variables
  
• password - Generate random passwords
• first_found - Find the first existing file from a list
• csvfile - Read data from CSV files
• dig - Perform DNS lookups
• pipe - Execute shell commands and return output

Things to try

• Try using the csvfile lookup to read structured data from a CSV file
• Use the pipe lookup to execute system commands and process their output with filters
• Combine dig lookup with map filter to resolve multiple hostnames
• Experiment with vars lookup to dynamically access variable names

Creating a Custom Filter

1. While Jinja and Ansible provide a wide range of filters, there may be a desire to extend the based set of features. Fortunately, Ansible provides the mechanism for users to create their own filters. In this section, we will create a custom filter that appends the trademark symbol (™) at the end of the string.

Custom filters are python scripts that contain the logic to format text appropriately. Ansible will look for custom filters in a filter_plugins directory relative to the executing playbook.

2. First, lets create the filter_plugins directory within the lab03 folder we are already in

3. Create a new file called trademark.py in the filter_plugins directory with the following content:

class FilterModule(object):
    def filters(self):
        return {
            'trademark': self.format_trademark
        }
  
    def format_trademark(self, input_text):
        return input_text + u"\u2122"

The actual formatting logic occurs in the format_trademark method. The unicode value for the trademark symbol (™) is utilized, otherwise an error would occur. Within the return statement in the filters method, the dictionary key trademark represents the name of the filter that end users can ultimately use. The dictionary value signifies the method to invoke.

4. Now, create a playbook called custom-filter.yml in the lab03 directory that makes use of this new filter. We will set a custom fact containing Ansible as the value of the variable trademark_string. Within the debug module, we will make use of the newly created filter plugin to format the variable with the trademark symbol.

- hosts: localhost
  gather_facts: no
  connection: local
  tasks:
    - name: Custom Fact
      set_fact:
        trademark_string: "Ansible"
    - name: Print Trademark
      debug:
        msg: "{{ trademark_string | trademark }}"

5. Execute the playbook

Sample Output

    PLAY [localhost] ***************************************************************************************************************************************************************************************************************************
    
    TASK [Gathering Facts] *********************************************************************************************************************************************************************************************************************
    ok: [localhost]
    
    TASK [Custom Fact] *************************************************************************************************************************************************************************************************************************
    ok: [localhost]
    
    TASK [Print Trademark] *********************************************************************************************************************************************************************************************************************
    ok: [localhost] => {
        "msg": "Ansible™"
    }
    
    PLAY RECAP *********************************************************************************************************************************************************************************************************************************
    localhost                  : ok=3    changed=0    unreachable=0    failed=0

Notice how the trademark symbol appears appended to the Ansible text. This validates that the custom filter plugin is being executed successfully.

Return to Exercises