|
| 1 | +# Migrating from V1 to V2 of `cloudera-deploy` |
| 2 | + |
| 3 | +## In Summary |
| 4 | + |
| 5 | +1. Don't change your `definition.yml` or `cluster.yml` files. |
| 6 | +2. Create a playbook within your project to run your setup. You can start by referencing the following: |
| 7 | + * [Public Cloud](public-cloud/aws/datalake/main.yml) |
| 8 | + * Private Cloud (coming soon!) |
| 9 | +3. Create an `ansible-navigator.yml` configuration in your project. You can start by referencing the following: |
| 10 | + * [Public Cloud](public-cloud/aws/datalake/ansible-navigator.yml) |
| 11 | + * Private Cloud (coming soon!) |
| 12 | +4. Run your playbook by using `ansible-navigator` vs. `ansible-playbook`. |
| 13 | + * All other arguments apply, so continue to use `-e` and `-t` as needed, e.g. `ansible-navigator run your_playbook.yml -e key=value -t infra,plat,another_tag` |
| 14 | + |
| 15 | +## In Detail |
| 16 | + |
| 17 | +So, you may ask yourself, "How do I run my `cloudera-deploy` V1 playbooks in `ansible-navigator`?" <cue [The Talking Heads](https://youtu.be/5IsSpAOD6K8?si=K4vEs-b3kvZimM5X&t=49)> |
| 18 | + |
| 19 | +Previously, you would execute the `quickstart.sh` script to bootstrap the `cldr-runner` image into a shell and then run your scripts _from the container shell_, e.g. `ansible-playbook /opt/cloudera-deploy/main.yml -e "definition_path=examples/sandbox" -t run,default_cluster -vvv`. While this mode is still certainly possible, the introduction of `ansible-navigator` simplifies these action. |
| 20 | + |
| 21 | +**The most significant change**: the legacy definitions only contain configuration files -- the `definition.yml`, `cluster.yml`, `application.yml`, and `inventory_*` files -- and the legacy `cloudera-deploy` has local playbooks that orchestrated the whole run by calling a "sequence" role in `cloudera.exe`... No longer! |
| 22 | + |
| 23 | +So, what to do? First off... |
| 24 | + |
| 25 | +**Your existing platform configurations -- `definition.yml` and `cluster.yml`, specifically -- remain as they are. No changes are needed.** |
| 26 | + |
| 27 | +What does need to change? |
| 28 | + |
| 29 | +**You need to provide an entrypoint playbook.** |
| 30 | + |
| 31 | +Your project now needs a playbook, ala `main.yml`, to coordinate execution. This change allows for considerable flexibility as to how and when infrastructure and platform runlevels execute - frankly, how and when _any_ tasks, runlevel or otherwise, are run. |
| 32 | + |
| 33 | +In short, we have moved the responsiblity of managing key sections of the "runlevel" from the `cloudera_deploy` _application_ to the project _itself_. This allows you, on a per-project basis, to define _exactly_ what you want, when you need it. Yet, you still can call on the common, shared order-of-operations for installing Cloudera Manager or spinning up a CDP Public Cloud Datalake that the legacy `cloudera-deploy` once had, rather forced you to have. A simple `ansible.builtin.import_playbook` pragma will include these _collection playbooks_ from the updated `cloudera.exe` collection. |
| 34 | + |
| 35 | +Here is an example. The previous `main.yml` file eventually calls the `cloudera.exe.sequence` role, which in turn calls the _runlevel_ roles. |
| 36 | + |
| 37 | +```yaml |
| 38 | +# cloudera.exe.sequence/tasks/main.yml |
| 39 | + |
| 40 | +- name: Validate Infrastructure Configuration |
| 41 | + ansible.builtin.include_role: |
| 42 | + name: cloudera.exe.infrastructure |
| 43 | + tasks_from: validate |
| 44 | + # Truncated for clarity |
| 45 | + |
| 46 | +- name: Validate Platform Configuration |
| 47 | + ansible.builtin.include_role: |
| 48 | + name: cloudera.exe.platform |
| 49 | + tasks_from: validate |
| 50 | + # Truncated for clarity |
| 51 | + |
| 52 | +- name: Validate Runtime Configuration |
| 53 | + ansible.builtin.include_role: |
| 54 | + name: cloudera.exe.runtime |
| 55 | + tasks_from: validate |
| 56 | + # Truncated for clarity |
| 57 | +``` |
| 58 | + |
| 59 | +([See this file in its entirety.](https://github.com/cloudera-labs/cloudera.exe/blob/v1.7.5/roles/sequence/tasks/main.yml)) |
| 60 | + |
| 61 | +The _v2.x_ of `cloudera.exe` (and via proxy, `cloudera-deploy`) moves this code from the role _into_ a playbook within `cloudera.exe`. |
| 62 | + |
| 63 | +Here is a _v2.x_ entrypoint playbook. It assumes that you want to handle infrastructure - say, for a sandbox install - as well as the CDP Public Cloud setup. (There is an explicit playbook to teardown.) |
| 64 | + |
| 65 | +```yaml |
| 66 | +# cloudera-deploy/public-cloud/aws/datalake/main.yml |
| 67 | + |
| 68 | +- name: Set up the cloudera-deploy variables |
| 69 | + hosts: localhost |
| 70 | + connection: local |
| 71 | + gather_facts: yes |
| 72 | + tasks: |
| 73 | + - name: Read definition variables |
| 74 | + ansible.builtin.include_role: |
| 75 | + name: cloudera.exe.init_deployment |
| 76 | + public: yes |
| 77 | + when: init__completed is undefined |
| 78 | + tags: |
| 79 | + - always |
| 80 | + |
| 81 | +- name: Set up CDP Public Cloud infrastructure (Ansible-based) |
| 82 | + ansible.builtin.import_playbook: cloudera.exe.pbc_infra_setup.yml |
| 83 | + |
| 84 | +- name: Set up CDP Public Cloud (Env and DL example) |
| 85 | + ansible.builtin.import_playbook: cloudera.exe.pbc_setup.yml |
| 86 | +``` |
| 87 | +
|
| 88 | +And the new `cloudera.exe` playbooks? |
| 89 | + |
| 90 | +```yaml |
| 91 | +# cloudera.exe/playbooks/pbc_infra_setup.yml |
| 92 | +
|
| 93 | +- name: Set up CDP Public Cloud infrastructure (Ansible-based) |
| 94 | + hosts: "{{ target | default('localhost') }}" |
| 95 | + environment: "{{ globals.env_vars }}" |
| 96 | + gather_facts: yes |
| 97 | + tasks: |
| 98 | + - name: Validate CDP Public Cloud infrastructure configuration |
| 99 | + ansible.builtin.import_role: |
| 100 | + name: cloudera.exe.infrastructure |
| 101 | + tasks_from: validate |
| 102 | + tags: |
| 103 | + - validate |
| 104 | + - initialize |
| 105 | + - infra |
| 106 | +
|
| 107 | + - name: Initialize CDP Public Cloud infrastructure setup |
| 108 | + ansible.builtin.import_role: |
| 109 | + name: cloudera.exe.infrastructure |
| 110 | + tasks_from: initialize_setup |
| 111 | + tags: |
| 112 | + - initialize |
| 113 | + - infra |
| 114 | +
|
| 115 | + - name: Set up CDP Public Cloud infrastructure |
| 116 | + ansible.builtin.import_role: |
| 117 | + name: cloudera.exe.infrastructure |
| 118 | + tasks_from: setup |
| 119 | + tags: |
| 120 | + - infra |
| 121 | +``` |
| 122 | + |
| 123 | +```yaml |
| 124 | +# cloudera.exe/playbooks/pbc_setup.yml |
| 125 | +
|
| 126 | +- name: Set up CDP Public Cloud |
| 127 | + hosts: "{{ target | default('localhost') }}" |
| 128 | + environment: "{{ globals.env_vars }}" |
| 129 | + gather_facts: yes |
| 130 | + tasks: |
| 131 | + - name: Validate Platform configuration |
| 132 | + ansible.builtin.import_role: |
| 133 | + name: cloudera.exe.platform |
| 134 | + tasks_from: validate |
| 135 | + tags: |
| 136 | + - validate |
| 137 | + - initialize |
| 138 | + - plat |
| 139 | + - run |
| 140 | +
|
| 141 | + - name: Validate Data Services configuration |
| 142 | + ansible.builtin.import_role: |
| 143 | + name: cloudera.exe.runtime |
| 144 | + tasks_from: validate |
| 145 | + tags: |
| 146 | + - validate |
| 147 | + - initialize |
| 148 | + - run |
| 149 | +
|
| 150 | + - name: Initialize Platform setup |
| 151 | + ansible.builtin.import_role: |
| 152 | + name: cloudera.exe.platform |
| 153 | + tasks_from: initialize_setup |
| 154 | + tags: |
| 155 | + - initialize |
| 156 | + - plat |
| 157 | + - run |
| 158 | +
|
| 159 | + - name: Set up Platform |
| 160 | + ansible.builtin.import_role: |
| 161 | + name: cloudera.exe.platform |
| 162 | + tasks_from: setup |
| 163 | + tags: |
| 164 | + - plat |
| 165 | + - run |
| 166 | +
|
| 167 | + - name: Initialize Data Services setup |
| 168 | + ansible.builtin.import_role: |
| 169 | + name: cloudera.exe.runtime |
| 170 | + tasks_from: initialize_setup |
| 171 | + tags: |
| 172 | + - initialize |
| 173 | + - run |
| 174 | +
|
| 175 | + - name: Set up Data Services |
| 176 | + ansible.builtin.import_role: |
| 177 | + name: cloudera.exe.runtime |
| 178 | + tasks_from: setup |
| 179 | + tags: |
| 180 | + - run |
| 181 | +``` |
| 182 | + |
| 183 | +You can see that instead of calling the role and passing Ansible tags, you call the playbook, which now has _the very same code_ but without the need for some of the tags or the intermediate role, `cloudera.exe.sequence`. In fact, the playbooks in `cloudera.exe` have become the `cloudera.exe.sequence` role. |
| 184 | + |
| 185 | +You don't want to use the infrastructure playbook because you have your own process for establishing infrastructure? Great! Remove the `import_playbook` and call whatever is necessary! So long as you have run `cloudera.exe.init_deployment` in your project's playbook(s) _prior_ to importing any of the _collection playbooks_, you can use the collection playbooks anytime in your project playbooks. |
| 186 | + |
| 187 | +Need to discuss this further? Stop by the [Discussions > Help](https://github.com/cloudera-labs/cloudera.exe/discussions/categories/help)! |
0 commit comments