gcp_storage_object Python3 UnicodeDecodeError: 'utf-8' codec can't decode #266

Akasurde · 2020-08-02T14:13:15Z

From @bucklander on Jul 31, 2020 22:41

SUMMARY

While using Python3, gcp_storage_object module is unable to upload binary files (and perhaps other file types) without throwing:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x98 in position 522: invalid start byte

This is possibly related to Ansible python3's explicit string conversion (or not). When using other file types (E.g. simple text files) or just using Python2 with any file type, the problem does not exist.

Below are two example playbooks; One which uploads a gzip tarball file, and another which uploads a simple text file. Each playbook is executed with Python2 and again with Python3.

I'm certainly no expert in file compression or file types, however in my particular case the gzip'd file format seems to cause the issue. I'm also seeing this problem with binary files within regular tarball (no gzip compression). You could also replace the compressed file examples below with seemingly any randomly generated binary file, and the result appears to be the same.

ISSUE TYPE

Bug Report

COMPONENT NAME

gcp_storage_object

ANSIBLE VERSION

$ ansible --version
ansible 2.9.11
  config file = None
  configured module search path = ['${HOME}/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible
  executable location = /Library/Frameworks/Python.framework/Versions/3.8/bin/ansible
  python version = 3.8.5 (v3.8.5:580fbb018f, Jul 20 2020, 12:11:27) [Clang 6.0 (clang-600.0.57)]

CONFIGURATION

$ ansible-config dump --only-changed
$

OS / ENVIRONMENT

This can be reproduced on a few different systems. Here are a few specific examples:

macOS Mojave 10.14.6
Python 2.7.15
Python 3.8.5

CentOS Linux release 8.1.1911 (Core) 
Python 3.6.8

STEPS TO REPRODUCE

Execute the following example playbooks with a simple, flat text file as well as a tarball.

echo "hello" > foobar.txt
tar -zcf binary.tar foobar.txt

Example playbook to upload binary file to GCS:

# upload-bin-file.yml
---

- name: Upload a binary file to Google Cloud Storage bucket using gcp_storage_object module
  hosts: all
  connection: local
  tasks:
      - name: Upload Latest Backup(s) to GCS Bucket
        gcp_storage_object:
          action: upload
          overwrite: yes
          bucket: gcs-bucket-name
          src: binary.tar
          dest: "binary.tar"
          project: google-cloud-project-name
          auth_kind: serviceaccount
          service_account_file: "jwt.json"
          state: present

Example playbook to upload text file to GCS:

# upload-text-file.yml
---

- name: Upload a text file to Google Cloud Storage bucket using gcp_storage_object module
  hosts: all
  connection: local
  tasks:
      - name: Upload Latest Backup(s) to GCS Bucket
        gcp_storage_object:
          action: upload
          overwrite: yes
          bucket: gcs-bucket-name
          src: foobar.txt
          dest: "foobar.txt"
          project: google-cloud-project-name
          auth_kind: serviceaccount
          service_account_file: "jwt.json"
          state: present

Run upload-text-file.yml using Python 2 (modify Python envs per your system):

ansible-playbook --connection=local --inventory localhost, upload-text-file.yml -e ansible_python_interpreter=`which python2`

Run upload-text-file.yml using Python 3:

ansible-playbook --connection=local --inventory localhost, upload-text-file.yml -e ansible_python_interpreter=`which python3`

Run upload-bin-file.yml using Python 2:

ansible-playbook --connection=local --inventory localhost, upload-bin-file.yml -e ansible_python_interpreter=`which python2`

Run upload-bin-file.yml using Python 3:

ansible-playbook --connection=local --inventory localhost, upload-bin-file.yml -e ansible_python_interpreter=`which python3`

EXPECTED RESULTS

It's expected that all files upload successfully to GCS, regardless of using Python 2, or Python 3 or file type.

ACTUAL RESULTS

Python 2 Text File Upload (successful):

$ ansible-playbook --connection=local --inventory localhost, upload-text-file.yml -e ansible_python_interpreter=/Library/Frameworks/Python.framework/Versions/2.7/bin/python2

PLAY [Upload a file to GCS using gcp_storage_object module] ********************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************
ok: [localhost]

TASK [Upload Latest Backup(s) to GCS Bucket] ***********************************************************************************************
changed: [localhost]

PLAY RECAP *********************************************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Python 3 Text File Upload (successful):

$ ansible-playbook --connection=local --inventory localhost, upload-text-file.yml -e ansible_python_interpreter=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3

PLAY [Upload a file to GCS using gcp_storage_object module] *****************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************
ok: [localhost]

TASK [Upload Latest Backup(s) to GCS Bucket] ********************************************************************************************************************************
changed: [localhost]

PLAY RECAP ******************************************************************************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Python 2 Binary File Upload (successful):

$ ansible-playbook --connection=local --inventory localhost, upload-bin-file.yml -e ansible_python_interpreter=/Library/Frameworks/Python.framework/Versions/2.7/bin/python2

PLAY [Upload a file to GCS using gcp_storage_object module] *****************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************
ok: [localhost]

TASK [Upload Latest Backup(s) to GCS Bucket] ********************************************************************************************************************************
changed: [localhost]

PLAY RECAP ******************************************************************************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Python 3 Binary File Upload (fails):

$ ansible-playbook --connection=local --inventory localhost, upload-bin-file.yml -e ansible_python_interpreter=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3 -vvvv
ansible-playbook 2.9.11
  config file = None
  configured module search path = ['${HOME}/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible
  executable location = /Library/Frameworks/Python.framework/Versions/3.8/bin/ansible-playbook
  python version = 3.8.5 (v3.8.5:580fbb018f, Jul 20 2020, 12:11:27) [Clang 6.0 (clang-600.0.57)]
No config file found; using defaults
setting up inventory plugins
Set default localhost to localhost
Parsed localhost, inventory source with host_list plugin
Loading callback plugin default of type stdout, v2.0 from /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible/plugins/callback/default.py

PLAYBOOK: upload-bin-file.yml ***********************************************************************************************************************************************************************************
Positional arguments: upload-bin-file.yml
verbosity: 4
connection: local
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('localhost,',)
extra_vars: ('ansible_python_interpreter=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3',)
forks: 5
1 plays in upload-bin-file.yml

PLAY [Upload a file to GCS using gcp_storage_object module] *****************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
task path: ${HOME}/Desktop/upload-bin-file.yml:3
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: bwallander
<localhost> EXEC /bin/sh -c 'echo ~bwallander && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ${HOME}/.ansible/tmp `"&& mkdir ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500 && echo ansible-tmp-1596236353.566118-46540-87177398296500="` echo ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500 `" ) && sleep 0'
Using module file /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible/modules/system/setup.py
<localhost> PUT ${HOME}/.ansible/tmp/ansible-local-46536i64et516/tmpj1i25bxy TO ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500/AnsiballZ_setup.py
<localhost> EXEC /bin/sh -c 'chmod u+x ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500/ ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500/AnsiballZ_setup.py && sleep 0'
<localhost> EXEC /bin/sh -c '/Library/Frameworks/Python.framework/Versions/3.8/bin/python3 ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500/AnsiballZ_setup.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500/ > /dev/null 2>&1 && sleep 0'
ok: [localhost]
META: ran handlers

TASK [Upload Latest Backup(s) to GCS Bucket] ********************************************************************************************************************************************************************
task path: ${HOME}/Desktop/upload-bin-file.yml:7
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: bwallander
<localhost> EXEC /bin/sh -c 'echo ~bwallander && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ${HOME}/.ansible/tmp `"&& mkdir ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804 && echo ansible-tmp-1596236354.939475-46572-195465935008804="` echo ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804 `" ) && sleep 0'
Using module file /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible/modules/cloud/google/gcp_storage_object.py
<localhost> PUT ${HOME}/.ansible/tmp/ansible-local-46536i64et516/tmp1a31n78a TO ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py
<localhost> EXEC /bin/sh -c 'chmod u+x ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/ ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py && sleep 0'
<localhost> EXEC /bin/sh -c '/Library/Frameworks/Python.framework/Versions/3.8/bin/python3 ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py", line 102, in <module>
    _ansiballz_main()
  File "${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible.modules.cloud.google.gcp_storage_object', init_globals=None, run_name='__main__', alter_sys=True)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py", line 286, in <module>
  File "/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py", line 190, in main
  File "/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py", line 206, in upload_file
  File "/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/module_utils/gcp_utils.py", line 99, in post_contents
  File "/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/module_utils/gcp_utils.py", line 159, in full_post
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/auth/transport/requests.py", line 448, in request
    response = super(AuthorizedSession, self).request(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1039, in _send_output
    for chunk in chunks:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 994, in _read_readable
    datablock = readable.read(self.blocksize)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x98 in position 522: invalid start byte
fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible.modules.cloud.google.gcp_storage_object', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py\", line 207, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py\", line 286, in <module>\n  File \"/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py\", line 190, in main\n  File \"/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py\", line 206, in upload_file\n  File \"/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/module_utils/gcp_utils.py\", line 99, in post_contents\n  File \"/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/module_utils/gcp_utils.py\", line 159, in full_post\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py\", line 578, in post\n    return self.request('POST', url, data=data, json=json, **kwargs)\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/auth/transport/requests.py\", line 448, in request\n    response = super(AuthorizedSession, self).request(\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py\", line 530, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py\", line 643, in send\n    r = adapter.send(request, **kwargs)\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/adapters.py\", line 439, in send\n    resp = conn.urlopen(\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 670, in urlopen\n    httplib_response = self._make_request(\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 392, in _make_request\n    conn.request(method, url, **httplib_request_kw)\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py\", line 1255, in request\n    self._send_request(method, url, body, headers, encode_chunked)\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py\", line 1301, in _send_request\n    self.endheaders(body, encode_chunked=encode_chunked)\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py\", line 1250, in endheaders\n    self._send_output(message_body, encode_chunked=encode_chunked)\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py\", line 1039, in _send_output\n    for chunk in chunks:\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py\", line 994, in _read_readable\n    datablock = readable.read(self.blocksize)\n  File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/codecs.py\", line 322, in decode\n    (result, consumed) = self._buffer_decode(data, self.errors, final)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0x98 in position 522: invalid start byte\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

PLAY RECAP ******************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Copied from original issue: ansible/ansible#71034

The text was updated successfully, but these errors were encountered:

Akasurde · 2020-08-02T14:13:18Z

From @bucklander on Jul 31, 2020 22:55

FWIW, besides my local macbook, this is also repeatable on an AWX server container:

bash-4.4# cat /etc/redhat-release 
CentOS Linux release 8.1.1911 (Core) 
bash-4.4# python3 -V
Python 3.6.8
bash-4.4# ansible --version
ansible 2.9.7
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.8 (default, Nov 21 2019, 19:31:34) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]

Akasurde · 2020-08-02T14:13:19Z

From @bucklander on Jul 31, 2020 23:23

cc @erjohnso @GoogleCloudPlatform @rambleraptor

bucklander · 2020-08-04T21:15:13Z

FWIW this bug can be resolved by simply changing gcp_storage_object.py#L254 from...

with open(src, "r") as file_obj: to with open(src, "rb") as file_obj:

This ensures there's no decoding attempt (or subsequent error).

This approach appears similar to download_file's use of the open func at gcp_storage_object.py#L243, where it specifically is configured with the b (binary) mode specifier.

Not only does this resolve the issue with uploading binary files, but the change doesn't appear to harm text uploads. Unsure though if this is a safe strategy overall.

bucklander · 2020-08-04T21:45:50Z

Have opened #268 with a proposed fix, but have no real way to regression test it.

bucklander · 2020-12-02T22:32:18Z

@Akasurde Any idea how to get attention to this?

gavenkoa · 2021-04-18T22:55:51Z

Until it fixed I stay with Python 2 ansible.cfg:

[defaults]
interpreter_python = /usr/bin/python2

An alternative approach is to use uri module and handle authentication / error checking manually: #268 (comment)

CodeForcer · 2022-01-08T12:08:09Z

This error is still persisting in January 2022 more than a year later.

Safe to say at this point the Google Cloud ansible collection should be considered legacy

gavenkoa · 2022-01-08T17:04:49Z

Safe to say at this point the Google Cloud ansible collection should be considered legacy

There is an incentive to create new products inside Google (to get the promotion). There is no incentive to keep it working.

Still I see new commits in the repository...

I use gcloud cli + Bash for automation, GCP Ansible modules were unreliable for automation: I need work done, not to waste time debugging Python code of google.collection (most problems with lack of idempotency).

smemsh · 2022-01-09T03:12:05Z

I think this collection is only a downstream output of https://github.com/GoogleCloudPlatform/magic-modules

gavenkoa · 2022-01-09T12:38:31Z

downstream output

It explains lack of idempotency, idiomatic for Ansible. They robo-generate code around REST API instead of providing convenient tools. That explains the wide coverage of API & difficulty to fix bugs or add extra functionality beside dumb translation to HTTP call.

Here is on the bug tracker another person recommended to use Google API directly using JSON API key and uri module. You can list & update resources, building idempotency themselves.

Akasurde mentioned this issue Aug 2, 2020

gcp_storage_object Python3 UnicodeDecodeError: 'utf-8' codec can't decode ansible/ansible#71034

Closed

bucklander mentioned this issue Aug 4, 2020

Fixing issue with UTF-8 conversion in Python3 for GCS uploads #268

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gcp_storage_object Python3 UnicodeDecodeError: 'utf-8' codec can't decode #266

gcp_storage_object Python3 UnicodeDecodeError: 'utf-8' codec can't decode #266

Akasurde commented Aug 2, 2020

Akasurde commented Aug 2, 2020

Akasurde commented Aug 2, 2020

bucklander commented Aug 4, 2020 •

edited

Loading

bucklander commented Aug 4, 2020

bucklander commented Dec 2, 2020

gavenkoa commented Apr 18, 2021

CodeForcer commented Jan 8, 2022

gavenkoa commented Jan 8, 2022 •

edited

Loading

smemsh commented Jan 9, 2022

gavenkoa commented Jan 9, 2022 •

edited

Loading

gcp_storage_object Python3 UnicodeDecodeError: 'utf-8' codec can't decode #266

gcp_storage_object Python3 UnicodeDecodeError: 'utf-8' codec can't decode #266

Comments

Akasurde commented Aug 2, 2020

SUMMARY

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

Akasurde commented Aug 2, 2020

Akasurde commented Aug 2, 2020

bucklander commented Aug 4, 2020 • edited Loading

bucklander commented Aug 4, 2020

bucklander commented Dec 2, 2020

gavenkoa commented Apr 18, 2021

CodeForcer commented Jan 8, 2022

gavenkoa commented Jan 8, 2022 • edited Loading

smemsh commented Jan 9, 2022

gavenkoa commented Jan 9, 2022 • edited Loading

bucklander commented Aug 4, 2020 •

edited

Loading

gavenkoa commented Jan 8, 2022 •

edited

Loading

gavenkoa commented Jan 9, 2022 •

edited

Loading