ansible-galaxy fails to install collection from GalaxyNG when there are many versions

[watsonb]

Summary

Given I have installed GalaxyNG and published custom content and synchronized content from galaxy.ansible.com, when I try to install collections listed in a requirements.yml via a modern version of ansible-galaxy configured to use GalaxyNG, ansible-galaxy fails when encountering the community.vmware collection.

If you comment out community.vmware from requirements.yml, all other collections (custom published and synchronized) install fine.

If you uncomment community.vmware and use an older version of ansible-galaxy (e.g., Ansible 2.9.12), all collections install fine.

Issue Type

Bug Report

Component Name

ansible-galaxy

Ansible Version

$ ansible --version

# works (Ansible 2.9.12 installed via pip)
ansible 2.9.12
  config file = /home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg
  configured module search path = ['/home/ben/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ben/venv3_ansible-2.9.12/lib/python3.8/site-packages/ansible
  executable location = /home/ben/venv3_ansible-2.9.12/bin/ansible
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]

# fails (Ansible 4.2.0 installed via pip)
ansible [core 2.11.2] 
  config file = /home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg
  configured module search path = ['/home/ben/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ben/venv3_ansible-4.2.0/lib/python3.8/site-packages/ansible
  ansible collection location = /home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections:/home/ben/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/ben/venv3_ansible-4.2.0/bin/ansible
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 3.0.1
  libyaml = True

# fails (Ansible 5.8.0 installed via pip)
ansible [core 2.12.6]
  config file = /home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg
  configured module search path = ['/home/ben/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ben/venv3_ansible-5.8.0/lib/python3.8/site-packages/ansible
  ansible collection location = /home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections:/home/ben/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/ben/venv3_ansible-5.8.0/bin/ansible
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True

Configuration

$ ansible-config dump --only-changed -t all
ANSIBLE_NOCOWS(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = True
CACHE_PLUGIN(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = jsonfile
CACHE_PLUGIN_CONNECTION(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = $HOME/ansible/facts
CACHE_PLUGIN_TIMEOUT(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = 600
CALLBACKS_ENABLED(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = ['timer', 'profile_roles', 'profile_tasks', 'junit']
COLLECTIONS_PATHS(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = ['/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections', '/home/ben/.ansible/collections', '/usr/share/ansible/col>
DEFAULT_FORKS(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = 20
DEFAULT_GATHERING(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = smart
DEFAULT_INVENTORY_PLUGIN_PATH(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = ['/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/plugins/inventory', '/home/ben/.ansible/plugins/inventory',>
DEFAULT_LOAD_CALLBACK_PLUGINS(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = True
DEFAULT_ROLES_PATH(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = ['/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/roles', '/home/ben/.ansible/roles', '/usr/share/ansible/roles', '/etc/>
DEFAULT_STDOUT_CALLBACK(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = yaml
DEFAULT_VAULT_IDENTITY_LIST(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = ['vcenter@~/workspace/kiewit/ansible_vaults/vcenter', 'azure@~/workspace/kiewit/ansible_vaults/azure', 'ansible_use>
DEFAULT_VAULT_PASSWORD_FILE(env: ANSIBLE_VAULT_PASSWORD_FILE) = /home/ben/workspace/kiewit/ansible_vaults/molecule
GALAXY_SERVER_LIST(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = ['kiewit_published', 'kiewit_community']
HOST_KEY_CHECKING(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = False
INVENTORY_ENABLED(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = ['host_list', 'script', 'auto', 'yaml', 'ini', 'toml', 'vmware_vm_inventory', 'azure_rm', 'ldap_inventory']
RETRY_FILES_ENABLED(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = True
VARIABLE_PRECEDENCE(/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg) = ['all_plugins_inventory', 'groups_plugins_inventory', 'all_inventory', 'groups_inventory', 'all_plugins_play', 'groups_plug>

OS / Environment

Ubuntu 20.04 running Ansible installed via pip into a virtual environment.

Steps to Reproduce

---
# requirements.yml

collections:
- name: azure.azcollection
  version: 1.7.0
- name: community.vmware
  version: 1.17.0
- name: community.general
  version: 4.5.0
- name: community.windows
  version: 1.9.0

# custom published collection
- name: kiewit.content
  version: 1.4.0

ansible-galaxy collection install -r collections/requirements.yml -p collections/ -vvvvv --force

GalaxyNG installed, custom collection published, remote collections synchronized to GalaxyNG via the following requirements.yml

---
collections:
- name: azure.azcollection
- name: community.vmware
- name: community.general
- name: community.windows

Notice there are 316 community.vmware versions:

Expected Results

I expect all listed requirements to be forcefully re-installed.

Actual Results

#
# Failing with `ansible-galaxy` via Ansible 5.8.0
#
ansible-galaxy [core 2.12.6]
  config file = /home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg
  configured module search path = ['/home/ben/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ben/venv3_ansible-5.8.0/lib/python3.8/site-packages/ansible
  ansible collection location = /home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections:/home/ben/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/ben/venv3_ansible-5.8.0/bin/ansible-galaxy
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True
Using /home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg as config file
Reading requirement file at '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/requirements.yml'
Starting galaxy collection install process
Found installed collection azure.azcollection:1.7.0 at '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/azure/azcollection'
Found installed collection ansible.windows:1.10.0 at '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/ansible/windows'
Found installed collection community.vmware:2.5.0 at '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/community/vmware'
Found installed collection community.general:4.5.0 at '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/community/general'
Found installed collection community.windows:1.9.0 at '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/community/windows'
Found installed collection kiewit.content:1.4.0 at '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/kiewit/content'
Process install dependency map
Initial connection to galaxy_server: https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/
Found API version 'v3' with Galaxy server kiewit_published (https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/)
Opened /home/ben/.ansible/galaxy_token
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/v3/collections/azure/azcollection/
Initial connection to galaxy_server: https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/
Found API version 'v3' with Galaxy server kiewit_community (https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/)
Opened /home/ben/.ansible/galaxy_token
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/v3/collections/azure/azcollection/
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/v3/collections/community/vmware/
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/v3/collections/community/vmware/
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/v3/collections/community/vmware/versions/?limit=100
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/community/vmware/versions/?limit=100&offset=100
[WARNING]: Skipping Galaxy server https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/. Got an unexpected error when getting available versions of collection community.vmware:
'/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/community/vmware/versions/'
ERROR! Failed to resolve the requested dependencies map. Could not satisfy the following requirements:
* community.vmware:2.5.0 (direct request)

#
# Working via ansible-galaxy provided in Ansible 2.9.12
#
ansible-galaxy 2.9.12
  config file = /home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg
  configured module search path = ['/home/ben/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ben/venv3_ansible-2.9.12/lib/python3.8/site-packages/ansible
  executable location = /home/ben/venv3_ansible-2.9.12/bin/ansible-galaxy
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
Using /home/ben/workspace/kiewit/ansible/playbooks/ap_foo/ansible.cfg as config file
Reading requirement file at '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/requirements.yml'
Found installed collection azure.azcollection:1.7.0 at '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/azure/azcollection'
Found installed collection ansible.windows:1.10.0 at '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/ansible/windows'
Found installed collection community.vmware:2.5.0 at '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/community/vmware'
Found installed collection community.general:4.5.0 at '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/community/general'
Found installed collection community.windows:1.9.0 at '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/community/windows'
Found installed collection kiewit.content:1.4.0 at '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/kiewit/content'
Process install dependency map
Initial connection to galaxy_server: https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/
Opened /home/ben/.ansible/galaxy_token
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/
Found API version 'v3' with Galaxy server kiewit_published (https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/)
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/v3/collections/azure/azcollection/versions/1.7.0/
Initial connection to galaxy_server: https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/
Opened /home/ben/.ansible/galaxy_token
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/
Processing requirement collection 'azure.azcollection'
Collection requirement 'azure.azcollection' is the name of a collection
Collection 'azure.azcollection' is not available from server kiewit_published https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/
Found API version 'v3' with Galaxy server kiewit_community (https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/)
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/v3/collections/azure/azcollection/versions/1.7.0/
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/v3/collections/community/vmware/versions/2.5.0/
Collection 'azure.azcollection' obtained from server kiewit_community https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/
Processing requirement collection 'community.vmware'
Collection requirement 'community.vmware' is the name of a collection
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/v3/collections/community/vmware/versions/2.5.0/
Collection 'community.vmware' is not available from server kiewit_published https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/v3/collections/community/general/versions/4.5.0/
Collection 'community.vmware' obtained from server kiewit_community https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/
Processing requirement collection 'community.general'
Collection requirement 'community.general' is the name of a collection
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/v3/collections/community/general/versions/4.5.0/
Collection 'community.general' is not available from server kiewit_published https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/v3/collections/community/windows/versions/1.9.0/
Collection 'community.general' obtained from server kiewit_community https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/
Processing requirement collection 'community.windows'
Collection requirement 'community.windows' is the name of a collection
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/v3/collections/community/windows/versions/1.9.0/
Collection 'community.windows' is not available from server kiewit_published https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/v3/collections/kiewit/content/versions/1.4.0/
Collection 'community.windows' obtained from server kiewit_community https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/
Processing requirement collection 'kiewit.content'
Collection requirement 'kiewit.content' is the name of a collection
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/v3/collections/ansible/windows/versions/
Collection 'kiewit.content' obtained from server kiewit_published https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/
Processing requirement collection 'ansible.windows' - as dependency of community.windows
Collection requirement 'ansible.windows' is the name of a collection
Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/ansible/windows/versions/?limit=10&offset=10
Collection 'ansible.windows' obtained from server kiewit_community https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/
Starting collection install process
Installing 'azure.azcollection:1.7.0' to '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/azure/azcollection'
Downloading https://kneawxalp311.kiewitplaza.com/api/galaxy/v3/plugin/ansible/content/community/collections/artifacts/azure-azcollection-1.7.0.tar.gz to /home/ben/.ansible/tmp/ansible-local-20811912wtbig7j/tmp7pjnjmnu
Validating downloaded file hash 28041b95da141c55e01cc491f004b64cd9a7c7d2d28e622825543309744f689a with expected hash 28041b95da141c55e01cc491f004b64cd9a7c7d2d28e622825543309744f689a
Installing 'community.vmware:2.5.0' to '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/community/vmware'
Downloading https://kneawxalp311.kiewitplaza.com/api/galaxy/v3/plugin/ansible/content/community/collections/artifacts/community-vmware-2.5.0.tar.gz to /home/ben/.ansible/tmp/ansible-local-20811912wtbig7j/tmp7pjnjmnu
Validating downloaded file hash 4e3aaabdf4802af8ba9c08c77e464c1e0a52abcd7dec7d3aa51b1f865eadfe2a with expected hash 4e3aaabdf4802af8ba9c08c77e464c1e0a52abcd7dec7d3aa51b1f865eadfe2a
Installing 'community.general:4.5.0' to '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/community/general'
Downloading https://kneawxalp311.kiewitplaza.com/api/galaxy/v3/plugin/ansible/content/community/collections/artifacts/community-general-4.5.0.tar.gz to /home/ben/.ansible/tmp/ansible-local-20811912wtbig7j/tmp7pjnjmnu
Validating downloaded file hash c191817d1c19fef2ec2927d65dbec65f3f0bbba60523b95ec163a48ba39dbcf0 with expected hash c191817d1c19fef2ec2927d65dbec65f3f0bbba60523b95ec163a48ba39dbcf0
Installing 'community.windows:1.9.0' to '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/community/windows'
Downloading https://kneawxalp311.kiewitplaza.com/api/galaxy/v3/plugin/ansible/content/community/collections/artifacts/community-windows-1.9.0.tar.gz to /home/ben/.ansible/tmp/ansible-local-20811912wtbig7j/tmp7pjnjmnu
Validating downloaded file hash 63909e16fc055e39266127098f7c11e9da5e6000b231be5bdac74c1758ef9e9b with expected hash 63909e16fc055e39266127098f7c11e9da5e6000b231be5bdac74c1758ef9e9b
Installing 'kiewit.content:1.4.0' to '/home/ben/workspace/kiewit/ansible/playbooks/ap_foo/collections/ansible_collections/kiewit/content'
Downloading https://kneawxalp311.kiewitplaza.com/api/galaxy/v3/plugin/ansible/content/published/collections/artifacts/kiewit-content-1.4.0.tar.gz to /home/ben/.ansible/tmp/ansible-local-20811912wtbig7j/tmp7pjnjmnu
Validating downloaded file hash 5c72fee203e0e927ab7451cd0d7206c549f4280208f464ec0d5078711c70beda with expected hash 5c72fee203e0e927ab7451cd0d7206c549f4280208f464ec0d5078711c70beda
Skipping 'ansible.windows' as it is already installed

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibot]

Files identified in the description:

• lib/ansible/galaxy

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[watsonb]

I think the big difference between the more modern ansible-galaxy and the older (working) version is this:

5.8.0

Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/community/vmware/versions/?limit=100&offset=100

2.9.12

Calling Galaxy at https://kneawxalp311.kiewitplaza.com/api/galaxy/content/published/v3/collections/community/vmware/versions/2.5.0/

It seems the newer versions have an HTTP query string limiting the results whereas the older version does not. I cannot find a way to coerce ansible-galaxy to get past this limit of 100. I don't think this is a configuration/limitation imposed by GalaxyNG as it appears to work for older versions of ansible-galaxy.

[watsonb]

I've worked around this by:

1. delete community.vmware from GalaxyNG
2. refine the requirements.yml used to configure GalaxyNG remote sync
3. re-sync

My specific refinement uses a version qualifier for the community.vmware collection like so:

---
collections:
  - name: community.vmware
    version: '>= 1.17.0'

This greatly pared down the number of versions of the collection synced to GalaxyNG (I don't use anything older than 1.17.0) and now ansible-galaxy can install all collections specified in a requirements.yml in both older (Ansible 2.9.12) and more modern (Ansible 4.2.0+) versions of Ansible.

[s-hertel]

@watsonb Would you be able to curl https://kneawxalp311.kiewitplaza.com/api/galaxy/content/community/v3/collections/community/vmware/versions/?limit=100 and share the value of links.next? I've been unable to reverse-engineer that to add a test that matches your output (well, unless links.next is actually /api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/community/vmware/versions/?limit=100&offset=100, but I'm guessing not...).

needs_info

[watsonb]

@s-hertel whilst using curl doesn't yield any results (not returning JSON for some reason), I am able to paste the URL into a browser and leverage the buil-in API viewer. Here's what comes back (mind you, this is after re-sync with a version qualifier to limit the number of sync'd versions).

{
    "meta": {
        "count": 9
    },
    "links": {
        "first": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/community/vmware/versions/?limit=100&offset=0",
        "previous": null,
        "next": null,
        "last": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/community/vmware/versions/?limit=100&offset=0"
    },
    "data": [
        {
            "version": "2.5.0",
            "href": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/community/vmware/versions/2.5.0/",
            "created_at": "2022-05-25T19:30:47.878318Z",
            "updated_at": "2022-05-25T19:30:47.878337Z",
            "requires_ansible": ">=2.11.0"
        },
        {
            "version": "2.4.0",
            "href": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/community/vmware/versions/2.4.0/",
            "created_at": "2022-05-25T19:30:47.878318Z",
            "updated_at": "2022-05-25T19:30:47.878337Z",
            "requires_ansible": ">=2.11.0"
        },
        {
            "version": "2.3.0",
            "href": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/community/vmware/versions/2.3.0/",
            "created_at": "2022-05-25T19:30:47.878318Z",
            "updated_at": "2022-05-25T19:30:47.878337Z",
            "requires_ansible": ">=2.11.0"
        },
        {
            "version": "2.2.0",
            "href": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/community/vmware/versions/2.2.0/",
            "created_at": "2022-05-25T19:30:47.878318Z",
            "updated_at": "2022-05-25T19:30:47.878337Z",
            "requires_ansible": ">=2.11.0"
        },
        {
            "version": "2.1.0",
            "href": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/community/vmware/versions/2.1.0/",
            "created_at": "2022-05-25T19:30:47.878318Z",
            "updated_at": "2022-05-25T19:30:47.878337Z",
            "requires_ansible": ">=2.11.0"
        },
        {
            "version": "2.0.0",
            "href": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/community/vmware/versions/2.0.0/",
            "created_at": "2022-05-25T19:30:47.878318Z",
            "updated_at": "2022-05-25T19:30:47.878337Z",
            "requires_ansible": ">=2.11.0"
        },
        {
            "version": "1.18.0",
            "href": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/community/vmware/versions/1.18.0/",
            "created_at": "2022-05-25T19:30:47.878318Z",
            "updated_at": "2022-05-25T19:30:47.878337Z",
            "requires_ansible": ">=2.9.10"
        },
        {
            "version": "1.17.1",
            "href": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/community/vmware/versions/1.17.1/",
            "created_at": "2022-05-25T19:30:47.878318Z",
            "updated_at": "2022-05-25T19:30:47.878337Z",
            "requires_ansible": ">=2.9.10"
        },
        {
            "version": "1.17.0",
            "href": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/community/vmware/versions/1.17.0/",
            "created_at": "2022-05-25T19:30:47.878318Z",
            "updated_at": "2022-05-25T19:30:47.878337Z",
            "requires_ansible": ">=2.9.10"
        }
    ]
}

That being said, I do have the full ansible.netcommon collection sync'd, which has 152 versions. I'll include a shortened version of that output below:

{
    "meta": {
        "count": 152
    },
    "links": {
        "first": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/ansible/netcommon/versions/?limit=100&offset=0",
        "previous": null,
        "next": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/ansible/netcommon/versions/?limit=100&offset=100",
        "last": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/ansible/netcommon/versions/?limit=100&offset=52"
    },
    "data": [
        {
            "version": "3.0.0",
            "href": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/ansible/netcommon/versions/3.0.0/",
            "created_at": "2022-05-13T15:55:58.913107Z",
            "updated_at": "2022-05-13T15:55:58.913121Z",
            "requires_ansible": ">=2.9.10"
        },
        {
            "version": "2.6.1",
            "href": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/ansible/netcommon/versions/2.6.1/",
            "created_at": "2022-05-13T15:55:58.913107Z",
            "updated_at": "2022-05-13T15:55:58.913121Z",
            "requires_ansible": ">=2.9.10"
        },
        {
            "href": "snipped for brevity"
        },
        {
            "version": "1.0.1-dev7",
            "href": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/ansible/netcommon/versions/1.0.1-dev7/",
            "created_at": "2022-05-13T15:55:58.913107Z",
            "updated_at": "2022-05-13T15:55:58.913121Z",
            "requires_ansible": ">=2.9.10,<2.11"
        },
        {
            "version": "1.0.1-dev6",
            "href": "/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/ansible/netcommon/versions/1.0.1-dev6/",
            "created_at": "2022-05-13T15:55:58.913107Z",
            "updated_at": "2022-05-13T15:55:58.913121Z",
            "requires_ansible": ">=2.9.10,<2.11"
        }
    ]
}

This 100 limit "feels" to me like one tool ansible-galaxy or the other GalaxyNG isn't iterating over pages correctly to find the specified version of a collection that happens to be on a later page. It seems that GalaxyNG is serving these oldest to newest, and since community.vmware==1.17.0 wasn't on the first page (when I had all 316 versions), ansible-galaxy said it couldn't satisfy the requirements.

Now I feel like I should mention that I went back and did some experimenting with my older GalaxyNG 4.2.1 installation. I installed GalaxyNG 4.5.0 in a new VM "side-by-side" with my old install, because I could never get GalaxyNG 4.2.1 to sync my requirements.yml from galaxy.ansible.com. But I went back to 4.2.1 and started with smaller/simpler requirements.yml files. Lo and behold, with 1 or 2 collections in requirements.yml, GalaxyNG 4.2.1 was able to sync. However, if requirements.yml grew to include several collections (and I don't know what the real number is here), GalaxyNG 4.2.1 would fail to fully sync, often throwing some cryptic timeout and/or SSL errors.

I began resource monitoring GalaxyNG 4.2.1 via htop and noticed it was thrashing quite a bit when trying to sync the larger list of requirements. I went back to the GUI to edit the sync requirements again and noticed under the Advanced options, you could adjust the Download Concurrency (default is 10). I reduced that to 2, and kept the larger list of requirements for the sync and kicked it off again. I again monitored resource use via htop and it wasn't nearly as thrasy and, after quite some time, the sync process completed in Success.

Now here's the important bit that might help with root cause? GalaxyNG 4.2.1 apparently doesn't honor the version qualifiers in a requirements.yml from a sync point of view. I used the same requirements.yml that I used in 4.5.0 to limit the number of synced community.vmware versions, and 4.2.1 happily sync'ed all 316 versions anyway. Initially I was happy the sync worked, but then I thought, "Here we go with the version count issue again". Only, that isn't an issue with GalaxyNG 4.2.1. ansible-galaxy was able to install requirements from this 4.2.1 server using ansible==2.9.12, ansible==4.2.0, and ansible==5.8.0.

Below is a little table showing my success/failures across versions of things:

Ansible/GalaxyNG	4.2.1	4.5.0
2.9.12	pass	pass
4.2.0	pass	fail
5.8.0	pass	fail

So this is a very version dependent issue and I'm not sure which tool is actually causing the problem, I'm merely reporting my experiences. Unfortunately, I cannot find a way to open an issue with the GalaxyNG team directly, as they use a different issue tracker.

Thanks for your time in looking!

[bcoca]

for any issues with galaxy.ansible.com go to https://github.com/ansible/galaxy/issues (which also hosts the older galaxy code, but it is still main interface for issues with the site).

https://github.com/ansible/galaxy_ng is the repo for the new code, but as you can see issues are disabled in github, but README has a link to https://issues.redhat.com/projects/AAH/issues

In any case this does look like an issue with the client, which means this is the correct repo to open it in.

[sivel]

Based on the next link, I'd have to guess there is either a misconfiguration in your galaxy_ng config, or a bug in galaxy_ng.

The next link from your curl seems to match what is failing:

/api/galaxy/content/community/v3/plugin/ansible/content/community/collections/index/ansible/netcommon/versions/?limit=100&offset=100

I believe based on what the galaxy team provided us with, that should have instead been:

/api/v3/plugin/ansible/content/community/collections/index/ansible/netcommon/versions/?limit=100&offset=100