From 5d3d7f6ddeebabaeda5154c7bd6e96655d9e8bef Mon Sep 17 00:00:00 2001 From: Aitor Velez Date: Thu, 30 Jan 2025 14:19:16 +0100 Subject: [PATCH 1/4] Update how to deployment Signed-off-by: Aitor Velez --- .../20-orchestrator-deployment.rst | 4 +-- .../walkthrough/30-backend-deployment.rst | 26 ++++++++++++++++++- .../walkthrough/40-connect-organizations.rst | 25 +++++++++++++++--- .../walkthrough/50-frontend-deployment.rst | 5 ++-- 4 files changed, 52 insertions(+), 8 deletions(-) diff --git a/docs/source/how-to/deploying-substra/walkthrough/20-orchestrator-deployment.rst b/docs/source/how-to/deploying-substra/walkthrough/20-orchestrator-deployment.rst index 25584b63d..b93af137f 100644 --- a/docs/source/how-to/deploying-substra/walkthrough/20-orchestrator-deployment.rst +++ b/docs/source/how-to/deploying-substra/walkthrough/20-orchestrator-deployment.rst @@ -126,11 +126,11 @@ To setup TLS, follow these steps: rm orchestrator-cert.csr orchestrator-ca.srl -#. Create a Kubernetes ConfigMap for the CA certificate: +#. Create a Kubernetes Secret for the CA certificate: .. code-block:: bash - kubectl create configmap orchestrator-tls-cacert --from-file=ca.crt=orchestrator-ca.crt + kubectl create secret generic orchestrator-tls-cacert --from-file=ca.crt=orchestrator-ca.crt #. Create a Kubernetes Secret for the orchestrator TLS key and certificate: diff --git a/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst b/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst index 225d51344..96febc3f8 100644 --- a/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst +++ b/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst @@ -49,9 +49,12 @@ To configure your values: host: ORCHESTRATOR_HOSTNAME port: ORCHESTRATOR_PORT mspID: ingen + sameCluster: ORCHESTRATOR_SAME_CLUSTER + | ``ORCHESTRATOR_HOSTNAME`` should be ``orchestrator.cluster-1.DOMAIN`` if you are _outside_ the cluster, but if we are working on ``cluster-1`` we should use its local name ``orchestrator-server.orchestrator`` (following the ``service-name.namespace`` convention). | ``ORCHESTRATOR_PORT`` should be ``443`` if TLS is enabled, otherwise ``80``. + | ``ORCHESTRATOR_SAME_CLUSTER`` should be ``true`` if the backend is in the same cluster as the orchestrator, otherwise ``false``. .. _backend-channel-config: @@ -126,4 +129,25 @@ Deploy the Chart .. code-block:: javascript - {"detail":"Authentication credentials were not provided."} \ No newline at end of file + {"detail":"Authentication credentials were not provided."} + +Execution Problems +================ + +Once everything is deployed, if there are execution problems when adding a function to substra, it can be related with the network policy. + +#. Check the log of the pod ``backend-substra-backend-builder-0`` + + .. code-block:: bash + kubectl logs backend-substra-builder-0 -n ingen + +#. If there there is ```HTTPSConnectionPool(host='10.43.0.1', port=443)``` error, modify the next network policies: + + Remove except content inside ```substra-backend-internet-egress``` network policy + + Add the next lines inside the to section for the ```substra-backend-api-server-egress``` network policy + .. code-block:: yaml + - to: + - ipBlock: + cidr: 0.0.0.0/0 + diff --git a/docs/source/how-to/deploying-substra/walkthrough/40-connect-organizations.rst b/docs/source/how-to/deploying-substra/walkthrough/40-connect-organizations.rst index f2c590e0e..1335bcac2 100644 --- a/docs/source/how-to/deploying-substra/walkthrough/40-connect-organizations.rst +++ b/docs/source/how-to/deploying-substra/walkthrough/40-connect-organizations.rst @@ -10,7 +10,7 @@ Update your Helm values ======================= This guide assume that you have two organization in your network, one named ``ingen`` and the other ``biotechnica``. -The process would be the same if you have more organizations as we have to pair them. For example with three orgs you would repeat it for (org1, org2), (org1, org3) and (org2, org3). +The process would be the same if you have more organizations as we have to pair them. We have also to pair the organization with itself. For example with three orgs you would repeat it for (org1, org1), (org2, org2), (org3, org3), (org1, org2), (org1, org3) and (org2, org3). In this setup we want ``ingen`` to exchange assets with ``biotechnica``. We are assuming that you have two values files with the configuration for your existing deployment, one for each organization named: ``backend-ingen-values.yaml``, ``backend-biotechnica-values.yaml``. @@ -28,7 +28,7 @@ Configure matching values for your 2 :term:`Organizations `: secret: SECRET_ORG1_ORG2 channel: our-channel - | ``SECRET_ORG1_ORG2`` is a password ``biotechnica`` needs to download assets from ``ingen``. + | ``SECRET_ORG1_ORG2`` is a pbkdf2_sha2 encrypted password ``biotechnica`` needs to download assets from ``ingen``. | ``our-channel`` was defined in the :ref:`backend channel configuration ` -- both ``ingen`` and ``biotechnica`` are members of it. #. Create an account for ``ingen`` on ``biotechnica``. @@ -51,7 +51,7 @@ Configure matching values for your 2 :term:`Organizations `: - name: biotechnica secret: SECRET_ORG2_ORG1 - | ``SECRET_ORG2_ORG1`` must naturally be the same as earlier. + | ``SECRET_ORG2_ORG1`` must naturally be the same as earlier, but without pbkdf2_sha2 encryption. #. Configure ``biotechnica`` to use the right password when connecting to ``ingen``. In ``backend-biotechnica-values.yaml`` add the following content under the ``addAccountOperator`` key: @@ -93,6 +93,25 @@ in ``backend-ingen-values.yaml``, and: in ``backend-biotechnica-values.yaml``. +For linking organizations we also provide a small using the small utility on the Substra backend server. Follow these steps on each organization to achive the same result: + +#. Connect to the Substra backend pod: + + .. code-block:: bash + + kubectl exec -it $(kubectl get pod -l "app.kubernetes.io/name=substra-backend-server" -o name) -- /bin/bash + + This opens a shell on the backend server pod. + +#. Create incoming and outgoing organization: + + .. code-block:: bash + + ./manage.py create_incoming_organization "" "" + ./manage.py create_outgoing_organization "" "" + + The utility will encrypt the password authomatically. + Deploy the updated chart ======================== diff --git a/docs/source/how-to/deploying-substra/walkthrough/50-frontend-deployment.rst b/docs/source/how-to/deploying-substra/walkthrough/50-frontend-deployment.rst index b1002e9d4..9601711c9 100644 --- a/docs/source/how-to/deploying-substra/walkthrough/50-frontend-deployment.rst +++ b/docs/source/how-to/deploying-substra/walkthrough/50-frontend-deployment.rst @@ -16,8 +16,9 @@ Tell the backend to set the proper headers for cross-origin resources, by adding .. code-block:: yaml config: - CORS_ORIGIN_WHITELIST: '["substra.cluster-1.DOMAIN"]' # this is a string parsed as a JSON list - CORS_ALLOW_CREDENTIALS: True + CORS_ORIGIN_WHITELIST: '["https://substra.cluster-1.DOMAIN"]' # this is a string parsed as a JSON list + CSRF_TRUSTED_ORIGINS: '["https://substra.cluster-1.DOMAIN"]' + CORS_ALLOW_CREDENTIALS: 'true' # you should already have ALLOWED_HOSTS under "config" Prepare your Helm values From e0626986a188d9a7fe48b51ba8d1c4b9be861035 Mon Sep 17 00:00:00 2001 From: SdgJlbl Date: Thu, 13 Feb 2025 09:30:31 +0100 Subject: [PATCH 2/4] Apply suggestions from code review --- .../deploying-substra/walkthrough/30-backend-deployment.rst | 2 +- .../deploying-substra/walkthrough/40-connect-organizations.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst b/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst index 96febc3f8..36e562273 100644 --- a/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst +++ b/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst @@ -143,7 +143,7 @@ Once everything is deployed, if there are execution problems when adding a funct #. If there there is ```HTTPSConnectionPool(host='10.43.0.1', port=443)``` error, modify the next network policies: - Remove except content inside ```substra-backend-internet-egress``` network policy + Remove all the network policies except the ```substra-backend-internet-egress``` network policy. Add the next lines inside the to section for the ```substra-backend-api-server-egress``` network policy .. code-block:: yaml diff --git a/docs/source/how-to/deploying-substra/walkthrough/40-connect-organizations.rst b/docs/source/how-to/deploying-substra/walkthrough/40-connect-organizations.rst index 1335bcac2..0624818fc 100644 --- a/docs/source/how-to/deploying-substra/walkthrough/40-connect-organizations.rst +++ b/docs/source/how-to/deploying-substra/walkthrough/40-connect-organizations.rst @@ -93,7 +93,7 @@ in ``backend-ingen-values.yaml``, and: in ``backend-biotechnica-values.yaml``. -For linking organizations we also provide a small using the small utility on the Substra backend server. Follow these steps on each organization to achive the same result: +For linking organizations we also provide a small utility on the Substra backend server. Follow these steps on each organization to achieve the same result: #. Connect to the Substra backend pod: From 9b5ddd350b44bfb983e6a066396da1cd657c2c0f Mon Sep 17 00:00:00 2001 From: SdgJlbl Date: Thu, 13 Feb 2025 09:53:32 +0100 Subject: [PATCH 3/4] Fix warnings for CI --- .../deploying-substra/walkthrough/30-backend-deployment.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst b/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst index 36e562273..e1d352aef 100644 --- a/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst +++ b/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst @@ -132,13 +132,14 @@ Deploy the Chart {"detail":"Authentication credentials were not provided."} Execution Problems -================ +================== Once everything is deployed, if there are execution problems when adding a function to substra, it can be related with the network policy. #. Check the log of the pod ``backend-substra-backend-builder-0`` .. code-block:: bash + kubectl logs backend-substra-builder-0 -n ingen #. If there there is ```HTTPSConnectionPool(host='10.43.0.1', port=443)``` error, modify the next network policies: @@ -147,6 +148,7 @@ Once everything is deployed, if there are execution problems when adding a funct Add the next lines inside the to section for the ```substra-backend-api-server-egress``` network policy .. code-block:: yaml + - to: - ipBlock: cidr: 0.0.0.0/0 From 76839b360d97023667bfc59d34863e11182f9978 Mon Sep 17 00:00:00 2001 From: SdgJlbl Date: Thu, 13 Feb 2025 10:04:42 +0100 Subject: [PATCH 4/4] Fix warnings for CI --- .../deploying-substra/walkthrough/30-backend-deployment.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst b/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst index e1d352aef..5ef2c1a9d 100644 --- a/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst +++ b/docs/source/how-to/deploying-substra/walkthrough/30-backend-deployment.rst @@ -146,7 +146,8 @@ Once everything is deployed, if there are execution problems when adding a funct Remove all the network policies except the ```substra-backend-internet-egress``` network policy. - Add the next lines inside the to section for the ```substra-backend-api-server-egress``` network policy + Add the next lines inside the to section for the ```substra-backend-api-server-egress``` network policy: + .. code-block:: yaml - to: