fix: Tiltfile refactoring to improve performance and reduce maintenance

README.md

@@ -18,17 +18,24 @@ This document provides guidance on running OpenCRVS both locally (on your PC or
 - [terraform-templates](terraform-templates), configuration templates to build cloud environments for OpenCRVS on different providers
 - [examples](examples), pre-defined values for helm charts and additional documentation and deployment scenarios for OpenCRVS
 
-# Quickstart localhost
+# Demo / Quickstart system requirements
 
 Check quickstart instructions how to deploy OpenCRVS to existing docker-desktop or minikube Kubernetes cluster running on your laptop at [charts/opencrvs-services](charts/opencrvs-services/README.md#-quickstart)
 
-# Examples
 
-Single server deployment flow is example is describe at [examples/dev](./examples/dev/README.md)
+# Development environment setup
 
-# System requirements
+Kubernetes is the easiest option to run OpenCRVS locally on your PC or Laptop and test all features and functionality.
+Before running make sure all hardware and software requirements are met.
 
-## Desktop / Quickstart system requirements
+Once you make sure your development environment is ready for running OpenCRVS we are recommending you start from "For OpenCRVS DevOps" configuration and get familiar with all tools used to deploy OpenCRVS locally (tilt, kubectl, helm). In that particular configuration all docker images are pulled from our registry and OpenCRVS application is starting with Falajaland demo data. No additional actions are needed from your side.
+
+
+The OpenCRVS team uses [Tilt](https://tilt.dev/) to manage the local development environment. Depending on your role and development needs, the following configurations (Tiltfiles) are available:
+
+- [DevOps developers](#for-opencrvs-devops), This basic configuration is designed for Helm chart development. Tilt uses official OpenCRVS release images along with the Farajaland demo data. Docker images are pulled from the OpenCRVS container registry.
+- [Country config developers](#for-opencrvs-country-config-developers), In this setup, OpenCRVS Core images are pulled from the OpenCRVS container registry. The Country Config image is built locally using Tilt's live update feature, so your code changes are reflected almost immediately. Typically, you’ll be working with your own fork of the Country Config repository.
+- [Core developers](#for-opencrvs-core-developers), This configuration builds OpenCRVS Core images locally with live updates enabled, allowing near-instant reflection of code changes. By default, the Country Config image is pulled from the OpenCRVS container registry. If you maintain your own fork of the Country Config repository and container registry, you should update the Tiltfile to use your own registry.
 
 ### Hardware requirements
 - 16G RAM
@@ -51,7 +58,7 @@ Single server deployment flow is example is describe at [examples/dev](./example
 
 ---
 
-### Docker engine with Kubernetes cluster
+### Docker engine and Kubernetes cluster
 
 #### Docker Desktop (with Kubernetes enabled)
 
@@ -78,15 +85,90 @@ fs.inotify.max_user_watches = 65536
 If you already have minikube cluster running, please recreate it (delete/start) to apply changes properly/
 
 Start minikube with unlimited amount of memory:
-```
-minikube start --memory=max
+```bash
+minikube start --cpus=8 --memory=max --ports=80:30080
 ```
 
-Start load balancer (tunnel) on localhost to proxy requests from your browser inside minikube cluster:
-```
-minikube tunnel -c --bind-address='127.0.0.1'
-```
+- `--cpus`: Number of CPUs allocated to Kubernetes. Use "max" to use the maximum number of CPUs
+- `--memory`: Amount of RAM to allocate to Kubernetes (format: <number>[<unit>], where unit = b, k, m or g).
+- `--ports`: List of ports that should be exposed (docker and podman driver only)
+
 
 ---
 
 **NOTE:** Any other Kubernetes solution for desktop should work as well. Please check to LoadBalancer and kubernetes services setup if you are not able to access service.
+
+
+## For OpenCRVS DevOps
+
+1. Clone this repository:
+   ```
+   git clone https://github.com/opencrvs/infrastructure
+   ```
+2. Run:
+   ```
+   tilt up
+   ```
+3. Navigate to [http://localhost:10350/](http://localhost:10350/)
+4. Run [Data seed](#initial-data-seeding-with-tilt) resource
+5. Once all container images are up and running your environment will be available at http://opencrvs.localhost
+
+
+## For OpenCRVS Country Config Developers
+
+1. Clone OpenCRVS Country Config repository:
+
+    For county config use:
+    ```bash
+    git clone https://github.com/opencrvs/opencrvs-countryconfig
+    ```
+    For your own fork use:
+    ```bash
+    git clone [email protected]:<your-github-account>/<your-repository>.git
+    ```
+2. Run Tilt:
+    ```bash
+    tilt up
+    ```
+3. Navigate to [http://localhost:10350/](http://localhost:10350/)
+4. Run [Data seed](#initial-data-seeding-with-tilt) resource
+5. Once all container images are up and running your environment will be available at http://opencrvs.localhost
+
+
+## For OpenCRVS Core Developers
+
+1. Clone the OpenCRVS Core repository:
+    ```bash
+    git clone [email protected]:opencrvs/opencrvs-core.git
+    ```
+2. Run Tilt:
+    ```bash
+    tilt up
+    ```
+3. Navigate to [http://localhost:10350/](http://localhost:10350/)
+4. Run [Data seed](#initial-data-seeding-with-tilt) resource
+5. Once all container images are up and running your environment will be available at http://opencrvs.localhost
+
+---
+
+## Initial data seeding with tilt
+
+This task should run only once on fresh environment after environment installation.
+
+1. Navigate to [http://localhost:10350/](http://localhost:10350/)
+2. Scroll to section `2.Data-tasks` and find resource `Reset database`
+3. Run resource using reload button
+   ![](doc/images/seed-data.png)
+4. Once data seeding completed you will be able to login using default credentials, see [4.1.4 Log in to OpenCRVS locally](https://documentation.opencrvs.org/setup/3.-installation/3.1-set-up-a-development-environment/3.1.4-log-in-to-opencrvs-locally)
+
+## Reset database and Seed data with tilt
+
+1. Navigate to [http://localhost:10350/](http://localhost:10350/)
+2. Scroll to section `2.Data-tasks` and find resource `Reset database`
+3. Run resource using reload button
+   ![](doc/images/reset-data.png)
+4. Once data reset completed you will be able to login using default credentials, see [4.1.4 Log in to OpenCRVS locally](https://documentation.opencrvs.org/setup/3.-installation/3.1-set-up-a-development-environment/3.1.4-log-in-to-opencrvs-locally).
+
+# Server setup
+
+## [Single server deployment steps](./examples/dev/README.md)

Tiltfile

@@ -1,38 +1,31 @@
+############################################################
 # Tiltfile for OpenCRVS
-
 ############################################################
 # ℹ️ Please check readme at: https://github.com/opencrvs/infrastructure/tree/develop
-############################################################
-# Load extensions for namespace and helm operations
-load('ext://namespace', 'namespace_create', 'namespace_inject')
-load("../infrastructure/tilt/lib.tilt", "copy_secrets", "reset_environment", "seed_data")
-include('./tilt/common.tilt')
+load('../infrastructure/tilt/opencrvs.tilt', 'setup_opencrvs')
 
 ############################################################
 # ⚙ CONFIGURATION SECTION: User-defined Variables
 ############################################################
 # NOTE: You could take any value from https://github.com/orgs/opencrvs/packages
 # Be careful, because not all core + countryconfig images are compatible with each other,
 # especially if you are using "develop" tag.
-# Usually, official release images, e/g v1.8.0, are compatible with each other
+# Usually, official release images, e/g v1.9.0, are compatible with each other
 # One of working combination is:
-# - core: da3ab7b
-# - opencrvs/ocrvs-farajaland: efc9b7a
+# - core: v1.9.0-beta-6
+# - opencrvs/ocrvs-countryconfig: v1.9.0-beta-6
 
-core_images_tag = "develop"
+core_images_tag = "v1.9.0-beta-6"
 
 # Countryconfig/Farajaland image repository and tag
 # Usually image repository value (countryconfig_image_name) is your repository on DockerHub
 # If for some reason you don't have DockerHub account yet, please create you local registry
 # (see: https://medium.com/@ankitkumargupta/quick-start-local-docker-registry-35107038242e)
 # If you would like to use Farajaland demo image, please use:
-countryconfig_image_name="opencrvs/ocrvs-farajaland"
-# Mosip integration is hardcoded in the Farajaland demo image,
-# Please also set mosip_enabled to True in the configuration file
-mosip_enabled = True
+countryconfig_image_name="opencrvs/ocrvs-countryconfig"
 # If you would like to start with sample countryconfig image, please use:
 # countryconfig_image_name="opencrvs/ocrvs-countryconfig"
-countryconfig_image_tag="develop"
+countryconfig_image_tag="v1.9.0-beta-6"
 
 # Namespaces:
 # - opencrvs-deps-dev, dependencies namespace
@@ -45,7 +38,7 @@ opencrvs_namespace = 'opencrvs-dev'
 # - Setup MinIO admin user and password
 # - Configure Redis users
 # - Sync passwords between dependencies and OpenCRVS services
-security_enabled = True 
+security_enabled = False 
 # Monitoring enabled:
 # Enable Kibana and all other components
 monitoring_enabled = False
@@ -57,69 +50,12 @@ update_settings(max_parallel_updates=5)
 ############################################################
 # Deploy workloads:
 ############################################################
-
-# Create namespaces:
-namespace_create(dependencies_namespace)
-namespace_create(opencrvs_namespace)
-
-# Select configuration files for dependencies and OpenCRVS services
-print("🔑 Deploying OpenCRVS and Dependencies in {} mode".format('Secure' if security_enabled else 'Non-secure'))
-if not security_enabled:
-  print("""  - Minio admin user/password: minioadmin/minioadmin
-  - Authentication for Redis, MongoDB and Elasticsearch is disabled""")
-if security_enabled:
-    deps_configuration_file = './examples/localhost/dependencies/values-dev-secure.yaml'
-    opencrvs_configuration_file = './examples/localhost/opencrvs-services/values-dev-secure.yaml'
-else:
-    deps_configuration_file = './examples/localhost/dependencies/values-dev.yaml'
-    opencrvs_configuration_file = './examples/localhost/opencrvs-services/values-dev.yaml'
-
-######################################################
-# OpenCRVS Dependencies Deployment
-print("🚀 Deploying dependencies: mongo, minio, elasticsearch...")
-dependencies_chart_path = './charts/dependencies'
-k8s_yaml(helm(dependencies_chart_path,
-  namespace=dependencies_namespace,
-  values=[deps_configuration_file],
-  set=[
-    "monitoring.enabled={}".format(monitoring_enabled).lower()
-  ]))
-
-######################################################
-# OpenCRVS Deployment
-print("🚀 Deploying services: auth, events, gateway...")
-opencrvs_chart_path = './charts/opencrvs-services'
-opencrvs_values = [opencrvs_configuration_file]
-if mosip_enabled:
-    opencrvs_values.append('./examples/localhost/opencrvs-services/values-mosip.yaml')
-k8s_yaml(
-  helm(opencrvs_chart_path,
-       namespace=opencrvs_namespace,
-       values=opencrvs_values,
-       set=[
-        "image.tag={}".format(core_images_tag),
-        "countryconfig.image.name={}".format(countryconfig_image_name),
-        "countryconfig.image.tag={}".format(countryconfig_image_tag)
-        ]
-      )
+setup_opencrvs(
+    core_images_tag=core_images_tag,
+    countryconfig_image_name=countryconfig_image_name,
+    countryconfig_image_tag=countryconfig_image_tag,
+    security_enabled=security_enabled,
+    monitoring_enabled=monitoring_enabled,
 )
 
-if mosip_enabled:
-    print("🚀 Deploying mosip services...")
-    mosip_chart_path = './charts/opencrvs-mosip'
-    k8s_yaml(
-      helm(mosip_chart_path,
-           namespace=opencrvs_namespace,
-           values=[opencrvs_configuration_file])
-    )
-
-#######################################################
-# Add Data Tasks to Tilt Dashboard
-reset_environment(opencrvs_namespace, opencrvs_configuration_file)
-
-seed_data(opencrvs_namespace, opencrvs_configuration_file)
-
-if security_enabled:
-    copy_secrets(dependencies_namespace, opencrvs_namespace)
-
 print("✅ Tiltfile configuration loaded successfully.")

charts/opencrvs-services/templates/postgres-on-update-analytics-job.yaml

@@ -1,4 +1,4 @@
-{{- if or (eq .Values.postgres.auth_mode "auto") (eq .Values.postgres.auth_mode "disabled") }}
+---
 apiVersion: batch/v1
 kind: Job
 metadata:
@@ -81,4 +81,3 @@ spec:
             name: postgres-on-update-script
             defaultMode: 0755
       restartPolicy: "OnFailure"
-{{- end }}

charts/opencrvs-services/templates/postgres-on-update-configmap.yaml

@@ -1,4 +1,4 @@
-{{- if or (eq .Values.postgres.auth_mode "auto") (eq .Values.postgres.auth_mode "disabled") }}
+---
 apiVersion: v1
 data:
   on-deploy.sh: |
@@ -13,4 +13,3 @@ metadata:
     "helm.sh/hook": pre-install,pre-upgrade
     "helm.sh/hook-weight": "0"
   name: postgres-on-update-script
-{{- end }}

charts/opencrvs-services/templates/postgres-on-update-core-job.yaml

@@ -1,4 +1,4 @@
-{{- if or (eq .Values.postgres.auth_mode "auto") (eq .Values.postgres.auth_mode "disabled") }}
+---
 apiVersion: batch/v1
 kind: Job
 metadata:
@@ -80,4 +80,3 @@ spec:
             name: postgres-on-update-script
             defaultMode: 0755
       restartPolicy: "OnFailure"
-{{- end }}

examples/localhost/opencrvs-services/values-dev.yaml → examples/localhost/opencrvs-services/values.yaml

@@ -14,4 +14,7 @@ pdb:
 # Dashboards is disabled by default to allow easy deployment on Apple Silicon
 # If you are using x86 device, feel free to enable:
 dashboards:
-  enabled: false
+  enabled: false
+
+# Expose services on Node port
+service_type: LoadBalancer

examples/localhost/traefik/values.yaml

@@ -2,16 +2,16 @@
 namespaceOverride: "traefik"
 logs:
   general:
-    level: "DEBUG"
-    format: "common"
+    # "TRACE", "DEBUG", "INFO", "WARN", "ERROR", "FATAL", "PANIC"
+    level: "INFO"
+    # format: "common" # For local environment
+    format: "json" # For server environment
   access:
     # -- To enable access logs
     enabled: true
 ingressRoute:
   dashboard:
     enabled: false
-    entryPoints: ["web"]
-    matchRule: Host(`traefik.opencrvs.localhost`) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))
 # Be explicit that we only use CRDs, not ingress/gw support
 providers: 
   kubernetesCRD:
@@ -23,18 +23,13 @@ providers:
 service:
   enabled: true
   single: false
-  type: LoadBalancer
+  type: NodePort
 
 ports:
-  traefik:
-    hostPort: 8080
   web:
-    asDefault: true
-    hostPort: 8000
-  websecure:
-    asDefault: false
-    expose:
-      default: false
-    tls:
-      enabled: false
+    port: 8000
+    hostPort: 80
+    protocol: TCP
+    nodePort: 30080
+
 tlsStore: {}