The Data Gateway Application is compatible with the following system configurations:

• docker tag SOURCE_IMAGE_ID LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/IMAGE_NAME:TAG
• Replace SOURCE_IMAGE_ID, LOCATION, PROJECT_ID, REPOSITORY, IMAGE_NAME, and TAG with appropriate values.
• Example command: docker tag 9ed55ef2c3f5 us-central1-docker.pkg.dev/my-gcp-project/my-repo/datagateway-ui:latest

• kubectl create secret docker-registry <secret-name> –docker-server=<registry-url> –docker-username=<your-username> –docker-password=<your-password> -n namespace_name
• example command: kubectl create secret docker-registry my-docker-secret –docker-server=https://index.docker.io/v1/ –docker-username=mydockeruser –docker-password=mydockerpass  -n datagateway
• example command: kubectl create secret docker-registry my-docker-secret –docker-server=https://index.docker.io/v1/ –docker-username=mydockeruser –docker-password=mydockerpass  -n datagateway-secure

3. Include the required password details in the api-secret.yaml file provided in the Helm chart. These should cover the database password, the API keystore password (created earlier), and the SMTP password. These secrets will be used during the deployment process.

 

• kubectl create secret generic <secret-name> –from-file=<filename> -n <namespace>
• Replace <secret-name> with your preferred secret name
• <filename> with the path to your file (e.g., datagateway-secure-keystore.jks)
• <namespace> with the target namespace where the secret should be created.
• Example command: kubectl create secret generic ftps-keystore-secret –from-file=datagateway-secure-keystore.jks -n datagateway-secure

This will generate two files:

• dg_key – the private key

• dg_key.pub – the public key

The output file is named datagateway-server-keystore.jks, but you may use any name as required.

• keytool -importkeystore -srckeystore datagateway-server.pfx -srcstoretype PKCS12 -srcstorepass <<PFX_PASSWORD>> -destkeystore datagateway-server-keystore.jks -deststoretype JKS
  -deststorepass <<JKS_PASSWORD>>

3. Create a Kubernetes secret using the generated keystore file. Replace the filename and namespace with your actual values as needed:

• kubectl create secret generic datagateway-server-keystore-secret –from-file=datagateway-server-keystore.jks -n <namespace>

4. Create a Kubernetes secret from the generated private SSH public key using the following command.

• kubectl create secret generic secret_name –from-file=public_key_name -n datagateway-namespace-name

Replace the placeholders with your actual values:

• secret_name – the name you want to assign to the secret

• private_key_name – the file name of your private key

• datagateway-namespace-name – the target namespace (e.g., where the datagateway component is deployed)

5. Specify the necessary passwords in the server secret file provided in the Helm chart, which will be used during the deployment process.

 

Configuring Helm Charts & Deploying

1. Traverse to the location where the helm chart was untarred, and open values.yaml file in any text editor.
2. Accept the license by setting it to "true".

UI Component

1. Provide the container image name along with the necessary details as shown in the snapshot below. Additionally, define the resource specifications by setting:

   • Resource Limits – the maximum CPU and memory the container can use

   • Resource Requests – the minimum guaranteed CPU and memory to reserve for the container

   Ensure the configuration aligns with your workload requirements and available cluster resources.

2. Add any additional labels you wish to apply to the UI pod by specifying the labels directly below.

 

Affinity controls how pods are scheduled onto nodes based on certain rules and preferences. If you wish to enable affinity, remove the curly braces, uncomment the configurations, and update the settings according to your cluster configuration.

3. Specify the server type and port configuration.

• For SSL-enabled servers, use port 443.

• For non-SSL servers, you may use any preferred port.

• To disable a port, leave the port details blank.

If you wish to add annotations, remove the curly braces and provide the desired configuration below.

4. To use Ingress instead of LoadBalancer or NodePort service types, provide the Ingress configuration details and set enabled to true. This will allow external access to the service through the configured Ingress rules.

 

 

5. To enable SSL for the UI:

• Set enabled to true.

• Provide the Host, along with the TLS certificate and key details.

• Specify the TLS certificate file name and the secret created from it, as configured earlier during the UI prerequisites.

• Similarly, provide the TLS key file name and the corresponding secret name.

• Include the certificate name used and the secret that was created in previous steps.

• Additionally, define the maximum file upload size for the UI using the max_file_size parameter.

Ensure all referenced secrets are available in the correct namespace before deploying.

 

 

6. Provide the Persistent Volume configuration for storing log data. A single Persistent Volume Claim (PVC) is used to collect logs from both the API and server pods; therefore, the access mode must be set to ReadWriteMany.

If you intend to use an existing Persistent Volume, specify its name under the preExistingVolume parameter. Ensure that the volume is correctly configured and accessible within the target namespace.

Note: The storage class used should have RWX access mode access and the volumes should be based on block or network storage.

API Component

1. Provide the container image name along with the necessary details as shown in the snapshot below. Additionally, define the resource specifications by setting:

   • Resource Limits – the maximum CPU and memory the container can use

   • Resource Requests – the minimum guaranteed CPU and memory to reserve for the container

   Ensure the configuration aligns with your workload requirements and available cluster resources.

 

2. Add any additional labels you wish to apply to the API pod by specifying the labels directly below.

 

 

Affinity is used to control how pods are scheduled onto nodes based on certain rules and preferences. If you wish to enable affinity, remove the curly braces, uncomment the configurations, and update the settings according to your cluster configuration.

3. Specify the service type and port configuration:

• To disable a port, leave the port field blank.

• If you wish to add annotations, remove the curly braces and provide the configuration below.

This will allow you to adjust the service settings and customize it as needed for your deployment.

4. Set acceptLicense to true and specify the desired profile for your API to run on.

• If the profile is SAML, ensure that valid SAML details are provided in the upcoming API configurations. This setup the application to use SAML authentication using IDP.

• If the profile is ADG, you can skip providing SAML details as they are not required.

Provide the maximum and minimum heap size values for the API:

• The recommended maximum heap size (-Xmx) should be 50% of the total available RAM.

• Set the minimum heap size (-Xms)Based on your workload requirements, typically starting from a reasonable value (e.g., 1000m).

Ensure that both values are set according to the available resources in your environment for optimal performance.

 

 5. To enable SSL for the API:

• Set enabled to true.

• Provide the Host along with the keystore details generated from the SSL certificates in the API prerequisites.

• Specify the keystore name and the secret name created using the keystore.

• Additionally, provide the secret name containing the keystore password, which was created during the API prerequisites.

Ensure the secrets and certificates are correctly configured

6. Configure the database settings:

• Set enabled to true if the installation is targeting an empty database schema.

• Provide the necessary database information, including the database host, name, and port.

• Specify the database secret name containing the database password, which was created as part of the prerequisites.

Ensure the database details and secrets are correctly configured

 

7. Configure the SMTP details:

• Provide the necessary SMTP server information, such as the server address, port, and any other relevant configuration.

• For the SMTP password, specify the secret name that contains the SMTP password, which was created as part of the API prerequisites.

Ensure the SMTP configuration and secret are properly set for the email functionality to work as expected.

 

9. To enable MFA for user authentication, set ‘enable’ to true and specify the expiration time.

10. Configure the SAML settings to enable Multi-Factor Authentication (MFA) for the application:

• Provide the metadata file name containing the SAML configuration.

• Specify the secret name created from the metadata file.

• Include any other relevant SAML-related details.

• Ensure the profile is set to saml if you are using it for authentication.

Make sure the metadata and secret are correctly configured to integrate MFA with your application.

• kubectl create secret generic secret-name –from-file CustomLogo.png
• Example Command:  kubectl create secret generic logo-secret –from-file CustomLogo.png

Server Component

1. Provide the container image name along with the necessary details as shown in the snapshot below. Additionally, define the resource specifications by setting:

   • Resource Limits – the maximum CPU and memory the container can use

   • Resource Requests – the minimum guaranteed CPU and memory to reserve for the container

   Ensure the configuration aligns with your workload requirements and available cluster resources.

 

 

2. Provide extra labels to the server pod by  directly adding the labels below

 

Affinity is used to control how pods are scheduled onto nodes based on certain rules and preferences. If you wish to enable affinity, remove the curly braces, uncomment the configurations, and update the settings according to your cluster configuration.

3 .Specify the server type and ports for the following protocols:

• SFTP

• FTP

• FTPS

Additionally, provide the passive port range for FTP/FTPS data transfers.

• If you do not wish to use a specific protocol, leave the corresponding details blank.

• Set session affinity to ClientIP for FTP/FTPS.

• For SFTP, set session affinity to None.

Ensure that the configuration aligns with your network and protocol requirements.

4.Provide the maximum and minimum heap size values for the API:

• The recommended maximum heap size (-Xmx) should be 50% of the total available RAM.

• Set the minimum heap size (-Xms)Based on your workload requirements, typically starting from a reasonable value (e.g., 1000m).

Ensure that both values are set according to the available resources in your environment for optimal performance.

 

5. Configure the SMTP details:

• Provide the necessary SMTP server information, such as the server address, port, and any other relevant configuration.

• For the SMTP password, specify the secret name that contains the SMTP password, which was created as part of the server prerequisites.

Ensure the SMTP configuration and secret are properly set for the email functionality to work as expected.

 

6.Provide the following details:

• The public key name that was created as part of the server prerequisites.

• The SSH public key secret name associated with the public key, which was also created during the server prerequisites.

7. Provide the following details:

• The keystore name created for the application.

• The secret name generated from the keystore.

• The secret name containing the keystore password, which was created as part of the server prerequisites.

Ensure that the keystore and associated secrets are correctly configured

 

8. Provide the following details for the Proxy SFTP Tunnel:

• The SFTP Tunnel username.

• The secret name that contains the tunnel password, which was created as part of the server prerequisites.

Ensure that both the username and secret are correctly configured

 

Secure Component

1. Provide the container image name along with the necessary details as shown in the snapshot below. Additionally, define the resource specifications by setting:

   • Resource Limits – the maximum CPU and memory the container can use

   • Resource Requests – the minimum guaranteed CPU and memory to reserve for the container

   Ensure the configuration aligns with your workload requirements and available cluster resources.

 

2. Provide extra labels to the proxy pod by adding the labels below

 

Affinity is used to control how pods are scheduled onto nodes based on certain rules and preferences. If you wish to enable affinity, remove the curly braces, uncomment the configurations, and update the settings according to your cluster configuration.

 

3.Specify the server type and port configurations for the following protocols:

• SFTP

• FTP

• FTPS

Additionally, define the passive port range for FTP/FTPS data transfers.

• If you do not intend to use a particular protocol, leave the corresponding configuration fields blank.

• Set session affinity to "ClientIP" for FTP and FTPS.

• For SFTP, set session affinity to "None".

Ensure all settings are aligned with your network and security requirements.

4. Provide the maximum and minimum heap size values for the API:

• The recommended maximum heap size (-Xmx) should be 50% of the total available RAM.

• Set the minimum heap size (-Xms)Based on your workload requirements, typically starting from a reasonable value (e.g., 1000m).

Ensure that both values are set according to the available resources in your environment for optimal performance.

 

5. Provide the following details:

• The keystore file name and the secret name created from the keystore file.

• The public key file name and the secret name associated with the public key, which were created as part of the secure prerequisites.

Ensure these values are correctly referenced for secure communication

 

6. Provide the Tunnel Username and the corresponding secret name that contains the tunnel password.
These credentials should have been created as part of the secure prerequisites to enable secure tunneling.

7.Provide the following user login policy settings:

• Maximum false login attempts: The number of consecutive failed login attempts allowed before action is taken.

• Maximum reset attempts: The number of times a user can attempt to reset their credentials within a given period.

 

8. Provide the Auto Scaling Configurations:

• Set the target CPU utilization percentage and target memory utilization percentage.
  Based on these thresholds, pods will automatically scale by creating additional replicas as needed.

• You can enable or disable autoscaling for any specific component within the Data Gateway deployment, depending on performance requirements.

Ensure the values reflect your expected load and available cluster resources.

 

9. After making all necessary configurations, save and close the values.yaml file.

Then, begin the installation using the following command:

• helm install <release-name> <chart-directory> -n <namespace> –set datagateway_ui.replicas=0

Replace <release-name>, <chart-directory>, and <namespace> with your actual values.

 

 

10. After all the pods are up except the UI pod, perform the following steps:

1. Map the Load Balancer IPs of the API and UI services to their respective DNS hostnames as defined in the configuration files.

2. Once the DNS mapping is complete and SSL is enabled on the applications, scale up the UI pod using the command below:

• kubectl scale deployment <ui-deployment-name> –replicas=1 -n <namespace>

Replace <ui-deployment-name> and <namespace> with the actual values from your deployment.

 

The deployment is complete.
Please wait a few minutes to allow the applications to initialize and complete their setup.

 

Accessing the Application

1. Open your browser and traverse to the URL which would be the HOST and PORT provided in the configuration of the UI Component.

         2. You can login with your factory credentials which would be superadmin and Expl0re@123 as the Username and Password respectively. 

Note: You can change the password upon login using Change Password.

Astrocyte Data Gateway Monitor is an application designed for monitoring both platform and application performance. It ensures that pods are running as expected and provides visibility into system statistics such as RAM, CPU usage, and thread activity—enabling proactive planning and resource optimization.

Pre-Requisites

• Kubernetes Cluster 

• Container images 

• Helm packages 

• Kibana 

• Grafana 

• Elasticsearch 

• SSL Certificates 

Elasticsearch deployment procedure: Elastic Stack Helm Chart | Elastic Cloud on Kubernetes [2.16] | Elastic 

Kibana deployment procedure: Elastic Stack Helm Chart | Elastic Cloud on Kubernetes [2.16] | Elastic 

Grafana deployment procedure:  Deploy Grafana using Helm Charts | Grafana documentation 

Container Images

The Product team would provide you the Data Gateway bucket details, which can be used to download the images. 

Download the DG Monitor  .tar image file: DG_MONITOR.tar. Upload them to a Linux jump server of your choice. Once uploaded, load each image into Docker on the jump server using the following command: 

                               docker load -i DG_MONITOR.tar 

Helm Details

The Data Gateway helm package is a tar file, and the contents of the tar file include various files and directories required for installation and configuration, such as: 

• Chart.yaml file is a component that defines metadata about the chart. It provides information that helps Helm manage, package, and deploy the product.  

• product.info file is a custom metadata file that defines the product-related information. 

• values.yaml file is the configuration file that defines customizable parameters for the data gateway chart’s deployment. It acts as the input file where users can override default settings for their applications without modifying the chart itself. 

• LICENSE file contains detailed information about the product license, including the terms and conditions governing the use, distribution, and modification of the application.  

• README.md file provides comprehensive step-by-step instructions for deploying the Data Gateway application in a Kubernetes environment. It includes information on the required prerequisites, installation procedures, configuration guidelines, and best practices to ensure a seamless deployment. The document outlines how to set up the necessary Kubernetes resources such as deployments, services, ingress controllers, and environment-specific configurations. 

• The Secrets folder contains sensitive files used by the Data Gateway application to securely store confidential information such as passwords and authentication credentials. 

• The Secrets folder contains three secret files:  dg-monitor-secret. Each file includes fields to input passwords and other sensitive information required by the application. These secrets are securely injected into the Data Gateway monitor application through Kubernetes Secrets, ensuring sensitive data is not exposed in plain text. 

• The Templates folder contains the Kubernetes resource definitions written in YAML format. These templates are used to dynamically generate Kubernetes manifests based on the values specified in the values.yaml file. Helm templates allow developers to create reusable, configurable, and parameterized deployments without hardcoding environment-specific configurations. 

Cluster Setup

Refer to this for more information about Cluster Setup. 

Installing CLI

Refer to this for more information about CLI Installation. 

Data Gateway monitor – Image Details

Image Extracting and Loading

1. Download the Data Gateway Images tar files from the provided location and move it to the Jump Server. 
2. Load the images to Docker using the below command. Refer to docker push | Docker Docs for more information on Docker load to image repository. 

• docker load DG_monitor.tar

Once loaded, push it to the image repo of your choice

3. Upload the helm chart to the jump server and  Untar the helm charts using the command below 

• tar -xvf helm-package-name

4.Create a namespace in the Kubernetes cluster for DG Monitor.

        5. Set the Kubernetes current context to the datagateway monitor namespace using the following                   command:

• kubectl config set-context –current –namespace=dg-monitor

Creating Secrets

1. This command will create an image pull secret in the dg-monitor namespace, allowing Kubernetes to pull images from the specified registry.

• kubectl create secret docker-registry <secret-name> –docker-server=<registry-url> –docker-username=<username> –docker-password=<password> -n dg-monitor

Replace the following placeholders with your actual values:

• <secret-name>: Desired name for the secret.

• <registry-url>: URL of your Docker registry.

• <username>: Your Docker registry username.

• <password>: Your Docker registry password.

2 provide the required credentials for the configuration:

• Datagateway Database Password

• Elasticsearch Login Password

• Swagger API Password (to secure access to Swagger)

• Keystore Password (if SSL is enabled for the Swagger API)

• Elasticsearch Server Integration Password (used by the server application to send data to Elasticsearch)

provide the following credentials and details required for configuration:

• Elasticsearch Password used by the client application to send data to Elasticsearch.

• FTP, SFTP, and FTPS Passwords used by the Adapter Monitor for authenticating with the Datagateway during monitoring.

• Namespace in which the Datagateway server application is deployed.

To enable Datagateway Monitor Swagger with SSL, follow the steps below:

Create a PFX file using your domain certificate.
Provide the private key, certificate, and a desired password using the placeholders below:

• <<PRIVATE_KEY>>: Path to your private key file

• <<CERT_FILE>>: Path to your certificate file

• <<PASSWORD>>: Password to secure the PFX file

You can name the output file as desired; in this example, we’re using datagateway-monitor.pfx.

• openssl pkcs12 -export -out datagateway-monitor.pfx -inkey <<PRIVATE_KEY>> -in <<CERT_FILE>> -passout pass:<<PASSWORD>>
• Replace the placeholders with your actual file paths and password. Ensure the password is stored securely for later use in your Kubernetes secrets.

Create a Java Keystore (JKS) file using the previously generated PFX file.

Use the datagateway-monitor.pfx file created in the earlier step. You can name the output JKS file as desired; in this example, we’ll use datagateway-monitor-keystore.jks.

• keytool -importkeystore -srckeystore datagateway-monitor.pfx  -srcstoretype PKCS12  -srcstorepass <<PFX_PASSWORD>> -destkeystore datagateway-monitor-keystore.jks  -deststoretype JKS -deststorepass <<JKS_PASSWORD>>

Make sure <<PFX_PASSWORD>> and <<JKS_PASSWORD>> are consistent with the values used in your Kubernetes secrets a

create a secret with the keystore (jks) using the below command

• kubectl create secret generic dg-keystore-secret –from-file=keystore.jks=datagateway-keystore.jks

Replace dg-keystore-secret, keystore.jks, datagateway-keystore.jks with your actual secret name, desired key name, file path, and namespace.

Helm configuration

1. Open values.yaml in any text editor of your choice
2. Provide the image pull secret created as a part of prerequisites
3. provide the image name, image tag, pull policy, and restart policy for the container configuration.
4. Please provide the resource requests and limits for the DG Monitor Server pod. 

5. Define the service type used to expose the Datagateway Monitor Server pod’s Swagger API, along with the ports it should be accessible on.
   By default, the service type is set to ClusterIP.
   If you wish to access the Swagger API externally, consider using NodePort or LoadBalancer.

   Additionally, to apply any required annotations (e.g., for ingress controllers or cloud provider configurations), remove the curly braces and insert the appropriate key-value pairs in the annotations section.

   

6. Provide the keystore configuration details required for enabling SSL for Swagger:

   • Specify the name of the .pfx keystore file created during the prerequisites step.

   • Provide the name of the Kubernetes secret that was created from this .pfx file.

   • Include the name of the Kubernetes secret that contains the password for the keystore.

   • Specify the keystore type (default is PKCS12, unless a different format was used).

7. Set liquibase to true if you are installing the DG Monitor for the first time. Provide the database details such as the JDBC URL, username, and the secret name containing the DB_PASS, which should have been created as part of the prerequisites. Also, specify the time zone (e.g., EST) in which the application should run. 
8. Set enable to true if you wish to store DG Monitor logs. If enabled, provide the log storage capacity, storageClass, and access modes. Alternatively, if you prefer to use a pre-existing persistent volume, you can do so by specifying the preExistingVolume name. 
9. The DG Monitor client requires a service account to monitor Kubernetes metrics. You can set enable to true to automatically create a new service account. Alternatively, set enable to false to use an existing service account, and provide its name using the serviceAccountName field. 
10. Provide the namespace where DataGateway is deployed, as DG Monitor depends on the DataGateway server logs PVC to monitor the application logs. Ensure the correct DataGateway namespace is specified. And provide the resource limits and requests. 
11. Provide the Elasticsearch details, including the index name, URL, username, SSL configuration, and the operating system on which Elasticsearch is running. Also, specify the secret name containing the Elasticsearch credentials, which should have been created as part of the prerequisites. 
12. Set enable to true to monitor the FTP endpoint. If enabled, provide the FTP hostname, port, username, the ftpsPasswordSecret (created as part of the prerequisites), and the mailbox path. Also, specify schedule_ftp_seconds to define how frequently the FTP endpoint should be monitored. 
13. Set enable to true to monitor the FTPS endpoint. If enabled, provide the FTPS hostname, port, username, the ftpsPasswordSecret (created as part of the prerequisites), and the mailbox path. Also, specify schedule_ftps_seconds to define how frequently the FTPS endpoint should be monitored. 
14. Set enable to true to monitor the SFTP endpoint. If enabled, provide the SFTP hostname, port, username, the sftpPasswordSecret (created as part of the prerequisites), and the mailbox path. Also, specify schedule_sftp_seconds to define how frequently the SFTP endpoint should be monitored. 
15. Provide the filename of the DataGateway server log file. DG Monitor uses this log file to collect data points and generate monitoring insights based on the keywords defined in proxy_keyword_list. Default values are already included—do not modify them. If you wish to monitor additional error logs in the DataGateway server logs, you can add custom keywords to proxy_keyword_list, separated by commas and enclosed in double quotes. Provide the names of the schedulers to be monitored in scheduler_list, enclosed in single quotes and separated by commas. Provide the schedule_thread_seconds value, which defines how frequently the system should check for changes in thread count. Also, provide the schedule_datapoints_seconds value, which determines how often data points should be monitored.) and provide the username for the DataGateway Monitor server application, along with the secret name that contains the Swagger password, which should have been created as part of the prerequisites. 
16. Provide the scheduler_seconds value under the schedulers section to specify how frequently schedulers should be monitored. Similarly, provide the scheduler_seconds under the certificate_manager section to define how often certificates and related logs should be checked. In the authenticated_user section, specify the schedule_seconds value to determine how frequently the script should check for authenticated users. Also, provide the server log file name. Do not modify the error_keyword_list.
17. In the datapoint section, provide the namespace and the name of the DataGateway logs PVC. 
18. Provide the namespaces that DG Monitor should monitor for pods. Also, specify the schedule_system_stats_seconds value to define how frequently system stats should monitor the pods at regular intervals.
19. Remove the curly braces and uncomment the configuration. Then, add any additional configuration below securityContext to include custom session affinity settings. 
20. Add any affinity-related configurations under the affinity section. If you want to use the predefined configuration, simply remove the curly braces, uncomment it, and modify it as needed to fit your requirements. And save the values .yaml file. 
21. Add any affinity-related configurations under the affinity section. If you want to use the predefined configuration, simply remove the curly braces, uncomment it, and modify it as needed to fit your requirements. And save the values .yaml file.
22. Install the dg monitor using the below command 

• helm install dg-monitor <path-to-chart-directory> -n <namespace>

replace the placeholder with

• dg-monitor: A release name you want to assign (you can choose a different one).

• <path-to-chart-directory>: The local path to your DG Monitor Helm chart.

• <namespace>: The namespace you created for the monitor, e.g., dg-monitor.

To validate the installation of the DG Monitor and check the logs inside the pod for any errors, you can follow these steps:

Get the Pod name: First, get the name of the DG Monitor pod by running the following command:

• kubectl exec -it <pod-name> -n <namespace> — /bin/bash

Check the logs: Once inside the pod, navigate to the log directory and check for any errors: