11. Endpoint Management Module

The Endpoint Management Module in Data Gateway is a robust feature that enables users to seamlessly set up partnerships using a diverse range of protocols and cloud storage options. This module supports protocols such as SFTP, FTP, FTPS, along with various cloud storage providers including GCS, S3, AZURE, and IBM-OBJECT-STORAGE.

11.1 Create Endpoint

Endpoints can be set up through industry certified protocols and Data Gateway enhances it by providing cloud storage options tailored to the requirements. The application takes care of setting up the appropriate cloud storage buckets. 1. Endpoint can be created from Endpoint Management -> Create Endpoint

2. Provide the Endpoint details.

  • Endpoint Name: Provide the Endpoint Name.
  • Endpoint ID: Provide a valid Endpoint ID to identify the Endpoint.
  • Email: Provide the email ID of the Endpoint contact for reference.
  • Phone: Provide the phone number of the Endpoint Contact.
  • Protocol: Choose from FTP, SFTP, FTPS, etc.
  • Status: Provide the status of the endpoint, whether it is Active or Inactive.
  • Storage Type: Choose from GCS, S3, AZURE, IBM-OBJECT-STORAGE.
  • Service Account Name: Choose from the account which was set up earlier. Refer to 8. Cloud Configurations for more information.
  • Field 1, Field 2: You can provide additional custom metadata which you would like to store.
3. The Protocol selection would provide the screen with additional details for the protocol transmission. The information related to the protocol details are provided in the upcoming topic. You can click here to go directly to the Protocol Section.

11.2 Manage Endpoint

1. Endpoints can be managed from Endpoint Management -> Manage Endpoint 2. It gives you the list of all Endpoints, from which you can activate/deactivate, edit, delete using the provided action buttons on the right side

11.3 Protocols

Data Gateway supports multiple protocols, including FTP, FTPS, and SFTP, facilitating secure and reliable file transfers.

11.3.1 FTP (File Transfer Protocol)

FTP (File transfer Protocol) is a communication protocol utilizing the client server connection model. In Data Gateway, it can be used to act both as a Server(Push-Pull) and Client(Pull-Push). The configuration can be done for all combinations of PULL and PUSH. The connection is performed from the Cloud to the External Parties in Data Gateway through secure layer implemented in the Secure Component. 

Pull-Push

This is where Data Gateway acts as the Client and connects to the Endpoint provided.

  • Connection Type: Select between FTP Active and Passive modes, facilitating server-client data transfers. 
  • Remote Host: Host Name of the FTP Server to which the connection is established. 
  • Remote Port: Port number of the FTP Server for the connection. 
  • Transfer Type: Type of the transfer (Binary / ASCII). 
  • Username: Username of the remote FTP server. 
  • Password: Password of the remote FTP server. 
  • No. Of Retries: Number of times a user can attempt to connect to the server. 
  • Retry Interval: Time gap to attempt to access the server if the connection fails. 
  • Remote file pattern: Regex pattern of the files to be pulled from the remote server. 
  • Delete after collection: Check box to delete the file or not in remote FTP server once it is pulled. 
  • Pickup Directories: Path of the directories in remote FTP server from where the files need to be pulled. (Pull from Partner) 
  • Drop Directories: Path of the directories in remote FTP server to which the files need to be pushed. (Push to Partner) 
  • Polling Interval: Interval in which the session to be created and files need to be collected. 
  • Archive File Interval (Days): No of days the file should be archived on deletion. The files will be archived in the cloud storage as non-current version and would be used for file replays. 
Push-Pull

This is where Data Gateway acts as the Server and allows External Parties to connect for File Exchanges.

  • Username: Username using which the session to be opened to data gateway. 
  • Password: Password for authentication. 
  • IP Range: Range of IP addresses to allow. 
  • Rate Limit: No of concurrent sessions to be opened. 
  • Virtual Root: Provide the Path which would act as the first level of the directory path for the user. 
  • Pickup Directory: Directory from the where the files need to be pulled by the Partner. (Pull from Gateway) 
  • Drop Directory: Directory to which the files need to be pushed. (Push to Gateway) 
  • Permission: Select user folder permissions, like READ DATA, WRITE DATA, APPEND DATA. 
  • Archive File Interval (Days): No of days the file should be archived on deletion. The files will be archived in the cloud storage as non-current version and would be used for file replays. .
Push – Push Scenario

This is where Data Gateway acts as the Server and allows External Parties to connect for receiving File Exchanges and acts as the Client to send Files back to External.
This configuration would be the combination of the above two configurations (Push-Pull and Pull-Push).

11.3.2 FTPS (FTP Secure)

FTPS (File transfer Protocol Secure) is a communication protocol utilizing the client server connection model, along with SSL Certificates. In Data Gateway, it can be used to act both as a Server(Push-Pull) and Client(Pull-Push). The configuration can be done for all combinations of PULL and PUSH. The connection is performed from the Cloud to the External Parties in Data Gateway through secure layer implemented in the Secure Component. 

Pull-Push

This is where Data Gateway acts as the Client and connect to the Endpoint provided.

  • Connection Type: Select between FTPS Active and Passive modes, facilitating server-client data transfers. 
  • Remote Host: Host Name of the FTPS Server to which the connection is established. 
  • Remote Port: Port number of the FTPS Server for the connection. 
  • Transfer Type: Type of the transfer (Binary / ASCII). 
  • Username: Username of the remote FTPS server. 
  • Password: Password of the remote FTPS server. 
  • No. Of Retries: Number of times a user can attempt to connect to the server. 
  • Retry Interval: Time gap to attempt to access the server if the connection fails. 
  • Remote file pattern: Regex pattern of the files to be pulled from the remote server. 
  • Delete after collection: Check box to delete the file or not in remote FTPs server once it is pulled. 
  • Pickup Directories: Path of the directories in remote FTPs server from where the files need to be pulled. (Pull from Partner) 
  • Drop Directories: Path of the directories in remote FTPs server to which the files need to be pushed. (Push to Partner) 
  • Polling Interval: Interval in which the session to be created and files need to be collected. 
  • Archive File Interval (Days): No of days the file should be archived on deletion. The files will be archived in the cloud storage as non-current version and would be used for file replays. 
  • Upload Cert: Upload remote FTPS server’s certificate. 
Push-Pull:
  • Username: Username using which the session to be opened to data gateway. 
  • Password: Password for authentication. 
  • IP Range: Range of IP addresses to whitelist. 
  • Rate Limit: No of concurrent sessions to be opened. 
  • Virtual Root: Mapping for the endpoint that serves as the root directory. 
  • Pickup Directory: Directory from the where the files need to be pulled by the Partner. (Pull from Gateway) 
  • Drop Directory: Directory to which the files need to be pushed. (Push to Gateway) 
  • Archive File Interval (Days): No of days the file should be archived on deletion. The files will be archived in the cloud storage as non-current version and would be used for file replays. 
  • Permission: Select user folder permissions, like READ DATA, WRITE DATA, APPEND DATA. 
Push-Push Scenario

This is where Data Gateway acts as the Server and allows External Parties to connect to our FTPS Server for receiving File Exchanges and acts as the Client to send Files back to External by connecting to External Parties FTPS Server. 

This configuration would be the combination of the above two configurations (Push-Pull and Pull-Push).

11.3.3 SFTP (SSH File Transfer Protocol)

SFTP (SSH File transfer Protocol) is a widely used communication protocol utilizing the client server connection model, involving Private/Public Key Authentication Mechanism. In Data Gateway, it can be used to act both as a Server(Push-Pull) and Client(Pull-Push). The configuration can be done for all combinations of PULL and PUSH. The connection is performed from the Cloud to the External Parties in Data Gateway through secure layer implemented in the Secure Component. 

 

Pull-Push

This is where Data Gateway acts as the Client and connect to the Endpoint provided.

  • Remote Host: Host Name of the SFTP Server to which the connection is established. 
  • Remote Port: Port number of the SFTP Server for the connection. 
  • Preferred Mac Algorithm: Select the mac algorithm from the drop down (HMAC-MD5, HMAC-SHA1, HMAC-SHA2-256) MAC algorithm is used to ensure data integrity and authenticity during file transfer. 
  • Remote User: Username of the remote SFTP server. 
  • SSH Password:  Password of the remote SFTP server. 
  • Character Encoding: In SFTP ensures the text data is correctly interpreted and displayed across different systems and platforms. Usually UTF-8 or UTF-16. 
  • Connection Retry count: Number of times a user can attempt to connect to the server. 
  • Retry Delay: Time gap to attempt to access the remote server if the connection fails. 
  • Preferred cipher: The preferred cipher in SFTP specifies the encryption algorithm used to secure data during transfer. Select the preferred ciphers from drop down list. 
  • Auth Type: Select the authentication type as Password or Public Key. 
  • In case of Public Key, Data Gateway will create the Key and share the public part to the Endpoint email address.  
  • Response timeout: Response timeout in SFTP defines the duration the client waits for a response from the server before considering the connection unresponsive. 
  • Remote file pattern: Regex pattern of the files to be pulled from the remote server. 
  • Delete after collection: Check box to delete the file or not in remote SFTP server once it is pulled. 
  • Pickup Directories: Path of the directories in remote SFTP server from where the files need to be pulled. (Pull from Partner) 
  • Drop Directories: Path of the directories in remote SFTP server to which the files need to be pushed. (Push to Partner) 
  • Polling Interval: Interval in which the session to be created and files need to be collected. 
  • Archive File Interval (Days): No of days the file should be archived on deletion. 
  • Upload Cert: Upload remote SFTP server’s public key when auth type is selected as Public Key. 
Push-Pull
 
  • Username: User ID used to connect to the SFTP server. 
  • Password: Password used to connect to the SFTP server. 
  • Authentication Type: Select auth type as password or public key. If it is public key, then the authentication key will be sent to the mail. 
  • IP Range: Range of IP addresses to allow. 
  • Rate Limit: No of concurrent sessions to be opened. 
  • Virtual Root: Mapping for the endpoint that serves as the root directory. 
  • Delete after Collection: Option to delete the file after collecting it from the connected server. 
  • Pickup Directory: Directory from the where the files need to be pulled by the Partner. (Pull from Gateway) 
  • Drop Directory: Directory to which the files need to be pushed. (Push to Gateway) 
  • Archive File Interval (Days): No of days the file should be archived on deletion. 
  • Permission: Select user folder permissions, like READ DATA, WRITE DATA, APPEND DATA etc., 
Push-Push Scenario

This is where Data Gateway acts as the Server and allows external parties to connect to our SFTP Server for receiving file exchanges and acts as the Client to send files back to external parties by connecting to external parties’ SFTP Server.

This configuration would be the combination of the above two configurations

This is where Data Gateway acts as the Server and allows External Parties to connect to our SFTP Server for receiving File Exchanges and acts as the Client to send Files back to External by connecting to External Parties SFTP Server.  This configuration would be the combination of the above two configurations (Push-Pull and Pull-Push). 

11.3.4 API Based File Transfers

The API-based File Transfer feature allows end-users to automate and manage file transfers programmatically through REST APIs. This functionality enables seamless integration with your file transfer systems, providing a way to handle file transfers without the need for manual intervention
11.3.4.1 Pull-Push
The Endpoint creation is the same as how it is explained in the above section. In case of API based transfer, the Polling Interval can be selected as “ApiEndPoint”, which means the connection would be established only when the API invocation is made.
The API based transfer can also be utilized independently, irrespective of the Polling Interval. The Polling Interval can be anything based on which the schedulers will establish the connection. Meanwhile, API can also be invoked to connect to the remote server for file operations.
11.3.4.2 Scenario: File Transfer through API, where You connect to Remote Server
The initial process is to generate the token for the API execution. The basic authentication using username and password can also be used. Below is the API for generating token POST:  https:// <api-host>/adg/endpoint/access/generate-token?partnerId=<EndPoint-Id>&username=<remote-server-username>&renewalTime=<Time> The API would give the token as the response, which can be used in the subsequent APIs for authentication. Below is the API used to pick the file and start the transfer. POST: https://<api-host>/adg/endpoint/access/transfer-files?partnerId=<endpoint-id>&username=<remote-username>&pickUpDir=<pickup-dir>&remoteFilePattern=<remote-file-pattern> You could see the successful API response, which means the session is started to connect to a remote server and pick up the file and process it. This would pick all the files available from the source and move them to their destination. Similar process can be applied for other cases as well, sending files to remote servers, moving files from/to our server for external parties to connect and pull/drop files. You can see the file activity as well in the File Transfer module
11.3.4.3 Scenario: File Transfer through API, where Partner connects to Your Server.

The initial process is to generate the token for the API execution. The basic authentication using username and password or public key.
Below is the API for generating token
POST:http://<api-host>/adg/endpoint/access/generate-token-partnerToHub?username=<username>&passwordOrKey=<passwordOrkey>&renewalTime=<time> 

The API would give the token as the response, which can be used in the subsequent APIs for authentication. Below is the API used to list all the files from the cloud. GET: http:// <api-host> /adg/endpoint/access/List-Files- pullFromHub?folderPath=<pickupDir>&username=<username>

You can get the file names from the API response and use it in the subsequent APIs for file movement.
Below is the API used to upload files to the cloud. This is the case of Inbound, where the Partner drops the file in Data Gateway Server. In this process, files are directly sent through API.

In API Client, select “form-data” as the body type and provide the required details.
File : <upload file>
username: <username>
path: <dropDir>

POST: http://<api-host>/adg/endpoint/access/transfer-files-pushFromPartner

You could see the successful API response, which means that the file given to the API is moved to the Cloud Storage. You can see the file activity as well in the File Transfer module.

Below is the API used to download file from the cloud. This is the case of Outbound, where the Partner connects to Data Gateway Server to pick the file. In this process, files are directly picked through API. GET: http:// <api-host> /adg/endpoint/access/download-files-pullFromHub?folderPath=<pickupDir>&fileName=<filename>&username=<username> 

The API successful response shows that the file is picked from Cloud Storage and given as the API response. If delete after collection is enabled, then the download file will be deleted from the Cloud Storage. You can see the file activity as well in the File Transfer module.

11.3.4.4 Push-Pull
Enable the Api Trigger option Once this option is enabled, you cannot directly connect to WinSCP or transfer files manually. You must first generate a token by providing the Endpoint ID, Password, and Renewal Time in the POST API. After generating the token, you can able to connect to winscp and successfully transfer files until the renewal time expires.

11.3.5 AS2 (Applicability Statement 2)

AS2 (Applicability Statement 2) is a specification that defines secure and reliable data transmission over the Internet utilizing digital certificates and encryption. In Data Gateway, AS2 facilitates B2B (Business-to-Business) message exchange with trading partners through HTTP/HTTPS protocols. The AS2 implementation consists of Organizations (your entity), Endpoints (trading partners), and Relationships (the connection configuration between them).
11.3.5.1 AS2 Organizations
An AS2 Organization represents your company or business unit in AS2 transactions. Each organization has its own identity, certificates, and cryptographic credentials used for signing outgoing messages and decrypting incoming messages.
  • Endpoint Name: Provide the name of the organization being created (e.g., “MyCompany_Finance”).
  • Endpoint ID: Provide a unique identifier for the organization in the system.
  • Protocol: Select “AS2” from the protocol dropdown.
  • Storage Type: Select the storage type.
  • Storage Service Name: Select the configured storage service instance. This references the bucket or container name, based on this service name if secret manager is enable the passwords and certs will be stored in cloud.
  • Status: Set the status as Active or Inactive to enable or disable the organization.
  • Email: Email address associated with the organization for notifications and contact purposes.
  • Phone: Contact phone number for the organization (optional).
  • AS2 Identifier: Provide a unique AS2 identifier used in AS2 message headers (AS2-From/AS2-To). This identifier must match what your trading partners use to identify your organization.
  • Exchange Certificate: Upload the certificate used for decrypting incoming messages. This is typically a Keystore certificate in .jks or .p12 format.
  • Signing Certificate: Upload the certificate used for signing outgoing messages. This ensures message integrity and authenticates the sender.
  • Exchange Key Alias: Provide the alias name of the private key used for decrypting incoming AS2 messages from the keystore.
  • Exchange Key Phrase:Provide the passphrase/password to unlock the exchange (encryption) private key.
  • Signing Key Alias: Provide the alias name of the private key used for signing outgoing AS2 messages from the keystore.
  • Signing Key Phrase: Provide the passphrase/password to unlock the signing private key.
Functional Purpose:
  • The Signing Keystore is used to sign all outgoing AS2 messages, ensuring message integrity and sender authenticity. Recipients can verify the signature using your public signing certificate.
  • The Exchange Keystore is used to decrypt incoming AS2 messages received at your endpoint. Only your private key can decrypt messages encrypted with your public certificate.
  • Each Organization must have a unique AS2 Identifier that matches the identifier used by your trading partners in their endpoint configurations.
  • Certificates must be valid and not expired. Regular certificate renewal is essential for maintaining secure communications.
11.3.5.2 AS2 Endpoints
An AS2 Endpoint represents an external trading partner with whom your organization exchanges AS2 messages. Each endpoint configuration includes the partner’s URL, certificates, security settings, and communication preferences.
  • Endpoint Name: Provide the name of the trading partner endpoint being created (e.g., “Partner_ABC_Trading”).
  • Endpoint ID: Provide a unique identifier for the endpoint in the system.
  • Protocol:Select “AS2” from the protocol dropdown.
  • Storage Type: Select the storage platform where pickup and drop directories will be created.
  • Storage Service Name: Select the configured storage service instance. This references the bucket or container name where AS2 files will be stored.Based on this service name if secret manager is enable the passwords and certs will be stored in cloud.
  • Status:Set the status as Active or Inactive to enable or disable communication with this endpoint.
  • Email:Email address of the trading partner for notifications and contact purposes.
  • Phone:Contact phone number for the trading partner (optional).
  • AS2 Identifier:Provide the unique AS2 identifier of the trading partner. This identifier appears in AS2-From (when receiving) or AS2-To (when sending) headers.
  • URL:Provide the trading partner’s AS2 receiving endpoint address (e.g., https://partner.com:8443/as2/receive). This is where AS2 messages will be sent.
  • Response Timeout (seconds):Maximum time to wait for an HTTP response after sending an AS2 message. Typical value: 60-300 seconds.
  • Socket Timeout (seconds):Maximum idle time allowed for a network connection before it is closed. Typical value: 60-300 seconds.
  • Exchange Certificate:Upload the trading partner’s public certificate used for encrypting outgoing messages to them. Only they can decrypt messages with their private key.
  • Signing Certificate:Upload the trading partner’s public certificate used for verifying signatures on incoming messages. This ensures messages received are authentic.
  • Compress Data: Indicates whether payload compression should be applied before transmission. When enabled, the AS2 message payload is compressed using compression algorithm to reduce file size, conserve bandwidth, and improve transmission speed. This is particularly beneficial for large files.
  • Payload Type:Specifies the security and structure format applied to the AS2 message payload. Available options include Plain (no encryption or signing), Encrypted (payload encrypted Signed (payload digitally signed with sender’s private key), Signed and Encrypted (payload both signed and encrypted), and Signed Detached (signed payload)
  • MIME Type:Defines the primary content type category of the payload according to MIME standards. Standard values include text (for plain text content), application (for application-specific data formats like EDI, XML, JSON, or binary files), audio (for audio files), video (for video content), message (for encapsulated messages), image (for image files), and multipart (for messages containing multiple parts).
  • Sub MIME Type: Specifies the detailed subtype within the MIME type classification.
  • SSL Type:Defines whether SSL/TLS is required for the HTTPS connection.
  • Encryption Algorithm: Specifies the algorithm used to encrypt AS2 message payload data.
  • Signing Algorithm: Specifies the algorithm used to digitally sign AS2 messages.
  • MDN Required (Yes/No): Specifies whether a Message Disposition Notification (MDN) is requested after sending messages. MDN serves as a delivery receipt and processing acknowledgment.
  • MDN Type:Defines how the MDN acknowledgment is returned:
  • MDN Signing Algorithm:Specifies the hashing/signature algorithm used to protect the MDN.
Functional Purpose:
  • The Signing Certificate (partner’s public key) ensures message authenticity. Your system verifies that incoming messages were truly signed by the configured endpoint.
  • The Encryption Certificate (partner’s public key) is used when sending messages to the endpoint. The payload is encrypted so only the endpoint can decrypt it with their private key.
  • MDN configuration determines the acknowledgment mechanism. Synchronous MDN is simpler but requires the partner to respond immediately. Asynchronous MDN is better for high-volume or slow-processing scenarios.
11.3.5.3 AS2 Relationships
An AS2 Relationship defines the connection between an Organization (your entity) and an Endpoint (trading partner), including directory paths, polling intervals, and file exchange settings. Each relationship represents a unique file exchange channel.
  • AS2 Organisation: Select the local AS2 Organization that will participate in this relationship. This determines which signing/decryption keys and AS2 Identifier will be used for outgoing and incoming messages.
  • AS2 Endpoint: Select the trading partner Endpoint that represents the external AS2 entity. This defines which endpoint certificate, AS2 Identifier, and URL are used for encryption, signature verification, and message transmission.
  • Pickup Directory: The directory path in the storage bucket where the AS2 system will look for files to send to the Endpoint.
  • Drop Directory: The directory path in the storage bucket where files received from the Endpoint will be stored after successful file transfer
  • Polling Interval: Defines how frequently the system scans the pickup directory for new files to send. This ensures controlled scheduling of outbound message transfers.
  • Append GUID: Option to append unique global unique identifier to file name for the files transferred through the relationship channel.
Functional Purpose: The AS2 Relationship serves as the operational bridge between your organization and trading partners, providing:
  • Automatic Directory Creation: When a relationship is created, the system automatically creates the specified pickup and drop directories in the configured cloud storage bucket under the path structure /as2/{orgName}/{endpointName}/.
  • File Monitoring: A polling mechanism continuously monitors the pickup directory at the specified interval. When files are detected, the system automatically initiates AS2 transmission using the relationship’s configured settings.
  • Bi-Directional Communication:
    • Outbound (Send): Files placed in the pickup directory are processed according to the configuration and sent to the endpoint’s URL.
    • Inbound (Receive): Files received from the endpoint processed and stored in the drop directory.

11.4 GUID

The GUID (Globally Unique Identifier) functionality in the Data Gateway application ensures unique filenames during file transfers, preventing overwriting or duplication in the target system.  This functionality is applicable to file transfers via: 
  • SFTP 
  • FTP 
  • FTPS 
  • Cloud-to-Cloud transfers 
  • Other file transfer operations supported by the application 
The GUID is appended to the filename before initiating the transfer when the “Enable GUID” option is selected in the Application UI. Key Benefits:
  • Guarantees filename uniqueness across transfers 
  • Preserves original file extension 
  • Compatible with both internal and external uploads 
  • Lightweight and negligible performance impact 

11.4.1 Enabling GUID

To use the GUID functionality: 
  1. Navigate to the Application UI. 
  1. Locate the Enable GUID checkbox for the desired file transfer. 
  1. Check the box to enable GUID generation. 
  1. Leave the box unchecked to transfer files with their original filenames.

11.4.2 How GUID Works

  1. File Upload Triggered: The user initiates a file transfer. 
  2. Check GUID Setting: The system verifies whether the Enable GUID option is checked. 
  3. Generate GUID: If enabled, the system calls the Append GUID to Filename:  1. The GUID is appended to the original filename, separated by an underscore.
    1. File extensions are preserved.
Example:
Condition Input Filename Output Filename
GUID Disabled azurefile2.txt Azurefile.txt
GUID Enabled Azurefile2.txt Azurefile2.txt_8d85648f-2761-4a5a-8de1-a7e6d6f4018e

11.4.3 Using GUID – Step-by-Step

  1. Select File for Transfer Choose the file(s) to transfer via SFTP, FTP, FTPS, or Cloud. 
  1. Enable GUID Check the Enable GUID checkbox in the transfer settings. 
  1. Initiate Transfer The system generates a GUID and appends it to the filename. 
  1. File Transfer The file is transferred to the target location with the GUID appended. 
  1. Logging & Audit Transfer logs record the original filename and the GUID-modified filename for traceability. 
In the Destination cloud storage as well we are able to see the File name with Guid was appended.  
Astrocyte, ai, artificial intelligence, deep learning, machine learning, data, data transformation, robotics, robotics automation, automation,
Your AI assistant – Unlock value from data and AI to deliver exceptional user experience.
Linkedin-in Twitter Facebook-f Instagram Youtube

Our Services

Contact us

  • 12724 Gran Bay Parkway West, Suite 410, Jacksonville, 32258
  • info@astrocyte.ai
  • +1 844-534-8883

Subscribe to Newsletter

Scroll to Top