Azure Integration Service: Event Driven Architecture with sFTP and SMB
Azure
Integration Service: Event Driven Architecture with sFTP and SMB
In today’s interconnected enterprise
environments, seamless data exchange between vendors, cloud services, and
on-premises systems is critical. This blog outlines a comprehensive integration
solution that automates the flow of employee timesheet data from external
sources to invoice generation and multi-system distribution, using Azure
Integration Services and Azure Storage capabilities.
The solution demonstrates how a secure,
event-driven architecture can be implemented to handle timesheet processing and
invoice distribution across multiple stakeholders. Leveraging Azure Storage
SFTP, Event Grid/Event Hub, Logic Apps, Azure Functions, and NFS-enabled file
shares, this design supports both modern cloud-native applications and
traditional on-premises systems.
This guide is divided into three parts,
focusing on configuring secure file exchange, orchestrating integration
workflows, and enabling hybrid access to generated invoice files.
Architecture
Overview
The architecture is designed to support secure,
scalable, and automated processing of timesheets and invoice distribution
across both cloud-based and on-premises systems. It utilizes various components
within the Azure ecosystem to facilitate file ingestion, transformation, and
distribution, while maintaining strict security and access controls.
Key
Components
- Azure Storage (SFTP-enabled)
- Acts as the entry point for
timesheets submitted by Vendor 1.
- Configured with IP
whitelisting, SSH key authentication, and user-based access control.
- Azure Event Grid / Event Hub
- Monitors the storage account
for new file uploads.
- Triggers downstream processing
by emitting events when a new timesheet is received.
- Azure Integration Services
- Logic Apps / Azure Functions handle the orchestration
and transformation of data.
- Timesheets (in XML format) are
parsed, validated, and transformed into invoices (in JSON format).
- Azure Blob Storage
(SFTP-enabled)
- Stores the generated invoices
and allows secure download by external systems such as Vendor 2 and
Application 1.
- Azure Files (NFS-enabled)
- Provides a network file share
for systems requiring direct file system access.
- Enables access via SMB
protocol for Application 2 and the on-premises SQL Server.
- Downstream Consumers
- Vendor 2 accesses invoices via
SFTP for payment gateway processing.
- Application 1 reads files from the
same SFTP endpoint (segregated via subnet).
- Application 2 and SQL Server consume
invoices from the NFS-enabled file share using SMB.
Integration Flow Summary
Technical Solution Overview
This solution outlines an integration scenario involving multiple vendors
and systems that process employee timesheets and generate invoices using Azure
services. The solution is divided into three key components:
Scenario Workflow
1. Timesheet
Submission by Vendor 1
o Vendor
1 collects employee (consultant) timesheets from an external website.
o Each
timesheet is securely uploaded to an Azure Storage SFTP endpoint.
o The
SFTP endpoint is protected using IP whitelisting, SSH keys, and user
credentials to ensure secure access.
2. Timesheet
Processing and Invoice Generation
o Upon
successful upload of a timesheet to the SFTP location, an event is triggered in
the associated Azure Storage Account.
o This
event is propagated through Azure Event Grid or Event Hub and consumed by Azure
Integration Services (Logic Apps or Azure Functions).
o The
timesheet (in XML format) is processed, and an invoice (in JSON format) is
generated.
o The
generated invoice is saved to both:
§ A
Blob container (with SFTP enabled)
§ A
Storage File Share (NFS-enabled) for broader accessibility
3. Invoice
Consumption by Downstream Systems
o The
generated invoice is a key input for four different downstream systems, each
accessing the file via different protocols and locations:
§ Vendor
2 connects to the Azure SFTP endpoint and downloads the invoice for payment
processing.
§ Application
1 accesses the file through SFTP (from a separate subnet).
§ Application
2 reads the invoice from the NFS-enabled Storage File Share via the SMB
protocol.
§ On-premises
SQL Server also accesses the NFS-enabled file share via SMB, ingests the
invoice data, and stores it for auditing purposes.
Blog Structure
This blog is organized into three parts:
1. Configuring
Azure SFTP Access
o Set
up SFTP on Azure Storage
o Configure
secure access for Vendor 1 and Vendor 2 using proper authentication and
authorization mechanisms
2. Designing
Azure Integration Services
o Use
Logic Apps and Azure Functions to process timesheets (XML format)
o Generate
invoices (JSON format) based on processed data
3. Setting
Up Storage File Share with NFS Support
o Enable
NFS on Azure Storage File Share
o Allow
on-premises systems to read files using SMB protocol
Stay tuned for Part 1, where we'll walk through setting up Azure Storage SFTP and configuring secure access for Vendor 1 and Vendor 2
Comments
Post a Comment