Senior Azure Data Migration & Dataverse Integration Engineer

Role: Senior Azure Data Migration & Dataverse Integration Engineer

Company: Arch Systems

Location: Remote

Type: Full-time

Position Summary

Seeking a Senior Azure Data Migration & Dataverse Integration Engineer to design, develop, implement, and support enterprise data migration and synchronization capabilities between Microsoft SQL Server and Microsoft Dataverse .

The position requires a senior hands-on engineer capable of owning the complete data pipeline lifecycle, including source analysis, data mapping, extraction, transformation, migration, synchronization, validation, exception handling, monitoring, performance optimization, and automated deployment.

The engineer will develop and maintain repeatable pipelines supporting initial bulk migration, incremental migration, Change Data Capture (CDC), ongoing synchronization, and transition/cutover activities between legacy SQL Server applications and modern Microsoft Power Platform/Dataverse solutions.

Primary Responsibilities

The Data Migration Engineer will:

• Design and implement end-to-end SQL Server-to-Dataverse data migration pipelines .
• Develop automated pipelines using Azure Data Factory (ADF) and supporting Azure services.
• Analyze legacy SQL Server databases, tables, views, relationships, stored procedures, and data dependencies.
• Map legacy SQL Server data structures to Dataverse tables, columns, relationships, choices, lookups, and alternate keys.
• Develop repeatable processes for:
• Initial/bulk data migration
• Incremental data migration
• Change Data Capture
• Delta processing
• Scheduled synchronization
• Near-real-time synchronization
• Legacy-to-Dataverse synchronization
• Dataverse-to-legacy synchronization where required
• Cutover and final synchronization
• Develop and maintain ADF pipelines, datasets, linked services, integration runtimes, triggers, parameters, variables, and reusable pipeline components.
• Develop transformation and data-cleansing logic required to convert legacy data into Dataverse-compatible structures.
• Implement controlled sequencing to maintain parent/child relationships and referential integrity.
• Develop lookup resolution and cross-reference processes for Dataverse GUIDs and legacy SQL identifiers.
• Implement restartable and recoverable migration processes.
• Develop pipeline logging, audit, exception handling, retry, and reconciliation capabilities.
• Validate migrated data for completeness, accuracy, integrity, and consistency.
• Perform source-to-target record-count and field-level reconciliation.
• Troubleshoot failed records, pipeline failures, API limitations, data-quality issues, and performance problems.
• Develop automated deployment processes for data pipeline components across DEV, TEST, UAT, and PROD environments.

Azure Data Factory / Pipeline Responsibilities

The candidate must have advanced hands-on experience designing and developing Azure Data Factory solutions.

Responsibilities include:

• ADF pipeline architecture and development
• Copy Activities
• Lookup Activities
• Stored Procedure Activities
• ForEach and Until processing
• Conditional processing
• Pipeline dependencies
• Parameterized pipelines
• Dynamic expressions
• Metadata-driven pipeline development
• Linked Services
• Datasets
• Integration Runtime configuration
• Managed Identity authentication
• Azure Key Vault integration
• Pipeline triggers and scheduling
• Incremental load processing
• Watermark processing
• CDC patterns
• Retry and failure handling
• Pipeline logging and auditing
• Performance tuning and parallelization
• Environment-specific configuration
• Infrastructure and pipeline deployment automation

The engineer should be capable of developing reusable, metadata-driven pipeline patterns rather than creating independent hard-coded pipelines for every table.

SQL Server Responsibilities

Strong SQL Server development and data engineering skills are required, including:

• Complex T-SQL development
• Stored procedures
• Views
• Common Table Expressions
• Temporary and staging tables
• Data profiling
• Data cleansing
• Data transformation
• Joins and relationship analysis
• Primary and foreign key analysis
• Indexing and query optimization
• SQL Server Change Data Capture
• Change Tracking
• Watermark-based incremental extraction
• Data validation queries
• Reconciliation queries
• Duplicate detection
• Referential-integrity valid…