Custom Software Engineer

Key Responsibilities

• Data Pipeline Development
• Design and implement robust, scalable batch and streaming data pipelines using Python and GCP services.
• Build and maintain data transformation workflows in Dataform (SQLX) integrated with BigQuery.
• Develop event-driven and real-time data processing solutions using Pub/Sub and Cloud Functions / Cloud Run.
• Manage large-scale structured and unstructured data stored in GCS, BigQuery, and Cloud Spanner.
• Python & Google SDK
• Write clean, well-structured Python code leveraging GCP client libraries (google-cloud-bigquery, google-cloud-storage, google-cloud-pubsub, google-cloud-spanner, etc.).
• Develop reusable Python modules, utilities, and SDKs for internal data platform tooling.
• Ensure all code meets quality standards through thorough unit testing with pytest and code reviews.
• Infrastructure & DevOps
• Provision and manage GCP infrastructure using Terraform, following infrastructure-as-code (IaC) best practices.
• Maintain CI/CD pipelines in GitLab for automated testing, linting, and deployment of data workloads.
• Understanding of monitoring, alerting, and observability for data pipelines and GCP resources.
• Manage containerised workloads on Cloud Run, including Dockerfiles and container registries.

Skills

• 5+ years of experience in data engineering with at least 2 years on GCP.
• Strong Python programming skills clean code, OOP, packaging, and performance optimization.
• Hands-on experience with BigQuery: query optimization, partitioning, clustering, and data modelling.
• Experience with Pub/Sub for event-driven architectures and streaming ingestion.
• Proficiency in Cloud Run and/or Cloud Functions for serverless and containerized workloads.
• Experience writing unit tests with pytest mocking, fixtures, coverage reporting.
• Hands-on Terraform experience for GCP resource provisioning (projects, IAM, networking, data services).
• Experience with GitLab CI/CD pipeline authoring, environment management, secrets handling.
• Solid understanding of GCS for data lake patterns file formats (Parquet, Avro, JSON), lifecycle policies.

Nice to Have

• Experience with Dataform for SQL-based transformation pipelines in BigQuery.
• Familiarity with Cloud Spanner for globally distributed relational workloads.
• Knowledge of data mesh or data lakehouse architecture patterns.
• Knowledge/Experience with Apache Beam / Dataflow for large-scale streaming or batch processing.
• GCP Professional Data Engineer or Cloud Architect certification.
• Familiarity with Apache Airflow / Cloud Composer, or similar orchestration tools.

Additional Information:

• The candidate should have minimum 3 years of experience in Google Cloud Platform Architecture.