🎁 Before you apply, rehearse this interview. Create your free WorkMundi account and get an Interview Training on HelpsYouSpeak — no cost, no card. I want my training →
We are looking for an experienced Airflow Orchestration and Ingestion Engineer to support the migration of legacy data workflows from Apache Oozie to Apache Airflow, as part of a broader transition from an on-prem Hadoop setting to a modern cloud-based data platform on Databricks and AWS . This is a critical role focused on reengineering data pipeline orchestration, automation, and deployment within a cloud-native framework. Key Responsibilities: Workflow Migration: - Convert Oozie workflows to Airflow DAGs using Python. - Build reusable, modular Airflow pipelines for ingestion, transformation, and scheduling. - Ensure accurate migration and one-to-one workflow alignment without disrupting business processes. Cloud Data Platform Transition: - Work with engineering teams to migrate Hadoop workloads to Databricks on AWS. - Leverage Airflow to orchestrate data pipelines across AWS services (S3, EMR, Glue, Redshift). Pipeline Optimization: - Enhance pipeline performance for throughput and latency in the AWS ecosystem. - Integrate Airflow with Databricks for transformation and analytics tasks. Monitoring & Error Handling: - Implement retry logic, exception handling, and alerting in Airflow. - Set up observability tools like CloudWatch, Prometheus, or Airflow s native monitoring. Collaboration & Documentation: - Collaborate with data architects, DevOps, and cloud teams. - Document orchestration logic, best practices, and the migration process. CI/CD and Infrastructure Automation: - Develop CI/CD pipelines using Jenkins and Terraform. - Automate DAG deployment and infrastructure provisioning via IaC. - Integrate validation steps in deployment workflows. Required Skills and Experience: - Strong expertise in Apache Airflow , including complex DAG design and orchestration. - Prior experience with Apache Oozie and workflow migration. - Proficiency in Python for Airflow DAG development. - Hands-on experience with Hadoop ecosystems (e.g., HDFS, Hive, Spark). - Knowledge of CI/CD tools such as Jenkins and Infrastructure as Code (IaC) with Terraform. - Experience with Databricks (preferably on AWS) and big data orchestration. - Solid understanding of AWS services : S3, EMR, Glue, Lambda, Redshift, IAM. - Familiarity with container tools such as Docker or Kubernetes . Preferred Qualifications: - Experience with large-scale cloud migrations , especially Hadoop-to-Databricks. - Proficiency in Spark / PySpark for big data transformation. - AWS or Databricks certifications are a plus. - Familiarity with Git and workflow monitoring platforms. .
Here's how to pick the right one and stand out in your application.
144.883Jobs
31.687IN
81%EN
That number is real. WorkMundi's database shows 144,883 open engineer roles across the world. India has the most with 31,687 jobs, followed by the United States with 30,084. If you just finished reading one job ad and felt paralyzed by choice, you're not alone—but this scale is actually an advantage. It means you can afford to be selective.
Start by geography and language. The majority of engineer ads—117,837 of them—have the job posting text written in English. Use that as one filter, but remember: the ad text language tells you nothing about whether the role actually requires you to speak English day-to-day. Read the job description carefully. Then check which countries have the volume you're targeting. Singapore, Poland, and Australia round out the top five after India and the US.
Next, learn who's hiring. Accenture has posted 2,801 engineer roles. andurilindustries, speechify, and jobgether are also actively recruiting. If you're applying to one of these names, research their hiring patterns and interview style before you apply. That homework pays off.
When you interview, expect the question every engineer hears: 'Tell me about a time you had to debug a problem that wasn't in your job description.' Have a specific story ready—not a general one. Name the tools, the deadline pressure, and what you learned. Hiring managers listen for whether you see problem-solving as part of the role itself, not a favour.