🎁 Before you apply, rehearse this interview. Create your free WorkMundi account and get an Interview Training on HelpsYouSpeak — no cost, no card. I want my training →
Back to All Jobs Senior Web Crawling Engineer Engineering Ahmedabad / Remote Full-time as per the market standards About The Role We're looking for a Senior Web Crawling Engineer to help build and maintain the web crawling infrastructure that powers our data extraction and analytics platforms. You'll work on developing robust, scalable crawlers that extract valuable data from complex websites and APIs. Key Responsibilities Architect, design, and implement scalable web crawlers using Scrapy framework as the primary technology Build reusable Scrapy spiders, middlewares, pipelines, extensions, and custom schedulers Develop distributed crawling solutions using Scrapy Cluster / Redis / Kafka-based queue systems Optimize Scrapy performance (auto-throttling, concurrency tuning, memory management) Implement advanced request fingerprinting, retry logic, proxy rotation, and user-agent rotation Implement strategies to bypass anti-scraping mechanisms (rate limiting, CAPTCHA handling, fingerprinting) Work with headless browsers (Playwright/Selenium) integrated with Scrapy for JS-heavy websites Required Qualifications 5+ years of strong Python development experience Extensive hands-on experience with Scrapy (core framework, pipelines, middleware, extensions) Deep understanding of Scrapy internals (Twisted, asynchronous networking) Strong experience with XPath, CSS selectors, JSONPath Solid understanding of HTTP/HTTPS protocols, headers, cookies, sessions Experience handling dynamic websites and API-based scraping Nice to Have Experience integrating Scrapy with Playwright or Selenium Knowledge of bot detection techniques and mitigation Experience building REST APIs for crawl orchestration Familiarity with cloud deployment and DevOps practices What We Offer Competitive compensation package Hybrid work model with flexible hours Professional development opportunities Team building events and activities Generous PTO and holiday schedule Apply for this Position Full Name Email Phone LinkedIn Resume PDF, DOC, or DOCX (max 5MB) Cover Letter .
Here's how to pick the right one and stand out in your application.
144.883Jobs
31.687IN
81%EN
That number is real. WorkMundi's database shows 144,883 open engineer roles across the world. India has the most with 31,687 jobs, followed by the United States with 30,084. If you just finished reading one job ad and felt paralyzed by choice, you're not alone—but this scale is actually an advantage. It means you can afford to be selective.
Start by geography and language. The majority of engineer ads—117,837 of them—have the job posting text written in English. Use that as one filter, but remember: the ad text language tells you nothing about whether the role actually requires you to speak English day-to-day. Read the job description carefully. Then check which countries have the volume you're targeting. Singapore, Poland, and Australia round out the top five after India and the US.
Next, learn who's hiring. Accenture has posted 2,801 engineer roles. andurilindustries, speechify, and jobgether are also actively recruiting. If you're applying to one of these names, research their hiring patterns and interview style before you apply. That homework pays off.
When you interview, expect the question every engineer hears: 'Tell me about a time you had to debug a problem that wasn't in your job description.' Have a specific story ready—not a general one. Name the tools, the deadline pressure, and what you learned. Hiring managers listen for whether you see problem-solving as part of the role itself, not a favour.