We are redirecting you to the source. If you are not redirected in 3 seconds, please click here.

Web Scraping Engineer (Python) (Remote - UK) at Jobgether. This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Web Scraping Engineer (Python) in the United Kingdom.. This role offers the opportunity to design and operate high-scale web and AI/LLM scraping systems that deliver critical data to drive business insights. You will build resilient pipelines capable of handling anti-bot defenses and upstream changes while ensuring reliable, cost-efficient cloud deployments. Collaborating closely with Data Engineering, DevOps, and Product teams, you will influence architecture, performance, and system observability. The environment is fast-paced, innovative, and highly collaborative, allowing for autonomy and ownership. Your work will have immediate, tangible impact on global products and customers, while providing opportunities for technical growth and innovation.. . Accountabilities. . Design, build, deploy, and maintain scalable web and AI/LLM scraping systems with high reliability and performance.. . Develop and implement anti-bot and evasion strategies, including proxies, headless browsers, CAPTCHAs, and request fingerprinting.. . Integrate scraped data into pipelines, including cleaning, deduplication, schema validation, and delivery to data warehouses.. . Ensure system resilience through monitoring, alerting, retries, and fallback strategies.. . Collaborate with cloud infrastructure and DevOps teams for containerized deployments and cost optimization.. . Proactively adapt systems to platform changes, anti-bot measures, and evolving AI data sources.. . Maintain strong documentation, best practices, and code hygiene, and communicate effectively across distributed teams.. . . . Bachelor’s degree in Computer Science or a related technical field, or equivalent practical experience.. . Minimum 5 years of experience building and running production web scraping systems at scale.. . Strong Python skills, with experience in frameworks such as Scrapy, Playwright, or Selenium.. . Practical anti-bot experience: proxies, CAPTCHAs, JavaScript rendering, and fingerprinting.. . Experience with distributed systems and pipelines (Spark, Airflow) integrated with Redshift, PostgreSQL, or similar.. . Experience scraping LLM/AI sources (Perplexity, ChatGPT, Claude, Gemini).. . Experience deploying and managing workloads in cloud environments (AWS/GCP) using Docker and Kubernetes.. . Proven ability to build observable, resilient systems using Grafana, Prometheus, ELK, or similar monitoring stacks.. . Strong debugging, performance optimization, and reliability focus.. . Independent self-starter with excellent communication, able to take end-to-end ownership of projects.. . Bonus / Desirable Skills:. . Awareness of legal, ethical, and compliance considerations in scraping (e.g., GDPR, terms of service).. . Familiarity with messaging/queuing systems (Kafka, RabbitMQ).. . Knowledge of Infrastructure-as-Code (Terraform, Ansible) and CI/CD pipelines (Jenkins, GitHub Actions, TeamCity).. . Experience with cloud cost optimization and scaling strategies for large-scale scraping.. . Company Location: United Kingdom.