Lead Engineer - Alternative Data & Web Extraction (Python / AI)

About the Client / Who are you joining?

On behalf of our client - a premier, global US-headquartered financial technology and proprietary market analytics enterprise operating at massive scale - we are seeking an experienced Lead Engineer.

Our client operates cutting-edge engineering hubs delivering market-defining technological capabilities powered by big data, distributed systems, and applied machine learning. The Warsaw engineering group is directly responsible for designing high-throughput systems that ingest, transform, and structure petabytes of Alternative Data - unconventional, real-time data streams (web, digital platforms, multimedia, APIs) that feed algorithmic decision engines and research platforms.

Role Overview:

As a Lead Engineer, you will combine hands-on architecture with technical and team leadership. You will guide a dedicated engineering squad building high-capacity, low-latency automated data acquisition systems while driving the adoption of Agentic AI and LLM-assisted workflows (autonomous subagents, MCPs, intelligent extractors) to scale data ingestion and standardization.

You will act as the technical anchor for web extraction engineering, balancing system design, mentoring, and close collaboration with data researchers, quantitative analysts, and data science teams.

Key Responsibilities:

• Technical Guidance & Mentorship: Lead, coach, and support a talented group of engineers. Foster engineering excellence, establish coding standards, and run high-quality code reviews.

• Large-Scale Data Ingestion: Architect resilient, scalable, and low-latency extraction frameworks capable of ingesting massive, unstructured datasets from web, APIs, and multimedia channels.

• Agentic AI & Automation: Design and deploy modern agent-driven pipelines (leveraging LLMs, Model Context Protocol / MCPs, subagents, and automated workflows) to collect, parse, and normalize complex information.

• Pipeline Infrastructure: Build and maintain fault-tolerant streaming and orchestration architectures using modern queueing systems and workflow engines (Kafka, Airflow, relational/NoSQL datastores).

• Data Product Delivery: Partner closely with data researchers, quantitative analysts, and domain experts to turn raw market signals into reliable, production-grade data products.

• Human-in-the-Loop Coordination: Collaborate with distributed validation teams to build efficient quality-assurance loops and human-in-the-loop workflows for dataset verification.

• Continuous Innovation: Track emerging trends in anti-bot mitigation, browser automation, and LLM-powered extraction to maintain a competitive technical edge.

Required Qualifications:

• Professional Experience: 7–10+ years of commercial software engineering / data engineering experience, including demonstrated experience leading or managing engineering teams.

• Education: Bachelor’s or Master’s degree in Computer Science, Software Engineering, Mathematics, Physics, or related STEM discipline.

• Python Mastery: Expert-level proficiency in Python, its data manipulation ecosystem (e.g., Pandas, NumPy), and advanced web scraping / extraction toolkits.

• Data & Networking Fundamentals: Strong command of SQL, relational/non-relational database design, network protocols, web standards (HTML, DOM, REST/GraphQL APIs, browser automation), and anti-scraping mitigation techniques.

• Agentic Coding & LLMs: Practical familiarity with modern agentic workflows (e.g., Claude Code, MCPs, tool-use, subagents, LLM-assisted data extraction/parsing).

• Data Pipelines: Solid practical experience with data streaming and pipeline orchestration (e.g., Apache Airflow, Kafka).

• Communication: Excellent verbal and written English (C1/C2) with strong interpersonal skills and stakeholder management capability.

Nice to Have:

• Experience in high-scale FinTech, high-throughput digital platforms, or systems serving data directly to Quant/Data Science teams.

• Hands-on knowledge of containerization and orchestration (Docker, Kubernetes) and modern cloud infrastructure (AWS preferred).

• Experience architecting data validation loops and managing human-in-the-loop validation teams.

What the Client Offers:

• High-impact technical leadership role with direct visibility into mission-critical systems handling massive data volumes.

• Substantial innovation freedom and access to the latest tech stack (enterprise LLM tooling, modern cloud infrastructure).

• Highly competitive remuneration package with attractive bonus schemes.

• Premium office space in Warsaw, flexible hybrid setup, and international career trajectory.

• Comprehensive benefits: private healthcare, life insurance, professional development allowance.