Cloud engineer for streaming platform (Kafka) (m/f/n)

We are looking for a skilled Cloud engineer for streaming platform (Kafka, Pulsar)

The ideal candidate will design, deploy, and benchmark Pulsar clusters across CleverCloud (sovereign) and AWS (public cloud), assess protocol compatibility with existing Kafka and JMS workloads, and validate geo-replication, fault tolerance, and observability capabilities. This role requires strong experience with distributed messaging systems, cloud-native architecture design, and the ability to produce clear architectural assessments and recommendations that drive business decisions.

Project Description:

The project focuses on evaluating Apache Pulsar as a next-generation event streaming platform in a complex enterprise environment spanning sovereign cloud, public cloud, and on-premises infrastructure.

The successful consultant will be responsible for designing, deploying, operating, and benchmarking Apache Pulsar clusters while validating interoperability with existing Kafka and JMS workloads. The engagement includes architecture assessment, protocol compatibility analysis, geo-replication validation, multi-tenancy testing, performance benchmarking, observability integration, and production-readiness evaluation.

The outcome of the project will be a comprehensive technical assessment and recommendation regarding Apache Pulsar's suitability as the organization's future event streaming platform.

Responsibilities:

Architecture Design & Evaluation

• Lead the architectural assessment of Apache Pulsar as a replacement/extension of the existing Kafka-based event streaming platform.

• Design target-state architectures for intra-cloud, cross-cloud, and hybrid (cloud ↔ on-premises) messaging scenarios.

Pulsar Deployment & Configuration

• Deploy, configure, and operate Apache Pulsar clusters on CleverCloud (managed) and on-premises (containerised on OpenShift/Kubernetes).

Performance Benchmarking

• Design and execute performance tests at varying message sizes (KB/MB), load levels (messages/second), and tenant configurations.

• Compare results against existing on-premises Kafka cluster baselines (non-prod and prod).

Protocol Compatibility Assessment

• Evaluate Kafka-on-Pulsar protocol handlers (Starlight for Kafka, StreamNative KSN) and Pulsar JMS support.

• Prototype migration of an existing Kafka application and a JMS-based application to Pulsar with minimal code changes.

Geo-Replication & Hybrid Architecture

• Configure and validate Pulsar geo-replication between sovereign cloud, public cloud, and on-premises clusters.

• Assess latency, disaster recovery, and cross-environment read/write scenarios.

Observability & Monitoring

• Integrate Pulsar monitoring with existing corporate tools (Dynatrace, Grafana, Splunk, OpenSearch).

• Validate that operational visibility is sufficient for production readiness.

Multi-Tenancy Validation

• Configure and test Pulsar's native multi-tenancy model (tenants, namespaces, access policies) to confirm suitability for a shared-cluster enterprise model.

Documentation & Recommendations

• Produce clear POC findings, Architecture Decision Records (ADRs), and a final recommendation on Pulsar's suitability for the corporate event streaming platform.

Collaboration

• Work closely with the Kafka operations team, solution architects, and application teams for migration prototyping.

Training & Knowledge Sharing

• Contribute to the knowledge base and provide technical guidance to stakeholders on event streaming architecture patterns and Pulsar capabilities.

Requirements:

Must Have:

• 5+ years of experience in Cloud Architecture, distributed systems engineering, or messaging platform design and operations.

• Fluent English (written and spoken) is required; any additional European language will be considered an asset.

• Higher education degree in Computer Science, Engineering, Information Technology, or a related field.

Apache Kafka

• Strong hands-on experience with Apache Kafka (architecture, operations, tuning, troubleshooting), including understanding of brokers, partitions, replication, consumer groups, and Kafka Connect.

Apache Pulsar

• Knowledge of Apache Pulsar (architecture concepts: brokers, BookKeeper, topics, tenants, namespaces, geo-replication) or demonstrated ability to ramp up quickly on comparable distributed systems.

Multi-Cloud & Hybrid Architecture

• Experience designing multi-cloud and hybrid architectures integrating services across public cloud (AWS), sovereign cloud, and on-premises infrastructure.

Containerised Platforms

• Proficiency in containerised deployments, including Kubernetes/OpenShift, Helm charts, and container lifecycle management.

Automation & Infrastructure as Code

• Proficiency in scripting and automation using Python, Bash, and infrastructure-as-code tools such as Terraform and Ansible.

Monitoring & Observability

• Experience with monitoring and observability tools including Dynatrace, Grafana, Splunk, and OpenSearch.

CI/CD & GitOps

• Experience with CI/CD and GitOps using GitLab CI, ArgoCD, or similar solutions, including understanding of GitOps principles for infrastructure configuration.

Messaging Technologies

• Understanding of messaging protocols including JMS 2.0, Kafka protocol, and AMQP, with experience in protocol bridging or migration.

Pulsar Protocol Handlers

• Knowledge of Apache Pulsar protocol handlers including Starlight for Kafka (DataStax), StreamNative KSN, and Pulsar JMS, or willingness to evaluate them hands-on.

Event Streaming & Processing

• Knowledge of event-driven architectures and stream processing technologies including Kafka Streams, Flink, and Spark Streaming.

Streaming Platforms

• Experience with Confluent Kafka or other managed Kafka/streaming solutions.

Performance Testing

• Performance testing methodology, including the ability to design, execute, and interpret load tests for distributed systems.

Documentation & Communication

• Strong analytical and documentation skills with the ability to produce concise technical assessments, architecture descriptions, comparisons, and Architecture Decision Records (ADRs).

• Excellent problem-solving and communication skills.

• Strong communication skills (oral and written) to interact effectively with both technical teams and management stakeholders, including making presentations and driving architectural discussions.

Additional Requirements:

• EU citizenship is required.

• Ability to work in an international and multicultural environment.

• Ability to work both independently and collaboratively within a team.

Nice to Have:

Additional Technologies & Qualifications

• Experience with CleverCloud or other European sovereign cloud providers.

• Experience with Redis and key-value store patterns.

• Certifications in Kafka, Kubernetes, or cloud platforms.

• ITIL qualification.

Offer:

• B2B contract.

• Daily rate: 530-550 EUR/MD net.

• 6-month project with the possibility of extension.

• Fully remote work with occasional business trips to Luxembourg.

• Onboarding in Luxembourg (1-2 days on-site).

• Opportunity to work in an international and multicultural environment.

• Collaboration with experienced architects, platform engineers, and cloud specialists.

• Exposure to modern cloud-native, hybrid cloud, distributed messaging, and event-driven architecture projects.