Data Platform Developer (m/f/d)

Für unseren Kunden suchen wir ab Mitte April einen Data Platform Developer (m/f/d) Big Data / K8s - remote & Berlin

Tasks

• Development and Maintenance of Data Architecture Objective: Create and sustain a robust and scalable data architecture.

• Development of infrastructure encompassing databases, data lakes, and data warehouses

• Maintaining of databases, data lakes and data warehouses Recommendations for Data Storage, Retrieval, and Processing Objective: Optimize data handling by following best practices

• Give recommendations on implementing efficient data storage, retrieval, and processing practices. Adoption of Data Mesh Principles Objective: Promote decentralized data management and architecture

• Data Mesh Implementation: Recommend strategies for adopting Data Mesh principles to encourage decentralized data ownership and architecture. Conceptualization and Implementation of Data Mesh Proof of Concepts (PoCs) Objective: Validation of decentralized data architectures through practical application.

• Providing designs recommendations on decentralized data architectures

• Implementation of PoCs to validate conceptual decentralized data architectures. Implementation of a Comprehensive Data Catalog Objective: Enhancement of data transparency and usability.

• Creation of a comprehensive data catalog documenting metadata, data lineage, and data dictionaries for all data assets.

Skills

• The contractor must be at a senior level combining a strong technical background in data engineering with expertise in implementing data architectures, having collaboration skills, and an innovative mindset to contribute effectively to the organization's data strategy

• 5+ years of general IT experience

• 3+ years of Big Data experience

• Proven experience as a Data Engineer with a focus on designing and implementing scalable data architectures

• Extensive experience in developing and maintaining databases, data lakes, and data warehouses

• Hands-on experience with ETL processes and data integration from various sources

• Familiarity with modern data technologies and cloud services

• Proficient in designing and implementing data models to meet business requirements

• Extensive experience with Data Catalogue technology in conjunction with Data Mesh

Must have experience:

• Proven hands-on software development experience

• Proficiency in data processing languages such as SQL, Java, Python or Scala

• Knowledge and experience with at least some of the Data technologies/frameworks:

• RDBMS (PostgreSQL/MySql etc.)

• NoSQL Storages (MongoDB, Cassandra, Neo4j etc.)

• Timeseries (InfluxDB, OpenTSDB, TimescaleDB, Prometheus etc.)

• Workflow orchestration (AirFlow/Oozie etc.)

• Data integration/Ingestion (Flume etc)

• Messaging/Data Streaming (Kafka/RabbitMQ etc.)

• Data Processing (Spark, Flink etc.)

• And/Or with their Cloud provided counterparts, i.e., Cloud Data/Analytics services (GCP, Azure, AWS)

• Familiarity with reference Big Data architectures (Warehouse, Data Lake, Data Lakehouse) and their implementation.

• Experience in implementing and operating data intensive applications

• Strong focus on DataOps/DevOps

• Good k8s knowledge and experience

• Proficiency in both speech and writing in English (at least C1

• Deeper K8s skills and experience, e.g. k8s operators’ development experience and/or k8s operators for Big Data technologies)

• In-depth knowledge of best practices in data privacy and data protection

• Proven experience with DataMesh principles in practice

• Data platform development and/or operations experience

• Knowledge and experience in lifecycle management in Data (e.g. CD4ML, MLOps)