BCBS
|Sr Databricks Engineer
, FL, US
Summary
Leads the design and implementation of scalable data solutions for enterprise customer data and personalized experiences within Adobe Experience Platform (AEP), Adobe Customer Data Platform (CDP), Real-Time CDP, and Adobe Journey Optimizer (AJO) initiatives.
Highlights
Designed and optimized high-performance data engineering solutions for customer profiles, segmentation, identity resolution, Customer 360, and real-time customer activation across the Adobe Experience Platform ecosystem.
Built enterprise-scale ETL/ELT pipelines using Databricks, Apache Spark, PySpark, Python, SQL, Delta Lake, Azure Data Lake Storage (ADLS), and AWS S3 to ingest, transform, cleanse, enrich, and publish large volumes of structured and semi-structured customer data.
Implemented Delta Lake architecture for reliable data storage and processing, leveraging ACID transactions, schema enforcement, schema evolution, time travel, optimized data layouts, and incremental processing patterns.
Developed and maintained Azure Data Factory pipelines for data ingestion, transformation, scheduling, and dependency management, integrating enterprise source systems with ADLS, Databricks, and downstream platforms.
Integrated streaming and event-driven customer data using Kafka/Event Streaming to support near-real-time ingestion, customer profile updates, segmentation, identity processing, and activation use cases.
Implemented robust data quality, validation, reconciliation, error handling, and monitoring mechanisms across ETL/ELT pipelines, ensuring accuracy, completeness, and reliability of critical customer data.
Utilized Unity Catalog within Databricks to support enterprise data governance, centralized access control, data discovery, permissions management, lineage, and secure management of data assets.
Developed ML feature engineering pipelines using Databricks, Spark/PySpark, Python, and SQL to prepare and manage machine-learning-ready customer features for analytics and predictive use cases.