Databricks Engineer

May 14, 2026 Auto Import

Overview: We are seeking a Databricks Engineer to design, build, and operate a Data & AI platform with a strong foundation in the Medallion Architecture (raw/bronze, curated/silver, and mart/gold layers). This platform will orchestrate complex data workflows and scalable ELT pipelines to integrate data from enterprise systems such as PeopleSoft, D2L, and Salesforce, delivering high-quality, governed data for machine learning, AI/BI, and analytics at scale. You will play a critical role in engineering the infrastructure and workflows that enable seamless data flow across the enterprise, ensure operational excellence, and provide the backbone for strategic decision-making, predictive modeling, and innovation.<div align="center" style="margin-left:24px;text-align:center;"><hr align="center" size="2" width="100%"></div>Responsibilities: 1. Data & AI Platform Engineering (Databricks-Centric):<ul><li style="text-align:justify;">Design, implement, and optimize end-to-end data pipelines on Databricks, following the Medallion Architecture principles.</li><li style="text-align:justify;">This will be a complete Remote role in India.</li><li style="text-align:justify;">Build robust and scalable ETL/ELT pipelines using Apache Spark and Delta Lake to transform raw (bronze) data into trusted curated (silver) and analytics-ready (gold) data layers.</li><li style="text-align:justify;">Operationalize Databricks Workflows for orchestration, dependency management, and pipeline automation.</li><li style="text-align:justify;">Apply schema evolution and data versioning to support agile data development.</li></ul>2. Platform Integration & Data Ingestion:<ul><li style="text-align:justify;">Connect and ingest data from enterprise systems such as PeopleSoft, D2L, and Salesforce using APIs, JDBC, or other integration frameworks.</li><li style="text-align:justify;">Implement connectors and ingestion frameworks that accommodate structured, semi-structured, and unstructured data.</li><li style="text-align:justify;">Design standardized data ingestion processes with automated error handling, retries, and alerting.</li></ul>3. Data Quality, Monitoring, and Governance:<ul><li style="text-align:justify;">Develop data quality checks, validation rules, and anomaly detection mechanisms to ensure data integrity across all layers.</li><li style="text-align:justify;">Integrate monitoring and observability tools (e.g., Databricks metrics, Grafana) to track ETL performance, latency, and failures.</li><li style="text-align:justify;">Implement Unity Catalog or equivalent tools for centralized metadata management, data lineage, and governance policy enforcement.</li></ul>4. Security, Privacy, and Compliance:<ul><li style="text-align:justify;">Enforce data security best practices including row-level security, encryption at rest/in transit, and fine-grained access control via Unity Catalog.</li><li style="text-align:justify;">Design and implement data masking, tokenization, and anonymization for compliance with privacy regulations (e.g., GDPR, FERPA).</li><li style="text-align:justify;">Work with security teams to audit and certify compliance controls.</li></ul>5. AI/ML-Ready Data Foundation:<ul><li style="text-align:justify;">Enable data scientists by delivering high-quality, feature-rich data sets for model training and inference.</li><li style="text-align:justify;">Support AIOps/MLOps lifecycle workflows using MLflow for experiment tracking, model registry, and deployment within Databricks.</li><li style="text-align:justify;">Collaborate with AI/ML teams to create reusable feature stores and training pipelines.</li></ul>6. Cloud Data Architecture and Storage:<ul><li style="text-align:justify;">Architect and manage data lakes on Azure Data Lake Storage (ADLS) or Amazon S3, and design ingestion pipelines to feed the bronze layer.</li><li style="text-align:justify;">Build data marts and warehousing solutions using platforms like Databricks.</li><li style="text-align:justify;">Optimize data storage and access patterns for performance and cost-efficiency.</li></ul>7. Documentation & Enablement:<ul><li style="text-align:justify;">Maintain technical documentation, architecture diagrams, data dictionaries, and runbooks for all pipelines and components.</li><li style="text-align:justify;">Provide training and enablement sessions to internal stakeholders on the Databricks platform, Medallion Architecture, and data governance practices.</li><li style="text-align:justify;">Conduct code reviews and promote reusable patterns and frameworks across teams.</li></ul>8. Reporting and Accountability:<ul><li style="text-align:justify;">Submit a weekly schedule of hours worked and progress reports outlining completed tasks, upcoming plans, and blockers.</li><li style="text-align:justify;">Track deliverables against roadmap milestones and communicate risks or dependencies.</li></ul><div align="center" style="margin-left:24px;text-align:center;"><hr align="center" size="2" width="100%"></div>Required Qualifications:<ul><li style="text-align:justify;">Hands-on experience with Databricks, Delta Lake, and Apache Spark for large-scale data engineering.</li><li style="text-align:justify;">Deep understanding of ELT pipeline development, orchestration, and monitoring in cloud-native environments.</li><li style="text-align:justify;">Experience implementing Medallion Architecture (Bronze/Silver/Gold) and working with data versioning and schema enforcement in enterprise grade environments.</li><li style="text-align:justify;">Strong proficiency in SQL, Python, or Scala for data transformations and workflow logic.</li><li style="text-align:justify;">Proven experience integrating enterprise platforms (e.g., PeopleSoft, Salesforce, D2L) into centralized data platforms.</li><li style="text-align:justify;">Familiarity with data governance, lineage tracking, and metadata management tools.</li></ul><div align="center" style="margin-left:24px;text-align:center;"><hr align="center" size="2" width="100%"></div>Preferred Qualifications:<ul><li style="text-align:justify;">Prior UMGC or USM experience preferred.</li><li style="text-align:justify;">Experience with Databricks Unity Catalog for metadata management and access control.</li><li style="text-align:justify;">Experience deploying ML models at scale using MLFlow or similar MLOps tools.</li><li style="text-align:justify;">Familiarity with cloud platforms like Azure or AWS, including storage, security, and networking aspects.</li><li style="text-align:justify;">Knowledge of data warehouse design and star/snowflake schema modeling.</li></ul>

Back to blog