Databricks for Data Engineers: advanced techniques

Learn best practices for using the Databricks Platform as a data engineer.

Description

In this training, you build on your foundational Databricks knowledge and develop a data platform using professional best practices within a realistic mock‑up scenario. You gain hands‑on experience with connecting new data sources, configuring catalogs, setting up security, and working with Git and Databricks Asset Bundles. You apply ingestion techniques such as Merge Into, Lakeflow Connect, and streaming ingestion to reliably process data. Throughout the labs, you put each concept into practice, giving you concrete experience with both batch and streaming workloads.

With declarative Lakeflow Pipelines, you transform data and combine sources into meaningful use cases. You also learn how to monitor your environment using system tables and SQL alerts to detect anomalies in production processes early. In addition, you create metric views and lightweight dashboards that clearly communicate results to end users. By the end of the training, you can deliver a full end‑to‑end data flow (from source to dashboard) including alerting and operational visibility.

Learning Goals

• Describe core Databricks workspace and platform concepts. [Remember]
• Explain how catalogs, schemas, and permissions work in Unity Catalog. [Understand]
• Apply ingestion techniques such as Merge Into, Lakeflow Connect, and streaming ingestion. [Apply]
• Implement data contracts, catalog structures, and basic security settings. [Apply]
• Construct declarative Lakeflow Pipelines for transforming and combining datasets. [Apply]
• Analyze ingestion and transformation runs using system tables, logs, and SQL alerts. [Analyze]
• Produce metric views and dashboards to present operational insights. [Apply]
• Interpret monitoring signals to identify anomalies in batch or streaming workloads. [Understand]
• Implement an end‑to‑end data flow from source ingestion to dashboarding. [Apply]

Subjects

1. Introduction & Databricks Environment
2. Ingestion
3. Transformation
4. Monitoring
5. Serving

• Introduction
• Catalog configuration
• Grants & security
• Git integration
• Lab: Load repository, prepare mock-up data platform, use Databricks CLI

• Data contracts
• Merge Into
• Connecting a Parquet source
• Lakeflow Connect
• Cluster configuration tuning & serverless
• Lab: Connect Parquet source
• Lab: Connect streaming source

• Lakeflow Declarative Pipelines
• SQL Alerts
• Lab: Build pipelines based on existing and new sources

• System tables
• System dashboarding
• Lab: Review pipeline results

• Metric Views
• Dashboarding
• Genie Space
• Power BI Desktop
• Lab: Build metric views to support dashboards
• Lab: Create dashboards and monitor end‑to‑end operations