Data Engineering on Azure (DP-203)

Doe hands-on en diepgaande ervaring op met de Azure Data Engineering toolbox

Beschrijving

In deze training leer je over data engineering op het Azure data platform.

Hierbij begint je met de compute en storage technologiën die je gebruikt om een analytics-oplossing te bouwen. Vervolgens ontdek je hoe je analytics serving layers ontwerpt en hoe je bronbestanden en -systemen kunt inzetten.

Je gaat interactief data exploration toepassen op bestanden in een data lake middels Serverless SQL pools of Apache Spark pools. Deze bestanden leer je vervolgens op verschillende manieren ontsluiten op Azure Synapse Analytics of Azure Databricks, of met Azure Data Factory en Azure Synapse pipelines. Je maakt daarbij gebruik van COPY, Polybase en andere technologiën om ook bij big data performant te blijven. Deze data leer je daarna te transformeren met dezelfde technologiën.

Je leert over het belang van security om te zorgen dat data zowel at rest als in transit volledig beschermd is.

Uiteindelijk leer je de data in te zetten voor real-time analytics oplossingen.

Deze training geldt ook als voorbereiding op het examen DP-203: Data Engineering on Microsoft Azure. Heb je dit examen gehaald dan ben je Microsoft Certified: Azure Data Engineer Associate. Examenvoucher is niet inbegrepen.

Onderwerpen

In most organizations, a data engineer is the primary role responsible for integrating, transforming, and consolidating data from various structured and unstructured data systems into structures that are suitable for building analytics solutions. An Azure data engineer also helps ensure that data pipelines and data stores are high-performing, efficient, organized, and reliable, given a specific set of business requirements and constraints.

Lessons

• Introduction to data engineering on Azure
• Introduction to Azure Data Lake Storage Gen2
• Introduction to Azure Synapse Analytics

If you have large volumes of data stored as files in a data lake, you'll need a convenient way to explore and analyze the data they contain. Azure Synapse Analytics enables you to apply the SQL skills you use in a relational database to files in a data lake.

Lessons

• Use Azure Synapse serverless SQL pool to query files in a data lake
• Use Azure Synapse serverless SQL pool to transform data in a data lake
• Create a lake database in Azure Synapse Analytics

Apache Spark is a highly scalable distributed processing solution for big data analytics and transformation. You can leverage its power in Azure Synapse Analytics by using Spark pools.

Lessons

• Analyze data with Apache Spark in Azure Synapse Analytics
• Transform data with Spark in Azure Synapse Analytics
• Use Delta Lake in Azure Synapse Analytics

Relational data warehouses are at the heart of many business intelligence and enterprise analytics solutions. You can use Azure Synapse Analytics to implement highly scalable data warehouses in the cloud.

Lessons

• Analyze data in a relational data warehouse
• Load data into a relational data warehouse

Azure Synapse Analytics enables data integration through the use of pipelines, which you can use to automate and orchestrate data transfer and transformation activities.

Lessons

• Build a data pipeline in Azure Synapse Analytics
• use Spark Notebooks in an Azure Synapse Pipeline

Hybrid Transactional and Analytical Processing (HTAP) is a technique for near real time analytics without a complex ETL solution. In Azure Synapse Analytics, HTAP is supported through Azure Synapse Link.

Lessons

• Plan hybrid transactional and analytical processing using Azure Synapse Analytics
• Implement Azure Synapse Link with Azure Cosmos DB
• Implement Azure Synapse Link for SQL

Stream processing enables you to capture and analyze data in real-time. Azure Stream Analytics is a cloud-based stream processing engine that you can use to build highly scalable real-time analytics solutions.

Lessons

• Get started with Azure Stream Analytics
• Ingest streaming data using Azure Stream Analytics and Azure Synapse Analytics
• Visualize real-time data with Azure Stream Analytics and Power BI

Use Microsoft Purview to register and scan data, catalog data artifacts, find data for reporting, and manage Power BI artifacts to improve data governance in your organization.

Lessons

• Introduction to Microsoft Purview
• Integrate Microsoft Purview and Azure Synapse Analytics

Learn how to harness the power of Apache Spark and powerful clusters running on the Azure Databricks platform to run large data engineering workloads in the cloud.

Lessons

• Explore Azure Databricks
• Use Apache Spark in Azure Databricks
• Use Delta Lake in Azure Databricks
• Use SQL Warehouses in Azure Databricks
• Run Azure Databricks Notebooks with Azure Data Factory