Data Engineering on Microsoft Azure (DP-203)

Module 1: Get started with data engineering on Azure

In most organizations, a data engineer is the primary role responsible for integrating, transforming, and consolidating data from various structured and unstructured data systems into structures that are suitable for building analytics solutions. An Azure data engineer also helps ensure that data pipelines and data stores are high-performing, efficient, organized, and reliable, given a specific set of business requirements and constraints.

Lessons:

• Introduction to data engineering on Azure
• Introduction to Azure Data Lake Storage Gen2
• Introduction to Azure Synapse Analytics

Module 2: Build data analytics solutions using Azure Synapse serverless SQL pools

If you have large volumes of data stored as files in a data lake, you'll need a convenient way to explore and analyze the data they contain. Azure Synapse Analytics enables you to apply the SQL skills you use in a relational database to files in a data lake.

Lessons:

• Use Azure Synapse serverless SQL pool to query files in a data lake
• Use Azure Synapse serverless SQL pools to transform data in a data lake
• Create a lake database in Azure Synapse Analytics
• Secure data and manage users in Azure Synapse serverless SQL pools

Module 3: Perform data engineering with Azure Synapse Apache Spark Pools

Apache Spark is a highly scalable distributed processing solution for big data analytics and transformation. You can leverage its power in Azure Synapse Analytics by using Spark pools.

Lessons:

• Analyze data with Apache Spark in Azure Synapse Analytics
• Transform data with Spark in Azure Synapse Analytics
• Use Delta Lake in Azure Synapse Analytics

Module 4: Transfer and transform data with Azure Synapse Analytics pipelines

Azure Synapse Analytics enables data integration through the use of pipelines, which you can use to automate and orchestrate data transfer and transformation activities.

Lessons:

• Build a data pipeline in Azure Synapse Analytics
• Use Spark Notebooks in an Azure Synapse Pipeline

Module 5: Implement a Data Analytics Solution with Azure Synapse Analytics

Lessons:

• Introduction to Azure Synapse Analytics
• Use Azure Synapse serverless SQL pool to query files in a data lake
• Analyze data with Apache Spark in Azure Synapse Analytics
• Use Delta Lake in Azure Synapse Analytics
• Analyze data in a relational data warehouse
• Build a data pipeline in Azure Synapse Analytics

Module 6: Work with Data Warehouses using Azure Synapse Analytics

Relational data warehouses are at the heart of many business intelligence and enterprise analytics solutions. You can use Azure Synapse Analytics to implement highly scalable data warehouses in the cloud.

Lessons:

• Analyze data in a relational data warehouse
• Load data into a relational data warehouse
• Manage and monitor data warehouse activities in Azure Synapse Analytics
• Secure a data warehouse in Azure Synapse Analytics

Module 7: Work with Hybrid Transactional and Analytical Processing Solutions using Azure Synapse Analytics

Hybrid Transactional and Analytical Processing (HTAP) is a technique for near real time analytics without a complex ETL solution. In Azure Synapse Analytics, HTAP is supported through Azure Synapse Link.

Lessons:

• Plan hybrid transactional and analytical processing using Azure Synapse Analytics
• Implement Azure Synapse Link with Azure Cosmos DB
• Implement Azure Synapse Link for SQL

Module 8: Implement a Data Streaming Solution with Azure Stream Analytics

Stream processing enables you to capture and analyze data in real-time. Azure Stream Analytics is a cloud-based stream processing engine that you can use to build highly scalable real-time analytics solutions.

Lessons:

• Get started with Azure Stream Analytics
• Ingest streaming data using Azure Stream Analytics and Azure Synapse Analytics
• Visualize real-time data with Azure Stream Analytics and Power BI

Module 9: Implement a data lakehouse analytics solution with Azure Databricks

Learn how to harness the power of Apache Spark and powerful clusters running on the Azure Databricks platform to run large data engineering workloads in the cloud.

Lessons:

• Explore Azure Databricks
• Perform data analysis with Azure Databricks
• Use Apache Spark in Azure Databricks
• Manage data with Delta Lake
• Build data pipelines with Delta Live Tables
• Deploy workloads with Azure Databricks Workflows