Apache Spark for Data Engineers Masterclass

Verdiep je kennis van Apache Spark om je dataworkflow te optimaliseren.

Beschrijving

In deze cursus leer je technieken en best practices voor het optimaliseren van Apache Spark-toepassingen. Je bestudeert de architectonische elementen van Spark en werkt met de Spark UI. Je identificeert en pakt veelvoorkomende prestatieproblemen veroorzaakt door shuffles en skew aan. Daarnaast leer je geavanceerde optimalisatiestrategieën voor join-, union- en merge-operaties, gegevensformaten, cachingmechanismen, garbage collector-instellingen, gegevenspartitionering, bucketing en Delta Lake-optimalisaties. Je verkent ook reguliere onderhoudstaken voor Spark-toepassingen en leert hoe je Spark-sessieconfiguraties kunt aanpassen voor optimale prestaties.

Leerdoelen

• Describe the architecture of a spark application. [Remember]
• Explain the structure and functionality of the Spark UI. [Understand]
• Predict common performance issues casued by shuffling and data skew. [Apply]
• Optimize join, union, and merge operations in Spark. [Analyze]
• Change the data format for optimal performance. [Apply]
• Implement caching mechanisms and garbage collector settings for enhanced performance. [Apply]
• Use data partitioning and bucketing in Spark workloads. [Apply]
• Apply Delta Lake optimizations for better performance in Spark. [Apply]
• Describe regular maintenance tasks for Spark applications. [Understand]
• Customize Spark session configurations for optimal performance. [Apply]

Onderwerpen

1. Introduction to Spark Architecture and Ecosystem
2. Understanding the Spark UI
3. Common Performance Issues in Spark
4. Optimizing Data Operations in Spark
5. Data Formats and Performance
6. Caching and Garbage Collection in Spark
7. Data Partitioning and Bucketing
8. Delta Lake Optimizations
9. Maintenance of Spark Applications
10. Customizing Spark Session Configurations

• Overview of Spark architecture
• Key components: Driver, Executors, Cluster Manager
• The ecosystem: JVM, Kubernetes, Yarn, HDFS, Hive Metastore

• Structure of the Spark UI
• Functionality of different tabs (Jobs, Stages, Storage, Environment, Executors)
• Monitoring and diagnosing Spark applications

• Shuffles and Data Skew
• Sorting
• Narrow and Wide transformations

• Join operations: broadcast joins, shuffle joins
• Union and merge operations

• Common data formats such as json, csv and parquet
• Impact of data format on performance
• Making optimal use of data formats for Spark applications

• Caching mechanisms in Spark (cache(), persist())
• Data persistence
• Garbage collection settings and their impact on performance

• Partitioning strategies and impact in Spark
• Bucketing techniques and their benefits

• Introduction to Delta Lake
• Performance optimization in Delta Lake
• Delta Lake housekeeping

• Regular maintenance tasks for Spark applications
• Monitoring and diagnostics tools

• Spark session configurations and their impact on performance
• Common spark session parameters
• Customizing configurations for specific workloads