Développeur Big Data Hadoop et Spark - eLearning

- Introduction to Big Data and Hadoop
- Introduction to Big Data
- Big Data Analytics
- What is Big Data?
- Four vs of Big Data
- Case Study Royal Bank of Scotland
- Challenges of Traditional System
- Distributed Systems
- Introduction to Hadoop
- Components of Hadoop Ecosystem Part One
- Components of Hadoop Ecosystem Part Two
- Components of Hadoop Ecosystem Part Three
- Commercial Hadoop Distributions
- Demo: Walkthrough of Simplilearn Cloudlab
- Key Takeaways
- Knowledge Check

- Architecture Hadoop Stockage Distribué (HDFS) et YARN
- Qu'est-ce que HDFS
- La nécessité de HDFS
- Système de fichiers classique vs HDFS
- Caractéristiques de HDFS
- Architecture et composants de HDFS
- Mise en œuvre de clusters à haute disponibilité
- Espace de noms du système de fichiers des composants HDFS
- Fractionnement des blocs de données
- Topologie de réplication des données
- Ligne de commande HDFS
- Démonstration : Commandes HDFS courantes
- Projet pratique : Ligne de commande HDFS
- Introduction à Yarn
- Cas d'utilisation de Yarn
- Yarn et son architecture
- Gestionnaire de ressources
- Fonctionnement du gestionnaire de ressources
- Maître d'application
- Comment Yarn exécute une application
- Outils pour les développeurs Yarn
- Démonstration : Visite guidée de la première partie du cluster
- Démonstration : Visite guidée de la deuxième partie du cluster
- Points clés Vérification des connaissances
- Projet pratique : Architecture Hadoop, stockage distribué (HDFS) et Yarn

- Data Ingestion Into Big Data Systems and Etl
- Data Ingestion Overview Part One
- Data Ingestion Overview Part Two
- Apache Sqoop
- Sqoop and Its Uses
- Sqoop Processing
- Sqoop Import Process
- Sqoop Connectors
- Demo: Importing and Exporting Data from MySQL to HDFS
- Practice Project: Apache Sqoop
- Apache Flume
- Flume Model
- Scalability in Flume
- Components in Flume’s Architecture
- Configuring Flume Components
- Demo: Ingest Twitter Data
- Apache Kafka Aggregating User Activity Using Kafka
- Kafka Data Model
- Partitions
- Apache Kafka Architecture
- Demo: Setup Kafka Cluster
- Producer Side API Example
- Consumer Side API
- Consumer Side API Example
- Kafka Connect
- Demo: Creating Sample Kafka Data Pipeline Using Producer and Consumer
- Key Takeaways
- Knowledge Check
- Practice Project: Data Ingestion Into Big Data Systems and ETL

- Distributed Processing Mapreduce Framework and Pig
- Distributed Processing in Mapreduce
- Word Count Example
- Map Execution Phases
- Map Execution Distributed Two Node Environment
- Mapreduce Jobs
- Hadoop Mapreduce Job Work Interaction
- Setting Up the Environment for Mapreduce Development
- Set of Classes
- Creating a New Project
- Advanced Mapreduce
- Data Types in Hadoop
- Output formats in Mapreduce
- Using Distributed Cache
- Joins in MapReduce
- Replicated Join
- Introduction to Pig
- Components of Pig
- Pig Data Model
- Pig Interactive Modes
- Pig Operations
- Various Relations Performed by Developers
- Demo: Analyzing Web Log Data Using Mapreduce
- Demo: Analyzing Sales Data and Solving Kpis Using Pig Practice Project: Apache Pig

- Demo: Wordcount
- Key Takeaways
- Knowledge Check
- Practice Project: Distributed Processing - Mapreduce Framework and Pig

- Apache Hive
- Hive SQL over Hadoop MapReduce
- Hive Architecture
- Interfaces to Run Hive Queries
- Running Beeline from Command Line
- Hive Metastore
- Hive DDL and DML
- Creating New Table
- Data Types Validation of Data
- File Format Types
- Data Serialization
- Hive Table and Avro Schema
- Hive Optimization Partitioning Bucketing and Sampling
- Non-Partitioned Table
- Data Insertion
- Dynamic Partitioning in Hive
- Bucketing
- What Do Buckets Do?
- Hive Analytics UDF and UDAF
- Other Functions of Hive
- Demo: Real-time Analysis and Data Filtration
- Demo: Real-World Problem
- Demo: Data Representation and Import Using Hive
- Key Takeaways
- Knowledge Check
- Practice Project: Apache Hive

- NoSQL Databases HBase
- NoSQL Introduction
- Demo: Yarn Tuning
- Hbase Overview
- Hbase Architecture
- Data Model
- Connecting to HBase
- Practice Project: HBase Shell
- Key Takeaways
- Knowledge Check
- Practice Project: NoSQL Databases - HBase

- Les bases de la programmation fonctionnelle et Scala
- Introduction à Scala
- Démonstration : Installation de Scala
- Programmation fonctionnelle
- Programmation avec Scala
- Démonstration : Notions de base des littéraux et de la programmation arithmétique
- Démonstration : Opérateurs logiques
- Inférence de type, classes, objets et fonctions dans Scala
- Démonstration : Inférence de type, fonctions anonymes et classe
- Collections
- Types de collections
- Démonstration : Cinq types de collections
- Démonstration : Opérations sur les listes Scala REPL
- Démonstration : Fonctionnalités de Scala REPL
- Points clés
- Vérification des connaissances
- Projet pratique : Apache Hive

- Apache Spark Next-Generation Big Data Framework
- History of Spark
- Limitations of Mapreduce in Hadoop
- Introduction to Apache Spark
- Components of Spark
- Application of In-memory Processing
- Hadoop Ecosystem vs Spark
- Advantages of Spark
- Spark Architecture
- Spark Cluster in Real World
- Demo: Running a Scala Programs in Spark Shell
- Demo: Setting Up Execution Environment in IDE
- Demo: Spark Web UI
- Key Takeaways
- Knowledge Check
- Practice Project: Apache Spark Next-Generation Big Data Framework

- Introduction to Spark RDD
- RDD in Spark
- Creating Spark RDD
- Pair RDD
- RDD Operations
- Demo: Spark Transformation Detailed Exploration Using Scala Examples
- Demo: Spark Action Detailed Exploration Using Scala
- Caching and Persistence
- Storage Levels
- Lineage and DAG
- Need for DAG
- Debugging in Spark
- Partitioning in Spark
- Scheduling in Spark
- Shuffling in Spark
- Sort Shuffle Aggregating Data With Paired RDD
- Demo: Spark Application With Data Written Back to HDFS and Spark UI
- Demo: Changing Spark Application Parameters
- Demo: Handling Different File Formats
- Demo: Spark RDD With Real-world Application
- Demo: Optimizing Spark Jobs
- Key Takeaways
- Knowledge Check
- Practice Project: Spark Core Processing RDD

- Spark SQL Processing DataFrames
- Spark SQL Introduction
- Spark SQL Architecture
- Dataframes
- Demo: Handling Various Data Formats
- Demo: Implement Various Dataframe Operations
- Demo: UDF and UDAF
- Interoperating With RDDs
- Demo: Process Dataframe Using SQL Query
- RDD vs Dataframe vs Dataset
- Practice Project: Processing Dataframes
- Key Takeaways
- Knowledge Check
- Practice Project: Spark SQL - Processing Dataframes

- Modélisation Spark Mlib des grandes données avec Spark
- Rôle du Data Scientist et du Data Analyst dans les grandes données
- Analytique dans Spark
- Apprentissage automatique
- Apprentissage supervisé
- Démonstration : Classification avec SVM linéaire
- Démonstration : Régression linéaire avec des études de cas réelles
- Apprentissage non supervisé
- Démonstration : Clustering non supervisé avec K-means
- Apprentissage par renforcement
- Apprentissage semi-supervisé
- Vue d'ensemble de Mlib
- Pipelines Mlib
- Points clés à retenir
- Vérification des connaissances
- Projet pratique : Spark Mlib - Modélisation des grandes données avec Spark

- Streaming Overview
- Real-time Processing of Big Data
- Data Processing Architectures
- Demo: Real-time Data Processing Spark Streaming
- Demo: Writing Spark Streaming Application
- Introduction to DStreams
- Transformations on DStreams
- Design Patterns for Using Foreachrdd
- State Operations
- Windowing Operations
- Join Operations Stream-dataset Join
- Demo: Windowing of Real-time Data Processing Streaming Sources
- Demo: Processing Twitter Streaming Data
- Structured Spark Streaming-
- Use Case Banking Transactions
- Structured Streaming Architecture Model and Its Components
- Output Sinks
- Structured Streaming APIs
- Constructing Columns in Structured Streaming
- Windowed Operations on Event-time
- Use Cases
- Demo: Streaming Pipeline
- Practice Project: Spark Streaming
- Key Takeaways
- Knowledge Check
- Practice Project: Stream Processing Frameworks and Spark Streaming

- Spark GraphX
- Introduction aux graphes
- GraphX dans Spark
- Opérateurs GraphX
- Opérateurs de jointure
- Système parallèle GraphX
- Algorithmes dans Spark
- API Pregel
- Cas d'utilisation de GraphX
- Démonstration : Prédicat de sommet GraphX
- Démonstration : Algorithme de PageRank
- Points clés
- Vérification des connaissances
- Projet pratique : Assistance pour le projet Spark GraphX