Big data with Pyspark

Big data history The term ‘Big Data’ has

What is distributed and single system A distributed

HDFS overview HDFS is a distributed file system

Mapreduce Overview MapReduce is a distributed computing model

Spark Overview Apache Spark is a lightning-fast cluster

Spark Ecosystem Overview The Apache Spark ecosystem is an

SparkContext Spark SparkContext is an entry point to

Spark Architecture Overview Apache Spark has a well-defined

SPark RDD overview Resilient Distributed Dataset (RDD) is the

textFile using Text File we can read the

sortByKey it will sort the record using Key

count() rdd1 = sc.parallelize([1,2,3,4,5,3,2])Action count() returns the number

PySpark DataFrame DataFrame definition is very well explained

Spark Read CSV file into DataFrame Using spark.read.format("csv").load("path")

pyspark joins it is used to join the

PySpark SQL overview PySpark SQL is a module

Reading the data from HDFS File in this

pyspark S3 Connection For connecting Pyspark with S3

python JSON Overview PySpark JSON functions are used

Add Your Heading Text Here Lorem ipsum dolor

Python Window Function. PySpark Window functions are used

Add Your Heading Text Here Lorem ipsum dolor

Add Your Heading Text Here Lorem ipsum dolor

Spark Repartition() vs Coalesce() overview Spark repartition() vs