Shuffling is the act of redistributing data so that it’s grouped differently across partitions. This typically involves copying data across executors and machines, making the shuffle a complex and costly operation.
Writing in progress: If you have any suggestions for improving the content or notice any inaccuracies, please email me at [email protected]. Thanks!
What is shuffling in Spark?
Explore more like this
Partitions and Bucketing in Spark
Partitioning and bucketing are used to improve the reading of data by reducing the cost of shuffles, the need for serialization, and the amount of network traffic.
Senthil Nayagan
Jul 25, 2022 - 13 Mins Read
Jul 25, 2022 - 13 Mins Read
Need for Caching in Apache Spark
Caching is one of Spark's optimization strategies for reusing computations. It stores interim and partial results so they'll be utilised in subsequent computation stages.
Senthil Nayagan
Jul 24, 2022 - 2 Mins Read
Jul 24, 2022 - 2 Mins Read
Comments