https://samarthya.me/tags/spark/Recent content in Spark on SamarthyaHugo -- gohugo.ioen© Samarthya 2018-2026. All rights reserved.Thu, 09 Oct 2025 12:00:00 +0000https://samarthya.me/posts/beyond-the-dataframe-how-parquet-and-arrow-turbocharge-pyspark/Thu, 09 Oct 2025 12:00:00 +0000https://samarthya.me/posts/beyond-the-dataframe-how-parquet-and-arrow-turbocharge-pyspark/<figure><img class="my-0 rounded-md" loading="lazy" decoding="async" fetchpriority="auto" alt="" width="600" height="600" src="https://samarthya.me/posts/beyond-the-dataframe-how-parquet-and-arrow-turbocharge-pyspark/arrow-2.png" srcset="https://samarthya.me/posts/beyond-the-dataframe-how-parquet-and-arrow-turbocharge-pyspark/arrow-2.png 800w, https://samarthya.me/posts/beyond-the-dataframe-how-parquet-and-arrow-turbocharge-pyspark/arrow-2.png 1280w" sizes="(min-width: 768px) 50vw, 65vw" data-zoom-src="https://samarthya.me/posts/beyond-the-dataframe-how-parquet-and-arrow-turbocharge-pyspark/arrow-2.png"></figure> <p>In my <a href="https://samarthya.me/posts/pyspark-vs-pandas-choosing-right/" >last post</a>, we explored the divide between <strong>Pandas</strong> (single machine) and <strong>PySpark</strong> (distributed computing).</p> <blockquote><p>The conclusion: for massive datasets, PySpark is the clear winner.</p> </blockquote>https://samarthya.me/posts/pyspark-vs-pandas-choosing-right/Thu, 09 Oct 2025 12:00:00 +0000https://samarthya.me/posts/pyspark-vs-pandas-choosing-right/<figure><img class="my-0 rounded-md" loading="lazy" decoding="async" fetchpriority="auto" alt="" width="600" height="600" src="https://samarthya.me/posts/pyspark-vs-pandas-choosing-right/pysp-1.png" srcset="https://samarthya.me/posts/pyspark-vs-pandas-choosing-right/pysp-1.png 800w, https://samarthya.me/posts/pyspark-vs-pandas-choosing-right/pysp-1.png 1280w" sizes="(min-width: 768px) 50vw, 65vw" data-zoom-src="https://samarthya.me/posts/pyspark-vs-pandas-choosing-right/pysp-1.png"></figure> <p>As another project wraps up, bringing with it the bittersweet relief of an almost-closed chapter (and, yes, a few too many late-night coding sessions), I found myself reflecting on the tools that got us through.</p>https://samarthya.me/posts/apache-spark/Sun, 14 Sep 2025 12:00:00 +0000https://samarthya.me/posts/apache-spark/<figure><img class="my-0 rounded-md" loading="lazy" decoding="async" fetchpriority="auto" alt="" width="600" height="600" src="https://samarthya.me/posts/apache-spark/big-data.png" srcset="https://samarthya.me/posts/apache-spark/big-data.png 800w, https://samarthya.me/posts/apache-spark/big-data.png 1280w" sizes="(min-width: 768px) 50vw, 65vw" data-zoom-src="https://samarthya.me/posts/apache-spark/big-data.png"></figure> <p>So as part of the academic curriculum we just started re-exploring Apache Spark. It’s been one of the technology that I had long lost touch with. Ironical but true, in this age of big data and speed Spark was left somewhere way behind in my past projects working for Pitney Bowes.</p>