Categories

Database

algorithms and data structures

An Introduction to Algorithms and Data Structures

An algorithm is a series of instructions in a particular order for performing a specific task.

Senthil Nayagan
Aug 25, 2022 - 10 Mins Read

amazon emr

Overview of Amazon EMR

Amazon EMR is a managed cluster platform that makes it easier to run big data frameworks like Apache Hadoop and Apache Spark on AWS to process and analyze huge amounts of data.

Senthil Nayagan
Sep 5, 2022 - 21 Mins Read

apache pinot

Apache Pinot joins hands with Kafka and Presto to provide low-latency, high-throughput user-facing analytics

Senthil Nayagan
Nov 14, 2022 - 18 Mins Read

apache spark

Partitions and Bucketing in Spark

Partitioning and bucketing are used to improve the reading of data by reducing the cost of shuffles, the need for serialization, and the amount of network traffic.

Senthil Nayagan
Jul 25, 2022 - 13 Mins Read

Need for Caching in Apache Spark

Caching is one of Spark's optimization strategies for reusing computations. It stores interim and partial results so they'll be utilised in subsequent computation stages.

Senthil Nayagan
Jul 24, 2022 - 2 Mins Read

apache yarn

aws

AWS Command Line Interface (AWS CLI)

Senthil Nayagan
Aug 12, 2022 - 4 Mins Read

aws
aws-cli

aws glue

coding problem solving

data engineering

Introduction to Data Engineering

It's the process of designing and building systems for gathering vast quantities of raw operational data from a variety of sources and formats, analyzing, converting, and storing it at scale.

Senthil Nayagan
Jul 24, 2022 - 5 Mins Read

data files and formats

Let’s Know About the Parquet File

An open source file format for Hadoop that provides columnar storage and is built from the ground up with complex nested data structures in mind.

Senthil Nayagan
Jul 25, 2022 - 1 Min Read

data lake

data lake and lakehouse

data management

Data Product vs. Data as a Product

A data product is not the same as data as a product. A data product aids the accomplishment of the product's goal by using the data, whereas in data as a product, the data itself is seen as the actual product.

Senthil Nayagan
Aug 2, 2022 - 3 Mins Read

Data Catalog

A data catalog is an ordered inventory of an organization's data assets that makes it easy to find the most relevant data quickly.

Senthil Nayagan
Jul 25, 2022 - 3 Mins Read

Data Deluge

When the granularity of data increases, its complexity also increases. At some point, we will reach a point where we cannot handle the volume of fresh data being generated.

Senthil Nayagan
Jul 18, 2022 - 1 Min Read

data quality

data security and compliance

Data Governance

Data governance is the process of defining security guidelines and policies and making sure they are followed by having authority and control over how data assets are managed.

Senthil Nayagan
Jul 26, 2022 - 4 Mins Read

Envelope Encryption - Putting Your Encryption Key in an Envelope Is the Safer Option

Envelope encryption is a way of encrypting plaintext data using a key and then encrypting that key using an another key. This strategy is intended not just to make things more secure but also to enhance performance.

Senthil Nayagan
Jul 22, 2022 - 6 Mins Read

data streaming

database

design patterns and coding principles

Singleton Pattern

A singleton pattern limits the number of instances of a class to one.

Senthil Nayagan
Jul 21, 2022 - 2 Mins Read

Anti-Pattern

Anti-patterns at first seem to be quick and reasonable, they typically have adverse effects in the future. They are design and code smells. It affects our software badly and adds technical debt. We should avoid them at all costs.

Senthil Nayagan
Jul 20, 2022 - 2 Mins Read

elasticsearch

knowledge graph

kubernetes

lake formation

lakehouse

orchestration

How To Set SLA in Apache Airflow

Apache Airflow enables us to schedule tasks as code. In Airflow, a SLA determines the maximum completion time for a task or DAG. Note that SLAs are established based on the DAG execution date, not the task start time.

Senthil Nayagan
Jul 24, 2022 - 4 Mins Read

postgresql

rpc

rust

Rust’s Ownership and Borrowing Enforce Memory Safety

Senthil Nayagan
Jul 19, 2022 - 40 Mins Read

scala

Access Modifiers in Scala

Access modifiers, also known as access specifiers, determine the accessibility and scope of classes, methods, and other members. Scala's access modifiers closely resemble those of Java, although they provide more granular and powerful visibility control than Java.

Senthil Nayagan
Sep 5, 2022 - 4 Mins Read

Case Class in Scala

The case class represents immutable data. It is a type of class that is often used for data storage.