Lambda Architecture

Lambda architecture starts with a messaging layer where stream data is queued for processing. Often uses Kafka as a messaging layer.

The data flow is then split into two parallel paths:

• Batch layer - handles data in files or other batch processes
• Streaming layer (also known as speed layer) - can be consumed instantly with an API

Both layers then land in a serving layer (often a data warehouse) where analytics processes can access the data.

Lambda Architecture Flow

Data Processing Steps:

1. Data Sources → Kafka/Messaging Layer
2. Data Split → Two parallel paths:
   • Batch Layer: For historical data
   • Speed Layer: For real-time data
3. Merge Results → Serving Layer (Data Warehouse)
4. Final Output → Analytics & Queries

Advantages:

• Can use batch processing for reliable, comprehensive data processing
• Can use streaming only when needed for real-time requirements
• Enables fast access when needed but keeps the reliability of batch processing
• Batch process can do thorough validation and cleaning

Disadvantages:

• Complexity: Need to maintain two separate processing pipelines
• Timing challenges: Data timing is more difficult to manage as some processes run daily and others in real-time
• Data consistency: Potential for discrepancies between batch and streaming results

Kappa Architecture

In Kappa architecture, the message layer coordinates everything. All data gets passed through the speed layer as it’s received and lands directly in a serving layer.

Kappa Architecture Flow

Data Processing Steps:

1. Data Sources → Kafka/Messaging Layer
2. Single Stream → Speed Layer (Stream Processing)
3. Real-time Processing → Serving Layer (Data Warehouse)
4. Final Output → Analytics & Queries

Historical Data Handling:

• Historical Data → Replay from Kafka → Speed Layer

Advantages:

• Fast processing: Everything is handled in real-time
• Simpler architecture: Fewer components to maintain and debug
• Consistency: Single processing path eliminates data discrepancies
• Scalability: Easier to scale a single processing pipeline

Disadvantages:

• Implementation challenges: Difficult to achieve when files are used to transfer data
• Legacy compatibility: Legacy systems often can’t stream data
• Historical data: Requires replay capabilities from message queue for historical analysis

When to Use Each Architecture

Choose Kappa if:

• All your data sources can stream data
• You need real-time processing for all use cases
• You want a simpler, more maintainable architecture

Choose Lambda if:

• You have batch data sources that can’t stream
• You need the reliability and thoroughness of batch processing
• You’re dealing with legacy systems that require batch processing

Logging is an action of recording the events that happen in the program.

1. Print 📝

The simplest way to log is to use the print function. This is okay for initial development, but not so much for production. It’s not so easy to filter messages made by print. And when you spam too many prints in production, let’s say on a Jenkins instance, you are risking DDoSing it. Don’t ask how I know this. 😅

Print issues:

• No level filtering: Everything prints, even debug info in production
• Not so easy to test, write to files, etc.

2. Logging 🪵

This is the preferred way to capture events in any Python application that’s running in production.

import logging

# Configure logging
logging.basicConfig(
    level=logging.WARNING,
    format='%(asctime)s | %(levelname)s: %(message)s',
    handlers=[
        logging.StreamHandler() # Surfaces logs to console
    ]
)

# Won't show because level is set to WARNING
logging.debug("Processing request.")
logging.info("Application started successfully")

# These will show (level WARNING and above)
logging.warning("High memory usage detected")
error_code = 404
logging.error("Error %d occurred during processing", error_code)

• Note how %s was used instead of f-strings (f”{error_code} …”). This is because this way it’s lazy and won’t evaluate unless the log level for it is enabled.
• StreamHandler controls where your logs go. By default, logs go to the console, but you can send them to files or other destinations.

3. Log Levels

Python’s logging module provides different levels of severity:

• DEBUG (10): Detailed information for diagnosing problems
• INFO (20): General information about program execution
• WARNING (30): Something unexpected happened, but the program can continue
• ERROR (40): A more serious problem occurred
• CRITICAL (50): A critical problem that may prevent the program from running

✅ Delta Table Benefits:

• Tracks the changes to the data (deltas) and allows for easy rollback.
• Performance optimisations like collecting statistics on columns.
• Supports ACID transactions.
• Can enforce schema on writes and simple schema evolution.
• Works with both batch and streaming.

How does it work?

• Data is stored as Parquet files, while metadata and transaction history are tracked using log files stored in a hidden _delta_log/ directory inside the table directory.

The _delta_log/ contains:

• JSON log files: Each file represents a single atomic commit and contains metadata about added/removed files and schema changes.
• Checkpoint Parquet files: Periodically written to speed up reads, summarizing state up to a point (instead of scanning many JSON files).

When querying or writing to a Delta table:

• Delta reads the latest checkpoint + newer JSON logs to reconstruct the current table state.
• Changes (inserts, updates, deletes) are recorded as new log entries and new Parquet files — old files are not mutated, enabling versioning.
• This structure allows versioned reads, rollback to previous versions, and concurrent writes with isolation.

⚠️ Things to watch out for:

• Requires maintenance that need to be thought of when designing the pipelines. This includes VACUUM to delete the table backups that can take up a lot of storage and OPTIMIZE to reduce the number of files. Not having this may be okay to begin with but will eventually pile up as tech debt.
• Partitioning needs to be managed manually on write and needs to be well thought of. Liquid clustering in Databricks can help to simplify the challenge of choosing the right partition key and updating it as the data evolves.
• Monitor _delta_log growth and table versions. It’s possible to accidentally create a lot of versions that will take up 100 or 1000 times the storage of the current version of the table. Example for this is when you do an upsert that uses bad merge conditions, such as checking on equality of columns that will be different for every run- such as load_timestamp or updated_at.

References:

• Delta Lake Official Website
• Databricks Delta Lake Documentation
• Liquid clustering in Databricks

This is because of file pruning - an optimisation technique that is used by Spark. It allows the query engine to only look into files that match the expected criteria - based on column statistics like min/max values. If the filter condition is not met - the file is skipped, saving us time and money.

By default, only the first 32 columns get statistics collected. So if you have a column that’s frequently used for filters, make sure it’s not at the end of a very wide table!

• 🐌 Without optimisation: Spark reads every file, even when the filter would eliminate most data
• ⚡ With column reordering: Moving the frequently-filtered column to the first 32 positions enables file pruning
• 📈 Performance gain: Queries can be 10x faster or more, depending on data distribution

P.S. The default can be changed with spark.databricks.io.skipping.defaultNumIndexedCols but I think it makes a lot of sense to put the most frequently queried columns at the beginning when designing the table.

For another post

• Common Spark performance bottlenecks
• Data partitioning strategies

Ask your fav coding agent to build a tic-tac-toe for you. Easy. Gemini CLI, Cursor, Copilot they should all do it without much problem.

Exciting? Definitely. Groundbreaking? Not just yet.

The result that we get is not much different from a google search and cloning a repository for Github. These problems have been solved multiple times and made public exactly as per the prompt.

Level 2: a multi player snake game

How do you handle the game logic, graphics and the state? The requirements for the above start to become more ambiguous.

Shameless plug: You can have a go at my vibe coded snake game here.

To produce this I had a lot of back and forward with the agent that was making all kinds of mistakes, somehow going into hallucination loops or breaking existing feature when trying to implement the new ones. To say shortly, it was far from 1 shot prompt. The result of it is a bit of a mess but it kind of works. For now!

AI will accelerate our work and information retrieval but It will not replace knowing best patterns and practices in building applications in our domain. As the application grows beyond the complexity of a simple portfolio projects, relying on agents to do all the work is a recipe for disaster.

Final level: high scale enterprise projects

In the real world, the requirements are not always clear. Are we gonna choose the technology X or Y? What are the tradeoffs? Does our existing tech stack/ future roadmap/ non-functional requirements dictate a specific solution? Where do we source the system components from and how can they be validated?

For such cases I think we are very far from delegating the work to agents!

• It’s a smaller query defined on the top of the script that can be used in a larger query later on.
• It helps with readability and simplicity of SQL scripts.
• Abstracts away complex logic making larger queries more manageable.

Example: We want to find tennis players with more than average number of aces.

WITH avg_aces AS (
    SELECT AVG(aces) AS avg_val
    FROM players
)

SELECT player, aces
FROM players, avg_aces
WHERE aces > avg_val;

tips:

• Keep CTEs small and single purpose.
• If we keep coming back to the same query, creating a view or a table may be a more efficient way to store the result.

2. Use window functions

• They allow us to perform calculations and lookups across rows of a table.
• Unlike aggregate functions, they don’t reduce the number of rows
• Result in cleaner queries as we avoid excessive joins.
• Keep the granularity of individual rows while accessing summary-level insights.

Example: We want to find how many more or less aces a player had compared to their previous match.

SELECT player,
    match_date,
    aces,
    LAG(aces) OVER (
        PARTITION BY player ORDER BY match_date
    ) AS prev_match_aces
FROM player_stats;

3. Intentional JOINs

• Understand the data relationships before the join (one to one, one to many, many to many)
• Don’t assume keys are unique. Expect duplicates.
• Pick the right join type for the intended use case.

watch out for:

• Avoid unintended data duplication, always check before and after join unique counts.

ID comparison

Quite useful pattern is the ANTI JOIN that allows us to find ids that don’t have a match in the other table. This way we can find players that didn’t play any games.

SELECT *
FROM players
LEFT JOIN games
ON players.player_id = games.player_id
WHERE games.player_id IS NULL;

Row comparison

on the other hand, for quick full-row comparisons between two tables, we can use the EXCEPT operator. This will return all rows from the first table that are not present in the second table.

SELECT *
FROM games_1
EXCEPT
SELECT *
FROM games_2;

1. ⚖️ Skew

• Skew is an uneven distribution of data across partitions.
• In a dataset where one value occurs more frequently than others, this can lead to uneven distribution of tasks across the workers.
• It can lead to slow tasks (parallelism not used enough), or the task can timeout and fail.

Solution:

• Enable adaptive query execution that automatically detects and handles skew.
• Use salting to distribute data evenly. By creating an artificial column, you can split the frequently occurring values into multiple partitions.
• Filter out and process skewed values separately.

2. 🔄 Shuffle

• Most expensive operation in Spark.
• Costly data movement across the cluster
• Happens when you call an operation that requires an exchange of data between the nodes (e.g., groupby and join).

Solution:

• Filter early.
• Use broadcast join when possible (when one of the tables is small enough, you can send it to all the nodes for instant retrieval).
• Use partitioning to reduce the amount of data that needs to be shuffled.
• Use window functions instead of groupby & joins when possible.

3. 💾 Spill

• Happens when the data doesn’t fit in memory of a executor.
• Memory limits → disk usage -> slow tasks

Solution:

• Select only the columns you need, filter early.
• Reduce cardinality (the number of unique values in a column).
• Increase memory per core.

⚠️ Common Pitfalls

1. 🗂️ Too many/ too few partitions

• Too many small files cause issues with file reads and metadata management, negatively impacting query performance.
• Few large files result in poor parallelism.
• ✅ Use auto-compaction (e.g., when using Delta tables).
• On Databricks, you can use the OPTIMIZE command to compact small files into larger files and improve query performance.

2. Setting the wrong partition

• Best to partition on columns with low to moderate cardinality (dozens to thousands of values)
• When designing the data ingestion process, work from the end goal in mind.
• To pick the right column for partitioning, consider the consumption patterns, including most frequently used filters, joins, and aggregations.

3. Following Spark anti-patterns

• Iterating over rows. Is a no-no as it can’t be parallelised.
• Using UDFs when not needed. When possible, use Spark built-in functions instead.
• Calling collect() or .toPandas() on big datasets—defeats the purpose of using Spark.

4. Not using caching when you should

• Due to Spark’s lazy evaluation, whenever you call an action, the data is cleared from memory.
• To avoid expensive recomputation after you call an action, cache the DataFrame if you plan to use it multiple times; otherwise, it will be recomputed every time.

What exactly is Spark and why do we need it to process big data?

As of 2025, there is around 402 million terabytes of data created every single day.

Big Data Contributors 🌐

Big contributors are:

• 🌐 Search Data & general internet activity (Google Search and Analytics)
• 📱 Social Media Data
• 📡 Internet of Things (IoT) Data (your smart fridge monitoring the milk levels)
• 💸 Financial Transactions (banking, payments, crypto, etc.)

Companies capitalise on this, to make better decisions, improve their products and services and ultimately to increase their revenue.

In order to do this they have to think about:

• 💾 storage
• ⚙️ processing

Storage vs Processing 💾⚙️

Storage is now cheaper than ever, thanks to the cloud. SSDs got much cheaper and cloud tenants don’t need to worry about hardware maintenance and upgrades. Happy times for data hoarders.

Processing got more flexible and scalable. Want a supercomputer deployed in minutes? No problem. Want it in seconds? We have serverless.

Scaling Approaches 📈

What happens if we have more data than we can process with a single machine?

We have two options:

• 🦣 Increase the size of the machine (Vertical scaling)
• 🀟 Add more machines (Horizontal scaling)

Increasing the size of a machine can be expensive and limited. When this machine goes down for a second, the whole job fails.

Distributed Computing/ big data frameworks like Spark solved this problem by allowing parallel processing with a number of machines.

Spark Architecture ⚡

The architecture of Spark is:

• Driver
• Executors

Together they form a cluster.

Driver picks up tasks and shares it among executors. When all tasks are completed, the driver sends the results back to the user.

What is great about Spark 👍

• 🌩️ You can use it on any of the major clouds: AWS, Azure, GCP, or Databricks- created by the original Spark developers.
• 🆓 It’s open source and free- you can download it on your laptop and start developing and testing your code.
• 🚚 It allows you to process tons terabytes of data

What is not so great about Spark 👎

• 🧗 Has a steep learning curve: it’s not easy to get started due to configuration and dependencies.
• 🧩 It’s more complex than just SQL or pandas query- apart from business logic we need to think more about how the code is going to be executed.
• ⏳ It has a lot of overhead: spinning up the cluster, sending data back and forth etc.

Analogy ✈️

Although often marked as a golden bullet, the reality is that most companies don’t need Spark- they don’t have enough data to justify the cost and the added complexity.

A great analogy is that in theory, you could take a plane to travel within a city as long as it has 2 airports. But that wouldn’t be very efficient and you end up spending more time and energy.

On the other hand, if you travel across continents, you need a plane and don’t mind spending a few hours in the airport and checking in your luggage.

For another post 📝

• Common Spark issues and optimisations

• 🕵️‍♂️ Detecting data quality issues
• 📊 Understanding distributions of data
• 🚨 Monitoring processes and triggering alerts

Mean (average): The most commonly used measure of average. sum(n)/count(n)

Median: Middle value of a sorted list. [0, 5 , (24), 50, 100]. Useful when dealing with outliers.

Mode: Most frequent value in a list. [0, 3, 3, 1000] -> 3

Relationship between mean and median under different skewness

In a perfectly normal distribution, the mean, median, and mode are the same.

Histogram 📊 It’s good to be able to use a histogram to check the distribution of the data, detect outliers and see if the data is skewed which can help with sampling/ partitioning decisions.

Variance: Measure of how spread out a list of numbers is. Average of squared differences from mean. Not so easy to interpret—”On average, the squared difference from the mean is 2.67.”

Standard Deviation: Measure of how spread out a list of numbers is. Square root of variance. Easier to interpret—”The values on average are 1.63 units away from the mean.” Useful for monitoring systems where values deviating from the average might signal a problem.

Percentiles: The relative standing of a value within a dataset. For example, if you’re in the 90th percentile in a test, you scored higher than 90% of participants.

Quartiles: Like percentiles but simplified to 4 parts where values may stand:

• Q1 (25th percentile): 0% - 25%
• Q2 (50th percentile): 25% - 50%
• Q3 (75th percentile): 50% - 75%
• Q4 (100th percentile): 75% - 100%

Useful when specifying SLAs or when we need a simplified distribution summary.

Correlation vs Causation 🔗🚫: Correlation measures the relationship between two variables. Causation takes place when one variable directly influences another. Correlation does not imply causation.

Example: Drowning deaths and ice cream sales are correlated. Every year, in the summer ice cream sales go up and so do drowning deaths. But this does not mean that ice cream sales cause drowning deaths.

Correlation vs Causation

In the next part I will go through:

• 🧪 A/B testing

📌 Must have

1. SQL - a language for interacting with databases. Must have for querying and manipulating data.
   Used in all modern data platforms.

   Useful commands:

   • SELECT column_name FROM table_name - simple query to get data from a table
   • WHERE - filter the data
   • ORDER BY - sort the data, to answer questions like “what is the most popular product?”
   • GROUP BY - group the data, ex. for calculating user monthly spend
   • JOIN - join together two tables by a common column
   • WINDOW FUNCTIONS - aggregations over subsets of rows, without changing the granularity of the data

   Knowing the above should cover 90% of SQL use cases.

2. Python - a general purpose programming language that excels in data.
   It simplifies the complexity and makes it easier to focus on the business logic.

   Opens the door to many applications, including big data and AI.
   It’s power is in a super wide range of open source libraries and frameworks.

   You can use it for anything data related: transforming data, visualising it, training machine learning models, bu not only. It can also be used in web development and automation so its definitely worth learning.

   libraries: pandas, matplotlib, sci-kit learn, etc.

3. Linux - an entry point into running scripts and automation.
   Has a very steep learning curve, knowing the basics will make you more productive.

   Useful commands:

   • ls - list files and directories
   • cd - change directory
   • mkdir - make directory
   • rm - remove file
   • cp - copy file
   • mv - move file
   • cat - print file content
   • chmod - change file permissions

✨ Nice to have

1. git - a way to go for code versioning. Helps with staying sane over code changes and face the consequences of accidentally deleting your work. (we have all been there). Helps to keep track of changes introduced in our applications and make the quality of code easier to maintain through the process of Pull Requests.

2. Orchestration - a way to automate the execution of tasks and manage the dependencies between them.
   For simpler use cases you can use cron jobs that are built in linux tool for simple scheduling.
   For bigger workflows, requiring more complex dependencies, you will need something like Airflow or Dagster.
   Must have for any data platform.

3. Cloud - when you want to step away from your local laptop and have your data applications running live reliably, you need to deploy it on some kind of server. A natural choice is public cloud providers like AWS, GCP or Azure because buying and stacking hardware is not cool anymore.

   Not all data engineering roles interact with it directly, I found that operations heavy and data platform jobs tend to need it the most.

4. CI/CD & infrastructure as code - a way to automate the deployment of your code. Be able to reliably tell that your code is working as expected before it goes live. These are MUST HAVE considerations for enterprise scale applications but if you are just getting started, I wouldn’t worry about them at the start. This is rather a controversial take, but I have seen startup projects fail because they followed enterprise best practices too early before identifying the right problems to solve.

examples: Terraform, Github Actions

1. Big Data - Must have when data is too big to fit into memory of a single machine. Although it has a lot of use cases, a lot of companies don’t really need it due to the scale of their data.

   This is a broad topic so expect separate posts on this. example: Apache Spark.

Is that all? No! There are thousands of other frameworks and technologies that will come up in your projects! But mastering the fundamental tools that are not going anywhere soon seems the best time investment at the start. They keep coming up and not going anywhere.

TLDR, this is how I would prioritise learning the tools if I started my journey again: