In database systems, a transaction is a sequence of operations (say SQL commands) executed as a single logical unit of work, ensuring that either all operations succeed together or none are applied. This principle ensures that any intermediate or partial changes during the execution of a transaction are never visible to the system. By maintaining this all-or-nothing behavior, transactions ensure that complex data operations, involving multiple interdependent steps, do not leave the system in an inconsistent or unpredictable state. A transaction's primary goal is to maintain the integrity of the data, even in the presence of failures, concurrency, or unexpected interruptions.

Consider an online shopping scenario where a customer places an order for multiple items. This transaction might involve the following steps:

1. Deduct the stock for each purchased item from the inventory.
2. Calculate the total cost, including taxes and discounts.
3. Process the customer’s payment through an external gateway.
4. Generate a shipping label for the order.
5. Log the transaction in the order history.

If the system executes this process as a single atomic transaction, and something fails-for example, the payment gateway times out, all prior operations (like stock deductions or shipping label generation) are reverted. The database remains consistent, as though the transaction never occurred. This behavior ensures that no partial changes leave the system in an invalid state.

The Need for ACID Properties

Transactions lie at the heart of reliable data systems, but the complexity of ensuring correctness across all possible scenarios (for example, failures, concurrency, or distributed operations) necessitates a rigorous framework. This is where ACID properties come in, providing a set of strict principles to govern how transactions should behave.

Without strict guarantees, we may end up with:

1. Inconsistencies: Partial updates or failures can leave the database in an invalid state.
2. Data Corruption: Multiple transactions modifying the same records simultaneously without proper safeguards can corrupt data.
3. Lack of Reliability: Users lose trust in the system if they encounter incomplete or inconsistent results.

The ACID principles of Atomicity, Consistency, Isolation, and Durability address these issues by ensuring:

• Atomicity: A transaction is treated as a single unit. Either all operations succeed, or none do.
• Consistency: Transactions respect the rules and constraints of the system, preserving its integrity.
• Isolation: Transactions operate independently of one another, even if they are executed concurrently.
• Durability: Once committed, a transaction’s effects persist, even in the face of failures.

Now that we have an understanding of ACID properties (especially in OLTP systems), let's understand how other data systems support ACID properties in the following section.

ACID in OLAP (Data Warehouse)

While Online Transaction Processing (OLTP) systems focus on high-frequency, low-latency transactional operations, Online Analytical Processing (OLAP) systems are designed to handle complex analytical queries on large-scale datasets, often involving aggregations, joins, and multi-dimensional analysis. Historically, OLAP systems primarily focused on optimizing query performance over transactional guarantees. However, as analytical systems have evolved to include ingestion capabilities (including both batch and stream load) and concurrent querying, ACID compliance has become essential for ensuring data reliability and correctness. Let’s go over some of the factors for why ACID is important in OLAP systems.

• Consistency Across Data Ingestion and Querying: Modern OLAP systems ingest data from multiple sources, often in parallel. Without Atomicity and Consistency, partial writes from ingestion pipelines can lead to incomplete or corrupted datasets. ACID properties ensure ingestion jobs are treated as atomic transactions, fully completing all updates or rolling back changes in case of failures. This prevents issues like partially populated columns or violations of integrity constraints.
• Concurrent Query Isolation: OLAP systems often serve multiple users and processes - ETL jobs, ad-hoc queries, and reports all access the same datasets simultaneously. Without Isolation, concurrent operations can lead to dirty reads, race conditions, or inconsistencies. ACID-compliant systems provide snapshot isolation, allowing each query to operate on a consistent dataset without interference from other ongoing processes
• Durability for Recovery and Reliability: OLAP systems manage massive datasets that must remain recoverable even in the event of failures. Durability ensures that fully committed transactions persist, while partial writes or failed operations are rolled back, preserving the integrity of the dataset.

By implementing ACID properties, modern OLAP systems provide reliable analytics, concurrent access and recovery during failures.

ACID in Data Lakes

Data lakes are designed to store vast amounts of raw, unstructured, and semi-structured data at scale, offering flexibility and cost efficiency. Built on distributed file systems or object storage (e.g., HDFS, Amazon S3, etc.), data lakes typically store data in open file formats such as Apache Parquet, Apache ORC, or Avro. However, traditional data lakes are just storage systems and lack database primitives or components like a storage engine that implements ACID properties.

Unlike OLTP databases or warehouses (OLAP), data lakes do not provide built-in mechanisms to ensure transactional guarantees, which introduces fundamental gaps in reliability, especially in environments requiring concurrent operations or failure recovery. These limitations arise because the core functionality of data lakes is storage, not managing transactions, consistency, or isolation.

While data lakes introduced the separation of compute and storage components, enabling elastic compute scaling and eliminating the need for long-running components in the data read/write path, the absence of ACID guarantees introduced critical challenges that hinder their reliability for transactional and concurrent workloads. Here are some of the issues that are often seen in data lakes.

• Corrupted Data from Failed ETL Jobs: Ingestion pipelines often write data to data lakes in batches or streams. Without atomicity, partial failures can leave incomplete or invalid files. For example, a pipeline might write some Parquet files but fail midway, leaving the dataset incomplete and unsuitable for querying.
• Concurrent Access Problems: In distributed systems, multiple processes may write to or read from the same dataset simultaneously. Without isolation, readers can encounter dirty or inconsistent data, such as partially ingested files, leading to incorrect analytical results. Concurrent writes can also result in file conflicts, causing data corruption.
• Data Inconsistencies: With no enforcement of schema consistency, updates to a dataset (e.g., adding or removing columns) can result in mismatched schemas across files. This leads to downstream failures in analytics pipelines that rely on predictable schema structures.
• Difficulties in Upserts and Deletes: Operations such as upserts and deletes are cumbersome in data lakes. Since data lakes lack built-in transaction support, these operations often require rewriting entire partitions or datasets, which is both time-consuming and resource-intensive.

ACID in Data Lakehouse

The significant limitations of traditional data lakes, including the lack of ACID properties, were a driving force behind the emergence of Lakehouse architecture. While data lakes provided scalable, low-cost storage for large datasets, their inability to manage transactions, enforce consistency, or reliably handle concurrent workloads often led to operational inefficiencies, corrupted data, and reprocessing overhead. These limitations made data lakes unsuitable for applications requiring data integrity, schema evolution, or reliable analytics pipelines.

To address these gaps, lakehouses introduced storage engine capabilities that provide database-like functionality atop distributed storage systems. These capabilities bring ACID properties into file systems such as Amazon S3 and HDFS. With ACID compliance, lakehouses unify the flexibility of data lakes with the reliability and transactional power of databases and warehouses.

These capabilities are made possible by open table formats such as Apache Hudi, Apache Iceberg, and Delta Lake that form the backbone of a lakehouse architecture. Each format takes a slightly different approach to implementing these properties, reflecting their unique design choices and use cases. However, their shared goal is to bring the following critical capabilities to the lakehouse:

• Atomic Operations: Ensuring that writes, updates, or deletes are either fully applied or entirely rolled back.
• Consistent Views: Guaranteeing that readers always see a consistent snapshot of the dataset, even during concurrent writes.
• Concurrency Control: Providing mechanisms for isolation, allowing multiple users or processes to interact with the dataset without interference.
• Durability: Ensuring that once changes are committed, they persist, even in the event of system failures.

In the following sections, we’ll explore how Apache Hudi, Apache Iceberg, and Delta Lake support ACID properties to bring transactional capabilities to lakehouses.

Apache Hudi

Apache Hudi ensures ACID compliance for data lakes by employing a timeline-based approach powered by a Write-Ahead Log (WAL), which tracks all transactional activities on the dataset. Every operation-whether an insert, update, or delete is recorded as a structured event on the timeline, ensuring reliability and recoverability. Atomicity is achieved by treating each write operation as a transaction. Changes are either fully committed or rolled back atomically by publishing them to the timeline with instant times, including requested and completed instant times, following Google Spanner’s Truetime API, and providing unique monotonically increasing instant values. This ensures that changes are applied in the correct sequence and ensures that readers only see completed transactions, avoiding partial writes or corrupted states.

Consistency is enforced through primary key uniqueness, where every record is mapped to specific partitions and file groups. Hudi tracks these changes using a sequence of timeline "instants," following a strict flow from requested → inflight → completed. Only completed commits are visible to readers and subsequent writers, ensuring that no partial or failed transactions disrupt the dataset's integrity. In scenarios involving concurrent transactions, the instant time prevents overlapping modifications and resolves conflicts between writers, using a concurrency control method such as Optimistic Concurrency Control (OCC) to maintain transactional integrity.

To maintain isolation, Hudi uses a combination of Multi-Version Concurrency Control (MVCC) and Optimistic Concurrency Control (OCC). MVCC ensures that readers always access consistent snapshots of the dataset, even during ongoing write operations, thereby supporting Snapshot Isolation. MVCC is also used between writers and table services (e.g., compaction, clustering or cleaning) or between multiple table services, enabling them to run completely asynchronously and lock-free, even within the same job. 

For writers, Hudi implements a variant of Serializable Snapshot Isolation (SSI). Writers begin their transactions by reading the latest snapshot of the dataset and proceed optimistically, assuming no conflicts. At commit time, a short-duration table-level exclusive lock is acquired to detect and resolve conflicts by comparing the proposed changes with other committed transactions. If conflicts are detected, Hudi resolves them using strategies like timestamp ordering, allowing the earliest transaction to be committed while aborting conflicting ones. Aborted transactions must retry their operations, ensuring that only consistent changes are applied. This hybrid approach balances scalability with strong isolation guarantees. Hudi’s advanced concurrency control mechanisms also enable early conflict detection for overlapping transactions, aborting them faster without requiring the full transaction to execute, saving resources and ensuring faster retries.

OCC in Hudi for multiple writers operates at the file level, meaning that non-overlapping concurrent transactions can succeed independently without conflicts. This fine-grained control allows concurrent writes to proceed in parallel, provided they operate on separate file groups. 

Hudi also supports Non-Blocking Concurrency Control (NBCC), introduced in the version 1.0.0. NBCC provides the same guarantees as OCC but eliminates the need for explicit locks to serialize writes, except when updating the commit metadata on the Hudi timeline. The completion time of each commit determines the serialization order, with file slicing based on the commit's completion timestamp. Multiple writers can operate on the same file group, and any conflicts are automatically resolved by the query reader and the compactor. This non-blocking approach ensures that writers can proceed concurrently without locking, enhancing scalability and reducing write latencies for distributed workloads.

Hudi guarantees durability by persisting all transactional changes to its WAL before applying them to the dataset. In the event of a system failure, Hudi can recover uncommitted transactions or roll back incomplete operations by replaying the log. Both metadata and data files are stored in durable systems such as HDFS or cloud object stores, further ensuring resilience against failures.

Apache Iceberg

Apache Iceberg ensures ACID compliance through a metadata-driven architecture and a compare-and-swap (CAS) mechanism for atomic commits. Each operation, such as an insert or update, generates a new metadata file that represents the table’s updated state. Writers commit changes by atomically updating the catalog's metadata pointer, ensuring that changes are fully applied or rolled back. This prevents partial updates and maintains consistency by enforcing a linear history of changes, where conflicting updates are rejected to avoid inconsistencies. Readers always see valid, fully committed data, while incomplete or failed writes remain invisible.

Iceberg supports multiple concurrent writes using Optimistic Concurrency Control (OCC), where each writer assumes it is the only one making changes. If another writer successfully commits first, the swap fails, and the transaction is rejected. The failed writer must then retry by creating a new metadata tree based on the updated table state and reattempting the commit. This concurrency model is simplistic. Iceberg assumes that concurrent writes are infrequent and lets users handle contention through retries when necessary. This means that under high-contention scenarios, frequent retries may impact write throughput. 

Iceberg provides snapshot isolation for readers to access consistent snapshots unaffected by ongoing writes and serializable isolation for writers. For durability, Iceberg persists data and metadata in fault-tolerant storage systems like S3 or HDFS. Each snapshot represents a recoverable state, enabling Iceberg to revert to the latest valid version in case of failures, ensuring data reliability in distributed environments.

Delta Lake

Delta Lake ensures ACID compliance through its transaction log (delta log), which tracks every operation on the dataset, ensuring atomicity, consistency, isolation, and durability. For atomicity, all changes-whether adding new data or marking files for removal-are logged as a single transaction. If a commit fails at any point, none of the changes are visible, ensuring that the table is never left in a partially updated state. This guarantees that transactions are either fully applied or rolled back.

Consistency is maintained by validating every operation against the table’s current state. The delta log ensures schema constraints are respected, records remain valid, and all changes are appended as new files, with older files logically removed. This append-only approach prevents in-place modifications, ensuring that every table snapshot is consistent and based on committed transactions. Transactions that conflict with schema rules or concurrent updates are aborted, preserving the dataset's integrity.

Delta Lake provides snapshot isolation for readers and serializable isolation for writers. Readers access stable snapshots unaffected by ongoing writes, while writers use optimistic concurrency control (OCC) to detect conflicts before committing. If another writer modifies the same data, the transaction is rejected and must retry, ensuring isolation without locking. Durability is guaranteed by persisting both data and metadata in fault-tolerant storage like S3 or HDFS. The delta log enables recovery to the latest consistent state, protecting all committed data from system failures.

Conclusion

ACID transactions are fundamental to ensuring data reliability, consistency, and integrity in data systems. The limitations of traditional data lakes-particularly their inability to provide transactional guarantees-were a major catalyst for the evolution of the Data Lakehouse. By introducing transactional capabilities atop scalable storage systems, lakehouse architecture with open table formats offer a unified solution that combines the flexibility of data lakes with the reliability of transactional systems. With ACID compliance, the Lakehouse architecture empowers organizations to handle complex data workloads confidently, supporting consistent analytics, reliable real-time ingestion, and efficient concurrent operations.