Data Lakes: Flexible Storage That Requires Governance

Organizations generate tables, logs, events, images, audio, documents, and model data.

A data lake stores these diverse datasets in scalable object storage so several processing systems can use them.

A data lake provides flexible shared storage; cataloging, ownership, quality, and lifecycle turn it into a trustworthy platform.

Without governance, it becomes a collection of files nobody understands.

A concrete example: one analytical storage system

A company lands:

application events,
database exports,
support messages,
product images,
device telemetry,
model features.

SQL engines, notebooks, machine-learning jobs, and archive processes read governed subsets from the same storage foundation.

Object-storage foundation

Object storage provides:

large scale,
durable storage,
API access,
lifecycle tiers,
and separation of storage from compute.

Engines can start and stop without moving the underlying data.

Raw data

Raw zones preserve source-shaped input for:

replay,
audit,
new transformations,
and correction of pipeline bugs.

"Raw" should still have:

source,
ingestion time,
schema,
owner,
and access policy.

Processed zones

Data may progress through layers:

landing,
validated,
standardized,
curated,
product-specific.

Names vary. Each layer should communicate quality and intended use.

Do not create duplicate copies without lifecycle rules.

File formats

Formats include:

CSV,
JSON,
Avro,
Parquet,
ORC,
images,
and domain-specific files.

Columnar formats such as Parquet improve analytical scans and preserve types better than CSV.

Open formats and portability

Open, documented formats let several engines read the same stored data and reduce dependence on one compute vendor.

Portability still requires compatible catalogs, table metadata, encryption keys, identity, and SQL behavior. A Parquet file is portable at the byte level while the surrounding data product may remain tightly coupled to one platform.

Immutable landing

Landing data can be append-only so ingestion preserves exactly what the source delivered.

Corrections create a new version or a downstream clean table instead of silently editing evidence. Immutability improves replay and audit, but privacy deletion and legal correction still need controlled mechanisms that reach every copy.

Small-file problem

Millions of tiny files create:

listing overhead,
metadata cost,
slow planning,
and inefficient reads.

Compaction combines small outputs into appropriately sized files.

Streaming ingestion often needs scheduled compaction.

Partitioning

Directory prefixes can organize by:

date,
source,
tenant,
or region.

Query engines skip irrelevant partitions. Overpartitioning creates many tiny directories and files.

Partition by common filters with enough data per partition.

Schema-on-read

Data lakes are often described as schema-on-read: interpretation occurs when a tool reads the data.

This provides flexibility but does not eliminate schema. Every successful reader still needs field definitions, types, and version handling.

Unknown schemas create inconsistent results.

Catalog

A data catalog records:

dataset name,
location,
schema,
owner,
description,
sensitivity,
quality,
lineage,
and freshness.

Discovery is essential when storage contains thousands of datasets.

Metadata

Useful metadata includes:

source,
event time,
ingestion time,
version,
partition,
checksum,
and processing history.

Metadata makes files interpretable and supports reproducibility.

Governance

Governance defines:

who can publish,
who can read,
approved uses,
retention,
quality,
and correction.

Self-service data should not mean uncontrolled duplication of sensitive records.

Security

Use:

bucket and prefix policies,
workload identity,
encryption,
key separation,
audit logs,
and network controls.

One broad lake role can expose many domains at once.

Privacy

Raw data may contain fields later removed from curated tables.

Apply privacy controls at landing:

minimization,
restricted access,
tokenization,
retention,
and deletion propagation.

Raw cannot mean exempt from policy.

Discoverability and documentation

Every published dataset should explain:

purpose,
grain,
source,
update schedule,
owner,
quality status,
and example use.

Storage paths such as final_v2_new are not documentation. A searchable catalog and consistent naming help users select trusted products instead of making another private copy.

Data products

A governed lake can treat important datasets as products with users, objectives, contracts, support, and lifecycle.

This shifts attention from "the file landed" to whether consumers can understand, trust, and operate the data. Not every temporary intermediate needs full product ceremony; prioritize shared durable outputs.

Data quality

Validate:

schema,
completeness,
uniqueness,
ranges,
freshness,
and reconciliation.

Quarantine invalid data rather than silently mixing it with trusted datasets.

Expose quality status in the catalog.

Transactions

Plain object files do not automatically provide table transactions.

Lakehouse table formats add metadata for:

atomic commits,
snapshots,
schema evolution,
time travel,
update,
and delete.

Multiple engines must follow the same table protocol.

Lakehouse

A lakehouse combines low-cost open lake storage with warehouse-like table management and query behavior.

It can support analytics and machine learning over one data foundation.

Operational complexity remains in catalogs, compaction, optimization, and engine compatibility.

Compute separation

Different compute engines can read the same data:

batch processing,
SQL,
stream processing,
notebooks,
and ML.

Isolation prevents one workload from consuming every resource, while shared storage preserves reuse.

Lifecycle

Move data among:

hot,
infrequent,
archive,
and deletion.

Retention must include derivatives, snapshots, backups, and catalog entries.

Archive retrieval time should match future use.

Cost

Object storage is inexpensive per byte but cost also includes:

requests,
scans,
compute,
catalog,
transfer,
and duplicate retention.

Poor partitioning can cause every query to read terabytes unnecessarily.

Avoiding a data swamp

A lake becomes a swamp when datasets lack:

ownership,
definition,
quality,
discovery,
and lifecycle.

Require publication standards and remove abandoned datasets.

Flexibility needs accountable curation.

Knowledge check

Why is object storage suitable for a data lake?
What does the small-file problem cause?
Why does schema-on-read not mean schema-free?
What do lakehouse table formats add?
Which properties prevent a lake from becoming a data swamp?

The one idea to remember

A data lake stores diverse raw and processed datasets in scalable shared object storage. File formats, catalogs, schemas, quality, security, privacy, table transactions, ownership, and lifecycle are what make that flexible storage trustworthy and usable.