Data
-
Nobody Agrees What a CSV Is
CSV is simple but powerful. Values, separated by commas. It’s easy to understand and use.
It is also, by a wide margin, the one that destroys the most data.
That’s not a paradox. It’s cause and effect. A format simple enough that everyone writes their own parser is a format with as many dialects as it has parsers, and CSV’s defining property is that it carries no information about how to read it.
There Is No Standard
RFC 4180 exists. Yakov Shafranovich published it in October 2005, it registers the
text/csvmedia type, and it gives an ABNF grammar.It is also Informational, not Standards Track. It was written to describe what people were already doing, twenty-odd years after spreadsheets started emitting it. Section 2 says so outright: there is “no formal specification in existence,” and what follows documents “the format that seems to be followed by most implementations.”
By the time someone wrote it down, every spreadsheet, database, and scripting language had already shipped its own interpretation. The RFC didn’t settle anything. It just added one more dialect, with the distinction of having a number.
Eight years later, RFC 7111 added URI fragments for pointing at a row, column, or cell inside a
text/csvfile. Also Informational. CSV still had no standard, but you could now cite a specific cell of one.
Where Does a Record End?
The obvious answer is “at the newline,” and the obvious answer is wrong, because a quoted field is allowed to contain one.
name,notes Alice,"line one line two" Bob,fineThat’s a valid three-row CSV. Split it on newlines and you get four:
naive split gives 4 lines: 'name,notes' 'Alice,"line one' 'line two"' 'Bob,fine' a real CSV parser gives 3 rows: ['name', 'notes'] ['Alice', 'line one\nline two'] ['Bob', 'fine']Every
head,wc -l,split("\n"), and shell pipeline that assumes one record per line is wrong on this file. Not wrong on a malformed file. Wrong on a correct one.This is the single most common CSV bug, and it’s invisible in testing, because your test fixtures don’t have newlines in them until a user pastes an address into a form.
What’s the Delimiter?
In most of Europe the decimal separator is a comma.
12,50is twelve and a half euros. Which means a comma cannot also be a field separator, so those locales use semicolons.Feed a German CSV to an RFC 4180 parser and everything survives, in the sense that nothing throws:
input: produkt;preis Kaffee;12,50 Tee;9,90 parsed as comma-delimited: ['produkt;preis'] ['Kaffee;12', '50'] ['Tee;9', '90'] parsed as semicolon-delimited: ['produkt', 'preis'] ['Kaffee', '12,50'] ['Tee', '9,90']The first reading gives you two columns of nonsense with no error. The prices split down the middle of the decimal point. A pipeline that ingests this will happily compute statistics on the number 12 and the number 50.
Excel picks the delimiter based on your operating system’s regional settings, which means the same file opens differently on two machines in the same office.
Is the First Row a Header?
1,2,3 4,5,6Header or data? Nothing in the file says.
RFC 4180’s answer is that you put it in the MIME type:
text/csv; header=present. Which is a real answer, and also means the information lives outside the file, in a transport layer that gets stripped the moment someone saves the attachment to disk.So in practice every tool guesses, usually by checking whether the first row looks less numeric than the rest.
The Part That Destroys Data
Everything above is a parsing problem. This one is worse, because the file parses fine and the damage happens after.
CSV has no types. Every value is text. So every spreadsheet and dataframe library applies type inference on import, and type inference is lossy.
Here’s a file with four columns of identifiers, all of which are strings that happen to be made of digits:
gene,zip,card,accession SEPT7,02138,4532012345678901,0004928 MARCH1,01234,4111111111111111,0000071Read it with a type-inferring reader and:
gene zip card accession SEPT7 2138 4532012345678901 4928 MARCH1 1234 4111111111111111 71The ZIP code
02138is now2138. The accession number0004928is now4928. Nobody was asked. Nothing warned. Save that back to CSV and the original values are gone from disk.Spreadsheets are worse than this, because they store every number as an IEEE 754 double. Microsoft is blunt about the consequence:
Excel has a maximum precision of 15 significant digits, which means that for any number containing 16 or more digits, such as a credit card number, any numbers past the 15th digit are rounded down to zero.
The example they reach for is a card number:
typed into a cell 1234 5678 9087 6543 Excel shows 1.23E+15Microsoft calls that “truncating numerical data to 15 digits of precision and converting to a number displayed in scientific notation.” Note that this is the vendor describing its own product, not a bug report.
The card number in the file above is also 16 digits. pandas read it back intact. Excel would not.
Credit card numbers are 16 digits. Many national ID numbers are longer. They are not numbers in any meaningful sense, they’re strings of digits, and a format with no type information cannot tell the difference.
The Gene Name Problem
The best-documented case of this is genomics, because biologists name genes things like
SEPT1andMARCH1and spreadsheets read those as dates.In 2016 Ziemann and colleagues screened 35,175 supplementary Excel files from 18 journals covering 2005 to 2015. Among articles containing Excel gene lists, 19.6% had gene names corrupted this way. One in five.
A follow-up in 2021, “Gene name errors: Lessons not learned,” found 30.9% across a broader sample drawn from PubMed Central. Worth being careful comparing those two numbers directly, because the second study used a different sampling frame and also detected an additional error category the first one didn’t look for. The honest summary is that the problem did not go away in the five years after being loudly published.
The resolution is the remarkable part. The field did not fix the spreadsheets. It renamed the genes. The HUGO Gene Nomenclature Committee’s 2020 guidelines state that “all symbols that auto-converted to dates in Microsoft Excel have been changed,” giving
SEPT1becomingSEPTIN1andMARCH1becomingMARCHF1as examples.Human genes were renamed because a file format cannot say what type a column is.
What To Do About It
CSV isn’t going away, and mostly shouldn’t. It’s readable, streamable, diffable, and every tool on earth reads it.
The practical defenses are short:
- Quote everything. It’s never wrong and it removes a whole class of ambiguity.
- Treat identifiers as strings explicitly at the point of import. Every serious CSV reader lets you pin column types; use it.
- Never round-trip through a spreadsheet if the data contains identifiers. Opening and saving is a lossy operation.
- Say what you mean out of band. Delimiter, encoding, header presence, quoting style. The file will not.
- Use something else when you can. Parquet and even JSON Lines carry types. If the consumer is a program rather than a person, the readability argument for CSV mostly evaporates.
The lesson generalizes past CSV, and it’s the same one from the text file post. A format that carries no description of itself pushes that burden onto every reader, and readers guess. Usually well. Occasionally by silently deleting the leading zero from your ZIP code.
Sources
- RFC 4180 — the Informational spec that documents CSV rather than defining it
- RFC 7111 — URI fragment selectors for
text/csv, January 2014 - Ziemann et al., “Gene name errors are widespread in the scientific literature” — Genome Biology, 2016; the 19.6% figure (free full text)
- Abeysooriya et al., “Gene name errors: Lessons not learned” — PLOS Computational Biology, 2021; the 30.9% follow-up
- Bruford et al., “Guidelines for human gene nomenclature” — Nature Genetics, 2020; the renaming (free full text)
- Microsoft on Excel’s floating-point precision — Excel follows IEEE 754 and stores 15 digits of precision
- Microsoft on leading zeros and large numbers — “any numbers past the 15th digit are rounded down to zero,” with a credit card as the example
- Microsoft on importing and exporting text files — the CSV list separator comes from Windows Region settings
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Your Data Lake Has a Permissions Problem
Consolidating every business unit’s data into one giant lakehouse sounds like a win until you realize the security model from your old data warehouse can’t scale to it. You took ten silos, each with their own access rules, and merged them into one location. Now everyone wants in, and your security team is the bottleneck.
Let me walk through three places where the cracks usually show up.
RBAC Falls Over Faster Than You Think
Role-Based Access Control is the model most teams start with. Permissions are tied to a job function. Sales reps get read access to sales tables, data engineers get write access to staging, and so on. It works fine when you have ten roles.
It does not work when you have a thousand.
Say your sales reps should only see accounts in their territory, and only accounts they personally manage. Under pure RBAC, you need a unique role for every territory-by-account-owner combination. That’s role explosion, and it’s how compliance audits become impossible and legitimate access slows to a crawl. The roles list grows faster than anyone can review it, which means stale permissions sit there forever.
The answer is Attribute-Based Access Control. Instead of asking “what role is this user in,” the system asks “what attributes does this user have, what attributes does this data have, and what’s the policy at this exact moment.” Tag a column as
PII. Tag a schema asHR. Write one policy that says anyone outside the HR compliance group sees masked data when they touch a PII column. Done. That single policy replaces hundreds of bespoke roles.This is what Unity Catalog and Starburst Galaxy are built around, and it’s the model that will scale with the data.
Column and Row Security Should Be Boring
Once you have ABAC and a real metadata catalog, column-level masking and row-level filtering become a non-event. You write a SQL expression that masks the first five digits of an SSN for lower-privileged roles. You write a row filter that silently appends
WHERE region = 'user_region'to every executive’sSELECT *.The key word is silently. The user doesn’t see a different table. They don’t have a sanitized copy. The policy is enforced at the catalog layer, so it works the same whether they’re querying through Spark, Trino, a BI dashboard, or a pipeline. One source of truth, one policy, every engine.
If you’re still maintaining separate “sanitized” copies of tables for different audiences, you’re doing it the 2015 way and you’re going to drift.
The IAM Default Problem
Most cloud services ship with default IAM roles, and a surprising number of those defaults attach
AmazonS3FullAccessor something equally permissive.SageMaker does it. The Ray autoscaler role does it. There are more.
Picture the failure mode. An attacker compromises some peripheral app, maybe a forgotten Jupyter notebook, maybe a misconfigured Lambda. That workload has an IAM role attached because that’s how cloud workloads talk to S3 without hardcoded credentials. The attacker inherits the role. And because the role has full S3 access, they’re not constrained to the bucket the application actually uses. They can enumerate every bucket in the entire account.
That’s how a single compromised container becomes a full data lake breach. Researchers call it a bucket monopoly attack. I call it the most predictable incident in the industry.
The fix is not glamorous. Stop using
s3:*in any policy. Write resource-scoped policies that name the exact buckets and prefixes a workload needs. Audit the default roles every cloud service hands you and replace them. Use Security Lake or Detective to flag cross-service API calls that don’t match normal patterns. None of this is fun. All of it is necessary.And Then There’s the Agent Problem
The new wrinkle is that humans are no longer the primary consumers of your data. Autonomous agents are. They issue more queries, hit more tables, and move faster than any human team.
Long-lived credentials and static roles don’t fit that workload. The pattern emerging is Just-In-Time entitlements, where an agent gets a narrow, ephemeral permission for the duration of a single execution thread, then loses it. Pair that with declarative policy metadata baked into the data assets themselves, so the agent knows what it’s allowed to do with a dataset before it ever runs the query.
We’re early on this. Most organizations are still working through the basics, and that’s fine. But if you’re designing access controls today, design them assuming the next thing hitting your lake isn’t a person.
What to Actually Do
If you’re auditing your own data lake security, the order I’d work in:
- Find every IAM role with a wildcard permission. Replace them.
- Move from RBAC to ABAC at the catalog layer. Stop creating new roles.
- Pull your data lake off the public internet. PrivateLink, private endpoints, IP allowlists for the legacy stuff that can’t move.
- Then start thinking about agents.
The lakehouse pitch is unification. The lakehouse reality is that unification multiplies the cost of every bad permission. Get the basics right before you bolt on anything fancy.
Sources
- AWS Default IAM Roles Found to Enable Lateral Movement (The Hacker News) — SageMaker / Ray autoscaler default roles, bucket monopoly attacks
- What Is Fine-Grained Data Access Control? (TrustLogix) — RBAC role explosion, ABAC fundamentals
- Core concepts for ABAC (Databricks Unity Catalog docs) — Tag-driven policy enforcement
- Top 12 Data Governance Predictions for 2026 (Hyperight) — Just-in-time entitlements, declarative policy metadata
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
The Real Cost of Your Data Lake (It's Not the Storage)
If you’re sketching out a data platform on a whiteboard right now, I want you to do something. Stop calculating storage costs. They’re not the bill.
I pulled the public pricing for AWS, Azure, GCP, Databricks, and Snowflake and stacked them next to each other. Storage is the cheap part. The expensive part is everything that moves the data, and the expensive part is the part you’re least likely to model correctly when you’re picking a vendor.
Let me walk through what actually shows up on the invoice.
Raw Object Storage Is Basically Free
For hot, frequently accessed data, the big three are within a rounding error of each other:
- Azure Blob (LRS, Hot): $0.018 per GB/month
- Google Cloud Standard: $0.020 per GB/month
- AWS S3 Standard: $0.023 per GB/month (first 50 TB)
Drop into the cool tiers and AWS S3 takes the lead at $0.0125 per GB. Drop into deep archive and you’re paying $0.00099 per GB on either AWS Glacier Deep Archive or Azure Archive. That’s a tenth of a cent per gigabyte, per month, for data you almost never touch.
Good for you, but I think anyone leading with “per-GB storage cost” in a procurement deck is selling you a story. Storage capacity is roughly five percent of a typical Databricks bill. Five. The other 95% is the part nobody wants to talk about.
The Egress Trap
Ingress is free. Always. The cloud providers want your data in.
Getting it back out is where they collect.
- Azure Blob: $0.087/GB external egress
- AWS S3: $0.090/GB
- Google Cloud: $0.120/GB (but free if you stay inside Google’s ecosystem, which is the whole point of that pricing)
Then layer on API operations. A million GET requests on S3 costs about $0.40. The same million GETs on Google Cloud Storage can run closer to $5.00 because they classify operations differently. If your analytics workload is hammering small files, those API calls add up faster than the storage they’re reading.
Storing 10 TB? Maybe $200 a month. Storing 500 TB? You’re at $10,000 a month before a single byte leaves the region or a single query fires.
Databricks: Two Bills, One Headache
Databricks uses what’s commonly called a Two-Bill Model. You get one invoice from your cloud provider for the actual VMs and storage, and a separate invoice from Databricks for the software, measured in DBUs (Databricks Units).
In a typical mid-sized deployment around $18,000/month, the breakdown looks like this:
- VM compute from the cloud provider: ~55%
- Databricks DBU fees: ~30%
- Storage: ~5%
- Network egress: ~5%
The DBU rate changes based on what you’re doing. Automated jobs start at $0.15/DBU. Interactive notebooks for analysts start at $0.40/DBU. That’s not an accident. Databricks wants you running production workloads on cheap job clusters, not on the expensive all-purpose clusters your data scientists love to leave running over a weekend.
If you’re not actively pushing teams toward job clusters and ARM-based instances, you’re leaving real money on the table.
Snowflake: The Hidden Storage Multiplier
Snowflake’s pricing pitch sounds clean. Pass-through storage at $40/TB/month on-demand, dropping to $23/TB/month with a capacity commitment. Compute as Credits. Done.
Except it isn’t done. Snowflake stores data in immutable 16MB micro-partitions. Immutable. You can’t change them in place. Update a single row in a 1 TB table and Snowflake writes a new file and keeps the old one around.
Why keep the old one? Two features:
- Time Travel: query historical states of your data for up to 90 days
- Fail-Safe: a 7-day disaster recovery window you cannot turn off
This is the part that gets people. A 1 TB table that’s getting updated multiple times a day can balloon to 25 TB of billed storage because Snowflake is retaining every prior version of every micro-partition you’ve touched. Your dashboard says “1 TB table.” Your invoice says otherwise.
And compute? Virtual Warehouses bill per second, but with a 60-second minimum every single time you resume or resize. Aggressive auto-suspend sounds like a cost optimization. It’s not. If you’re spinning a warehouse up and down every 30 seconds, you’re paying the 60-second minimum every time and quietly multiplying your bill.
What I’d Actually Do
A few things I’d put on the wall before signing anything:
- Model egress, not storage. Run your worst-case query pattern through the calculator. Storage is noise.
- Lifecycle everything. Cool tier and archive pricing are 10x to 100x cheaper. If your data is older than 90 days and nobody’s queried it, it shouldn’t be in hot storage.
- For Databricks: push every recurring workload to job compute. Audit interactive cluster usage monthly.
- For Snowflake: if you have high-frequency update patterns, profile your actual storage footprint, not your logical table size. The gap will surprise you.
- For multi-cloud: don’t. Egress will eat the savings before you finish the architecture diagram.
The vendors all have a story about why their model is the cheap one. Read past the per-GB number on the slide. The bill is somewhere else.
Happy modeling.
Sources
- Databricks Pricing Explained (Dawiso) — Two-Bill Model, DBU breakdown
- Snowflake Pricing Explained (SELECT.dev) — Time Travel storage multiplier, micro-partition behavior
- Cloud & AI Storage Pricing Comparison 2026 (Finout) — AWS / Azure / GCP per-GB and tier pricing
- S3 vs GCS vs Azure Blob Storage (ai-infra-link) — Egress and API operation pricing
- Snowflake Pricing in 2026 (CloudZero) — Virtual Warehouse 60-second minimum behavior
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
The Data Lakehouse Won. Now Pick a Table Format.
If you’ve been ignoring the data infrastructure conversation for the last few years, here’s where we landed in 2026: the data lakehouse won. The data warehouse vendors will fight about it for another decade, but the architectural argument is over.
Let me back up.
The Quick History
At the bottom of every modern data stack is a cloud storage bucket. S3, Azure Blob, GCS. Pick your hyperscaler. A bucket is dumb on purpose. It stores files cheaply and durably and doesn’t care what’s in them. No schemas, no transactions, no relational anything. Just objects.
When you dump raw logs, IoT telemetry, and CSV exports into a bucket without any organizing layer, congratulations, you have a data lake. Cheap, flexible, and almost completely useless for analytics until someone builds a pipeline to make sense of it.
The traditional answer to that mess was a data warehouse. Snowflake, Redshift, BigQuery, the whole gang. You force your data through ETL, conform it to a strict schema, and pay a premium to keep it sitting in the vendor’s proprietary storage format. You get fast SQL, ACID transactions, and a vendor lock-in problem so severe that exporting your data becomes a major friction point.
The lakehouse is what happens when someone finally says: what if we kept the cheap object storage, but added the warehouse features as a layer on top?
What a Lakehouse Actually Is
The trick is decoupling. Storage stays in your bucket. Compute is whatever engine you point at it. Metadata lives in an open table format that turns a pile of Parquet files into something that behaves like a real database table.
One copy of the data. Multiple engines can query it. Schema evolution, time travel, ACID transactions, all without copying everything into a proprietary system. From what I’ve read, teams that move from a pure warehouse to a lakehouse are able to cut storage costs noticeably in the process, and they stop fighting their ML team about getting access to the same data.
That’s the pitch, and it’s a good one. The hard part is picking your table format.
The Four Formats Worth Knowing
Apache Iceberg
Iceberg is the one to bet on if you care about not getting locked in. It came out of Netflix and even Snowflake and Databricks have been forced to support it. The metadata is hierarchical, which sounds boring but matters: it lets query engines skip enormous chunks of irrelevant data without listing directories one by one. Iceberg also handles partition evolution gracefully, so you can change your partitioning strategy without rewriting petabytes of history.
If I’m starting a new lakehouse in 2026 and I don’t have a strong reason to pick something else, it’s Iceberg.
Delta Lake
Delta is what Databricks ships and what everyone using Spark already knows. It uses an append-only transaction log in a
_delta_logdirectory, and it’s beautifully integrated with the Databricks platform. Z-Ordering, native Spark performance, the whole ecosystem.If your team lives inside Databricks, Delta is the obvious answer. If you don’t, the calculus is harder, because Delta’s openness has improved a lot but it still feels most at home in the Databricks world.
Apache Hudi
Hudi came out of Uber and it was built for one thing: high-frequency upserts. If your problem is Change Data Capture, streaming ingestion, or constant record-level updates, Hudi is probably your answer. It gives you two storage modes. Copy-on-Write rewrites files on update so reads stay fast. Merge-on-Read writes deltas and reconciles at query time, which is what you want when writes are heavy and reads are flexible.
Hudi is the right pick when your pipeline is full of
UPSERTand you can’t afford to rewrite large files every time something changes.Apache Paimon
Paimon is the newest of the four and it’s worth keeping an eye on. It came from the Flink world and uses an LSM-tree style organization, which is what databases like RocksDB use under the hood. The whole point is unifying batch and streaming in a single format. If you’re doing real-time event-driven work and don’t want to maintain a separate streaming and batch stack, Paimon is interesting.
It’s not the safe choice yet, but it’s the one I’d watch most closely over the next two years.
So Which One?
Honestly, the answer depends less on the format and more on which ecosystem you’re already in.
- Mostly Spark and Databricks? Delta.
- Streaming-heavy with constant upserts? Hudi.
- Real-time event-driven and willing to bet on newer tech? Paimon.
- Anything else, or you want to keep your options open? Iceberg.
The format wars have mostly converged. Most major engines support multiple formats now, and the gap between them on raw query performance has shrunk. The choice is more about operational fit than performance ceilings.
The lakehouse pattern itself is the real story. The format is just plumbing.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
What Would Minimum Wage Be If It Kept Up With Housing?
I pulled the data and verified the math. The answer is… not great.
The Numbers We’re Working With
In 1950, the federal minimum wage was $0.75 per hour. A median owner-occupied single-family home cost $7,354.
In 2026, the federal minimum wage is $7.25 per hour — unchanged since 2009. The median U.S. family home price is $429,129.
These numbers come from multiple independent sources. Let’s see what happens when we put them side by side.
The Home-Labor Index
I’m using a simple metric here: how many hours of minimum-wage work does it take to buy a median home? No mortgages, no interest rates, no down payments; just raw labor hours versus home price.
1950:
- $7,354 ÷ $0.75/hr = 9,805 hours
- At 40 hrs/wk × 52 wks = 2,080 hrs/yr
- That’s 4.71 years of full-time minimum-wage work
2026:
- $429,129 ÷ $7.25/hr = 59,191 hours
- Same 2,080 hrs/yr
- That’s 28.46 years of full-time minimum-wage work
In 1950, a minimum-wage worker needed under 5 years of gross income to cover a median home. In 2026, that same worker needs over 28 years. The ratio has gotten roughly six times worse.
So What Should Minimum Wage Be?
If we wanted to preserve the same home-purchasing power that a minimum-wage worker had in 1950, we can work backwards:
- 2026 Median Home Price ÷ 1950 Home-Labor Index
- $429,129 ÷ 9,805 hours = $43.77 per hour
We can verify this another way. The 1950 ratio was 4.714 years of income to buy a home. To maintain that ratio in 2026:
- $429,129 ÷ 4.714 = $91,029/yr required income
- $91,029 ÷ 2,080 hours = $43.76/hr
Both methods land in the same place. To have the same relationship between minimum wage and housing that existed in 1950, the federal minimum wage would need to be roughly $43.77 per hour
You don’t need a PhD to look at these numbers and see the wage gap disparity. The gap between wages at the bottom and the cost of the most basic economic asset, a home, has grown dramatically. That the gap exists isn’t debatable.
The federal minimum wage has been $7.25 since 2009. That’s 17 years without an increase. Meanwhile, median home prices have roughly doubled in that same period.
If we cared about the citizens, we’d need to 6x the minmum wage while also working at more affordable housing for the middle class.