Stage 3 · Data Engineer · data platform case · public
Data on EKS
A Hadoop-era stack (Hive, Trino, Ranger, LDAP and a SQL IDE) rebuilt as Helm charts, so it deploys and is secured like any other Kubernetes service. The charts are public, and the security design was covered on the AWS Big Data Blog.
One query, two people
Every query passes the same two checks before it reaches an engine: who is this, and may they. Everything here is sample data.
analyst-aSELECT * FROM finance.balances_daily
analyst-bSELECT * FROM finance.balances_daily
How it is put together
One identity store
- LDAP holds the users and groups
- Every engine asks the same directory who is calling
One policy store
- Ranger holds who may see which table and column
- A rule is written once and applies to Hive and Trino alike
Charts, not hand-built servers
- Each component is a Helm chart on Amazon EKS
- Installing one is a command, and upgrading is a diff
Public, with the write-up
Before and after the merge
In production the same stack had been running twice. Putting both on one EKS platform changed the cost and the nightly schedule. Figures are approximate.
Two platforms doing the same job
Two copies of the same stack to run, patch and pay for.
One platform on EKS
One identity store, one policy store, one way to deploy.
The weekend an old node failed
The old platform lost a node. Because the new one already existed as charts, the switch was done in a single weekend instead of being planned for months.
What I took from it
- Put the access check in one place and every engine behind it inherits it.
- Infrastructure that is a chart can be rebuilt in a weekend. Infrastructure that is a person's memory cannot.
- Tests and monitoring are what make a quick cut-over safe.
One identity. One policy. Every engine.
Next
All work
See everything else I built
Grouped by the stage of the story, with shorter entries that open in place.