◆ Data & Analytics

What a data engineer
really does.

20 tasks, each one witnessed by the sources that watched the job — and behind every one, a prompt you can use tonight.

20evidenced tasks
67,140in the US (2025)
$139,500median pay / year
9systems it runs on
See the 20 tasks ↓Open in the atlasThe full career map →
This is what one task looks like here
Develop and test database modifications
Apply the schema changes to the staging user_activity and orders table…2 sources agree

The shape of the day

tap a movement to see its tasks

Which one is you, right now?

Pick the moment · no score, no sign-up
Which moment is you right now?
Whichever you pick, the task behind it opens below.

The work, task by task

20 tasks
Hands on the work16
Implement data backup and recovery procedures+
Document and implement nightly snapshot and point-in-time restore procedures for the production analytics cluster, run a full restore on the recent backup to a dev instance, validate row counts and critical reports, and hand over the runbook to oncall for first response.
jdonet2 agree
when the reply comes backPush once: ask it to sharpen the weakest part, and to say what it assumed. Helpful?
Create and manage data models+
Design canonical entity models for customer, product, and transaction, produce the physical table definitions with primary and surrogate keys, generate sample datasets, and run integration queries to confirm analytics and ETL jobs still produce expected totals.
jdonet2 agree
when the reply comes backPush once: ask it to sharpen the weakest part, and to say what it assumed. Helpful?
Create data models and schemas+
Create normalized entity-relationship models for order, inventory, and telemetry tables, produce the DDL for the canonical schemas, and publish sample datasets and field definitions to the analytics team before the architecture review on Monday.
escojdonetwiki4 agree
when the reply comes backPush once: ask it to sharpen the weakest part, and to say what it assumed. Helpful?
Optimize database performance+
Profile the sales and telemetry databases during peak hours, identify the top three slow queries, propose index or partition changes with estimated impact, and prepare a rollback-safe deployment plan for Thursday maintenance window.
jdonet2 agree
when the reply comes backPush once: ask it to sharpen the weakest part, and to say what it assumed. Helpful?
Express strategic data requirements+
Draft the strategic data requirements for the next fiscal product line: required source systems, record retention, SLAs, key entities and ownership, and the downstream consumers, then circulate to product, BI, and compliance for sign-off by end of month.
jdwiki2 agree
when the reply comes backPush once: ask it to sharpen the weakest part, and to say what it assumed. Helpful?
Provide technical guidance to team members+
Give the team a concise playbook this afternoon that defines our data exchange standards, preferred ingestion patterns, code review checklist, and two example API contracts so Priya and Marcos can implement it next sprint.
jdonet2 agree
when the reply comes backPush once: ask it to sharpen the weakest part, and to say what it assumed. Helpful?
Establish and calculate optimum values for database parameters, using manuals and calculators.+
Calculate optimal DB parameters for the OLAP cluster using vendor manuals and our workload samples: compute target memory, compaction thresholds, and parallelism settings, then document recommended values and expected impact.
onet
when the reply comes backPush once: ask it to sharpen the weakest part, and to say what it assumed. Helpful?
Ensure data integrity and security+
Audit the customer data pipeline for integrity and security: validate checksums, review access controls, confirm encryption in transit and at rest, and submit a remediation list to security by Monday.
jd
when the reply comes backPush once: ask it to sharpen the weakest part, and to say what it assumed. Helpful?
Watch and assess1
Keep it safe1
Grow the practice1
Work with people1

What the work runs on

named inside the evidenced tasks
7 tasksApache Hiveruns and validates table schema changes and migration queries for large datasets
4 tasksAmazon RedshiftData warehouse platform where query performance should be analyzed and tuned for analytics workloads
4 tasksApache HadoopMay host larger historical datasets requiring re-partitioning or pre-aggregation to reduce warehouse load
3 tasksApache AirflowOrchestrates pipelines that populate and validate the new schemas with sample data
2 tasksAmazon Web Services AWSorchestrates snapshots, backups, and restores across cloud storage and compute for disaster recovery
2 tasksApache Cassandramodels and tests wide-column data structures and their query behavior for scalable workloads
2 tasksAJAXenables interactive web-based query examples and live demo interactions during training sessions
1 taskAmazon Web Services AWS CloudFormationDescribes infrastructure as code which will capture required source systems and deployment expectations tied to the data strategy
Browse every evidenced software task →

The same task, four heights

this page is height one
ExecuteDo today's task, with fewer mistakesyou are here → ImproveMake it easy for the next person to acceptin the atlas → DecideWork out the right move when it is unclearin the atlas → BecomeLearn the pattern so it stops coming backin the atlas →

Can AI actually do this job?

the honest answer

It can

where it genuinely helps
  • Explain the theory behind the work
  • Draft, tidy and structure your writing
  • Rehearse a hard conversation before you have it
  • Build a study plan that fits your gaps

It cannot

where it stops, completely
  • Be in the room where a data engineer actually works
  • Carry the responsibility when the call is wrong — that weight stays yours
  • Notice what no one wrote down: the hesitation, the thing left unsaid
  • Live with the outcome

What the work pays

two countries, two different measures

United States

this exact occupation · BLS 2025
  • $139,500 a year — the middle: half earn more, half earn less
  • The lowest tenth earn near $86,240; the top tenth near $204,000
  • 67,140 people employed in this occupation

India

the occupation GROUP, not this job · PLFS via ILOSTAT 2025
  • ₹38,298 a month — the median for Professionals, the group this work sits in
  • India publishes pay by broad occupation group, so this covers many jobs besides this one. It is a shape, not a salary.
read this carefullyThese two numbers are not comparable and must not be converted into each other. One is a yearly figure for this job alone; the other is a monthly figure for a whole family of jobs. What travels between them is the pattern, not the amount: experience lifts pay almost everywhere.

Where the evidence lives

open any of it yourself
O*NETThe US Department of Labor's own record of this job9 of these tasks ESCOThe European skills and occupations register5 of these tasks Job descriptionsReal postings, in the wild11 of these tasks WikipediaHow the world describes this work5 of these tasks US Bureau of Labor StatisticsPay and employment, by occupation2025 figures UK NOS TECDT80851Design data processing systemsnational standard UK NOS TECDT80841Build and implement data processing systemsnational standard

Close to this work

12 nearby
Data & AnalyticsFinancial Analyst26 evidenced tasks Data & AnalyticsExperimentation Analyst26 evidenced tasks Data & AnalyticsData Governance Analyst26 evidenced tasks Data & AnalyticsR Analyst26 evidenced tasks Data & AnalyticsCrm Analyst26 evidenced tasks Data & AnalyticsWeb Analyst26 evidenced tasks Data & AnalyticsGrowth Analyst26 evidenced tasks Data & AnalyticsProduct Analyst25 evidenced tasks

Questions people actually ask

You usually split the day between coding, meetings, and monitoring. Mornings often start with checking Airflow jobs and AWS CloudWatch for failed ETL (extract-transform-load) tasks on Apache Airflow and Amazon EC2/Redshift clusters.

Afternoons go to building or changing data models in Hive or Redshift, tuning queries, and answering developer questions. You also spend time creating backup/recovery steps and documenting schemas so analysts and apps can use the data reliably.

Start with one cloud platform and one processing tool. Learn AWS basics (EC2, S3, IAM) and then Amazon Redshift for warehousing and Apache Hive for big-data SQL on Hadoop. Those cover most data storage and query needs listed in the tasks.

Add Apache Cassandra if you need low-latency, high-write workloads. Learn Airflow next to schedule ETL jobs. Knowing AWS CloudFormation helps automate deployments.

Use AI tools for repetitive work: generating SQL templates, producing data dictionaries, or suggesting query optimizations—but always review outputs. AI can hallucinate schema details or mis-handle PII (personally identifiable information).

Keep human review for data integrity and security steps, and do not feed raw PII into public AI services. Log and test any AI-driven changes in staging before applying to production.

The U.S. Bureau of Labor Statistics (BLS) reports 67,140 employed data engineers and a median pay of $139,500 per year. The lowest tenth earn about $86,240, and the top tenth about $204,000. (Source: BLS, 2025.)

Your pay will depend on location, experience with systems like AWS, Redshift, Hadoop, and responsibilities such as designing backup/recovery and leading data-modeling work.

Build a small ETL pipeline: ingest CSVs to S3, transform with a script or Spark, load to Redshift, and orchestrate with Airflow. Include schema definitions and a backup/recovery plan.

Also create a data model diagram, a sample inventory of datasets, and a short doc of access controls and data vocabulary. Show a performance tune example—indexing or Redshift distribution keys—to prove you can optimize databases.

A data engineer builds and maintains the data systems: schemas, ETL, backups, and performance tuning (think Hive, Hadoop, Redshift, Airflow). You make data reliable and fast for others to use.

Data analysts query and visualize data for reports; data scientists build models and experiments. All three collaborate, but engineers are responsible for storage, pipelines, security, and operational aspects.