22 tasks, each one witnessed by the sources that watched the job — and behind every one, a prompt you can use tonight.
You’ll spend mornings running and checking pipelines that ingest telescope or simulation data using tools like Apache Airflow, Spark, or Hadoop. Expect to clean raw data, merge sources (catalogs, telemetry) and ensure dataset consistency before analysis.
Afternoons often go to developing and testing models (Python, Spark MLlib), visualizing results with dashboards, and meeting cross-functional teams — instrument scientists, software engineers, and project managers — to turn a science question into code and plots.
Common backend systems are Apache Spark and Hadoop for large-scale processing, Apache Hive for queryable data, and Kafka for streaming telescope telemetry. Airflow schedules workflows; Alteryx may be used for ETL (extract, transform, load).
Teams track work in Atlassian JIRA and write documentation in Confluence. Expect Python and SQL for coding, and plotting libraries or dashboards for visualizations.
You train models on labeled observations or simulations, then monitor performance with holdout tests and validation metrics to avoid overfitting. Use explicit uncertainty estimates — e.g., probabilistic models or calibration curves — because false positives can waste telescope time.
Document assumptions in Confluence, version datasets and models, and run reproducibility checks (same input -> same output). Share results with domain experts before actioning them, so instrument limits and selection biases are considered.
The U.S. Bureau of Labor Statistics (BLS) reports 262,440 employed data scientists and a median wage of $120,230 per year. The lowest tenth earn about $67,240, and the top tenth about $199,130, according to BLS 2025 data.
Salaries vary with employer type: universities and observatories may pay less than private space companies or large tech firms, and location and experience change where you fall in that range.
Learn Python and SQL first; practice with small datasets (FITS files, CSVs) and Jupyter notebooks. Take introductory statistics and a basic machine learning course. Play with Astropy and simple Spark locally or on cloud free tiers.
Work a small project: download public survey data (e.g., SDSS), clean and merge catalogs, build a classifier or clustering model, and make visualizations. Put code on GitHub and write a short Confluence-like README.
Statistical thinking combined with coding: you must know how sampling, bias, and uncertainty affect results and code that logic reliably. That means being able to design surveys or sampling strategies, clean and merge data, and interpret model outputs.
Practical tools matter too — proficiency with Spark or Hadoop for big data, plus version control and documentation (JIRA/Confluence) — but the core is being able to turn noisy telescope data into defensible scientific conclusions.