Job Description
Job Description
We are seeking a Data Engineer to support the design, development, optimization, and maintenance of cloud-based data pipelines, platforms, and data stores used for large-scale analytics.
The ideal candidate will have hands-on experience building scalable ETL/ELT workflows, working with cloud-based data technologies, and transforming structured and unstructured data into reliable, analysis-ready formats. This role will collaborate closely with analysts, engineers, developers, and other stakeholders to develop data solutions that support reporting, analytics, modeling, and visualization.
Key Responsibilities
- Assist in designing, developing, and maintaining scalable ETL/ELT pipelines to ingest, transform, and deliver structured and unstructured data from multiple cloud-based sources.
- Build and optimize data storage solutions, including data warehouses, data lakes, and lakehouses, to support large-scale analytics, reporting, and business intelligence.
- Perform backend SQL and NoSQL database testing, including data validation, post-deployment testing, troubleshooting, and identification of ingestion or schema issues.
- Collaborate with analysts, engineers, developers, and stakeholders to translate business and technical requirements into data models, workflows, and pipeline specifications.
- Develop scripts and automation to transform diverse data types into usable, analysis-ready formats.
- Support the scaling, monitoring, maintenance, and operation of cloud-based data platforms.
- Validate data quality, integrity, and reliability throughout the data pipeline lifecycle.
- Troubleshoot data processing, pipeline, and storage issues as needed.
Required Qualifications
- 2+ years of experience in Data Warehouse development.
- Bachelor’s degree.
- Experience analyzing and processing aggregated data from multiple sources in cloud-based environments.
- Experience developing and maintaining scalable data stores that provide large datasets in formats suitable for business analysis.
- Hands-on experience with SQL and Python for retrieving, parsing, transforming, and processing structured and unstructured data.
- Experience developing scalable ETL or ELT workflows using cloud-based technologies for reporting and analytics.
- Experience with Cloudera and/or Hadoop technologies for data ingestion, transformation, and processing.
- Experience using generative AI coding tools to support software or data engineering development.
- Ability to perform backend SQL and NoSQL database testing, including post-deployment and data validation testing.
- Ability to obtain and maintain a Public Trust or Suitability/Fitness determination, based on project requirements.
- Must be a U.S. Citizen or Lawful Permanent Resident (Green Card Holder).
Preferred Qualifications
- Experience working in collaborative, cross-functional engineering environments.
- Experience with AWS Glue, AWS Athena, and PostgreSQL.
- Experience with distributed data and computing technologies such as Spark, Databricks, Hive, AWS EMR, or Kafka.
- Experience developing real-time data processing and streaming applications.
- Experience implementing NoSQL solutions using MongoDB or Cassandra.
- Experience with data warehousing technologies such as AWS Redshift, MySQL, or Snowflake.
- Experience with UNIX/Linux, including basic commands and shell scripting.
- Experience working within Agile engineering environments.