View all jobs

Sr. Data Engineer – Snowflake, ETL, ADO

  • Denver, CO

Role: Sr. Data Engineer – Snowflake, ETL, ADO
Duration: 6 Months (Contract) W2
Location: Denver, CO (On-Site)

Key Skills: Snowflake, Informatica CDGC (Cloud Data Governance and Catalog), Terraform (Infrastructure as Code), Azure DevOps, GIT, ETL, SQL

 
1. Scope of Responsibilities

  • Data Ingestion & Pipeline Engineering: Design, develop, test, and deploy automated ETL/data pipelines in Informatica and Snowflake to ingest data from partner state agencies and third parties into the SLDS common data model. 
  • Transformation & Analytics Layer Support: Implement transformation logic to convert raw agency feeds into harmonized, linked longitudinal records and analytics layer data models, adhering to core naming and modeling standards established with the Data Architect.
  • Record Linkage & Identity Matching: Execute the SLDS internal matching logic to link newly ingested agency records to master longitudinal student and workforce profiles, handling match exceptions and cross-agency reconciliations.
  • Production Pipeline Operations & Incident Response: Maintain operational ownership of scheduled feed runs, monitoring job execution, resolving run failures, reloading late or corrected files, reconciling row counts, and rebuilding downstream objects.
  • Data Quality & Governance Compliance: Implement L1 (structural/format), L2 (completeness/de-duplication), and L3 (referential integrity/logic) validation checks, layout-change detection, and audit trails; enforce data privacy, suppression rules, and compliance standards (FERPA, COPPA, ADA, and state statutes).
  • Performance Optimization & Platform Stewardship: Optimize query execution, warehouse compute, and pipeline performance for cost and latency; identify and tune heavily used models and performance hot spots in Snowflake.
  • Quality Assurance, Code Review & Promotion: Adhere to the unified release process across Development, Test, UAT, and Production environments using Azure DevOps and Azure Test Plans; author and execute test cases, conduct peer code reviews, and maintain release change logs.
 

 
2. Activities Performed
A. Ingestion & Pipeline Development (OIT-Led Feeds)
  • CDHE Courses Data Ingestion:
    • Ingest and validate annual postsecondary course-level data files from CDHE per data sharing agreements.
    • Build ETL mapping processes from CDHE course files into the SLDS common data model based on design specifications provided by 3rd party implementation partner.
    • Pair with 3 party implementation team  during ETL construction, validation, and promotion.
  • Postsecondary SES & Disability Data Ingestion (Conditional):
    • Ingest and validate demographic, Pell Grant eligibility, and disability classification codes from postsecondary sources.
    • Apply mandated privacy protections, cell suppression rules, and permissible-use governance requirements.
  • PSAT/SAT Assessment Data Ingestion (Conditional):
    • Ingest, validate, and map section/composite assessment scores and grade levels from the College Board or CDE.
    • Execute identity matching to connect assessment records to existing longitudinal student profiles.
B. Knowledge Transfer & Operational Transition (Jan–April 2027)
  • Shadowing & Co-running (Jan–Feb 2027):
    • Shadow 3rd party implementation partner engineers on day-to-day operations of existing data feeds (e.g., NSC, K-3, CDEC, and Phase 1 feeds).
    • Co-run live feed cycles, handle data loading, execute record matching, and troubleshoot exceptions alongside 3rd party implementation partner.
  • Readiness Gate Execution (Target: March 2027):
    • Run two consecutive feed cycles completely unaided to achieve formal operational hand-over sign-off.
  • Full Production Operations (March 2027 Onward):
    • Execute recurring feed schedules, rebuild downstream objects, reconcile row counts against source files, and triage failed jobs.
    • Address ad-hoc data inquiries and export requests from OIT leadership and partner agencies.
C. Data Stabilization & Analytics Layer Upkeep
  • Stabilization Backlog Support: Resolve identified data reliability defects, run variations, and de-duplication issues transitioned from the Phase 1 stabilization backlog.
  • Analytics Layer Hardening & Maintenance (April 2027 Onward):
    • Apply object modeling standards, naming conventions, and layout-change detection into the RAW layer.
    • Perform ongoing reconciliation of analytics layer outputs across non-production and production environments.
    • Implement metric and calculation enhancements prioritizing downstream use cases (program participation, completion, equitable access, and wage outcomes).
    • Conduct query profiling and performance tuning on dashboard back-end models.
D. Cross-Functional Collaboration & Testing
  • Testing & Release Management: Author manual and automated test cases in Azure Test Plans; record execution evidence and resolve defect bugs before promoting code.
  • Cross-Role Support:
    • Partner with the Data Architect to review and adapt physical and logical data models and pipeline designs.
    • Support the Data Scientist / Visualization Designer by integrating new data sources to power Tableau dashboard refreshes and preparing source tables for de-identified Institution of Higher Education (IHE) outcome files.
    • Coordinate with Operations Engineers on Snowflake compute management, warehouse auto-shutdown, user role permissions, and CI/CD promotion pipelines.
 

 
3. Deliverables Created
Technical Code & Pipeline Artifacts
  1. Production ETL/ELT Pipelines: Fully developed, scheduled Informatica workflows and Snowflake SQL/transformation pipelines for CDHE Courses and existing operational feeds.
  2. Cleansing & Validation Modules: Automated L1, L2, and L3 quality-check rules and Informatica layout-change detection routines in the RAW layer.
  3. Identity Matching & Linkage Routines: Documented internal matching scripts linking new partner agency student/workforce records into longitudinal IDs.
  4. Analytics Layer Transformation Objects: Production views, tables, and metric models in Snowflake aligned with longitudinal reporting use cases .
  5. De-Identified Data Extraction Scripts: Data extraction scripts that generate de-identified, suppressed outcome files for participating IHEs per specification.
Documentation & Engineering Specifications
  1. ETL Pipeline Design Documents: Technical architecture and transformation specifications detailing source-to-target logic, schemas, and dependencies.
  2. Mapping & Transformation Specifications: Contribution to enterprise data mapping documents aligning source fields to the SLDS common data model.
  3. Lineage Documentation & Dependency Maps: As-built data lineage records and dependency diagrams detailing object-level upstream and downstream flows.
  4. Operational Runbooks & Incident Procedures: Standard operating procedures for running feed cycles, handling late or malformed files, performing failover, and reconciling row counts.
Governance, Testing & Operational Reporting
  1. Azure Test Plans & Execution Records: Fully documented test plans, test suites, manual test run results, and defect logs for all pipeline modifications prior to production promotion .
  2. Release Change Logs: Detailed audit entries for all OIT-led production releases, detailing changes, rationales, linked Azure DevOps work items, test results, and deployment timestamps].
  3. Feed Operation Run Logs & Match Summaries: Run completion records, reconciliation totals, exception logs, and data-quality summaries produced for agency loads.
 
Required Qualifications & Technical Experience
  • Snowflake ETL/ELT: Minimum 5 years of demonstrated experience building, orchestrating, and optimizing complex ETL/ELT data pipelines and transformation logic natively within the Snowflake platform.
  • Informatica CDGC: Minimum 1 year of hands-on experience working with Informatica Cloud Data Governance and Catalog (CDGC) for metadata management, data profiling, cataloging, and automated data quality validation.
  • Infrastructure as Code (IaC): Minimum 1 year of experience utilizing Terraform to provision, configure, and maintain cloud infrastructure and resources.
  • CI/CD & DevOps: Demonstrated proficiency implementing automated build, test, and release deployment pipelines using Azure DevOps.
  • Version Control: Expert proficiency with Git / version control workflows (branching strategies, pull requests, automated merge checks, code reviews).
  • Location Constraint: Must currently reside within the State of Colorado for the duration of the engagement.
  •  
Preferred Skills & Experience
  • Prior experience working with public sector, longitudinal education, and workforce data systems (e.g., CDE, CDHE, CDLE, CDEC, National Student Clearinghouse).
  • Experience with record linkage, entity resolution, and deterministic/probabilistic identity matching across disparate public records 
  • Familiarity with federal and state student/workforce privacy regulations, including FERPA, COPPA, and cell suppression standards
  • Experience authoring and executing structured test plans in Azure Test Plans 
  • Position Type: 6-Month Contract (with possibility of extension based on project performance and state funding)
  • Work Arrangement: Remote / Telecommuting (Candidate must reside within the State of Colorado; strong preference for local / Denver Metro  candidates)
Email your resume to usjobs@mavensoft.com to apply!