View all jobs
2. Activities Performed
A. Ingestion & Pipeline Development (OIT-Led Feeds)
3. Deliverables Created
Technical Code & Pipeline Artifacts
Required Qualifications & Technical Experience
Sr. Data Engineer – Snowflake, ETL, ADO
Role: Sr. Data Engineer – Snowflake, ETL, ADO
Duration: 6 Months (Contract) W2
Location: Denver, CO (On-Site)
Key Skills: Snowflake, Informatica CDGC (Cloud Data Governance and Catalog), Terraform (Infrastructure as Code), Azure DevOps, GIT, ETL, SQL
1. Scope of Responsibilities
- Data Ingestion & Pipeline Engineering: Design, develop, test, and deploy automated ETL/data pipelines in Informatica and Snowflake to ingest data from partner state agencies and third parties into the SLDS common data model.
- Transformation & Analytics Layer Support: Implement transformation logic to convert raw agency feeds into harmonized, linked longitudinal records and analytics layer data models, adhering to core naming and modeling standards established with the Data Architect.
- Record Linkage & Identity Matching: Execute the SLDS internal matching logic to link newly ingested agency records to master longitudinal student and workforce profiles, handling match exceptions and cross-agency reconciliations.
- Production Pipeline Operations & Incident Response: Maintain operational ownership of scheduled feed runs, monitoring job execution, resolving run failures, reloading late or corrected files, reconciling row counts, and rebuilding downstream objects.
- Data Quality & Governance Compliance: Implement L1 (structural/format), L2 (completeness/de-duplication), and L3 (referential integrity/logic) validation checks, layout-change detection, and audit trails; enforce data privacy, suppression rules, and compliance standards (FERPA, COPPA, ADA, and state statutes).
- Performance Optimization & Platform Stewardship: Optimize query execution, warehouse compute, and pipeline performance for cost and latency; identify and tune heavily used models and performance hot spots in Snowflake.
- Quality Assurance, Code Review & Promotion: Adhere to the unified release process across Development, Test, UAT, and Production environments using Azure DevOps and Azure Test Plans; author and execute test cases, conduct peer code reviews, and maintain release change logs.
2. Activities Performed
A. Ingestion & Pipeline Development (OIT-Led Feeds)
- CDHE Courses Data Ingestion:
- Ingest and validate annual postsecondary course-level data files from CDHE per data sharing agreements.
- Build ETL mapping processes from CDHE course files into the SLDS common data model based on design specifications provided by 3rd party implementation partner.
- Pair with 3 party implementation team during ETL construction, validation, and promotion.
- Postsecondary SES & Disability Data Ingestion (Conditional):
- Ingest and validate demographic, Pell Grant eligibility, and disability classification codes from postsecondary sources.
- Apply mandated privacy protections, cell suppression rules, and permissible-use governance requirements.
- PSAT/SAT Assessment Data Ingestion (Conditional):
- Ingest, validate, and map section/composite assessment scores and grade levels from the College Board or CDE.
- Execute identity matching to connect assessment records to existing longitudinal student profiles.
- Shadowing & Co-running (Jan–Feb 2027):
- Shadow 3rd party implementation partner engineers on day-to-day operations of existing data feeds (e.g., NSC, K-3, CDEC, and Phase 1 feeds).
- Co-run live feed cycles, handle data loading, execute record matching, and troubleshoot exceptions alongside 3rd party implementation partner.
- Readiness Gate Execution (Target: March 2027):
- Run two consecutive feed cycles completely unaided to achieve formal operational hand-over sign-off.
- Full Production Operations (March 2027 Onward):
- Execute recurring feed schedules, rebuild downstream objects, reconcile row counts against source files, and triage failed jobs.
- Address ad-hoc data inquiries and export requests from OIT leadership and partner agencies.
- Stabilization Backlog Support: Resolve identified data reliability defects, run variations, and de-duplication issues transitioned from the Phase 1 stabilization backlog.
- Analytics Layer Hardening & Maintenance (April 2027 Onward):
- Apply object modeling standards, naming conventions, and layout-change detection into the RAW layer.
- Perform ongoing reconciliation of analytics layer outputs across non-production and production environments.
- Implement metric and calculation enhancements prioritizing downstream use cases (program participation, completion, equitable access, and wage outcomes).
- Conduct query profiling and performance tuning on dashboard back-end models.
- Testing & Release Management: Author manual and automated test cases in Azure Test Plans; record execution evidence and resolve defect bugs before promoting code.
- Cross-Role Support:
- Partner with the Data Architect to review and adapt physical and logical data models and pipeline designs.
- Support the Data Scientist / Visualization Designer by integrating new data sources to power Tableau dashboard refreshes and preparing source tables for de-identified Institution of Higher Education (IHE) outcome files.
- Coordinate with Operations Engineers on Snowflake compute management, warehouse auto-shutdown, user role permissions, and CI/CD promotion pipelines.
3. Deliverables Created
Technical Code & Pipeline Artifacts
- Production ETL/ELT Pipelines: Fully developed, scheduled Informatica workflows and Snowflake SQL/transformation pipelines for CDHE Courses and existing operational feeds.
- Cleansing & Validation Modules: Automated L1, L2, and L3 quality-check rules and Informatica layout-change detection routines in the RAW layer.
- Identity Matching & Linkage Routines: Documented internal matching scripts linking new partner agency student/workforce records into longitudinal IDs.
- Analytics Layer Transformation Objects: Production views, tables, and metric models in Snowflake aligned with longitudinal reporting use cases .
- De-Identified Data Extraction Scripts: Data extraction scripts that generate de-identified, suppressed outcome files for participating IHEs per specification.
- ETL Pipeline Design Documents: Technical architecture and transformation specifications detailing source-to-target logic, schemas, and dependencies.
- Mapping & Transformation Specifications: Contribution to enterprise data mapping documents aligning source fields to the SLDS common data model.
- Lineage Documentation & Dependency Maps: As-built data lineage records and dependency diagrams detailing object-level upstream and downstream flows.
- Operational Runbooks & Incident Procedures: Standard operating procedures for running feed cycles, handling late or malformed files, performing failover, and reconciling row counts.
- Azure Test Plans & Execution Records: Fully documented test plans, test suites, manual test run results, and defect logs for all pipeline modifications prior to production promotion .
- Release Change Logs: Detailed audit entries for all OIT-led production releases, detailing changes, rationales, linked Azure DevOps work items, test results, and deployment timestamps].
- Feed Operation Run Logs & Match Summaries: Run completion records, reconciliation totals, exception logs, and data-quality summaries produced for agency loads.
Required Qualifications & Technical Experience
- Snowflake ETL/ELT: Minimum 5 years of demonstrated experience building, orchestrating, and optimizing complex ETL/ELT data pipelines and transformation logic natively within the Snowflake platform.
- Informatica CDGC: Minimum 1 year of hands-on experience working with Informatica Cloud Data Governance and Catalog (CDGC) for metadata management, data profiling, cataloging, and automated data quality validation.
- Infrastructure as Code (IaC): Minimum 1 year of experience utilizing Terraform to provision, configure, and maintain cloud infrastructure and resources.
- CI/CD & DevOps: Demonstrated proficiency implementing automated build, test, and release deployment pipelines using Azure DevOps.
- Version Control: Expert proficiency with Git / version control workflows (branching strategies, pull requests, automated merge checks, code reviews).
- Location Constraint: Must currently reside within the State of Colorado for the duration of the engagement.
- Prior experience working with public sector, longitudinal education, and workforce data systems (e.g., CDE, CDHE, CDLE, CDEC, National Student Clearinghouse).
- Experience with record linkage, entity resolution, and deterministic/probabilistic identity matching across disparate public records
- Familiarity with federal and state student/workforce privacy regulations, including FERPA, COPPA, and cell suppression standards
- Experience authoring and executing structured test plans in Azure Test Plans
- Position Type: 6-Month Contract (with possibility of extension based on project performance and state funding)
- Work Arrangement: Remote / Telecommuting (Candidate must reside within the State of Colorado; strong preference for local / Denver Metro candidates)