UK Data Engineering Stack — 2026 Recommendations
| Layer | ClickMasters Default | When to Use Alternatives | Regulated Sector Note |
|---|---|---|---|
| Ingestion | AWS Glue (managed ETL) or Lambda (event-driven) | Fivetran for SaaS source connectors | NHS: CloudTrail audit of all ingestion |
| Storage (raw) | S3 eu-west-2 (data lake — parquet) | GCS if GCP stack | NHS: eu-west-2 only, KMS encryption |
| Transformation | dbt (warehouse transforms) | Databricks for > 100M rows or ML features | FCA: dbt SQL reviewable by compliance team |
| Warehouse | Redshift Serverless (NHS/FCA) or Snowflake EU | BigQuery if GCP-heavy | NHS DTAC: Redshift same AWS account strongest |
| Orchestration | AWS MWAA (Managed Airflow) or Step Functions | Dagster, Prefect for smaller teams | NHS: Airflow audit log in CloudWatch |
| Visualisation | QuickSight (AWS — same account) or Metabase | Looker, Power BI for enterprise BI | FCA Board reports: PDF export from QuickSight |
| Data Quality | Great Expectations (automated data tests) | dbt tests for warehouse quality | NHS DTAC: data quality evidence for Domain 4 |
| Lineage | dbt DAG + OpenLineage (Marquez) | Atlan, Alation for enterprise data catalogue | UK GDPR Article 30: data processing record |
NHS Health Data Engineering — UK Specifics
NHS DTAC Domain 4 (Interoperability) data engineering evidence: ClickMasters provides dbt DAG as data lineage documentation, data quality test results from dbt tests, and AWS Glue job history as ingestion audit trail — all three submitted as DTAC Domain 4 evidence.