Data Engineer

Data Engineer (Demographic & Geospatial Data) Location: Remote working Type: Contract or Permanent Day Rate: Up to £550 per day Inside IR35 Salary: Up to £90,000 package Security Clearance: SC Eligible (Mandatory) / Active SC Preferred Start Date: Immediate or up to 2 weeks' notice The Opportunity We are supporting a major programme seeking an experienced Data Engineer with strong Databricks and Geospatial Data expertise. You will play a key role in designing and delivering scalable data pipelines that combine demographic datasets with geographic reference data to support population analysis, reporting, service planning, and operational decision-making. This is an excellent opportunity for a Data Engineer who enjoys working with large-scale data platforms, geospatial processing, and modern cloud-based technologies in a highly collaborative environment. Key Responsibilities Design, build and support data pipelines using Databricks, Python, PySpark and SQL Develop reusable ingestion, transformation, validation and enrichment processes Ingest demographic datasets, postcode data, geographic boundaries and reference datasets Build and maintain Delta Lake datasets and manage data lineage Perform postcode enrichment and geographic assignment processing Implement spatial joins, point-in-polygon matching and location-based calculations Manage geographic reference data versions and historical reproducibility Develop automated data quality controls and reconciliation processes Optimise Spark workloads and pipeline performance Support production releases, monitoring and troubleshooting activities Work closely with analysts, data owners and platform teams to define business rules and acceptance criteria Essential Skills & Experience Strong commercial experience with: Databricks Python PySpark SQL Proven experience building and supporting production-grade data pipelines Strong knowledge of Delta Lake, schema management, merge logic and incremental processing Experience handling large datasets within cloud-based environments Practical experience working with geospatial data and spatial processing Understanding of coordinate reference systems, including: British National Grid (BNG) WGS84 Experience with spatial joins, geometry validation and geographic aggregation Strong data quality, reconciliation and testing experience Experience with Git, CI/CD and automated deployment pipelines Desirable Experience Apache Sedona or Databricks Spatial Functions GeoPandas, Shapely, pyproj or QGIS NHS or healthcare data NHS Postcode Directory (NHSPD) ONS Postcode Directory (ONSPD) National Statistics Postcode Lookup (NSPL) Experience working with: LSOAs MSOAs Local Authorities Integrated Care Boards (ICBs) GeoJSON, GeoParquet, Shapefiles, WKT/WKB formats Population, census or demographic datasets Experience working within regulated or secure environments What We're Looking For The ideal candidate will have a strong Data Engineering background combined with experience working with geographic and location-based datasets. You will be comfortable building scalable Databricks solutions, implementing automated processes, and ensuring the accuracy and reliability of demographic and geospatial data. Candidates must be eligible to obtain SC Clearance , with active clearance highly advantageous. What's on Offer Contract Up to £550 per day Inside IR35 Predominantly remote working Permanent Up to £90,000 total package Base salary up to approximately £81,000 Performance bonus Predominantly remote working Apply Now Due to the urgency of this requirement, we are particularly interested in candidates who are immediately available or on a notice period of no more than two weeks. Apply today with your latest CV for immediate consideration.