Koniag Government Services (KGS) is an Alaska Native Owned corporation supporting the values and traditions of our native communities through an agile employee and corporate culture that delivers Enterprise Solutions, Professional Services and Operational Management to Federal Government Agencies.
Senior Databricks Engineer
Location
United States
Posted
16 days ago
Salary
$140K - $160K / year
Seniority
Senior
Job Description
Senior Databricks Engineer
Koniag Government Services, LLC
Role Description Koniag Emerging Technologies, LLC, a Koniag Government Services company, is seeking a Senior Databricks Engineer with a Secret security clearance to support KET and our government customer in Washington, DC. This is a remote position. The Senior Databricks Engineer will be responsible for designing, developing, and deploying scalable data infrastructure while driving key data engineering initiatives. By leveraging the Databricks platform and deep expertise in modern data warehousing and the Medallion Architecture, the engineer will build data pipelines supporting the Department of State’s “4Ps” data lake. The ideal candidate will be able to support the following functions: - Cross-Functional Collaboration: Work closely with project managers, team members, and business owners to gather requirements, communicate progress, and ensure data meets customer needs. - Architecture & Design: Architect and implement scalable, highly available data pipelines using Databricks, adhering to the Medallion Architecture (Bronze, Silver, Gold layers) to ensure data quality, consistency, and optimized reporting. - Data Quality Governance & Monitoring: Establish comprehensive data quality standards and implement monitoring dashboards to enable business owners and responsible parties to track and proactively address data issues. - Pipeline Development: Write clean, efficient, and production-ready code using PySpark and SQL for complex data extraction, transformation, and loading (ETL/ELT) processes. Expertise in DLT and structured streaming pipelines. - API Integration: Design and build robust data ingestion pipelines to reliably pull from various third-party external systems using RESTful API integrations. - Automated Deployments (CI/CD): Implement infrastructure and pipelines as code. Utilize Databricks Asset Bundles (DABs) integrated with Azure DevOps to manage and automate deployments across Development, Staging, and Production environments safely and reliably. - Code Reusability: Develop, package, and maintain custom Python libraries and modules to establish common frameworks used across multiple data projects and teams. - Data Warehousing: Apply foundational and advanced data warehousing concepts (e.g., dimensional modeling, slowly changing dimensions) to optimize data storage and retrieval for downstream analytics and business intelligence. - Performance Tuning: Monitor, troubleshoot, and optimize Databricks clusters and Spark jobs to ensure maximum efficiency and cost-effectiveness. - External BI & Native Dashboarding: Securely share data with external BI tools (such as PowerBI or SAP BusinessObjects) and develop dashboards and capabilities natively within the Databricks environment. - Mentorship & Leadership: Serve as a technical SME for the data engineering team, conducting code reviews, establishing best practices, and mentoring other staff members. Qualifications - Knowledge of Object Oriented Programming (OOP) concepts and proficiency in one or more programming or scripting languages (Java, C#, Python, JavaScript, PowerShell). - Experience: 8+ years of dedicated Data Engineering experience, with at least 3+ years functioning in a senior capacity focused on the Databricks ecosystem. - Core Languages: Expert-level proficiency in PySpark and complex SQL. - Python Library Management: Proven ability to create, package, and manage custom Python libraries (e.g., building Wheel files) for scalable reuse across multiple data pipelines and projects. - API Data Ingestion: Deep understanding of HTTP REST methods (GET, POST, PUT, etc.) with hands-on experience extracting, paginating, and processing data from external API sources. - Frameworks & Methodology: Deep theoretical and practical understanding of modern Data Warehousing concepts and hands-on experience building out the Medallion Architecture. - DevOps & CI/CD: Proven experience automating Databricks deployments. Hands-on experience configuring and deploying Databricks Asset Bundles (DABs) using Azure DevOps (creating YAML pipelines, managing service principals, and handling multi-workspace deployments). - Cloud Platforms: Strong familiarity with the broader cloud ecosystem surrounding Databricks (e.g., Azure Data Lake Storage Gen2, Azure Key Vault). - Version Control: Proficiency with Git-based version control workflows. - Concise and comprehensive written and oral communication skills for both technically understanding and customer usage support. Requirements - Secret Security clearance. Desired Skills - Experience with a wide range of data products, BI tools, and visualization libraries such as Tableau, PowerBI, Jupyter Notebooks, d3js, and Analytics Search Tools such as Elasticsearch, SOLR, Lucene. - Experience working with HR application (PeopleSoft) data for reporting purposes. - 3+ years of .NET, Java or similar OOP experience. - Experience writing Oracle PL/SQL queries, packages, and procedures. - Experience architecting Data Vault 2.0 solutions. Benefits - Competitive compensation. - Extraordinary benefits package including health, dental, and vision insurance. - 401K with company matching. - Flexible spending accounts. - Paid holidays. - Three weeks paid time off. - And more. Company Description Koniag Government Services (KGS) is an Alaska Native Owned corporation supporting the values and traditions of our native communities through an agile employee and corporate culture that delivers Enterprise Solutions, Professional Services, and Operational Management to Federal Government Agencies. As a wholly owned subsidiary of Koniag, we apply our proven commercial solutions to a deep knowledge of Defense and Civilian missions to provide forward-leaning technical, professional, and operational solutions. KGS enables successful mission outcomes for our customers through solution-oriented business partnerships and a commitment to exceptional service delivery. We ensure long-term success with a continuous improvement approach while balancing the collective interests of our customers, employees, and native communities.
Related Guides
Related Categories
Related Job Pages
More Engineer Jobs
• Conduct site reconnaissance, interviews, and independent research • Prepare deliverables with quality supporting documentation (site figures, appendices) • Travel on short notice (one week or less) will be required at times
• Design, build, and operate production-grade proxy infrastructure at scale • Own the full lifecycle of proxy platform • Serve as technical authority on traffic flows • Work across forward proxies, reverse proxies, API gateways, and protocol translators
Analytics Engineer
Hunt StWe help Aussie companies find top 3% remote talent in the Philippines & Nepal for a single finder's fee.
Role Description Looking for Philippines-based candidates. Job Role: Analytics Engineer Compensation range: $2,000 AUD - $3,000 AUD / Monthly Engagement type: Independent Contractor Agreement Work Schedule: This role is expected to align with the AU business hours (approx. 9 AM - 5 PM, Monday to Friday) for collaboration, but as a contractor, you’ll have flexibility in how you manage your time. The analytics engineer builds the Gold layer and dbt Semantic Layer for each dataset phase. Where the senior data engineer builds the pipeline plumbing, the analytics engineer builds the business metrics — translating the metrics register definitions into governed, tested Gold models and MetricFlow metric definitions. This role sits closest to the business, requiring both technical depth and the ability to work directly with business stakeholders during metric validation sessions. The analytics engineer must be able to explain metric calculation choices in plain English, not just write the SQL. In Run steady state, the analytics engineer role will take on support with AI monitoring responsibilities. New Gold model build volume drops significantly post go-live, freeing capacity for NLQ quality monitoring, LLM agent uptime, and Cortex ML model maintenance. Candidates should be comfortable with this evolution and ideally have some exposure to LLM API integration or ML monitoring. Key Responsibilities - Build Gold layer dbt models, including metric-level aggregations, KPI tables, and denormalized summary tables optimized for BI consumption in Sigma. - Author dbt Semantic Layer metric definitions using MetricFlow, including sell-through, out-of-stock (OOS) rate, conversion rate, average order value (AOV), weeks cover, and other metrics defined in the business metric register. - Populate and maintain the business glossary for each completed dataset, including metric definitions, calculation notes, known caveats, ownership, and data lineage. - Participate in business validation sessions with the onshore team and business stakeholders to ensure Gold metrics align with agreed metric register definitions prior to handover to Run operations. - Author and maintain dbt Explorer documentation, including model descriptions, column descriptions, and metric definitions to ensure the semantic layer remains fully self-documenting. - Support ongoing Run operations by managing metric change requests, implementing new Gold model additions, monitoring Cortex ML models, maintaining Claude NLQ context updates, and tracking AI agent uptime and performance. - Collaborate with stakeholders and cross-functional teams to validate business requirements, clarify KPI logic, and ensure consistent metric governance across the platform. Qualifications - Advanced experience with dbt Semantic Layer and MetricFlow, including metric definitions, dimensions, entities, measures, and time spines. - Advanced SQL expertise, including metric calculation logic, window functions, ratio metrics, and period-over-period comparisons. - Basic Python scripting skills for automation and monitoring-related tasks. - Intermediate experience with Power BI or other self-service BI platforms, with the ability to validate Gold layer outputs within BI environments. - Intermediate experience with Atlan or similar data cataloguing platforms, including catalogue administration, glossary authoring, and lineage review. - Strong retail domain knowledge, with a clear commercial understanding of metrics such as sell-through, OOS rate, clearance percentage, return rate, and AOV. - Strong communication and stakeholder management skills, with the ability to clearly present metric validation findings to non-technical business stakeholders. - Experience using Claude API or similar AI tooling to translate business metric register rules into dbt Gold model SQL. - Familiarity with Cortex Analyst YAML generation for scaffolding Semantic Layer definitions aligned with metric register requirements. - Experience monitoring Cortex ML models, including Forecasting and Anomaly Detection performance, and triggering retraining workflows when necessary. - Understanding of NLQ (Natural Language Query) context management, including maintaining governance and semantic layer context that supports AI-powered business user interfaces. - Exposure to LLM APIs, Snowflake Cortex, or similar AI-driven analytics platforms, whether in production, prototype, or personal project environments. Run Steady State Scope - Manage metric change requests by updating Gold models and Semantic Layer definitions as business rules and KPI definitions evolve. - Monitor AI agent performance, including NLQ answer quality, identifying model drift, escalating issues to the lead engineer, and updating AI system prompt context as new datasets are introduced. - Oversee Cortex ML maintenance by monitoring forecast accuracy and triggering retraining processes when performance falls below agreed thresholds. - Build and maintain new Gold model additions as the organization expands the Gold layer and introduces additional business metrics. What Good Looks Like - Demonstrates genuine enthusiasm for building a modern data platform from the ground up. - Can independently translate plain-English metric definitions (e.g., “net sales excluding shipping, gift cards, and staff purchases”) into dbt Gold model SQL and MetricFlow YAML. - Has prior experience in stakeholder-facing environments, including presenting data findings, answering questions from non-technical users, and defending calculation methodologies. - Brings curiosity and adaptability when working with emerging AI technologies, semantic layers, and modern analytics engineering practices. Work Arrangement & Expectations This is a remote role that will be set up as an independent contractor engagement. To ensure alignment and transparency, successful candidates will be expected to: - Disclose any existing ongoing roles or client work. - Reflect this engagement on their LinkedIn profile (clearly marked as “Independent Contractor”).
• Act as the primary technical authority on-site, becoming an expert in the customer’s domain to identify and resolve real-world friction points before they escalate. • Design and implement data architectures that accelerate AI/ML training pipelines, ensuring low-latency data access across hybrid-cloud environments. • Build custom integrations, automation scripts, and "last-mile" solutions that bridge the gap between Hammerspace’s core product and the client’s bespoke infrastructure. • Use your proximity to the action to identify new business problems, expansion opportunities, and emerging workloads that Hammerspace can solve. • Partner with our Rotational Sustaining Engineering team to translate field-discovered bugs into core product features, ensuring that local fixes become global platform strengths. • Translate complex technical hurdles into clear, actionable insights for both end-users (data scientists/IT admins) and executive leadership.


