Data Engineer

Duties & Role

The Data Engineer will be responsible for designing, developing, implementing, testing, and maintaining scalable data infrastructure and data pipeline solutions supporting data analytics, artificial intelligence, and operational reporting capabilities within NCIA environments. The contractor will work closely with Data Analysts, Data Scientists, Solution Architects, and other stakeholders to ensure efficient and secure processing of structured and unstructured data.

Key duties include:

• Design, develop, maintain, and optimize data pipelines and ETL/ELT processes for batch and near real-time data integration.
• Implement and support scalable data lake and data warehousing solutions in cloud and hybrid environments.
• Develop and maintain robust data ingestion, transformation, and storage mechanisms for structured and unstructured datasets.
• Collaborate with stakeholders to gather data requirements and translate them into technical solutions.
• Ensure data quality, integrity, availability, and security throughout the complete data lifecycle.
• Implement monitoring, logging, and alerting capabilities to ensure reliability and performance of data platforms and AI-related systems.
• Support the deployment and maintenance of AI, machine learning, and analytics workloads through reliable data engineering practices.
• Utilize containerization and orchestration technologies such as Docker and Kubernetes to support scalable deployments.
• Apply software engineering best practices including version control, CI/CD pipelines, automated testing, and documentation.
• Support cloud-based data engineering activities using platforms such as AWS, Microsoft Azure, or Google Cloud Platform.
• Ensure compliance with NATO security policies and data protection requirements, including encryption, authentication, authorization, and access control mechanisms.
• Provide technical support, troubleshooting, and performance optimization for existing data platforms and services.
• Produce and maintain technical documentation related to data architecture, pipelines, processes, and operational procedures.

Requirements

Education & Professional Experience

• A minimum requirement of a Bachelor’s degree at a nationally recognised/certified University in a related discipline such as Computer Science, Data Science, Software Engineering, Information Technology, or a related field, and relevant post-related experience.
• Exceptionally, the lack of a university degree may be compensated by the demonstration of particular abilities or experience of interest to NCIA, namely at least 6 years of extensive and progressive expertise in duties related to the function of the post.

Mandatory Experience & Skills

• Strong understanding of data engineering concepts including data warehousing, data lakes, ETL/ELT processes, and data governance.
• Proven experience designing, implementing, and maintaining data pipelines in both batch and near real-time processing environments.
• Experience with data engineering tools and frameworks supporting data analytics, data science, and AI solutions.
• Experience working with structured and unstructured data sources.
• Experience with monitoring, logging, and observability tools to ensure reliability and performance of data pipelines and AI systems.
• Solid understanding of containerization and orchestration technologies such as Docker and Kubernetes.
• Experience with cloud platforms including AWS, Microsoft Azure, and/or Google Cloud Platform.
• Solid understanding of software development best practices including Git/version control, CI/CD pipelines, automated testing, and code quality practices.
• Strong proficiency in Python programming for development and integration of data engineering solutions.
• Good understanding of data security best practices including encryption, authentication, authorization, access control, and compliance requirements.
• Ability to work effectively in an international and multidisciplinary environment.
• Strong analytical, problem-solving, and communication skills