Data Engineer

Home :: Data Engineer

Qualification: This is a professional position, and as such, we require, at minimum, a Bachelor’s degree in computer science, Computer Technology, computer information systems, information technology, or advanced educational background equating to the U.S. equivalent of a Bachelor’s degree in one of the aforementioned subjects.

Job Description: 

Designing, constructing, maintaining scalable data model and data pipelines:

Design, implement and improve processes, procedures and automation for all database centric areas; Build tools and scripts to develop, monitor, troubleshoot and automate our ETL’s; Involved in migrating the data models from Informatica PowerCenter to DBT; Work with the Data Modelers to understand the data for ingestion into the common Data Model, Extracting Data from the different RDBMS to the HDFS; Introduce new data management tools & technologies into the existing system to make it more efficient; Work as a Data Engineer which involves analyzing, Building the code in Snowflake/SQL; Take care of the complete ETL process.

Data Quality, Data Governance,Security and Optimization:

Investigate, analyze, correct and document reported data defects; Improve data quality, reliability & efficiency of the individual components & the complete system; Tune our data pipelines for performance and reliability; Optimaztion,Performance tuning and Capacity Planning; Establishing and maintaining data governance policies

Code Quality, Debugging and testing:

Perform scalability, latency, and availability tests on a regular basis; Perform code reviews and QA data imported by various processes; Perform root-cause analysis of any issues that arise post-implementation and work on solutions related to issue fixing; Build highly scalable, robust, fault-tolerant systems.

Continues Integration/Continues Deployemnt and Pipeline Orchestration:

Create a complete solution by integrating a variety of programming languages & tools together; Involved in multiple deployments through Jenkins/CICD pipeline tools; Build Dependencies and schedule jobs in Control M; Design, create, test, and maintain the complete data management; processing systems.

Application integration and testing:

Develop load and stress test designs, cases and procedures. Automated regression and functional test suites; Conduct testing as per the set standards and procedures in SQA/UAT/VAL environments; Script unique test plans, test scripts and processes to remove previously known redundancies and ensure predictable outcomes; Work with the QA team to ensure the product compliance with requirements; Work closely with performance engineering, network teams and apply techniques to create highly scalable applications.

Build team competencies and establish best practices:

Conduct knowledge sharing sessions with team members and walkthrough the existing software design/architecture and provide guidelines for building new application software; Train and mentor junior programmers in programming methodologies and best practices; Involved in Requirement Gathering, Map Gap Analysis, Data Modelling, Building and Deploying the code in Informatica using Cisco Jenkins Tool and IBM U-Deploy; U Release tools; Conduct peer code review and ensure design and development standards are strictly followed.

 

Scroll to Top