Top level Data scientist at Interswitch Group in Lagos

Position Objective

To build dependable data merging initiatives, clean, transform, and evaluate vast amounts of big data from various operational structures using Spark and other ETL resources to offer prepared-to-use dataset to data scientists and data analysts, while making sure data accuracy and honesty and reliability. Work alongside participants to design ability to scale and productive data initiatives that enable informed strategic judgment, and adherence with data administration.

Duties

• Formulate and execute productive data ingestion pipelines to obtain and extract large volumes of organized and unstructured data. Ensure information accuracy and quality during the ingestion process.
• Combine various data suppliers and data layouts into a unified data ecosystem.
• Design and implement information handling processes to clean, transform, and enrich unedited data.
• Formulate ability to scale information handling algorithms and methods to manage big data volumes productively.
• Enhance information handling pipelines for effectiveness and reliability.
• Record data engineering methods, processes, and system architectures for future letter of recommendation and expertise move.
• Organize system manuals, which includes data dictionaries, data lineage, and system requirements.
• Produce and manage records connected with data administration, adherence, and guard procedures.
• Produce and preserve data storage architectures that cater to the specific needs of big data applications. Execute reliable information management plans, which includes data partitioning, indexing, and compression methods.
• Ensure data guard, confidentiality, and adherence possessing applicable rules.
• Work alongside data scientists and analysts to comprehend their qualifications and convert them into ability to scale data models.
• Submit application data visualization methods to convey findings productively.
• Produce and preserve data storage architectures that cater to the specific needs of big data applications. Execute reliable information management plans, which includes data partitioning, indexing, and compression methods.
• Ensure data guard, confidentiality, and adherence possessing applicable rules.
• Work alongside data scientists and analysts to comprehend their qualifications and convert them into ability to scale data models. Submit application data visualization methods to convey findings productively.
• Work alongside cross-operational teams, which includes data scientists, analysts, and software engineers, to comprehend their data qualifications and offer IT assistance.
• Convey sophisticated technical notions and findings to non-technical participants in a clear and brief manner.
• Engage in information exchange initiatives and support to the ongoing development of data engineering procedures Recognize and execute plans to improve the effectiveness and effectiveness of big data applications and operational structures. Carry out effectiveness tuning, load running test, and capacity organizing to meet scalability and throughput qualifications.
• Supervise operations effectiveness and diagnose issues concerns connected with information handling, storage, and retrieval.
• Establish and ensure adherence to data administration rules, guidelines, and industry standards.
• Guarantee adherence with data rules, which includes GDPR or HIPAA, by executing suitable information security measures.
• Carry out data audits and execute data accuracy regulates to preserve data precision and uniformity.

General Expertise

A minimum of 3 years' conceptualizing, deploying, and coordinating strong ETL/ELT data initiatives, ideally in a reputable Financial Institution or Financial technology organisation.

Behavioural Skills:

• Possess robust logical thinking competencies to comprehend sophisticated data qualifications, recognize patterns and practices in data, and design productive and ability to scale data initiatives. Be capable to break down sophisticated challenges into manageable parts and formulate logical and efficient initiatives. Be capable to evaluate datarelated concerns, diagnose issues challenges, and execute suitable outcomes.
• Approach hurdles with a initiative-driven mentality and come up with creative and practical initiatives.
• Possess a sharp perception for description, making sure data precision, quality, and honesty and reliability through thorough data verification and confirmation methods. Remitting attention to effectiveness optimization and data guard measures.
• Be capable to operate productively within a collaborative environment, convey and work alongside team members (data scientists, analysts, software engineers, and other participants), and support your proficiency to attain common targets
• Have effective communication competencies to productively convey technical notions and qualifications to both technical and non-technical participants.
• Be capable to communicate effectively your creative thoughts, record your work, and produce clear and brief system manuals for future letter of recommendation.
• Follow ethical standard procedures and preserve a high level of competence.
• Prioritize data confidentiality, guard, and adherence possessing applicable rules.
• Show honesty and reliability, integrity, and responsibility in your work.
• Stay informed with the latest practices and improvement in data engineering technical solutions, resources, and industry standards. Actively pursue avenues for career advancement and self-advancement.
• Expect possibility concerns, design strong and ability to scale data architectures, and execute supervising and warning operational structures to identify and handle concerns proactively.

Competencies:

• Competence in operating with various Big Data technical solutions is crucial. This comprises Apache Hadoop, Apache Spark, Apache Kafka, Apache Hive, Apache Pig, and other associated frameworks. Comprehend the architecture, parts, and ecosystem of these technical solutions to design and execute reliable information processing pipelines.
• Competence in information handling and Extract, Transform, Load (ETL) methods.
• Must be proficient in designing and executing productive data pipelines to extract data from various suppliers, transform it into a appropriate document structure, and load it into target operational structures or data warehouses. Competence in PySpark and expertise with Microsoft Azure Databrick is essential.
• Competence in executing CI/CD procedures for data pipelines, which includes version regulate, automated running test, and deploying methods, using resources like Git, Jenkins, or similar channels to guarantee smooth and dependable deploying of data pipelines, for faster development cycles, upgraded code quality, and productive release management.
• Competence in python programming language and SQL, to adjust creatively and transform data, build data pipelines, and automate methods. Should comprehend data profiling, data cleansing, and data verification methods to guarantee the precision, thoroughness, and uniformity of the data. Expertise in data confidentiality and adherence rules is also significant.
• Strong insight of distributed operational structures and parallel processing notions.
• Should be knowledgeable about notions like data partitioning, parallel processing, fault tolerance, and cluster management.
• Should be knowledgeable about different data storage technical solutions and NoSQL databases like Apache HBase, Apache Cassandra, MongoDB, or Amazon DynamoDB. Should comprehend the trade-offs between different storage options and select the suitable one based on the application scenario and qualifications.
• Should be capable to design productive data schemas that enhance data storage, retrieval, and processing. Should comprehend notions which includes entity connection modeling, dimensional modeling, and schema evolution
• Should possess background in instantaneous information handling methods. Be knowledgeable about stream processing frameworks like Apache Flink, Apache Kafka Streams, or Apache Storm.
• Comprehend notions like event-driven architectures, message queues, and instantaneous data analysis for building instantaneous data pipelines.