Python Full-Stack Data Engineer

Basic Function

A data engineer is responsible for collecting, managing, and converting raw data into information that can be interpreted by data scientists and business analysts. Data accessibility is their ultimate goal, which is to enable organisations to utilise data for performance evaluation and optimization.

Roles and Responsibilities

• Write efficient, clean & maintainable code along with test cases.
• Own or assist in the end-to-end development of applications.
• Write reusable code & packages.
• Adhere to best engineering practices & design patterns during development.
• A self-starter who can align the technology solution with a business goal.

Qualifications and Experiences

• At least 4-6 years of experience.
• Strong hands-on experience in Python is a must.
• Good experience with open-source Python libraries like pandas, Beautiful Soup, Dask, Ray, and Poetry.
• Working knowledge of Java is a must.
• Good knowledge of Java Frameworks like Maven and Spring Boot.
• Good experience processing and transforming data in a variety of file types like xlsx, CSV, XML, and JSON.
• Good understanding of OOP, Algorithms, and Data Structures.
• Good experience in designing and implementing RESTful APIs using frameworks like FastAPI or Django Rest Framework.
• Good knowledge of ORM concepts/querying.
• Good experience with SQL and NoSQL databases like Postgres, Redis, or DynamoDB.
• Good knowledge of working with cloud services like AWS or Azure.
• Good knowledge of service virtualization/containerization with Docker.
• Experience working with agile and Software development lifecycle tools (e.g., JIRA, Confluence, Git) and Test-Driven Development (TDD).
• Experience with CI/ CD frameworks like Bamboo or Jenkins.
• Experience in working in a Linux environment.
• Knowledge of serverless technologies is a plus.
• Knowledge of container orchestration systems like Kubernetes or Docker – compose is a plus.
• Knowledge of Building APIs around machine learning models is a plus.
• Experience working with search solutions like Lucene, Solr, and Elasticsearch is a plus.