Bigdata Engineer

Job Title: Bigdata Engineer
Contract Type: Permanent
Location: Australia
Reference: 476785BR
Contact Name: Das, Rajib
Job Published: November 04, 2021 13:20

Job Description

At IBM, work is more than a job – it’s a calling: To build. To design. To code. To consult. To think along with clients and sell. To make markets. To invent. To collaborate. Not just to do something better, but to attempt things you’ve never thought possible. Are you ready to lead in this new era of technology and solve some of the world’s most challenging problems? If so, lets talk.

Your Role and Responsibilities
The Data Engineer is responsible for building and supporting the big data platform and data pipelines. As a Data Engineer – Big Data, your responsibilities include –

  • Design and implement complex Data Lake and Data mesh architectures utilising data fabric capabilities like streaming, event-driven and batch patterns.
  • Write high-quality code, perform code reviews, and take ownership of software delivery.
  • Responsible for building software development frameworks, best practices and guidelines for adoption.
  • Identify and resolve defects during test phases, handle deployments, and coordinate with multiple teams to provide ongoing improvements to the code, processes, and overall design.
  • Evangelise and democratise data both to internal as well as external stakeholders.
  • Knowledge of test-driven development and test automation frameworks:
  • Optionally you would have an Application Development background along with knowledge of front end development, Analytics libraries, open-source Natural Language Processing, statistical and big data computing libraries.

Key Skills

  • Strong python developer having a great understanding of Python features in version 3.7+
  • Experience in using jinja2 template in python
  • Java or Scala
  • Experience in designing and building data pipelines on Spark utilising PySpark
  • Experience in building REST services using python. fast API experience is highly desirable
  • Experience in Postgres database programming with python ORM framework like peewee
  • Experience in static type checking tools with python and CI/CD
  • Experience in AWS technology: Lambda, S3, SQS, SNS and event-sourced mapping
  • Experience in python performance optimization, multi-threaded programming is highly desirable
  • Front-end development experience with TS/Angular is desirable but not essential


  • Having a great understanding of Python features in version 3.7+ with a background working in data migration, data wrangling initiatives.
  • Experience in using jinja2 template in python or other similar templates.
  • Experience in building REST services using python. Fastapi experience is highly desirable or similar.
  • Experience in SQL database programming using python ORM framework like peewee.
  • Experience in static type checking tools with Python and experience in DevOps.
  • Experience in AWS technology: Lambda, S3, SQS, SNS and event sourced mapping
  • Design and development knowledge on Kafka and spark structured streaming (Databricks etc)
  • Good understanding of data modelling patterns
  • Working knowledge of Unix/Linux systems
  • Working knowledge of multiple database paradigms like RDBMS and NoSQL.
  • Strong knowledge of data security management policies and procedures
  • Experience with Agile Development methodologies
  • Strong troubleshooting and performance tuning skills

Good to have skills

  • Experience in Python performance optimization, multi-threaded programming
  • Front-end development experience with TS/Angular
  • 2+ years experience in the Financial Services domain

Being You @ IBM

IBM is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, pregnancy, disability, age, veteran status, or other characteristics. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.

Get similar jobs like these by email

By submitting your details you agree to our T&C's

Are you looking for better jobs?

Flexible Work . Equal Pay . Leadership Development

Join Our Movement

Are you looking for talents?

Join Us To Diversify Your Team!

Post A Job