Senior Data Engineer - Cloud ETL

Introduction
Software Developers at IBM are the backbone of our strategic initiatives to design, code, test, and provide industry-leading solutions that make the world run today - planes and trains take off on time, bank transactions complete in the blink of an eye and the world remains safe because of the work our software developers do. Whether you are working on projects internally or for a client, software development is critical to the success of IBM and our clients worldwide. At IBM, you will use the latest software development tools, techniques and approaches and work with leading minds in the industry to build solutions you can be proud of.

Your Role and Responsibilities
As a Data Engineer - Cloud ETL you are expected to be functionally knowledgeable in data domain areas with a good emphasis on Data architecture around ETL using cloud native Paas services. As a Data Engineer in our Data and AI team, you will provide support for data management and ETL for one or more projects; assist in defining scope and sizing of work; and work on Proof of Concept development. You will support the team in providing data engineering solutions based on the business problem, data integration with third party services, designing and developing complex ETL/ELT pipelines for clients’ business needs. You will collaborate with some of the best talent in the industry to create and implement innovative high quality solutions, participate in Pre-Sales and various pursuits focused on our clients’ business needs.

You will also contribute in a variety of roles in thought leadership, mentorship, systems analysis, architecture, design, configuration, testing, debugging, and documentation. You will challenge your leading edge solutions, consultative and business skills through the diversity of work in multiple industry domains.

Responsibilities

• Be responsible for requirements gathering, System Designing, Data modeling, and ETL design, development, Production Enhancements, Support and Maintenance
• Support the build of complex data processing pipeline in java/python using apache Beam/Glue/databricks for reading/writing data to /from Data warehouses/Lakes on cloud
• Write generic ETL flow templates, job scheduling and management using native services
• Developing and maintaining complex data ETL/ELT Pipelines, data models and standards for various data Integration & data warehousing projects from source to sinks on Saas/Paas platforms - snowflake, databricks, Azure Synapse, Redshift, BigQuery, etc
• Develop scalable, secure and optimized data transformation pipelines and integrate them with downstream sinks
• Develop and maintain documentation of the data flows and data models, integrations, pipelines etc
• Support the teams in providing technical solutions from data flow design and architecture perspective, ensure the right direction and propose resolution to potential data pipeline-related problems.
• Developing Proof of concepts (PoC) of key technology components to project stakeholders
• Collaborate with other members of the project team (Architects, Data Engineers) to support delivery of additional project components (like API interfaces, Search, visualization)
• Evaluate and create PoVs around the performance aspects of ETL/ELT tools in the market against customer requirements
• Working within an Agile delivery / DevOps methodology to deliver proof of concept and production implementation in iterative sprints.
• Assist in driving improvements to the Big Data and Cloud Data Warehouse technologies stack, with a focus on the digital experience for the user, as well as application performance & security to meet the needs of the business and customers, now & in the future
• Support technical investigations & proof of concepts, both individually and as part of a team, including being hands - on with code, to make technical recommendations
• Create documentation for Architecture Principles, design patterns & examples, technology roadmaps & future planning

Required Technical and Professional Expertise
ETL Developer with 7+ years of experience with following skills -

• Good hands-on experience in SQL
• Good experience in Data Modelling techniques
• Good experience in design & implementation of Enterprise Data Warehouse solutions
• Hands-on expertise in at least 1 projects (ETL) on Hyperscalars - AWS, Azure, GCP
• Experience in any of the following solutions - AWS Glue, Dataflow, Datafusion, Databricks, Azure Data Factory
• Exposure to Big Data and Cloud Data Warehouse technologies - Redshift, BigQuery, Synapse, Snowflake
• Experience in ETL technologies (in more than 1 area amongst Informatica, DataStage, SAS, Azure Data Factory, Talend, Matillion)
• Experience in BI technologies (in more than 1 areas amongst Cognos, BO, Power BI, Tableau)
• Experience in Database (in one or more amongst Teradata, Exadata, Netezza, Oracle)
• Meta Data Management and Data Governance Experience
• Hand on experience in Python, Scala, Spark, Kafka

Preferred Technical and Professional Expertise

• DevOps - CD/CI Implementations
• Framework Development and Automation Techniques
• Experience in implementation of Data Catalogue and Data Lake Implementations
• Experience in Data Management Solution Development with strong experience in SQL and NoSQL data bases

About Business UnitThis position currently sits within Global Technology Services (GTS) Infrastructure Services (IS) or a shared services function supporting GTS.
As announced in October 2020, IBM intends that its managed infrastructure services business of the GTS organization will become an independent company named Kyndryl by the end of 2021, creating two distinct and powerful market-leading companies.
Together, we will advance the vital systems that power the digital economy. Serving over 4,600 technology-intensive, highly regulated customers, including over 75% of the Fortune 100, our people will design, run, and manage the most modern and reliable technology infrastructure that the world depends on today.
We will work flexibly and in partnership with our customers to amplify business outcomes while always pushing ourselves to improve and meet all opportunities. Come join our team of diverse, devoted, and empathetic technology experts who are at the center of discovering what’s next.
Please note: The final decision if this position will transition from IBM to Kyndryl is yet to be confirmed.

Your Life @ IBMWhat matters to you when you’re looking for your next career challenge?

Maybe you want to get involved in work that really changes the world? What about somewhere with incredible and diverse career and development opportunities – where you can truly discover your passion? Are you looking for a culture of openness, collaboration and trust – where everyone has a voice? What about all of these? If so, then IBM could be your next career challenge. Join us, not to do something better, but to attempt things you never thought possible.

Impact. Inclusion. Infinite Experiences. Do your best work ever.

About IBMIBM’s greatest invention is the IBMer. We believe that progress is made through progressive thinking, progressive leadership, progressive policy and progressive action. IBMers believe that the application of intelligence, reason and science can improve business, society and the human condition. Restlessly reinventing since 1911, we are the largest technology and consulting employer in the world, with more than 380,000 IBMers serving clients in 170 countries.

Location StatementWhen applying to jobs of your interest, we recommend that you do so for those that match your experience and expertise. Our recruiters advise that you apply to not more than 3 roles in a year for the best candidate experience.

For additional information about location requirements, please discuss with the recruiter following submission of your application.

Being You @ IBMIBM is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, pregnancy, disability, age, veteran status, or other characteristics. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.