Hadoop Spark Engineer Expert

About Experian

Experian is the world’s leading global information services company. During life’s big moments — from buying a home or a car to sending a child to college to growing a business by connecting with new customers — we empower consumers and our clients to manage their data with confidence. We help individuals to take financial control and access financial services, businesses to make smarter decisions and thrive, lenders to lend more responsibly, and organizations to prevent identity fraud and crime.

We have 17,800 people operating across 44 countries, and every day we’re investing in new technologies, talented people and innovation to help all our clients maximize every opportunity. We are listed on the London Stock Exchange (EXPN) and are a constituent of the FTSE 100 Index.

The Spark Performance Tuning Engineer is to help drive performance analysis, resiliency initiatives, optimization and scalability of spark jobs running on the Big Data clusters.

The ideal candidate should have a good knowledge of performance engineering and have experience in Big Data Administrator role with a keen focus on performance and resilience. The candidate will be responsible for analyzing, debugging, measuring and recreating specific user/tenant issues in order to extract actionable data and model future capacity and scalability needs.

Required Experience:

• Ability to clearly characterize system and Spark Job performances.
• Passion for distilling large volumes of data, analyze scalability, capacity and performance issues in Big Data Platforms.
• Experience with Data Science and Big Data technologies, Hadoop, Hive, HBase, Solr, Spark, Kafka, Jupyter, Elasticsearch.
• Deep technical understanding of Spark Processing Framework.
• Ability to analyze and debug Spark applications written in Scala, Python and Java.
• Implement the spark performance tuning and resource optimization wherever required.
• Perform benchmarking/stress tests and document the best practices for different applications whenever required.
• Proactively work with multiple tenants on improving the application performance and ensure they are robust, resilient and scalable.
• Experience in tuning the spark applications on Kafka, HBase, Hive, HDFS and any other technologies.
• JVM Profiling and tuning experience.
• Application and architecture code profiling.
• Infrastructure and application performance tuning.
• Unix or Linux experience.
• Good understanding of OS Concepts & Resources Management.
• General operational expertise such as good troubleshooting skills, understanding of system’s capacity, bottlenecks, Memory, CPU, OS, storage, and networks.
• Proactively monitoring system performance and capacity planning.
• Automation experience (Shell/Python/Ansible).
• Good Understanding of Yarn Framework and Schedulers.

Required Knowledge / Skills / Experience:

• Bachelors or Master’s degree in Computer Science or equivalent experience
• At least 10 years of IT experience in Big Data/Hadoop environments
• Must have working knowledge of YARN, HBase, Hive, Spark, Flume, Kafka etc.
• Knowledge of programming & scripting languages a plus –Python, R, Perl, Hive, MapReduce, SQL etc.
• Experience with Open Source technologies ELK, Presto, Nodejs.
• Knowledge of Monitoring solutions like Dynatrace, Splunk.
• Spark and Java Applications Development Experience.
• Experience in automating deployments and monitoring/alerting tasks (preferably Ansible).
• Experience in working cloud environments, AWS, OCI etc.
• Experience with Performance, Load, Stress, and Scalability Testing and relevant tooling
• Experience of application software development life cycle. Should be able to influence application projects and provide guidance if needed.
• Strong Problem Solving and creative thinking skills
• Ability to multi-task and manage multiple assignments simultaneously
• Effective oral and written communications
• Experience working with geographically distributed teams

Our uniqueness is that we truly value yours. From India, across Asia to Japan and down to Australia, our 1800 people are what make our business special. Experian’s culture, people and environments are key differentiators. We focus on what truly matters; diversity and inclusion, work/life balance, development, innovation, collaboration, wellness, reward & recognition, volunteering... That’s why we’re an award winning organisation for our people-first practices, something we’re very proud of.