As an IA/BigData Developer: you will create custom batch-oriented and real-time streaming data pipelines working within the Hadoop ecosystem, creating new data workloads/applications and also migrating applications from On-Premise Ecosystem to Cloud Data lakes. Implement, troubleshoots, and optimizes distributed solutions based on modern big data technologies like Hive, Hadoop, Spark, Elastic Search, Kafka , etc. to solve large scale processing problems in a Cloud cluster enviornment. Ensure proper data governance policies are followed by implementing defining project specific quality checks, classification, etc. Apply technical expertise to challenging programming and design problems. Strong communication, interpersonal and teaming skills, including the ability to work effectively in agile / scrum environment. Demonstrated interest and ability to quickly learn new technologies.
- Proficient understanding of distributed computing principles
- Experience developing Data storage processing and streaming flows using technologies like Spark, HDFS, Yarn and Hadoop
- Experience with various messaging systems, such as Kafka or RabbitMQ
- Good knowledge of Big Data querying tools, such as Pig or Hive
- Good understanding of Lambda Architecture, along with its advantages and drawbacks
- Experience with integration of data from multiple data sources
- Ability to solve any ongoing issues with operating the cluster
- Experience with scripting languages and automation of processes
- Proven track record of development accomplishments in highly collaborative environment
- Translate complex functional and technical requirements into detailed design
- Be passionate about solving customer problems and develop solutions that result in a passionate customer/community following
- Experience working with Cloud as infrastructure (AWS/Azure) or other Cloud Platform based on IaaS and PaaS Solutions
- 2+ years’ hands-on Experience with various messaging systems, such as Kafka, Spark data manipulation, pipeline creation.
- NoSQL DB (Dynamo DB/Mongo DB knowledge is helpful)
- Strong understanding and experience with Apache technologies like Hadoop, KAFKA, Spark, Kerberos.
- Experience with Artificial Intelligence and Machine Learning is a plus
- Experience using with Code Management/CICD tools like GITHub, Jenkins is useful
- Prior experience with automation is helpful
- Prior experience working in a Linux/Unix enviornment is helpful
- Prior experience working in an enterprise technology environment is helpful