14,293 Apache Spark jobs in the United States

• 5+ years of experience working with Apache Spark (including Structured Streaming).
• Strong proficiency in Scala, with ability to write clean, performant code.
• 3+ years of experience with Apache Kafka for real-time streaming and messaging.
• Solid understanding of data modeling, ETL, and data pipeline orchestration.
• Familiarity with Hadoop ecosystem, Hive, Parquet, or similar.
• Experience with CI/CD, Docker, and Kubernetes is a plus.
• Exposure to cloud platforms like AWS (EMR, MSK), Azure, or GCP is preferred.
• Excellent problem-solving, communication, and analytical skills.

The Candiadte will be responsible for leading technical teams and projects related to apache spark, scala, and python. The role involves overseeing the design, development, and implementation of scalable and efficient solutions using these technologies.

Key Responsibilities
1. Design and implementation of solutions using apache spark, scala, and python
2. Provide technical expertise and guidance to team members in resolving complex technical issues
3. Collaborate with stakeholders to gather requirements and define project scope
4. Ensure adherence to best practices in coding, testing, and deployment processes
5. Conduct code reviews and performance optimization activities
6. Troubleshoot and debug technical issues to ensure seamless project delivery
7. Stay updated with the latest trends and advancements in apache spark, scala, and python technologies
8. Mentor team members and facilitate knowledge sharing within the team

Skill Requirements
1. Strong proficiency in apache spark, scala, and python
2. Experience in leading technical teams and projects
3. Excellent problem-solving skills and ability to think critically
4. Solid understanding of software development life cycle
5. Good communication and interpersonal skills
6. Ability to work effectively in a collaborative team environment
7. Strong organizational and leadership abilities
8. Experience in performance tuning and optimization techniques
9. Knowledge of big data concepts and tools is a plus

Certifications : Relevant certifications in apache spark, scala, or python are a plus

Elite Workforce is led by professionals with extensive industry
experience in IT and Consulting. Elite Workforce wishes to engage with
clients in the global market place as a trusted advisor and partner.

Our motto is to become an international professional services entity of
highest standards and help our clients and communities we work with to
grow and evolve using best practices and solutions thus building
sustainable, cleaner, safer and more prosperous world.

Position: Apache Spark Java Developer
Location: Brooklyn Park, MN
Duration: Fulltime / Permanent
Only US Citizen, Green Card, GC EAD

Must have experience in the below Skill set:

• 4 + Years of Experiance in Developing Apache spark Integrations & Framework
• Experiance in developing Data Streaming applications using Apache Spark & JAVA
• Experiance designing Apache based application using Java platform
• 6+ years of software development experience using Java/J2EE (Spring).
• Experience in web service/ REST API implementations using Spring/SpringBoot.
• Hands on experience in CICD automation using Git, Jenkins, Chef & Artifactory Repository.
• Experience in application server (Websphere, Tomcat).
• Experience in Build tools (Maven, Gradle, Ant).
• Experience with SQL (Oracle, DB2) and NoSQL (MangoDB, Postgres) databases.
• Experience in Agile Methodology, SCRUM, KANBAN.
• Proactively identifies opportunities for improvement and recommends technical solutions.
  

All your information will be kept confidential according to EEO guidelines.

Business Area:

Seniority Level:

Job Description:

At Cloudera, we empower people to transform complex data into clear and actionable insights. With as much data under management as the hyperscalers, we're the preferred data partner for the top companies in almost every industry. Powered by the relentless innovation of the open source community, Cloudera advances digital transformation for the world's largest enterprises.

Cloudera is seeking an experienced Principal Engineer with strong distributed systems expertise to work on the Cloudera distribution of Apache Spark. We are looking for senior engineers with experience in large-scale, distributed systems and data processing to help build our enterprise-grade system, designed for customers running Spark on thousands of nodes and processing petabytes of data.

We are looking for a passionate individual that is ready to be a team lead for a team that is already supporting production systems at many of the biggest companies - and is looking to expand and take on even more projects to drive the next gen Data Engineering experience. You will be working with a distributed team, spread across the United States and Hungary, including multiple committers on Apache Spark.

As a Principal Software Engineer, you will:

• Architect, design, and implement resilient and scalable solutions for distributed data processing at massive scale, with a focus on fault tolerance, performance optimization, query planning, and resource management

• Take ownership of critical distributed systems components, solving complex challenges related to network communication, concurrency, data consistency, and system reliability across clusters of thousands of nodes

• Develop advanced monitoring, debugging, and performance analysis tools for large-scale distributed systems.

• Act as a tech lead for Cloudera's Spark team

• Work with and contribute to the latest open source technologies, including Apache Spark, Iceberg, and Parquet

• Develop new features in Scala/Java/Python on a modern platforms

• Gain a solid understanding and deep technical knowledge of components across the Cloudera Data Engineering Experience stack, but focusing on Iceberg and Spark

• Debug system level deployment issues, root cause analysis, perform system test analysis and resolve failures

• Work on improving internal infrastructure

• Collaborate with other team members and stakeholders

We are excited about you if you have:

• Bachelor's degree in Computer Science or equivalent, and 10+ years of experience; OR Master's degree and 6+ years of experience; OR PhD and 4+ years of experience

• Experience with systems design and development specifically for large-scale distributed environments

• Experience leading and delivering complex product enhancements.

• We use Java/Scala/Python/GoLang in projects, you should have a strong understanding of at least one of the following languages: Java, Scala, GoLang, Rust, C++, Python. And interested to learn the languages we're using.

• Passionate about programming, clean coding habits, attention to detail, and focus on quality

• Strong oral and written communication skills.

• Strong ability to research and solve problems independently without constant supervision

• (Most importantly) Open-minded, desire to learn new things and build great products.

You may also have:

• Experience with large-scale, distributed systems design and development with an understanding of scaling, performance, and scheduling

• In-depth understanding of distributed query processing and planning

• Experience with using/developing Apache Spark or other related technologies.

• In-depth understanding of distributed systems concepts like consensus algorithms, distributed transactions, and fault tolerance

• Experience working with query automated query optimization

• Solid experience with at least one cloud service (AWS, Azure, GCP, OpenShift)

• Contributors to open-source projects.

This role is not eligible for immigrationsponsorship

What you can expect from us:

• Generous PTO Policy

• Support work life balance with Unplugged Days

• Flexible WFH Policy

• Mental & Physical Wellness programs

• Phone and Internet Reimbursement program

• Access to Continued Career Development

• Comprehensive Benefits and Competitive Packages

• Paid Volunteer Time

• Employee Resource Groups

EEO/VEVRAA

# LI-SZ1
#LI-Remote

• BS or MS in Computer Science
• 5+ years working as a professional Software Engineer with a focus on platform engineering
• Professional experience at a software startup strongly preferred
• Great communication skills so you can work directly with our Data Scientists, DevOps Engineers, and Product Managers
• Solid understanding of concurrency and distributed systems
• Professional experience building software with Scala and/or Java
• You can design and prototype scalable algorithms
• You can create your own efficient data structures and algorithms when the ones found in open source libraries are lacking
• Deep understanding of algorithm complexity and performance implications
• Experience with Apache Spark, HDFS, and/or general MapReduce technologies
• Professional experience writing unit tests and automated integration tests
• Experience working with code that runs in a JVM
• Familiarity with storage and caching solutions and the tradeoffs among them (Memcached, Redis, Postgres, etc.)
• You can build microservices with REST APIs for real-time query engines and web crawlers
• You know how to set up ETL pipelines for data streams
• Familiarity with Machine Learning techniques
• Must be legally authorized to work in the United States without the need for employer sponsorship, now or at any time in the future

• Competitive salary
• Company Equity
• Exceptional benefits package
• Flexible Vacation & Paid Time Off
• Employer-matched 401(k) plan
• A fun environment where work-life balance is valued

• Architect, design, and implement resilient and scalable solutions for distributed data processing at massive scale, with a focus on fault tolerance, performance optimization, query planning, and resource management
• Take ownership of critical distributed systems components, solving complex challenges related to network communication, concurrency, data consistency, and system reliability across clusters of thousands of nodes
• Develop advanced monitoring, debugging, and performance analysis tools for large-scale distributed systems.
• Act as a tech lead for Cloudera's Spark team
• Work with and contribute to the latest open source technologies, including Apache Spark, Iceberg, and Parquet
• Develop new features in Scala/Java/Python on a modern platforms
• Gain a solid understanding and deep technical knowledge of components across the Cloudera Data Engineering Experience stack, but focusing on Iceberg and Spark
• Debug system level deployment issues, root cause analysis, perform system test analysis and resolve failures
• Work on improving internal infrastructure
• Collaborate with other team members and stakeholders

• Bachelor's degree in Computer Science or equivalent, and 10+ years of experience; OR Master's degree and 6+ years of experience; OR PhD and 4+ years of experience
• Experience with systems design and development specifically for large-scale distributed environments
• Experience leading and delivering complex product enhancements.
• We use Java/Scala/Python/GoLang in projects, you should have a strong understanding of at least one of the following languages: Java, Scala, GoLang, Rust, C++, Python. And interested to learn the languages we're using.
• Passionate about programming, clean coding habits, attention to detail, and focus on quality
• Strong oral and written communication skills.
• Strong ability to research and solve problems independently without constant supervision
• (Most importantly) Open-minded, desire to learn new things and build great products.

• Experience with large-scale, distributed systems design and development with an understanding of scaling, performance, and scheduling
• In-depth understanding of distributed query processing and planning
• Experience with using/developing Apache Spark or other related technologies.
• In-depth understanding of distributed systems concepts like consensus algorithms, distributed transactions, and fault tolerance
• Experience working with query automated query optimization
• Solid experience with at least one cloud service (AWS, Azure, GCP, OpenShift)
• Contributors to open-source projects.

• Generous PTO Policy
• Support work life balance with Unplugged Days
• Flexible WFH Policy
• Mental & Physical Wellness programs
• Phone and Internet Reimbursement program
• Access to Continued Career Development
• Comprehensive Benefits and Competitive Packages
• Paid Volunteer Time
• Employee Resource Groups

Spark - Must have
Scala - Must Have
Hive & SQL - Must Have
Hadoop - Must Have
Communication - Must Have
Banking /Capital Markets Domain - Good to have

Job Description :

Scala/Spark
• Good Big Data resource with the below Skillset:
Spark
Scala
Hive/HDFS/HQL
• Linux Based Hadoop Ecosystem (HDFS, Impala, Hive, HBase, etc.)
• Experience in Big data technologies , real time data processing platform(Spark Streaming) experience would be an advantage.
• Consistently demonstrates clear and concise written and verbal communication
• history of delivering against agreed objectives
• bility to multi-task and work under pressure
• Demonstrated problem solving and decision-making skills
• Excellent analytical and process-based skills, i.e. process flow diagrams, business modelling, and functional design.

Join to apply for the Senior Software Engineer (Apache Spark) role at Quantifind

2 days ago Be among the first 25 applicants

Join to apply for the Senior Software Engineer (Apache Spark) role at Quantifind

Who You Are

Quantifind is seeking a Senior Platform Engineer to work on Quantifinds Apache Spark pipeline that generates risk signal results from diverse data sources such as Sanctions, PEP records, Negative News, Court & Arrest records, and much more. You will help define and deliver data services and machine learning infrastructure. The Platform Team owns the backend that powers the SaaS products we build for our customers. We process terabytes of data each day on our own dedicated Apache Spark clusters.

Who You Are

Quantifind is seeking a Senior Platform Engineer to work on Quantifinds Apache Spark pipeline that generates risk signal results from diverse data sources such as Sanctions, PEP records, Negative News, Court & Arrest records, and much more. You will help define and deliver data services and machine learning infrastructure. The Platform Team owns the backend that powers the SaaS products we build for our customers. We process terabytes of data each day on our own dedicated Apache Spark clusters.

You care deeply about your work and its impact on the work your team does. You hold yourself to a high standard when it comes to writing clean, performant, and maintainable code that powers well-designed microservices and enhances Quantifinds distributed data platform. You work well with cross-functional teams that include Data Scientists, DevOps Engineers, Front-end Engineers and Product Managers. You are constantly being proactive in finding new ways to improve the design, performance, and testing of the software that you build. You can effectively juggle immediate competing priorities while taking the long view to figure out how to build a bigger and better company and culture. You value the security of a company with a proven business model and enjoy the opportunities a startup offers.

Youre looking to explore a client-facing role at the intersection of software and consulting. You are obsessed with the idea of delighting clients at some of the worlds leading Financial Institutions helping them identify financial crimes.

You have always been a strong critical thinker and know how to problem solve.

You are an excellent communicator with the ability to distill complex concepts into clear and effective verbal and written presentations. You are comfortable working closely with Sales, Product and Technology teams to promote a Customer Centric culture.

You have an entrepreneurial spirit with a deep sense of personal accountability for work product quality and always take the initiative to propel work products to new heights. You maintain a positive attitude even in a deadline-driven environment, bring out the best in others, and enjoy contributing to the success of the team.

Who We Are

Quantifind is the Risk Intelligence company. We help some of the worlds biggest banks identify money laundering and fraud. Quantifind also works with government agencies, using the same platform, to uncover criminal networks and support national security. Unlike other players in this space, Quantifind delivers results through an AI-driven software-as-a-service (SaaS) solution with consumer-grade user experiences.

Quantifinds Graphyte platform uncovers signals of risk across disparate and unstructured text sources using artificial intelligence and natural language processing techniques. In financial crimes risk management, Quantifinds solution uniquely combines internal financial institution data with public domain data to assess risk in the context of Know Your Customer (KYC), Customer Due Diligence (CDD), Fraud Risk Management, and Anti-Money Laundering (AML) processes. Today these compliance processes are burdened by ever-increasing regulatory responsibilities and an expectation of frictionless transactions. Legacy technologies demand increasingly more human resources as the operations expand; Quantifinds solution offers a way to cut through the inefficiency and enhance effectiveness simultaneously. Demand for this solution is fueling our rapid growth.

To help you succeed, we provide a supportive environment that fosters collaboration between teams and team members, where learning and professional growth are considered a key part of your success, and of ours. We offer a flexible work environment with a family friendly work-life balance.

What A Great Candidate Looks Like

• BS or MS in Computer Science
• 5+ years working as a professional Software Engineer with a focus on platform engineering
• Professional experience at a software startup strongly preferred
• Great communication skills so you can work directly with our Data Scientists, DevOps Engineers, and Product Managers
• Solid understanding of concurrency and distributed systems
• Professional experience building software with Scala and/or Java
• You can design and prototype scalable algorithms
• You can create your own efficient data structures and algorithms when the ones found in open source libraries are lacking
• Deep understanding of algorithm complexity and performance implications
• Experience with Apache Spark, HDFS, and/or general MapReduce technologies
• Professional experience writing unit tests and automated integration tests
• Experience working with code that runs in a JVM
• Familiarity with storage and caching solutions and the tradeoffs among them (Memcached, Redis, Postgres, etc.)
• You can build microservices with REST APIs for real-time query engines and web crawlers
• You know how to set up ETL pipelines for data streams
• Familiarity with Machine Learning techniques
• Must be legally authorized to work in the United States without the need for employer sponsorship, now or at any time in the future
  
  

• Competitive salary
• Company Equity
• Exceptional benefits package
• Flexible Vacation & Paid Time Off
• Employer-matched 401(k) plan
• A fun environment where work-life balance is valued
  
  

• Seniority level Mid-Senior level

• Employment type Full-time

• Job function Engineering and Information Technology
• Industries Software Development

Referrals increase your chances of interviewing at Quantifind by 2x

Get notified about new Senior Software Engineer jobs in Washington, DC .

Arlington, VA $65,000.00- 304,000.00 3 days ago

Sterling, VA 130,355.00- 221,603.00 2 weeks ago

Washington, DC 130,000.00- 160,000.00 2 months ago

Fort Meade, MD 185,000.00- 207,000.00 1 week ago

District of Columbia, United States 37.00- 53.00 1 week ago

Gaithersburg, MD 104,650.00- 189,175.00 3 weeks ago

Chantilly, VA 112,320.00- 205,058.00 2 months ago

Washington, DC 135,000.00- 216,000.00 6 hours ago