79,916 Kafka Engineer jobs in the United States

Standing up and administer Kafka cluster. Strong experience with Confluent Kafka.
• Expertise in Kafka brokers, zookeepers, Kafka connect, schema registry, KSQL, Rest proxy, Replicator. Both Confluent and Apache Kafka.
• Solid understanding of Kafka architecture, how to save Kafka space.
• Working knowledge of Apache Kafka, Linux/Unix systems.
• Use automation tools like provisioning using Docker, Bamboo, Harness and GitLab.
• Hands on commands usage of Kafka and all components.
• Ability to perform data related benchmarking, performance analysis and tuning.
• Solid knowledge of monitoring tools and fine-tuning s on Splunk, Prometheus, Grafana, Splunk.
• Setting up security on Kafka (Setting different security mechanism in Kafka).
• Manage Kafka users and topics.
• Monitor, prevent and troubleshoot security related issues.
• Good knowledge on CI/CD pipelines
• Automation and scripting using Shell, python, Ansible, Salt stack

• The successful candidate will be responsible for developing and managing infrastructure as code (IaC), software development, continuous integration, system administration, and Linux.
• The candidate will be working with Confluent Kafka, Confluent cloud, Schema Registry, KStreams, and technologies like Terraform and Kubernetes to develop and manage infrastructure-related code on AWS platform.

Responsibilities
• Support systems engineering lifecycle activities for Kafka platform, including requirements gathering, design, testing, implementation, operations, and documentation.
• Automating platform management processes through Ansible, Python or other scripting tools/languages .
• Troubleshooting incidents impacting the Kafka platform.
• Collaborate with cross-functional teams to understand data requirements and design scalable solutions that meet business needs.
• Develop documentation materials.
• Participate in on-call rotations to address critical issues and ensure the reliability of data engineering systems.
• Monitor, troubleshoot, and optimize the performance and reliability of Kafka in AWS environments.

Experience
• Ability to troubleshoot and diagnose complex issues (e.g. including internal and external SaaS/PaaS, troubleshooting network flows).
• Able to demonstrate experience supporting technical users and conduct requirements analysis
• Can work independently with minimal guidance & oversight.
• Experience with IT Service Management and familiarity with Incident & Problem management
• Highly skilled in identifying performance bottlenecks, identifying anomalous system behavior, and resolving root cause of service issues.
• Demonstrated ability to effectively work across teams and functions to influence design, operations, and deployment of highly available software.
• Knowledge of standard methodologies related to security, performance, and disaster recovery
• Advanced understanding of agile methodologies such as CI/CD, Application Resiliency, and Security.

Required Technical Expertise
• Develop and maintain a deep understanding of Kafka and its various components.
• Strong Knowledge in Kafka Connect, KSQL and KStreams.
• Implementation experience in designing and building secure Kafka/streaming/messaging platform at enterprise scale and integration with other data system in hybrid multi-cloud environment.
• Experience in working with Confluent Kafka, Confluent Cloud, Schema Registry, and KStreams Infrastructure as code (IaC) using tools like Terraform.
• Strong operational background running Kafka clusters at scale.
• Knowledge of both physical/onprem systems and public cloud infrastructure.
• Strong understanding of Kafka broker, connect, and topic tuning and architectures.
• Strong understanding of Linux fundamentals as related to Kafka performance.
• Background in both Systems and Software Engineering.
• Strong understanding and working knowledge, experience of containers and Kubernetes cluster.
• Proven experience as a DevOps Engineer with a focus on AWS.
• Strong proficiency in AWS services such as EC2, IAM, S3, RDS, Lambda , EKS and VPC. Working knowledge of networking - VPCs, Transit Gateways, firewalls, load balancers, etc.
• Experience in monitoring and visualizing tools like Prometheus, Grafana, Kibana.
• Competent developing new solutions in one or more of high-level language Java, Python.
• Competent with configuration management in code/IaC including Ansible and Terraform
• Hands on experience delivering complex software in an enterprise environment.
• 3+ years of Python and Shell Scripting.
• 3+ years of AWS DevOps experience.
• Proficiency in distributed Linux environments.

Preferred Technical Experience
• Certification in Confluent Kafka and/or Kubernetes is a plus

• Good Knowledge In Kafka
• Good Communication Skill
• Design, develop, and manage Kafka-based data pipelines
• Work with other big data technologies such as Hadoop, Spark, and Storm
• Monitor and optimize Kafka clusters
• Troubleshoot Kafka related issues
• Handle customer queries and support

• 8+ years in platform engineering with 3+ years of hands-on experience with Apache Kafka.
• Expertise with Confluent Platform (Brokers, Schema Registry, Control Center, ksqlDB).
• Experience deploying and managing Kafka on AWS (including MSK or self-managed EC2-based setups).
• Solid understanding of Kubernetes, especially EKS, for microservices integration.
• Familiarity with monitoring and alerting stacks: Prometheus, Grafana, ELK, or similar.
• Management and support of Kafka Cloud and on-premise platforms.
• Capacity management.
• Good understanding of SRE principles and methodologies, and experience learning, adapting to, and automating new developments in the Kafka ecosystem.
• Experience troubleshooting integration platform issues (e.g., connectivity, schema management, producer/consumer, etc.) for Kafka, API gateway, etc.
• In-depth understanding of the Kafka producer and consumer client functionality.
• Experience troubleshooting custom Kafka client applications written in Java, .NET, Python, and Spring Boot.
• Hands-on experience with Kafka Connect, Kafka Streams, and Kafka Schema Registry components, understanding its underlying functionality and implementation.
• Preferably, at least 3 years' experience working in AWS, specifically EKS, EC2, IAM, Route53 and Terraform.
• Proficiency in deploying, scaling, and managing Kubernetes clusters, with a strong understanding of security best practices.
• Familiarity with Docker and Helm.
• Kafka, Kubernetes, Docker, or any Cloud certification.

• Manage and enhance existing Apache Kafka and Confluent Platform on AWS.
• Review existing implementations and recommend improvements.
• Collaborate with internal teams and respective stakeholders to understand user requirements and implement technical solutions.
• Collaborate with engineering and product teams to integrate new use cases and define scalable streaming patterns.
• Implement and maintain Kafka producers/consumers, Connectors, and Kafka Streams applications.
• Enforce governance around topic design, schema evolution, partitioning, and data retention.
• Monitor, troubleshoot, and tune Kafka clusters using Confluent Control Center, Prometheus, and Grafana.
• Use Kubernetes and Terraform to automate Kafka infrastructure deployment and scaling.
• Ensure high availability, security, and disaster recovery including participation in all DR exercises
• Responsibility and ownership for lifecycle management which includes upgrades, maintenance, restart, and migration projects.
• Analyze all platforms and ensure the environments are right-sized (i.e., capacity management), along with managing configurations, monitors, and alerts.
• Create guidelines, procedures, standards, conventions, and best practices for Kafka usage and administration.
• Develop, maintain, and troubleshoot Terraform IaC modules.
• Automate repetitive tasks using Terraform scripts.
• Guide and mentor team members on Terraform implementations.
• Partner with team members deploying releases to production, and support solution teams
• Coordinate with the Compute, Database, and other infrastructure teams to support weekend patching activities.
• Coordinate vendor support.
• Broad support with troubleshooting and resolving issues, and to be an escalation point for on-premise and cloud integration platform incidents.
• Remediate security vulnerabilities reported by VMAST across application runtimes, integration services, and messaging platforms.
• Support production incidents and outages for faster service restoration and provide required support to ensure application connectivity to API services.
• Work on service requests submitted by application teams requesting application integration team services to integration products (Kafka and any other messaging platforms).
• Create defects when services like tenant onboarding, resource provisioning, and other customer products are not functioning as per the documentation or working as intended.
• Analyze SN INCs and REQs to identify trends and opportunities for improvement.
• Consider implementing automation for manual and repetitive operations support functions.
• Troubleshoot operational issues related to the AWS infrastructure.

• 8+ years in platform engineering with 3+ years of hands-on experience with Apache Kafka.
• Expertise with Confluent Platform (Brokers, Schema Registry, Control Center, ksqlDB).
• Experience deploying and managing Kafka on AWS (including MSK or self-managed EC2-based setups).
• Solid understanding of Kubernetes, especially EKS, for microservices integration.
• Familiarity with monitoring and alerting stacks: Prometheus, Grafana, ELK, or similar.
• Management and support of Kafka Cloud and on-premise platforms.
• Capacity management.
• Good understanding of SRE principles and methodologies, and experience learning, adapting to, and automating new developments in the Kafka ecosystem.
• Experience troubleshooting integration platform issues (e.g., connectivity, schema management, producer/consumer, etc.) for Kafka, API gateway, etc.
• In-depth understanding of the Kafka producer and consumer client functionality.
• Experience troubleshooting custom Kafka client applications written in Java, .NET, Python, and Spring Boot.
• Hands-on experience with Kafka Connect, Kafka Streams, and Kafka Schema Registry components, understanding its underlying functionality and implementation.
• Preferably, at least 3 years' experience working in AWS, specifically EKS, EC2, IAM, Route53 and Terraform.
• Proficiency in deploying, scaling, and managing Kubernetes clusters, with a strong understanding of security best practices.
• Familiarity with Docker and Helm.
• Kafka, Kubernetes, Docker, or any Cloud certification.

• Manage and enhance existing Apache Kafka and Confluent Platform on AWS.
• Review existing implementations and recommend improvements.
• Collaborate with internal teams and respective stakeholders to understand user requirements and implement technical solutions.
• Collaborate with engineering and product teams to integrate new use cases and define scalable streaming patterns.
• Implement and maintain Kafka producers/consumers, Connectors, and Kafka Streams applications.
• Enforce governance around topic design, schema evolution, partitioning, and data retention.
• Monitor, troubleshoot, and tune Kafka clusters using Confluent Control Center, Prometheus, and Grafana.
• Use Kubernetes and Terraform to automate Kafka infrastructure deployment and scaling.
• Ensure high availability, security, and disaster recovery including participation in all DR exercises
• Responsibility and ownership for lifecycle management which includes upgrades, maintenance, restart, and migration projects.
• nalyze all platforms and ensure the environments are right-sized (i.e., capacity management), along with managing configurations, monitors, and alerts.
• Create guidelines, procedures, standards, conventions, and best practices for Kafka usage and administration.
• Develop, maintain, and troubleshoot Terraform IaC modules.
• utomate repetitive tasks using Terraform scripts.
• Guide and mentor team members on Terraform implementations.
• Partner with team members deploying releases to production, and support solution teams
• Coordinate with the Compute, Database, and other infrastructure teams to support weekend patching activities.
• Coordinate vendor support.
• Broad support with troubleshooting and resolving issues, and to be an escalation point for on-premise and cloud integration platform incidents.
• Remediate security vulnerabilities reported by VMAST across application runtimes, integration services, and messaging platforms.
• Support production incidents and outages for faster service restoration and provide required support to ensure application connectivity to API services.
• Work on service requests submitted by application teams requesting application integration team services to integration products (Kafka and any other messaging platforms).
• Create defects when services like tenant onboarding, resource provisioning, and other customer products are not functioning as per the documentation or working as intended.
• nalyze SN INCs and REQs to identify trends and opportunities for improvement.
• Consider implementing automation for manual and repetitive operations support functions.
• Troubleshoot operational issues related to the AWS infrastructure.

The position is described below. If you want to apply, click the Apply Now button at the top or bottom of this page. After you click Apply Now and complete your application, you'll be invited to create a profile, which will let you see your application status and any communications. If you already have a profile with us, you can log in to check status.

Need Help? (

If you have a disability and need assistance with the application, you can request a reasonable accommodation. Send an email to Accessibility ( ?subject=Accommodation%20request)

(accommodation requests only; other inquiries won't receive a response).

Regular or Temporary:

Regular

Language Fluency: English (Required)

Work Shift:

1st shift (United States of America)

Please review the following job description:

** This position is on-site 4 days per week **

The primary purpose of this role is to provide consultation and technical direction on translating business requirements and functional specifications into Middleware Messaging designs-particularly Kafka. This includes facilitating the implementation and maintenance of complex business and enterprise software solutions to ensure successful deployment of released applications. This role serves as a technical expert for project teams throughout the implementation and maintenance of enterprise Middleware solutions. In addition, this role personally develops and delivers automation, operational support, stable platforms, and integrated enterprise software solutions within various computing environments including on premise and cloud datacenters.

ESSENTIAL DUTIES AND RESPONSIBILITIES

Following is a summary of the essential functions for this job. Other duties may be performed, both major and minor, which are not mentioned below. Specific activities may change from time to time.

1. Resolves complex problems throughout the entire lifecycle of Kafka based platforms, from build to retirement. This includes monitoring, security, automation, troubleshooting, and deployment capabilities.

2. Provides consultation on business requirements and functional specifications in logical program designs, code modules, stable application systems, and software solutions designed around Kafka; facilitates the transition to high level design and supports the project lifecycle with input from executive leadership where needed.

3. Contributes to and leverages the technical direction for the development, configuration, or modification of integrated business and enterprise Kafka solutions within various computing environments by providing insight and guidance for the design and coding of Kafka infrastructure.

4. Supports systems integration testing and performance testing for large, complex, cross-functional application initiatives by providing insight to testing teams and ensuring the messaging platform is properly tuned and configured.

5. Mentors and advises others in all software development lifecycle phases by applying and sharing an in-depth understanding of company and industry methodologies, policies, standards, and controls.

6. Communicates changes in software architecture and coaches others to apply this understanding to software solutions; resolves escalated issues.

7. Leads efforts to improve engineering, test, and operational excellence best practices.

8. Solves complex cross-functional architecture/design and business problems; solutions are extensible; works to simplify, optimize, remove bottlenecks, etc.

9. Mentors and advises others, sharing an in-depth understanding of company and industry methodologies, policies, standards, and controls.

QUALIFICATIONS

Required Qualifications:

The requirements listed below are representative of the knowledge, skill and/or ability required. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.

1. Bachelor's degree in computer science, CIS, or related field

2. Five to seven years of experience in software development or a related field

3. Five to seven years of experience in database technologies

4. Five to seven years of experience working on project(s) involving the implementation of solutions applying development life cycles (SDLC)

Preferred Qualifications:

1. Master's degree in computer science, CIS, or related field

2. Five to seven years of IT experience developing and implementing business systems within an organization

3. Five to seven years of experience working with defect or incident tracking software

4. Five to seven years of experience writing technical documentation in a software development environment

5. Five to seven years of experience working with an IT Infrastructure Library (ITIL) framework

6. Three to five years of experience working with Kafka platforms

7. One to three years' experience working with Kubernetes based deployments

8. Experience with other messaging platforms like IBM MQ is a plus

9. Specific experience with OpenShift and/or EKS based Kubernetes is a plus

For this opportunity, Truist will not sponsor an applicant for work visa status or employment authorization, nor will we offer any immigration-related support for this position (including, but not limited to H-1B, F-1 OPT, F-1 STEM OPT, F-1 CPT, J-1, TN-1 or TN-2, E-3, O-1, or future sponsorship for U.S. lawful permanent residence status.)

General Description of Available Benefits for Eligible Employees of Truist Financial Corporation: All regular teammates (not temporary or contingent workers) working 20 hours or more per week are eligible for benefits, though eligibility for specific benefits may be determined by the division of Truist offering the position. Truist offers medical, dental, vision, life insurance, disability, accidental death and dismemberment, tax-preferred savings accounts, and a 401k plan to teammates. Teammates also receive no less than 10 days of vacation (prorated based on date of hire and by full-time or part-time status) during their first year of employment, along with 10 sick days (also prorated), and paid holidays. For more details on Truist's generous benefit plans, please visit our Benefits site (

. Depending on the position and division, this job may also be eligible for Truist's defined benefit pension plan, restricted stock units, and/or a deferred compensation plan. As you advance through the hiring process, you will also learn more about the specific benefits available for any non-temporary position for which you apply, based on full-time or part-time status, position, and division of work.

Truist is an Equal Opportunity Employer that does not discriminate on the basis of race, gender, color, religion, citizenship or national origin, age, sexual orientation, gender identity, disability, veteran status, or other classification protected by law. Truist is a Drug Free Workplace.

EEO is the Law (

Pay Transparency Nondiscrimination Provision (

E-Verify (