4,521 Big Data Technologies jobs in the United States

Job Description:

• The Data Science Engineering Analyst will play a pivotal role in analysing complex datasets, designing and implementing predictive models, and deriving actionable insights to drive business decisions. This position requires a combination of data engineering skills and data science expertise to ensure robust data pipelines and effective analytical solutions.

Responsibilities:

• Apply advanced statistical methods to large complex data sets to understand impact of abuse to the client ecosystem. Contribute strategy and development of new workflows against additional known vectors of abuse.

• Learn complex and technical concepts and systems and deliver meaningful results using them. Communicate highly technical results and methods clearly.

• Maintain, foster and promote quality by providing regular feedback metrics to the Global team. Manage technological solutions for streamlining quality assurance and produce scalable training solutions to emergent workflows.

• Perform deceptive/manipulated content and generative AI investigations using various data sources, identify product vulnerabilities and drive anti-abuse experiments to prevent abuse. Work with engineers and interact cross-functionally with stakeholders to improve workflows via process improvements, automation and anti-abuse system creation.

• Develop high quality training materials, create consensus and maintain communication for client with business stakeholders.

Experience:

• 3 years of experience in data management, metrics analysis, experiment design and automation.

• Experience with classification systems, ranking systems, or similar.

• Experience collecting, managing, and synthesizing large data sets and information from disparate sources, statistical modelling, data mining and data analysis.

US Tech Solutions is a global staff augmentation firm providing a wide range of talent on-demand and total workforce solutions. To know more about US Tech Solutions, please visit ( .

US Tech Solutions is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, colour, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.

• Design, develop, and maintain scalable backend services and APIs using Java, Python, Scala, Node.js, and GraphQL .

• Architect and implement big data solutions leveraging Hadoop, Hive, Spark (Scala), Presto/Trino, and Data Lake concepts .

• Develop and optimize data processing and streaming pipelines using Storm, Spark Streaming, Airflow, Luigi, and Automic .

• Collaborate with data scientists and ML engineers to integrate solutions with Vertex AI and other cloud-based AI/ML platforms.

• Implement containerized solutions with Docker & Kubernetes and manage deployment on cloud environments (AWS/GCP/Azure) .

• Ensure system reliability, scalability, and performance through monitoring, testing, and optimization.

• Partner with cross-functional teams including product managers, data engineers, and DevOps to deliver high-quality solutions.

• Troubleshoot production issues, optimize system performance, and ensure data consistency and security.

• Bachelor's/Master's degree in Computer Science, Engineering, or related field (or equivalent practical experience).

• 7–10 years of experience in backend and data engineering.

• Proficiency in Java, Python, Scala, and Node.js for backend and API development.

• Strong experience with GraphQL (GQL) schema design and implementation.

• Expertise in Hadoop ecosystem (HDFS, Hive, Spark with Scala) .

• Hands-on experience with Presto/Trino and Data Lake architectures .

• Practical knowledge of stream-processing frameworks such as Storm and Spark Streaming .

• Experience with orchestration & workflow tools : Airflow, Luigi, Automic.

• Proficiency with Kubernetes and containerized deployments.

• Strong understanding of cloud services (GCP/AWS/Azure) and Vertex AI .

• Excellent problem-solving, debugging, and optimization skills.

• Experience with machine learning integration and MLOps workflows.

• Knowledge of NoSQL databases (MongoDB, Cassandra, etc.).

• Exposure to observability/monitoring tools (Grafana, Prometheus, ELK, etc.).

• Familiarity with Agile/Scrum methodologies .

Manager, Data Engineering

Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative,inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who solve real problems and meet real customer needs. We are seeking Data Engineers who are passionate about marrying data with emerging technologies. As a Capital One Data Engineer, you’ll have the opportunity to be on the forefront of driving a major transformation within Capital One.

What You’ll Do:

• Collaborate with and across Agile teams to design, develop, test, implement, and support technical solutions in full-stack development tools and technologies

• Work with a team of developers with deep experience in machine learning, distributed microservices, and full stack systems

• Utilize programming languages like Java, Scala, Python and Open Source RDBMS and NoSQL databases and Cloud based data warehousing services such as Redshift and Snowflake

• Share your passion for staying on top of tech trends, experimenting with and learning new technologies, participating in internal & external technology communities, and mentoring other members of the engineering community

• Collaborate with digital product managers, and deliver robust cloud-based solutions that drive powerful experiences to help millions of Americans achieve financial empowerment

• Perform unit tests and conduct reviews with other team members to make sure your code is rigorously designed, elegantly coded, and effectively tuned for performance

Basic Qualifications:

• Bachelor’s Degree

• At least 4 years of experience in application development (Internship experience does not apply)

• At least 2 years of experience in big data technologies

• At least 1 year experience with cloud computing (AWS, Microsoft Azure, Google Cloud)

• At least 2 years of people management experience

Preferred Qualifications:

• 7+ years of experience in application development including Python, SQL, Scala, or Java

• 4+ years of experience with a public cloud (AWS, Microsoft Azure, Google Cloud)

• 4+ years experience with Distributed data/computing tools (MapReduce, Hadoop, Hive, EMR, Kafka, Spark, Gurobi, or MySQL)

• 4+ year experience working on real-time data and streaming applications

• 4+ years of experience with NoSQL implementation (Mongo, Cassandra)

• 4+ years of data warehousing experience (Redshift or Snowflake)

• 4+ years of experience with UNIX/Linux including basic commands and shell scripting

• 2+ years of experience with Agile engineering practices

At this time, Capital One will not sponsor a new applicant for employment authorization, or offer any immigration related support for this position (i.e. H1B, F-1 OPT, F-1 STEM OPT, F-1 CPT, J-1, TN, or another type of work authorization).

The minimum and maximum full-time annual salaries for this role are listed below, by location. Please note that this salary information is solely for candidates hired to perform work within one of these locations, and refers to the amount Capital One is willing to pay at the time of this posting. Salaries for part-time roles will be prorated based upon the agreed upon number of hours to be regularly worked.

Candidates hired to work in other locations will be subject to the pay range associated with that location, and the actual annualized salary amount offered to any candidate at the time of hire will be reflected solely in the candidate’s offer letter.

Capital One offers a comprehensive, competitive, and inclusive set of health, financial and other benefits that support your total well-being. Learn more at theCapital One Careers website. Eligibility varies based on full or part-time status, exempt or non-exempt status, and management level.

If you have visited our website in search of information on employment opportunities or to apply for a position, and you require an accommodation, please contact Capital One Recruiting at or via email at . All information you provide will be kept confidential and will be used only to the extent required to provide needed reasonable accommodations.

For technical support or questions about Capital One's recruiting process, please send an email to

Capital One does not provide, endorse nor guarantee and is not liable for third-party products, services, educational tools or other information available through this site.

Capital One Financial is made up of several different entities. Please note that any position posted in Canada is for Capital One Canada, any position posted in the United Kingdom is for Capital One Europe and any position posted in the Philippines is for Capital One Philippines Service Corp. (COPSSC).

Manager, Data Engineering

Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative,inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who solve real problems and meet real customer needs. We are seeking Data Engineers who are passionate about marrying data with emerging technologies. As a Capital One Data Engineer, you’ll have the opportunity to be on the forefront of driving a major transformation within Capital One.

What You’ll Do:

• Collaborate with and across Agile teams to design, develop, test, implement, and support technical solutions in full-stack development tools and technologies

• Work with a team of developers with deep experience in machine learning, distributed microservices, and full stack systems

• Utilize programming languages like Java, Scala, Python and Open Source RDBMS and NoSQL databases and Cloud based data warehousing services such as Redshift and Snowflake

• Share your passion for staying on top of tech trends, experimenting with and learning new technologies, participating in internal & external technology communities, and mentoring other members of the engineering community

• Collaborate with digital product managers, and deliver robust cloud-based solutions that drive powerful experiences to help millions of Americans achieve financial empowerment

• Perform unit tests and conduct reviews with other team members to make sure your code is rigorously designed, elegantly coded, and effectively tuned for performance

Basic Qualifications:

• Bachelor’s Degree

• At least 4 years of experience in application development (Internship experience does not apply)

• At least 2 years of experience in big data technologies

• At least 1 year experience with cloud computing (AWS, Microsoft Azure, Google Cloud)

• At least 2 years of people management experience

Preferred Qualifications:

• 7+ years of experience in application development including Python, SQL, Scala, or Java

• 4+ years of experience with a public cloud (AWS, Microsoft Azure, Google Cloud)

• 4+ years experience with Distributed data/computing tools (MapReduce, Hadoop, Hive, EMR, Kafka, Spark, Gurobi, or MySQL)

• 4+ year experience working on real-time data and streaming applications

• 4+ years of experience with NoSQL implementation (Mongo, Cassandra)

• 4+ years of data warehousing experience (Redshift or Snowflake)

• 4+ years of experience with UNIX/Linux including basic commands and shell scripting

• 2+ years of experience with Agile engineering practices

At this time, Capital One will not sponsor a new applicant for employment authorization, or offer any immigration related support for this position (i.e. H1B, F-1 OPT, F-1 STEM OPT, F-1 CPT, J-1, TN, or another type of work authorization).

The minimum and maximum full-time annual salaries for this role are listed below, by location. Please note that this salary information is solely for candidates hired to perform work within one of these locations, and refers to the amount Capital One is willing to pay at the time of this posting. Salaries for part-time roles will be prorated based upon the agreed upon number of hours to be regularly worked.

Candidates hired to work in other locations will be subject to the pay range associated with that location, and the actual annualized salary amount offered to any candidate at the time of hire will be reflected solely in the candidate’s offer letter.

Capital One offers a comprehensive, competitive, and inclusive set of health, financial and other benefits that support your total well-being. Learn more at theCapital One Careers website. Eligibility varies based on full or part-time status, exempt or non-exempt status, and management level.

If you have visited our website in search of information on employment opportunities or to apply for a position, and you require an accommodation, please contact Capital One Recruiting at or via email at . All information you provide will be kept confidential and will be used only to the extent required to provide needed reasonable accommodations.

For technical support or questions about Capital One's recruiting process, please send an email to

Capital One does not provide, endorse nor guarantee and is not liable for third-party products, services, educational tools or other information available through this site.

Capital One Financial is made up of several different entities. Please note that any position posted in Canada is for Capital One Canada, any position posted in the United Kingdom is for Capital One Europe and any position posted in the Philippines is for Capital One Philippines Service Corp. (COPSSC).

Manager, Data Engineering

Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who solve real problems and meet real customer needs. We are seeking Data Engineers who are passionate about marrying data with emerging technologies. As a Capital One Data Engineer, you'll have the opportunity to be on the forefront of driving a major transformation within Capital One.

What You'll Do:

• Collaborate with and across Agile teams to design, develop, test, implement, and support technical solutions in full-stack development tools and technologies

• Work with a team of developers with deep experience in machine learning, distributed microservices, and full stack systems

• Utilize programming languages like Java, Scala, Python and Open Source RDBMS and NoSQL databases and Cloud based data warehousing services such as Redshift and Snowflake

• Share your passion for staying on top of tech trends, experimenting with and learning new technologies, participating in internal & external technology communities, and mentoring other members of the engineering community

• Collaborate with digital product managers, and deliver robust cloud-based solutions that drive powerful experiences to help millions of Americans achieve financial empowerment

• Perform unit tests and conduct reviews with other team members to make sure your code is rigorously designed, elegantly coded, and effectively tuned for performance

Basic Qualifications:

• Bachelor's Degree

• At least 4 years of experience in application development (Internship experience does not apply)

• At least 2 years of experience in big data technologies

• At least 1 year experience with cloud computing (AWS, Microsoft Azure, Google Cloud)

• At least 2 years of people management experience

Preferred Qualifications:

• 7+ years of experience in application development including Python, SQL, Scala, or Java

• 4+ years of experience with a public cloud (AWS, Microsoft Azure, Google Cloud)

• 4+ years experience with Distributed data/computing tools (MapReduce, Hadoop, Hive, EMR, Kafka, Spark, Gurobi, or MySQL)

• 4+ year experience working on Real Time data and streaming applications

• 4+ years of experience with NoSQL implementation (Mongo, Cassandra)

• 4+ years of data warehousing experience (Redshift or Snowflake)

• 4+ years of experience with UNIX/Linux including basic commands and Shell Scripting

• 2+ years of experience with Agile engineering practices

At this time, Capital One will not sponsor a new applicant for employment authorization, or offer any immigration related support for this position (ie H1B, F-1 OPT, F-1 STEM OPT, F-1 CPT, J-1, TN, or another type of work authorization).

The minimum and maximum Full time annual salaries for this role are listed below, by location. Please note that this salary information is solely for candidates hired to perform work within one of these locations, and refers to the amount Capital One is willing to pay at the time of this posting. Salaries for part-time roles will be prorated based upon the agreed upon number of hours to be regularly worked.

Candidates hired to work in other locations will be subject to the pay range associated with that location, and the actual annualized salary amount offered to any candidate at the time of hire will be reflected solely in the candidate's offer letter.

Capital One offers a comprehensive, competitive, and inclusive set of health, financial and other benefits that support your total well-being. Learn more at the Capital One Careers website . Eligibility varies based on full or part-time status, exempt or non-exempt status, and management level.

If you have visited our website in search of information on employment opportunities or to apply for a position, and you require an accommodation, please contact Capital One Recruiting at or via email at (see below) . All information you provide will be kept confidential and will be used only to the extent required to provide needed reasonable accommodations.

For technical support or questions about Capital One's recruiting process, please send an email to (see below)

Capital One does not provide, endorse nor guarantee and is not liable for third-party products, services, educational tools or other information available through this site.

Capital One Financial is made up of several different entities. Please note that any position posted in Canada is for Capital One Canada, any position posted in the United Kingdom is for Capital One Europe and any position posted in the Philippines is for Capital One Philippines Service Corp. (COPSSC).

Job Title:

Manager, Data Engineering

Overview:

OVERVIEWThe Data and Analytics group within Services is seeking a highly experienced and strategic professional to manage the Mastercard Performance (MyMPA) product. MyMPA is a system designed to provide comprehensive data on Mastercard, Maestro, and Cirrus brands. It includes various data dimensions such as QMR and Processed Data, Country and Regional Totals, and Issuing, Acquiring, and Cross Border data. You will work closely with internal stakeholders, external partners, and subject matter experts to drive the growth of the product aligned with Mastercard's business goals and core values.

Services within Mastercard is responsible for acquiring, engaging, and retaining customers by managing fraud and risk, enhancing cybersecurity, and improving the digital payments experience. We provide value-added services and leverage expertise, data-driven insights, and execution.

Role
• Responsible for MyMPA application and build the team to achieve objectives defined around MyMPA delivery
• Develop high quality, secure and scalable data pipelines using spark, Scala/ python on Hadoop or object storage.
• Leverage new technologies and approaches to innovate with increasingly large data sets.
• Work closely with business stakeholders and embedded engineering teams within business teams in a collaborative manner to help them build scalable products in quick time.
• Contribute ideas to help ensure that required standards and processes are in place and actively look for opportunities to enhance standards and improve process efficiency.

All About You
• 10+ years of experience in Data Warehouse related projects or data oriented projects
• Experience of building data pipelines through Spark with Scala/Python/Java on Hadoop or Object storage
• Experience of building Nifi pipelines
• Understanding of Mastercard CORE datasets
• Experience of working with Databases like Oracle, Netezza would be added advantage
• Strong analytical skills required for debugging production issues, providing root cause and implementing mitigation plan
• Strong communication skills - both verbal and written - and strong relationship, collaboration skills and organisational skills
• Experience managing teams located across multiple locations
• Ability to be high-energy, detail-oriented, proactive and able to function under pressure in an independent environment along with a high degree of initiative and self-motivation to drive results

Our Purpose

Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we're helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.

Title and Summary

Director, Data Engineering

Overview:
The Mastercard Services Technology team is looking for Director of Data Engineering, to lead our mission of unlocking the full potential of Mastercard's data assets. In this high-impact role, you will drive innovation, reduce friction in the management of big data assets, and ensure seamless data accessibility across both public cloud (AWS, Azure) and on-premise environments. You will champion standards, best practices, and governance principles to enable modern, scalable, and secure data platforms.
As the Director, you will collaborate with cross-geography teams of data engineers and platform engineers, aligning efforts across multiple initiatives into a unified execution framework. Your expertise in technical program management will be critical in harmonizing delivery approaches, ensuring accountability, and enabling our teams to consistently deliver high-value data products and platforms.

Role:
• Lead the design and delivery of end-to-end cloud-native solutions with a focus on Lakehouse architectures (S3, Azure Data Lake, Databricks, EMR, Athena, etc.).
• Provide both technology and people leadership, mentoring engineers and fostering a culture of excellence, ownership, and continuous improvement.
• Drive CI/CD pipeline development, infrastructure-as-code, and automated environment provisioning across AWS and Azure.
• Set and enforce security best practices (IAM, encryption, network security) and ensure compliance with frameworks such as GDPR and HIPAA.
• Establish and enforce enterprise-grade data governance frameworks, including data cataloging, lineage, classification, and access control, leveraging tools like Lake Formation or Azure Purview.
• Design and lead implementation of robust data engineering pipelines, including ETL/ELT, data lakes, and data warehouses.
• Implement Lakehouse architectures using the medallion model (Bronze, Silver, Gold) with deep integration into AWS services (Glue, Lake Formation) and analytics engines (Databricks, EMR, Athena).
• Continuously monitor, troubleshoot, and optimize systems for performance, reliability, scalability, and cost efficiency.
• Work closely with product, engineering, and data stakeholders to translate business requirements into scalable technical solutions.
• Participate in architectural discussions, sprint planning, feature sizing, and Agile ceremonies to ensure seamless execution.
• Stay current with emerging technologies in cloud, data, and DevOps to keep our platforms future-ready.
• Effectively communicate technical vision, trade-offs, and roadmaps to diverse stakeholders.

All About You:
• Extensive experience in cloud and data platform architecture, DevOps, or related roles, with proven leadership of cross-functional teams.
• Hands-on expertise with AWS and Azure services: S3, EC2, Lambda, Glue, Redshift, Lake Formation, Azure Data Factory, Fabric, Synapse, and Databricks.
• Strong track record of building DevOps pipelines and infrastructure-as-code using Terraform, CloudFormation, ARM templates, Azure DevOps, or GitLab CI/CD.
• Solid understanding of cloud security concepts, IAM design, data encryption, and governance frameworks.
• Experience with data engineering frameworks (Spark, PySpark) and large-scale ETL systems.
• Proficiency in scripting and automation using Python, Bash, or PowerShell.
• Demonstrated success delivering production-ready medallion architecture solutions.
• Experience with containerization & orchestration (Docker, Kubernetes, ECS, AKS).
• Familiarity with data governance tools (Lake Formation, Purview) and observability stacks (CloudWatch, Azure Monitor, ELK).
• Knowledge of cloud cost optimization and financial management.
• Exposure to MLOps workflows and ML pipeline integrations.
• Comfortable working in Agile/Scrum environments with strong collaboration skills.
• Strong problem-solving skills, with the ability to identify discordant views and engage in constructive dialogue to resolve them.
• Excellent verbal and written communication skills, with the ability to explain technical problems clearly and succinctly and effectively sharing vision and roadmaps with diverse stakeholders.
• Bachelor's degree in Computer Science, Information Systems, Engineering, Data Analytics, or a related field. Equivalent practical experience will also be considered.

#LI-NF1

Mastercard is a merit-based, inclusive, equal opportunity employer that considers applicants without regard to gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law. We hire the most qualified candidate for the role. In the US or Canada, if you require accommodations or assistance to complete the online application process or during the recruitment process, please contact and identify the type of accommodation or assistance you are requesting. Do not include any medical or health information in this email. The Reasonable Accommodations team will respond to your email promptly.

Corporate Security Responsibility

All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:

• Abide by Mastercard's security policies and practices;

• Ensure the confidentiality and integrity of the information being accessed;

• Report any suspected information security violation or breach, and

• Complete all periodic mandatory security trainings in accordance with Mastercard's guidelines.

In line with Mastercard's total compensation philosophy and assuming that the job will be performed in the US, the successful candidate will be offered a competitive base salary and may be eligible for an annual bonus or commissions depending on the role. The base salary offered may vary depending on multiple factors, including but not limited to location, job-related knowledge, skills, and experience. Mastercard benefits for full time (and certain part time) employees generally include: insurance (including medical, prescription drug, dental, vision, disability, life insurance); flexible spending