15,584 AI Infrastructure jobs in the United States

Category:

Engineering

Employment Type:

Direct Hire

Reference:

BH-

AI Infrastructure Engineer

About the Position

We are looking for a senior AI Inference Infrastructure Software Engineer with strong hands-on experience building, optimizing, and deploying high-performance, scalable inference systems. This position is focused on designing, implementing, and delivering production-grade software that powers real-world applications of Large Language Models (LLMs) and Vision-Language Models (VLMs).
This is an exciting opportunity for an engineer who thrives at the intersection of AI systems, hardware acceleration, and large-scale robust deployment, and who wants to see their contributions ship in production, at scale.
In this role, you will directly shape the architecture, roadmap and performance of AI capabilities of our AIOS platform, driving innovations that make LLM/VLM systems fast, efficient, and scalable across cloud, edge, and hybrid edge-cloud environments. You will work closely with system, hardware, and product teams to deliver high-performance inference kernels for hardware accelerators, design scalable inference serving systems, and integrate optimizations such tensor parallelism and custom kernels into production pipelines. Your work will have immediate impact, powering intelligent automotive systems in the next generation of electric vehicles.

Roles and Responsibilities:

• Design and implement high-performance, scalable inference systems for LLMs and VLMs across cloud, edge, and edge-cloud hybrid platforms.
• Develop and optimize custom kernels and operators for specific hardware accelerators (GPU, NPU, DSP, etc.), improving throughput, latency, and memory efficiency.
• Integrate advanced optimization techniques such as KV-cache management, tensor/model parallelism, quantization, and memory-efficient execution into production inference systems.
• Partner with system and hardware teams to ensure tight hardware-software integration and optimal performance across diverse compute environments.
• Translate architectural requirements into robust, maintainable, production-ready software that meets performance, safety, and reliability standards.
• Define and drive the evolution roadmap for LLM/VLM inference in the AIOS stack, ensuring scalability and adaptability to new workloads.
• Stay ahead of industry trends and competitor solutions, applying best practices from both AI and large-scale systems engineering.

• 5+ years of hands-on software development experience in building and optimizing AI inference systems at scale.
• Direct experience in LLM/VLM model internals, including Transformer-based architectures, inference bottlenecks, and optimization techniques.
• Strong expertise in performance engineering: kernel development, parallelism strategies, memory optimization, and distributed inference systems.
• Proficiency with GPU/NPU programming (CUDA, or vendor-specific SDKs), compiler toolchains, and deep learning frameworks (PyTorch, or TensorFlow).
• Strong programming skills in C/C++, with a track record of delivering high-performance, production-grade software.
• Solid foundation in computer architecture, systems programming (CPU/GPU pipelines, memory hierarchy, scheduling), and embedded systems.
• BS/MS in Computer Science, Computer Engineering, or related technical field.
• Excellent communication and collaboration skills, with the ability to work across cross-functional teams.

• Master's or PhD degree in Computer Science, Electrical/Computer Engineering, or related fields, plus 5 years industry experience
• Experience building inference serving systems for large models, including batching, scheduling, caching, and load balancing.
• Expertise in hardware-aware model optimization (e.g., kernel fusion, mixed precision, quantization, pruning).
• Familiarity with edge and embedded AI, including real-time constraints and limited-resource optimization.
• Contributions to widely used AI frameworks, libraries, or performance-critical software (open source or proprietary).

• Medical, Prescription, Dental & Vision Benefits (for employees working 20+ hours per week)
• Health Savings Account (HSA) (for employees working 20+ hours per week)
• Life & Disability Insurance (for employees working 20+ hours per week)
• MetLife Voluntary Benefits
• Employee Assistance Program (EAP)
• 401K Retirement Savings Plan
• Direct Deposit & weekly epayroll
• Referral Bonus Programs
• Certification and training opportunities

LifeMD is a leading digital healthcare company committed to expanding access to virtual care, pharmacy services, and diagnostics by making them more affordable and convenient for all. Focused on both treatment and prevention, our unique care model is designed to optimize the patient experience and improve outcomes across more than 200 health concerns.

To support our expanding patient base, LifeMD leverages a vertically-integrated, proprietary digital care platform, a 50-state affiliated medical group, a 22,500-square-foot affiliated pharmacy, and a U.S.-based patient care center. Our company with offices in New York City; Greenville, SC; and Huntington Beach, CA is powered by a dynamic team of passionate professionals. From clinicians and technologists to creatives and analysts, we're united by a shared mission to revolutionize healthcare. Employees enjoy a collaborative and inclusive work environment, hybrid work culture, and numerous opportunities for growth. Want your work to matter? Join us in building a future of accessible, innovative, and compassionate care.

We're looking for an AI Infrastructure Engineer to own and maintain our AI infrastructure on GCP, which powers our voice assistants, message routing, and knowledge retrieval systems. This role will manage the technical backbone that enables our AI services to operate safely and effectively in healthcare settings. You will be the technical steward of our AI operations infrastructure and a partner to clinical and product stakeholders. Your job is to build new AI capabilities, maintain existing systems, and ensure our AI services meet LifeMD's standards. This role will operate across our entire AI stack: from service development and deployment to testing frameworks to production monitoring systems.

Responsibilities

• Own and maintain the AI infrastructure, including deployment pipelines, testing frameworks, and evaluation systems
• Build new AI capabilities and use cases while maintaining existing services and systems
• Develop and maintain AI evaluation models for regression testing
• Implement monitoring and alerting systems for AI service performance, cost management, and safety compliance
• Collaborate with engineering teams to integrate AI testing into deployment pipelines and ensure proper governance
• Manage BigQuery datasets, data pipelines, and cloud infrastructure supporting AI operations

• Design, build, and maintain the AI infrastructure and systems needed to support our AI applications. Examples include
• AI workflow scheduling system to improve GPU efficiency and throughput of our batch inference systems
• Model optimization to improve inference performance
• Auto Train systems to power our avatar models
• Large scale model evaluation systems
• Online model serving systems
• Collaborate with data scientists and machine learning engineers to understand their computational and data needs and provide efficient solutions.
• Stay up-to-date with the latest industry trends in AI infrastructure technologies and advocate for best practices and continuous improvement.
• Assist in budget planning and management of cloud resources and other infrastructure expenses.

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field
• 5+ years of experience
• Proven experience in managing infrastructure for large-scale AI or machine learning projects
• Excellent problem-solving skills and the ability to work independently or as part of a team.
• Proficiency in Python and C++
• Experience with GPU computing and optimizing computational workflows
• Familiarity with AI and machine learning frameworks like TensorFlow or PyTorch.

• Experience with CUDA
• Experience optimizing large deep learning model performance
• Experience building large scale batch inference system
• Prior experience in a startup or fast-paced tech environment.

• Competitive salary and benefits package.
• Dynamic and inclusive work environment.
• Opportunities for professional growth and advancement.
• Collaborative culture that values innovation and creativity.
• Access to the latest technologies and tools.

• Sr level + Building, training, and deploying ML models.
• Python.
• A combo of Pandas, Numpy, PyTorch, Tensorflow.
• Experience with transformer models - specifically GPT (Generative predictive transformer).
• Cloud experience- AWS, Azure, or GCP.
• CI/CD pipeline implementation.

• Kubernetes and Docker.

• Will work on both teams/projects.
• Responsible for assisting in design and implementation of ML models (need to be a do-er in addition to design).
• Should be able to look at the product and connect the business need with the technology capabilities.
• Split up into different teams - some teams focus on GenAI and ML, some teams focus on IaC.
• Collaborate with operational teams, Product Owners, Product engineering, and leadership on business needs and enabling various AI features.

• Building two horizontal AI platforms.
• MLOps - using predictive models.
• Detecting fraudulent transaction.
• Determine credit lines/amounts for small business loans.
• GenAI.
• Claims - upload documentation, AI read and analyze, auto-populate claims form, launch validation/classification for claims.
• Auto approvals.

• Design and implement high-performance, scalable inference systems for LLMs and VLMs across cloud, edge, and edge-cloud hybrid platforms.
  
• Develop and optimize custom kernels and operators for specific hardware accelerators (GPU, NPU, DSP, etc.), improving throughput, latency, and memory efficiency.
  
• Integrate advanced optimization techniques such as KV-cache management, tensor/model parallelism, quantization, and memory-efficient execution into production inference systems.
  
• Partner with system and hardware teams to ensure tight hardware-software integration and optimal performance across diverse compute environments.
  
• Translate architectural requirements into robust, maintainable, production-ready software that meets performance, safety, and reliability standards.
  
• Define and drive the evolution roadmap for LLM/VLM inference in the AIOS stack, ensuring scalability and adaptability to new workloads.
  
• Stay ahead of industry trends and competitor solutions, applying best practices from both AI and large-scale systems engineering.
  

• 5+ years of hands-on software development experience in building and optimizing AI inference systems at scale.
  
• Direct experience in LLM/VLM model internals, including Transformer-based architectures, inference bottlenecks, and optimization techniques.
  
• Strong expertise in performance engineering: kernel development, parallelism strategies, memory optimization, and distributed inference systems.
  
• Proficiency with GPU/NPU programming (CUDA, or vendor-specific SDKs), compiler toolchains, and deep learning frameworks (PyTorch, or TensorFlow).
  
• Strong programming skills in C/C++, with a track record of delivering high-performance, production-grade software.
  
• Solid foundation in computer architecture, systems programming (CPU/GPU pipelines, memory hierarchy, scheduling), and embedded systems.
  
• BS/MS in Computer Science, Computer Engineering, or related technical field.
  
• Excellent communication and collaboration skills, with the ability to work across cross-functional teams.
  

• Master's or PhD degree in Computer Science, Electrical/Computer Engineering, or related fields, plus 5 years industry experience
  
• Experience building inference serving systems for large models, including batching, scheduling, caching, and load balancing.
  
• Expertise in hardware-aware model optimization (e.g., kernel fusion, mixed precision, quantization, pruning).
  
• Familiarity with edge and embedded AI, including real-time constraints and limited-resource optimization.
  
• Contributions to widely used AI frameworks, libraries, or performance-critical software (open source or proprietary).
  

• Within the range, individual pay is determined by work location and additional factors, including job-related skills, experience, and relevant education or training.
  
• Please note that the compensation details listed in US role postings reflect the base salary only. It does not include discretionary bonus, equity, or benefits.
  

• CIGNA EPO, HSA, and Kaiser HMO medical plans with 0 for Employee Only Coverage.
• Dental (including orthodontic coverage) and vision plan. Both provide options with a 0 paycheck contribution covering you and your eligible dependents.
• Company Paid HSA (Health Savings Account) Contribution when enrolled in the High Deductible CIGNA medical plan
• Healthcare and Dependent Care Flexible Spending Accounts (FSA)
• 401(k) with Brokerage Link option
• Company paid Basic Life, AD&D, short-term and long-term disability insurance
• Employee Assistance Program
• Sick and Vacation time
• 13 Paid Holidays a year
• Paid Parental Leave for first 8 weeks at full pay (eligible after 90 days of employment with NIO)
• Paid Disability Leave for first 6 weeks at full pay (eligible after 90 days of employment with NIO)
• Voluntary benefits including: Voluntary Life and AD&D options for you, your spouse/domestic partner and dependent child(ren), pet insurance
• Commuter benefits
• Mobile Cell Phone Credit
• Healthjoy mobile benefit app supporting you and your dependents with benefit questions on the go & support with benefit billing questions
• Free lunch and snacks
• Onsite gym
• Employee discounts and perks program

As a Software Engineer on the ML Infrastructure team, you will design and build the platform for our agent sandboxing platform: the secure, high-performance code execution layer powering our agentic workflows. This system underpins critical applications and research initiatives, and is deployed across both internal and customer-managed environments.

This position requires deep expertise in systems engineering: operating systems, virtualization, networking, containers, and performance optimization. Your work will directly enable agents to execute untrusted or user-submitted code safely, efficiently, and repeatedly, and with fast startup times, strong isolation guarantees, and support for snapshotting and inspection.

• Design and build the sandboxing platform for code execution across containerized and virtualized environments.
• Ensure strong isolation, security, and reproducibility of execution across user sessions and workloads.
• Optimize for cold-start latency, memory footprint, and resource utilization at scale.
• Collaborate across security, infra, and product teams to support both internal research use cases and enterprise customer deployments.
• Lead architecture reviews and own projects from design through deployment in fast-paced, cross-functional settings.

• 3+ years of experience building high-performance systems software (e.g. OS, container runtime, VMM, networking stack).
• Deep understanding of Linux internals, process isolation, memory management, cgroups, namespaces, etc.
• Experience with containerization and virtualization technologies (e.g., Docker, Firecracker, gVisor, QEMU, Kata Containers).
• Proficiency in a systems programming language such as Go, Rust, or C/C++.
• Familiarity with networking, security hardening, sandboxing techniques, and kernel-level performance tuning.
• Comfort working across infrastructure layers, from kernel modules to orchestration frameworks (e.g., Kubernetes).
• Strong debugging skills and the ability to make performance/security tradeoffs in production systems.

• Familiarity with LLM agents and agent frameworks (e.g., OpenHands, Agent2Agent, MCP).
• Experience running secure workloads in multi-tenant or untrusted environments (e.g., FaaS, CI sandboxes, remote notebooks).
• Exposure to snapshotting and restore techniques (e.g., CRIU, VM snapshots, overlayfs).

Compensation packages at Scale for eligible roles include base salary, equity, and benefits. The range displayed on each job posting reflects the minimum and maximum target for new hire salaries for the position, determined by work location and additional factors, including job-related skills, experience, interview performance, and relevant education or training. Scale employees in eligible roles are also granted equity based compensation, subject to Board of Director approval. Your recruiter can share more about the specific salary range for your preferred location during the hiring process, and confirm whether the hired role will be eligible for equity grant. You'll also receive benefits including, but not limited to: Comprehensive health, dental and vision coverage, retirement benefits, a learning and development stipend, and generous PTO. Additionally, this role may be eligible for additional benefits such as a commuter stipend.

PLEASE NOTE: Our policy requires a 90-day waiting period before reconsidering candidates for the same role. This allows us to ensure a fair and thorough evaluation of all applicants.

About Us:

At Scale, our mission is to develop reliable AI systems for the world's most important decisions. Our products provide the high-quality data and full-stack technologies that power the world's leading models, and help enterprises and governments build, deploy, and oversee AI applications that deliver real impact. We work closely with industry leaders like Meta, Cisco, DLA Piper, Mayo Clinic, Time Inc., the Government of Qatar, and U.S. government agencies including the Army and Air Force. We are expanding our team to accelerate the development of AI applications.

We believe that everyone should be able to bring their whole selves to work, which is why we are proud to be an inclusive and equal opportunity workplace. We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability status, gender identity or Veteran status.

We are committed to working with and providing reasonable accommodations to applicants with physical and mental disabilities. If you need assistance and/or a reasonable accommodation in the application or recruiting process due to a disability, please contact us at Please see the United States Department of Labor's Know Your Rights poster for additional information.

We comply with the United States Department of Labor's Pay Transparency provision .

PLEASE NOTE: We collect, retain and use personal data for our professional business purposes, including notifying you of job opportunities that may be of interest and sharing with our affiliates. We limit the personal data we collect to that which we believe is appropriate and necessary to manage applicants' needs, provide our services, and comply with applicable laws. Any information we collect in connection with your application will be treated in accordance with our internal policies and programs designed to protect personal data. Please see our privacy policy for additional information.

• Design and build training, inference, and evaluation infrastructure to support our current autonomy stack development, orchestrating massive GPU clusters to process petabytes of multimodal sensor data
• Optimize multimodal data ingestion and preprocessing pipelines (LiDAR, camera, radar, map priors) to support cutting-edge perception and planning model development
• Work across cloud environments to support high-throughput distributed training
• Collaborate closely with the AI research team and autonomy teams
• Technologies: Pytorch, CUDA, Ray, Flyte, K8s

• Experience with building software components to address production, full-stack machine learning challenges.
• Opinions about building a company-wide platform for ML training, evaluation, and deployment
• Knowledge of the open source landscape with judgment on when to choose open source versus build in-house
• Excellent analytical and problem-solving skills

A stealth-stage AI infrastructure company is building a self-healing system for software that automates defect resolution and development. The platform is used by engineering and support teams to:

• Autonomously debug problems in production software
• Fix issues directly in the codebase
• Prevent recurring issues through intelligent root-cause automation

• Own mission-critical services end-to-end - from architecture and design reviews to deployment, observability, and service-level objectives.
• Scale LLM-driven systems : build RAG pipelines, vector indexes, and evaluation frameworks handling billions of events per day.
• Design data-heavy backends : streaming ETL, columnar storage, time-series analytics - all fueling the self-healing loop.
• Optimize for cost and latency across compute types (CPUs, GPUs, serverless); profile hot paths and squeeze out milliseconds.
• Drive reliability : implement automated testing, chaos engineering, and progressive rollout strategies for new models.
• Work cross-functionally with ML researchers, product engineers, and real customers to build infrastructure that actually matters.

• Have 2-5+ years of experience building scalable backend or infra systems in production environments
• Bring a builder mindset - you like owning projects end-to-end and thinking deeply about data, scale, and maintainability
• Have transitioned ML or data-heavy prototypes to production , balancing speed and robustness
• Are comfortable with data engineering workflows : parsing, transforming, indexing, and querying structured or unstructured data
• Have some exposure to search infrastructure or LLM-backed systems (e.g., document retrieval, RAG, semantic search)

• Experience with vector databases (e.g., pgvector, Pinecone, Weaviate) or inverted-index search (e.g., Elasticsearch, Lucene)
• Hands-on with GPU orchestration (Kubernetes, Ray, KServe) or model-parallel inference tuning
• Familiarity with Go / Rust (primary stack), with some TypeScript for light full-stack tasks
• Deep knowledge of observability tooling (OpenTelemetry, Grafana, Datadog) and profiling distributed systems
• Contributions to open-source ML or systems infrastructure projects

As a Software Engineer on the ML Infrastructure team, you will design and build the platform for our agent sandboxing platform: the secure, high-performance code execution layer powering our agentic workflows. This system underpins critical applications and research initiatives, and is deployed across both internal and customer-managed environments.

This position requires deep expertise in systems engineering: operating systems, virtualization, networking, containers, and performance optimization. Your work will directly enable agents to execute untrusted or user-submitted code safely, efficiently, and repeatedly, and with fast startup times, strong isolation guarantees, and support for snapshotting and inspection.
You will:

• Design and build the sandboxing platform for code execution across containerized and virtualized environments.
• Ensure strong isolation, security, and reproducibility of execution across user sessions and workloads.
• Optimize for cold-start latency, memory footprint, and resource utilization at scale.
• Collaborate across security, infra, and product teams to support both internal research use cases and enterprise customer deployments.
• Lead architecture reviews and own projects from design through deployment in fast-paced, cross-functional settings.

• 3+ years of experience building high-performance systems software (e.g. OS, container runtime, VMM, networking stack).
• Deep understanding of Linux internals, process isolation, memory management, cgroups, namespaces, etc.
• Experience with containerization and virtualization technologies (e.g., Docker, Firecracker, gVisor, QEMU, Kata Containers).
• Proficiency in a systems programming language such as Go, Rust, or C/C++.
• Familiarity with networking, security hardening, sandboxing techniques, and kernel-level performance tuning.
• Comfort working across infrastructure layers, from kernel modules to orchestration frameworks (e.g., Kubernetes).
• Strong debugging skills and the ability to make performance/security tradeoffs in production systems.

• Familiarity with LLM agents and agent frameworks (e.g., OpenHands, Agent2Agent, MCP).
• Experience running secure workloads in multi-tenant or untrusted environments (e.g., FaaS, CI sandboxes, remote notebooks).
• Exposure to snapshotting and restore techniques (e.g., CRIU, VM snapshots, overlayfs).