BentoML

BentoML is an inference platform built around the open-source BentoML framework, designed to package, deploy, and scale machine-learning and LLM workloads in production. It handles the messy parts of model serving — containerization, autoscaling, GPU scheduling, cold-start optimization, and observability — across any cloud, on-prem, or Kubernetes environment. The hosted product (Bento) adds a managed control plane with scale-to-zero, distributed GPU inference, and LLM-specific metrics on top of the OSS core.

It is aimed squarely at ML engineers and platform teams who are deploying their own models — open-source LLMs like Llama, DeepSeek, Qwen, and Flux, or proprietary architectures — rather than calling a third-party API. Pricing is the standard open-core split: the framework is free under Apache 2.0, with a freemium cloud tier and usage-based pricing for the managed platform. If you've outgrown SageMaker endpoints or are bolting together vLLM, Ray Serve, and Triton by hand, this is the obvious consolidation play.

BentoML is more inference infrastructure than 'agent platform' in the chatbot sense — its place in the agents category is as the substrate that runs the models behind compound AI systems and tool-calling workflows. Strong fit for teams that need real-time, async, and batch serving patterns from a single framework with first-class Kubernetes support.