关键字: [亚马逊云科技， EKS， 机器学习服务， 开源模型部署， Gpu集群扩缩， 可观测性监控， 服务网格控制]
高玉是亚马逊解决方案的架构师，他在演讲中介绍了基于亚马逊 EKS 的机器学习模型推理平台。该平台旨在解决开源机器学习模型部署的复杂性、扩展性和可观测性等痛点。它利用 Kubernetes 集群、负载均衡器、服务网格等技术，支持多种机器学习模型如 VRM、Healthy Face、TGI 和 Text-to-Image Web UI 等的部署和管理。该平台提供了用户界面、成本监控、指标监控和日志监控等功能，帮助客户高效部署和运维开源机器学习模型。
以下是小编为您整理的本次演讲的精华。
Gao Yu， an Amazon solutions architect， commenced his presentation by delving into the realm of machine learning models (MLMs) and their deployment on Amazon’s managed Kubernetes (K8S) platform， EKS. He acknowledged the prevalent use of closed-source models while simultaneously recognizing the persistent demand for open-source model deployment among numerous customers. This demand stemmed from various factors， including the perceived superior performance of open-source models， the flexibility to customize them， potential cost advantages in specific scenarios， and privacy concerns surrounding the transmission of data to pre-trained models. Gao Yu emphasized that these requirements transcended geographical boundaries， with customers in both China and overseas regions expressing a need for open-source model deployment.
展开剩余80%However， the deployment of open-source models presented a multitude of challenges. Firstly， the process itself was inherently complex， often involving GPU clusters of varying configurations， ranging from single GPU to multi-GPU setups， spanning across single or multiple machines. Scalability emerged as a critical consideration， particularly in large-scale cluster environments， where the ability to rapidly scale resources up or down became paramount. Observability was another crucial aspect， as customers sought to monitor the performance and health of their deployed models， akin to monitoring a conventional application. Key metrics such as concurrency levels， response times， and overall platform behavior were essential for effective management and optimization.
To address these challenges， Amazon unveiled a revolutionary architecture built upon EKS. This architecture encompassed a load balancer， Carped for scaling capabilities， and the Horizontal Pod Autoscaler (HPA) for triggering scaling events dynamically. The user-facing data plane consisted of an Application Load Balancer， a Service Mesh layer for traffic management， and the option to leverage Amazon’s custom Neuron Instances or NVIDIA GPUs for model deployment. Gao Yu mentioned that the presentation had a duration of only 5 minutes， necessitating a rapid overview of the solution.
Gao Yu highlighted the support for several popular open-source models within this architecture， including VRAM， Healthy Face， TGI， and the recently acclaimed Text-to-Image Web UI models， such as Stable Diffusion. These models catered to a wide range of use cases and customer requirements. For instance， customers could leverage the VRAM model for visual recognition tasks， the Healthy Face model for facial analysis applications， and the TGI model for image generation purposes. The Text-to-Image Web UI models， like Stable Diffusion， enabled customers to generate images from textual descriptions， unlocking new possibilities in creative and design workflows.
On the control plane side， Amazon provided a user-friendly interface， Coper Cost， enabling customers to monitor resource consumption at various levels， including pods， namespaces， and services. This feature allowed customers to gain insights into their resource utilization and associated costs， facilitating better resource management and cost optimization. Observability was further enhanced through a unified monitoring solution comprising Prometheus for metrics collection and comprehensive logging capabilities， empowering customers to proactively monitor and troubleshoot their deployed models.
During the demonstration， Gao Yu showcased a visualization of the deployed engines， including TGW， TGI， and VRAM， along with their respective traffic levels. The Service Mesh component played a crucial role in controlling traffic routing and weight distribution across these engines， ensuring optimal resource utilization and performance. This feature enabled customers to dynamically manage traffic flows and prioritize specific models based on their business requirements， further enhancing the flexibility and scalability of the solution.
In conclusion， Amazon’s revolutionary machine learning service， built upon the EKS platform， addressed the complexities associated with open-source model deployment， scalability， and observability. By leveraging Amazon’s cloud infrastructure and innovative architecture， customers could seamlessly deploy and scale open-source models while benefiting from enhanced performance， flexibility， and cost optimization. This groundbreaking solution empowered organizations to harness the power of open-source machine learning models while overcoming the traditional challenges posed by their deployment and management.
下面是一些演讲现场的精彩瞬间：
The speaker briefly introduces the business background， challenges， and solutions， aiming to cover the key points within the 5-minute time limit.
客户有部署开源语音模型的需求，以满足定制化、成本优化和隐私保护的要求。
如果我们想部署一个开业模型，需要考虑部署的复杂性、扩展性和可观测性等挑战。
亚马逊云科技推出了一项革命性的机器学习服务，旨在简化开源机器学习模型的部署和管理。该服务基于亚马逊弹性 Kubernetes 服务 (EKS) 构建，提供了一个完整的解决方案架构，包括负载均衡器、自动扩缩、服务网格等关键组件。它支持多种流行的开源模型，如大型语言模型、人脸识别模型和文本到图像生成模型。
该服务的核心优势在于简化了复杂的部署过程，提高了可扩展性和可观测性。用户可以轻松管理和监控模型的性能、并发量和响应时间。此外，它还提供了用户友好的界面，使非技术人员也能轻松操作。通过利用亚马逊自研的 Neuron GPU 实例和 NVIDIA GPU，该服务确保了高性能的推理能力。
总的来说，这项新服务为企业提供了一种高效、可扩展且经济实惠的方式来利用开源机器学习模型的强大功能，助力各行业的数字化转型。亚马逊云科技呼吁客户抓住这一机遇，充分释放人工智能的潜力，推动创新和增长。
亚马逊云科技（Amazon Web Services）是全球云计算的开创者和引领者。提供200多类广泛而深入的云服务，服务全球245个国家和地区的数百万客户。做为全球生成式AI前行者，亚马逊云科技正在携手广泛的客户和合作伙伴，缔造可见的商业价值 – 汇集全球40余款大模型，亚马逊云科技为10万家全球企业提供AI及机器学习服务，守护3/4中国企业出海。
发布于：新加坡华林优配提示：文章来自网络，不代表本站观点。
