AI-IN-A-BOX Performance Evaluation: Red Hat OpenShift Container Platform on AMD EPYC 9004 Series
Red Hat OpenShift is a hybrid cloud platform for developing, modernizing, and deploying applications, including AI-enabled apps powered by containers and Kubernetes. This provides full set of operations and developer services and tools, including Serverless, Service Mesh, and Pipelines.
Red Hat OpenShift Container Platform Version: 4.12.7
The AMD EPYC 9004 series, known as "Genoa," represents AMD's latest lineup of high-performance server processors, designed for the most demanding workloads across various industries.
AMD EPYC 9654: 4th Gen processor with 96 cores and 192 threads
This document provides a performance evaluation of an AI-IN-A-BOX solution running on Red Hat OpenShift Container Platform, powered by AMD EPYC 9004 series CPUs, for serving Large Language Model (LLM) inferencing with the open-source LLM, Llama2. It outlines the key findings from Infobell IT's comprehensive and independent performance benchmarking of the Llama2 model, conducted using Infobell IT’s innovative benchmarking tool, EchoSwift.
| LLM Model | SUT Config | Input and Output tokens combinations |
|---|---|---|
| Llama2 7B | 48 Core CPUs and 48GB memory |
|
| Llama2 13B | 48 Core CPUs and 96GB memory |
|
EchoSwift is a specialized tool for benchmarking inference in Large Language Models (LLMs). It offers comprehensive performance and scalability assessments, measuring key metrics like latency and throughput under varying parallel request loads. These insights help customers make informed decisions about the optimal configurations for their LLMs and aids in identifying potential bottlenecks in their deployments of LLM models under load. The insights gained from this benchmarking tool provide clear and actionable guidance, helping customers make informed decisions about the optimal configuration for their large language models (LLMs).