Belajar Performance Test Engineer - AI Workload Performance

#Pendahuluan

Setelah di episode 22 kita memahami continuous performance, kini saatnya membahas area yang semakin penting di 2026: AI workload performance. Dengan adopsi LLM, model inference, dan AI-powered features yang meningkat pesat, performance testing untuk AI workloads menjadi skill yang sangat dibutuhkan. Tetapi AI performance testing berbeda secara fundamental dari web application performance testing — latency yang diukur adalah inference time, bukan HTTP response time.

Episode ini membawa kalian memahami tantangan unik AI workload performance, cara mengukur GPU performance, dan strategi testing untuk LLM inference.

#Karakteristik AI Workload

#Perbedaan dengan Web Application

Aspek	Web Application	AI Workload
Bottleneck	CPU, network, database	GPU, VRAM, inference engine
Latency type	HTTP response time	Inference time
Throughput	Requests per second	Tokens per second
Scalability	Horizontal scaling	GPU scaling (mahal)
Caching	Cache results	Cache embeddings/activations

#GPU Performance Metrics

Metric	Deskripsi
GPU utilization	Persentase waktu GPU aktif
VRAM usage	Penggunaan GPU memory
Inference latency	Waktu untuk menghasilkan output
Tokens/sec	Throughput token output
Time to First Token (TTFT)	Waktu sampai token pertama muncul
Inter-Token Latency	Waktu antara token

#LLM Inference Performance

#Measuring Inference Latency

python

# Benchmark LLM inference latency
import time
from transformers import pipeline
 
pipe = pipeline("text-generation", model="meta-llama/Llama-3-8B")
 
prompt = "Explain quantum computing in simple terms"
 
# Warm up
pipe(prompt, max_new_tokens=100)
 
# Benchmark
latencies = []
for i in range(100):
    start = time.time()
    output = pipe(prompt, max_new_tokens=200)
    latencies.append(time.time() - start)
 
print(f"Mean latency: {sum(latencies)/len(latencies)*1000:.0f}ms")
print(f"p95 latency: {sorted(latencies)[95]*1000:.0f}ms")

#Throughput Testing

python

# Test concurrent inference throughput
import asyncio
import aiohttp
import time
 
async def inference_request(session, prompt):
    start = time.time()
    async with session.post('http://localhost:8080/v1/completions',
        json={'prompt': prompt, 'max_tokens': 100}) as resp:
        await resp.json()
        return time.time() - start
 
async def main():
    prompts = ["Explain AI"] * 50  # 50 concurrent requests
    async with aiohttp.ClientSession() as session:
        tasks = [inference_request(session, p) for p in prompts]
        latencies = await asyncio.gather(*tasks)
        print(f"Throughput: {len(prompts)/sum(latencies):.1f} req/s")
 
asyncio.run(main())

#Time to First Token (TTFT)

TTFT sangat penting untuk user experience — ini waktu yang user tunggu sebelum melihat output pertama. TTFT tinggi = user experience buruk.

python

# Measure TTFT
for chunk in model.generate(prompt, stream=True):
    start_time = time.time()  # First chunk = TTFT
    break

#GPU Performance Profiling

#NVIDIA SMI

bash

# Monitor GPU usage selama inference
nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total \
  --format=csv -l 1
 
# Detailed GPU profiling
nsys profile python inference.py

#GPU Bottlenecks

Bottleneck	Gejala	Solusi
VRAM overflow	OOM errors, swapping	Quantization, smaller model
GPU utilization low	GPU idle sering	Batch inference, async
Memory bandwidth	Latency tinggi, utilization tinggi	Model optimization
Compute bound	Utilization 100%, latency tinggi	Distillation, pruning

#Model Serving Performance

#Infrastructure Comparison

Platform	Latency	Throughput	Cost
Self-hosted GPU	Terendah	Tertinggi	Setup mahal
AWS SageMaker	Rendah	Tinggi	Per-invocation
OpenAI API	Sedang	Tinggi	Per-token

#Load Testing Model Serving

javascript

// k6: test model serving endpoint
export const options = {
  vus: 20,
  duration: '5m',
  thresholds: {
    http_req_duration: ['p(95)<5000'],  // 5s untuk inference
  },
};
 
export default function () {
  const res = http.post('http://localhost:8080/v1/completions', JSON.stringify({
    prompt: 'Explain quantum computing',
    max_tokens: 200,
  }), { headers: { 'Content-Type': 'application/json' } });
  check(res, {
    'inference status 200': (r) => r.status === 200,
    'TTFT < 2000ms': (r) => r.timings.waiting < 2000,
  });
}

#Batch Inference Optimization

#Mengapa Batch?

Batch inference menggabungkan banyak requests menjadi satu batch — memanfaatkan GPU parallelism secara lebih efektif. Ini meningkatkan throughput tetapi menambah latency per-request.

#Batch Size Tuning

plaintext

Batch size 1:  Throughput = 10 tokens/sec, Latency = 100ms
Batch size 8:  Throughput = 60 tokens/sec, Latency = 150ms
Batch size 32: Throughput = 150 tokens/sec, Latency = 300ms

Trade-off: batch lebih besar = throughput lebih tinggi tetapi latency lebih tinggi.

#Model Optimization for Performance

#Quantization

Quantization mengurangi precision model (dari FP32 ke INT8 atau INT4) — mengurangi VRAM usage dan inference time dengan trade-off akurasi yang minimal.

#Distillation

Knowledge distillation melatih model kecil dari model besar — menghasilkan model yang lebih ringan dengan performa mendekati model asli.

#Pruning

Pruning menghapus weights yang tidak penting dari model — mengurangi ukuran dan inference time.

#Penutup

Di episode 23 ini kalian telah memahami AI workload performance:

GPU performance: utilization, VRAM, inference latency.
LLM metrics: TTFT, tokens/sec, inter-token latency.
Profiling: nvidia-smi, nsys untuk GPU bottleneck detection.
Model optimization: quantization, distillation, pruning.
Serving performance: load testing model endpoints.

Di episode 24 selanjutnya, kita akan membahas Database Performance Engineering — deep dive ke query profiling, indexing strategies, dan database tuning untuk performance yang optimal. Siapkan pg_stat kalian!