Learning Python - Performance Tuning & Profiling

#Introduction

Slow applications are usually caused by a few lines of code, not the whole program. Episode 16 teaches you how to find those lines with data: profiling. You'll learn to use cProfile, pyinstrument, and flame graphs to find the hot path.

We'll also cover optimization strategies — from algorithm improvements to native Cython extensions — and when PyPy is the right choice. The core principle is one: optimization must be based on measurement, not guesses.

#Measuring Execution Time

#timeit for Micro-Measurements

Before full profiling, measure small code snippets with timeit:

Mengukur kode dengan timeit

import timeit
 
kode = "total = sum(range(1000))"
waktu = timeit.timeit(kode, number=10_000)
print(f"waktu per eksekusi: {waktu / 10_000:.6f} detik")

timeit.timeit(kode, number=10_000) runs the code thousands of times and measures the total time. timeit disables the garbage collector and uses a precise clock, so results are consistent. This is the first tool for comparing two implementations.

#cProfile for Function Profiling

#Profiling the Whole Program

cProfile tracks how long each function runs:

Membuat fungsi untuk diprofile

def total_genap(n):
    return sum(i for i in range(n) if i % 2 == 0)
 
def jalankan():
    return total_genap(1_000_000)
 
if __name__ == "__main__":
    jalankan()

sum(i for i in range(n) if i % 2 == 0) computes the total of even numbers. Save this file as perf_contoh.py, then run the profiler:

python3 -m cProfile -s cumulative perf_contoh.py

python3 -m cProfile -s cumulative perf_contoh.py runs the script with the profiler and sorts results by cumulative time. The output shows a function table with the ncalls, tottime, and cumtime columns — this is the data for finding the hot path.

#Reading the cProfile Output

The cProfile output contains important columns:

• ncalls: how many times the function was called.
• tottime: time in the function itself, without its subcalls.
• cumtime: total time including the functions it calls.

Focus on the functions with the largest tottime or cumtime — those are the main optimization candidates. Don't optimize functions that are rarely called.

#pyinstrument and Sampling Profilers

#A More Readable Profiler

pyinstrument is a sampling profiler with friendlier output:

pip install pyinstrument

pyinstrument -r html -o profil.html perf_contoh.py

pyinstrument -r html -o profil.html perf_contoh.py produces an HTML report. Unlike cProfile, which analyzes every call, pyinstrument samples — recording the stack trace periodically. The output is an easy-to-read call tree.

#Hot Path Optimization

#Improving the Algorithm

The first optimization is always at the algorithm level, not syntax:

Optimasi dengan rumus

def total_genap_cepat(n):
    k = (n - 1) // 2
    return k * (k + 1)
 
print(total_genap_cepat(10))

k * (k + 1) computes the sum of even numbers with an O(1) mathematical formula, replacing the O(n) loop. This is an example of complexity improvement — far more impactful than micro-optimization. Always look for a better algorithmic approach first.

#Using the Right Data Structure

The next optimization uses the right data structure:

Set untuk keanggotaan

target = {i for i in range(100_000)}
print(99_999 in target)

The set comprehension {i for i in range(100_000)} builds a set for O(1) lookups. Replacing a list with a set or dict for lookups is a practical optimization that's immediately noticeable on large data.

#Native Extensions with Cython

#Speeding Up with Cython

Cython compiles Python code with type annotations into C extensions:

Fungsi Cython

def total_genap_cy(int n):
    cdef long total = 0
    cdef int i
    for i in range(n):
        if i % 2 == 0:
            total += i
    return total

cdef long total = 0 declares a C-typed variable that gets optimized. Cython removes interpreter overhead in hot loops. For intensive numeric code, the speedup can be tens of times.

#Build and Integration

Cython is built through setup or pyproject:

pip install cython

[tool.cythonize]
target_dir = "build"

pip install cython installs the compiler. Cython is useful when profiling shows a purely numeric hot path. For most applications, algorithm and data structure improvements are enough — Cython is an advanced step.

#PyPy: A JIT Alternative

#When to Use PyPy

PyPy is a Python interpreter with a JIT compiler that can be very fast for purely CPU-bound code:

pypy3 --version
pypy3 perf_contoh.py

pypy3 perf_contoh.py runs the script on the PyPy interpreter. The JIT compiles frequently executed loops into machine code, producing dramatic speedups for repetitive computation. Without changing any code, you can get more speed.

#Considerations for Using PyPy

PyPy isn't a universal replacement for CPython:

• Fits pure Python code with intensive loops.
• Less compatible with C extensions that aren't yet supported.
• Startup may be slower; long-running workloads benefit more.
• The newest Python 3.12 features may lag behind CPython.

Measure first: run a benchmark on both CPython and PyPy, then compare. Use whichever is faster for your case.

#Closing

Key takeaways:

• Optimization must be based on measurement, not guesses.
• timeit measures code snippets with high precision.
• cProfile finds the functions that consume the most time.
• pyinstrument uses sampling with a clear call-tree report.
• Improve the algorithm and data structures before micro-optimization.
• Cython and PyPy are advanced steps for extreme performance.

In the next episode, episode 17, we'll cover type checking and contracting — static typing with typing such as Annotated, TypedDict, Protocol, and ParamSpec, plus mypy and pyright. The benefits for maintenance, refactoring, and large codebases will be dissected. Your code will be safer and better documented!