GGistDev

Performance Optimization in Python

Optimize by measuring first, choosing the right algorithms/data structures, and leveraging libraries.

Profile first

Identify hotspots before changing code.

python -m cProfile -o prof.out app.py
snakeviz prof.out

Line and memory profilers: line_profiler, memory_profiler.

Algorithmic improvements

Big-O dominates. Replace nested loops with maps/sets, early exits, and better structures.

# O(n) membership
s = set(items)
if x in s: ...

Data structures

Pick the right container for the job:

  • list for ordered, append-heavy sequences
  • deque for queue/stack ends
  • set/dict for O(1) membership/lookup
  • array/bytearray for compact numeric/byte data

Vectorization and C extensions

Use NumPy/Pandas for vectorized numeric/data ops; offload hot loops to C/Cython/Numba.

import numpy as np
x = np.arange(1_000_000)
y = x * 2

I/O and concurrency

Batch I/O, stream with iterators, and parallelize appropriately:

  • Async IO for many network operations
  • concurrent.futures for CPU bound (processes) or blocking I/O (threads)

Caching and memoization

Memoize pure functions; be mindful of memory keys.

from functools import lru_cache
@lru_cache(maxsize=1024)
def fib(n): ...

Micro-optimizations (sparingly)

  • Avoid repeated attribute lookups in tight loops (bind to locals)
  • Use comprehensions/generator expressions

Best practices

  • Write clear code first; optimize where profiling shows value
  • Add benchmarks/tests to prevent performance regressions
  • Consider alternative interpreters (PyPy) for long-running workloads

Summary

  • Measure, then optimize with algorithms/structures, vectorization, and appropriate concurrency
  • Cache wisely; keep code clear and validated with benchmarks
←PreviousDebugging