memory-optimization

Optimize Python code for reduced memory usage and improved memory efficiency. Use when asked to reduce memory footprint, fix memory leaks, optimize data stru...

Security Scan

This is an instruction-only Python memory-optimization guide whose requirements and instructions are consistent with its stated purpose and do not request unexplained credentials or installs.

Purpose & Capability

Name/description match the provided content: SKILL.md and references contain patterns, profiling commands, and transformations that are appropriate for reducing Python memory usage. Nothing in the files asks for access to unrelated services or credentials.

Instruction Scope

Instructions operate on local Python data and files (CSV, binary, arrays) and recommend standard profiling and transformation patterns. They reference third‑party Python libraries (numpy, pandas, pympler, memory_profiler, ijson, scipy) but the skill declares no install requirements — callers will need those libraries available to run examples. The document does not instruct reading unrelated system secrets or exfiltrating data.

Install Mechanism

No install spec and no code files — instruction-only skill. This minimizes installation risk because nothing is downloaded or written by the skill itself.

Credentials

No environment variables, credentials, or config paths are requested. The recommended techniques do not require secret access and the skill does not attempt to access unrelated system configuration.

Persistence & Privilege

always is false and the skill does not request persistent or elevated platform privileges. It does not modify other skills or system-wide agent settings.

Assessment

This skill is an advisory guide — it contains safe, standard suggestions for reducing Python memory usage. Before using it: ensure the required Python libraries (numpy, pandas, pympler, memory_profiler, ijson, scipy, etc.) are installed from trusted package sources; run transformed code in a test environment (verify correctness and performance); be cautious about operating on sensitive files (processing large local files may expose sensitive data if output is shared); and review any automatic code changes the agent proposes before applying them. The skill itself doesn't request credentials or download code, but following its examples may require installing third‑party packages and reading local data files.

Memory Optimization Skill

Transform Python code to minimize memory usage while maintaining functionality.

Workflow

1. Profile to identify memory bottlenecks (largest allocations, leak patterns)
2. Analyze data structures and object lifecycles
3. Select optimization strategies based on access patterns
4. Transform code with memory-efficient alternatives
5. Verify memory reduction without correctness loss

Memory Optimization Decision Tree

What's consuming memory?

Large collections:
├── List of objects → __slots__, namedtuple, or dataclass(slots=True)
├── List built all at once → Generator/iterator pattern
├── Storing strings → String interning, categorical encoding
└── Numeric data → NumPy arrays instead of lists

Data processing:
├── Loading full file → Chunked reading, memory-mapped files
├── Intermediate copies → In-place operations, views
├── Keeping processed data → Process-and-discard pattern
└── DataFrame operations → Downcast dtypes, sparse arrays

Object lifecycle:
├── Objects never freed → Check circular refs, use weakref
├── Cache growing unbounded → LRU cache with maxsize
├── Global accumulation → Explicit cleanup, context managers
└── Large temporary objects → Delete explicitly, gc.collect()

Transformation Patterns

Pattern 1: Class to slots

Reduces per-instance memory by 40-60%:

Before:

class Point:
    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z

After:

class Point:
    __slots__ = ('x', 'y', 'z')

    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z

Pattern 2: List to Generator

Avoid materializing entire sequences:

Before:

def get_all_records(files):
    records = []
    for f in files:
        records.extend(parse_file(f))
    return records

all_data = get_all_records(files)
for record in all_data:
    process(record)

After:

def get_all_records(files):
    for f in files:
        yield from parse_file(f)

for record in get_all_records(files):
    process(record)

Pattern 3: Downcast Numeric Types

Reduce NumPy/Pandas memory by 2-8x:

Before:

df = pd.read_csv('data.csv')  # Default int64, float64

After:

def optimize_dtypes(df):
    for col in df.select_dtypes(include=['int']):
        df[col] = pd.to_numeric(df[col], downcast='integer')
    for col in df.select_dtypes(include=['float']):
        df[col] = pd.to_numeric(df[col], downcast='float')
    return df

df = optimize_dtypes(pd.read_csv('data.csv'))

Pattern 4: String Deduplication

For repeated strings:

Before:

records = [{'status': 'active', 'type': 'user'} for _ in range(1000000)]

After:

import sys

STATUS_ACTIVE = sys.intern('active')
TYPE_USER = sys.intern('user')

records = [{'status': STATUS_ACTIVE, 'type': TYPE_USER} for _ in range(1000000)]

Or with Pandas:

df['status'] = df['status'].astype('category')

Pattern 5: Memory-Mapped File Processing

Process files larger than RAM:

import mmap
import numpy as np

# For binary data
with open('large_file.bin', 'rb') as f:
    mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
    # Process chunks without loading entire file

# For NumPy arrays
arr = np.memmap('large_array.dat', dtype='float32', mode='r', shape=(1000000, 100))

Pattern 6: Chunked DataFrame Processing

def process_large_csv(filepath, chunksize=10000):
    results = []
    for chunk in pd.read_csv(filepath, chunksize=chunksize):
        result = process_chunk(chunk)
        results.append(result)
        del chunk  # Explicit cleanup
    return pd.concat(results)

Data Structure Memory Comparison

Structure	Memory per item	Use case
list of dict	~400+ bytes	Flexible, small datasets
list of class	~300 bytes	Object-oriented, small
list of __slots__ class	~120 bytes	Many similar objects
namedtuple	~80 bytes	Immutable records
numpy.ndarray	8 bytes (float64)	Numeric, vectorized ops
pandas.DataFrame	~10-50 bytes/cell	Tabular, analysis

Memory Leak Detection

Common leak patterns and fixes:

Pattern	Cause	Fix
Growing cache	No eviction policy	@lru_cache(maxsize=1000)
Event listeners	Not unregistered	Weak references or explicit removal
Circular references	Objects reference each other	weakref, break cycles
Global lists	Append without cleanup	Bounded deque, periodic clear
Closures	Capture large objects	Capture only needed values

Profiling Commands

# Object size
import sys
sys.getsizeof(obj)  # Shallow size only

# Deep size with pympler
from pympler import asizeof
asizeof.asizeof(obj)  # Includes referenced objects

# Memory profiler decorator
from memory_profiler import profile
@profile
def my_function():
    pass

# Tracemalloc for allocation tracking
import tracemalloc
tracemalloc.start()
# ... code ...
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')

Verification Checklist

Before finalizing optimized code:

• Memory usage reduced (measure with profiler)
• Functionality preserved (same outputs)
• No new memory leaks introduced
• Performance acceptable (generators may add iteration overhead)
• Code remains readable and maintainable