Data Processing

Data loaders are critical yet understudied components of production ML systems. In ML engineering interviews, candidates must demonstrate practical mastery of data loading systems that handle real-world constraints like scale, training resumption, and complex sampling. This guide presents implementation patterns specifically tested in system design interviews, with complexity analysis and production considerations for each approach.

Core Knowledge

Core Concepts

Sampling

• Class imbalance solutions:

• Essential sampling techniques:

• Probability distributions in practice:

Performance & Reliability

• Critical state management:

Advanced Patterns

• Distributed data loading:

• Hybrid sampling approaches:

Key Questions

Status	Question	Category
	Memory-Mapped Dataset Iterator for Large Files	Data Processing
	Stratified Sampling for Training Data Iterator	Data Processing
	Implement Training Data Iterator with Class Balancing	Data Processing
	Implement Training Data Iterator with Weighted Sampling	Data Processing
	Implement a Resumable Training Data Iterator	Data Processing

Common Pitfalls

Extended Questions

Status	Question	Category
	Implement Prefetching with Parallel Loading in DataLoader	Parallel Data Loading
	Distributed Data Loading with Shard Synchronization	Distributed Data Loading
	Implement Streaming Shuffle Buffer for Online Learning	Streaming

Data Processing

Core Knowledge

Core Concepts

Sampling

Performance & Reliability

Advanced Patterns

Key Questions

Common Pitfalls

Forgetting to shuffle before splitting (data ordering bias)

Stratification errors with rare classes

Improper seeding leading to non-reproducible splits

Memory leaks from unoptimized batch loading

Naive weighted sampling with O(n) complexity

GPU underutilization due to poor prefetching

Incorrect state serialization in resumeable iterators

Failing to handle remaining samples in partial batches

Incorrect handling of dataset versions when resuming

Extended Questions

Framework-specific APIs for DataLoaders

Core Knowledge

Core Concepts

• Iterator protocol (iter/next)

• Batch processing mechanics (memory management, partial batches)

• Memory tradeoffs: lazy vs eager loading

• Streaming data handling

• Dataset versioning for continuous training

Sampling

◦ Oversampling (SMOTE, ADASYN)

◦ Undersampling (Tomek links, cluster-based)

◦ Hybrid approaches

◦ With/without replacement

◦ Weighted sampling (alias method)

◦ Stratified sampling (distribution preservation)

◦ Reservoir sampling (streaming)

◦ Dirichlet allocation for splits

• Stratification edge cases (empty classes, multi-label)

Performance & Reliability

• Efficient shuffling patterns

• Memory optimization (mmap, lazy transforms)

• Prefetching (parallel vs pipelined)

◦ RNG states (reproducibility)

◦ Checkpointing (resumeable training)

Advanced Patterns

◦ Sharding strategies

◦ Node-aware sampling

◦ Cross-worker synchronization

◦ Weighted stratified sampling

◦ Time-aware balancing

• Resampling methods (bootstrap, jackknife)

• Statistical validation during loading

Key Questions

Common Pitfalls

Forgetting to shuffle before splitting (data ordering bias)

Stratification errors with rare classes

Improper seeding leading to non-reproducible splits

Memory leaks from unoptimized batch loading

Naive weighted sampling with O(n) complexity

GPU underutilization due to poor prefetching

Incorrect state serialization in resumeable iterators

Failing to handle remaining samples in partial batches

Incorrect handling of dataset versions when resuming

Extended Questions

Framework-specific APIs for DataLoaders