Parallel Processing

Speed up KAST with multi-core parallel processing — 5-10x faster on large datasets!

What Is It?

Parallel processing distributes work across multiple CPU cores simultaneously, instead of using just one core.

Performance Gains:

Small datasets (< 10K molecules): Minimal improvement
Medium datasets (10K-100K): 3-5x faster
Large datasets (100K+): 5-10x faster

Quick Enable

Edit settings.py (Section 12):

# Section 12: PARALLEL PROCESSING CONFIGURATIONS

ENABLE_PARALLEL_PROCESSING = True    # On/Off
N_WORKERS = None                     # None = auto-detect (RECOMMENDED)
PARALLEL_BATCH_SIZE = 100000         # Molecules per batch
PARALLEL_MIN_THRESHOLD = 10000       # Min dataset size for parallel

Or configure interactively:

python main.py
→ [8] Advanced Options
→ [3] Configure CPU Cores
→ Choose [0] auto-detect or specific number

N_WORKERS Options

Setting	What Happens	Best For
`N_WORKERS = None`	Auto-detect optimal cores (N-1)	✅ RECOMMENDED
`N_WORKERS = -1`	Use ALL available cores	High-performance systems
`N_WORKERS = 4`	Use exactly 4 cores	Limited RAM (8GB)
`N_WORKERS = 1`	Single-core only (disable parallel)	Debugging

Memory Settings

Adjust PARALLEL_BATCH_SIZE based on available RAM:

RAM Available	Batch Size	N_WORKERS
4GB	25,000	2
8GB	50,000	4
16GB	100,000	6
32GB+	200,000	8+

Formula: (Available RAM in GB - 2) × 10,000 ≈ good batch size

Scripts That Support Parallel

Script	Speedup	When Enabled
`2_featurization.py`	5-10x	Dataset > 10K molecules
`4_1_cross_validation.py`	4-8x	Dataset > 10K molecules
`4_3_tanimoto_similarity.py`	3-5x	Always (if enabled)
`4_4_learning_curve.py`	4-8x	Always (if enabled)
`5_0_featurize_for_prediction.py`	5-10x	Dataset > 10K molecules
`5_1_run_prediction.py`	3-5x	Dataset > 10K molecules

Configuration Examples

Example 1: Small Dataset (8GB RAM)

ENABLE_PARALLEL_PROCESSING = True
N_WORKERS = 4                       # Use 4 cores
PARALLEL_BATCH_SIZE = 50000
PARALLEL_MIN_THRESHOLD = 10000      # Activate for 10K+ molecules

Example 2: Large Dataset (16GB+ RAM)

ENABLE_PARALLEL_PROCESSING = True
N_WORKERS = None                    # Auto-detect
PARALLEL_BATCH_SIZE = 100000        # Default, good for 16GB+
PARALLEL_MIN_THRESHOLD = 10000

Example 3: High-Performance (32GB+ RAM)

ENABLE_PARALLEL_PROCESSING = True
N_WORKERS = -1                      # Use ALL cores
PARALLEL_BATCH_SIZE = 200000
PARALLEL_MIN_THRESHOLD = 5000       # Activate earlier

Example 4: Debugging (Single Core)

ENABLE_PARALLEL_PROCESSING = False  # Or N_WORKERS = 1
PARALLEL_BATCH_SIZE = 100000
PARALLEL_MIN_THRESHOLD = 10000

Monitor Performance

During execution, KAST shows:

Parallel Processing ENABLED (6 workers)
Processing 150,000 molecules in batches of 100,000...
Batch 1/2: [████████████████] 100% 2m 15s
Batch 2/2: [████████████████] 100% 2m 10s
Total time: 4m 25s (would be 30m single-core!)

Troubleshooting

“Out of Memory” error

Cause: Batch size too large for your RAM
Fix: Reduce PARALLEL_BATCH_SIZE (try 50,000 or 25,000)

“Only using 1 core”

Cause: Dataset smaller than PARALLEL_MIN_THRESHOLD
Fix: Either use larger dataset or lower PARALLEL_MIN_THRESHOLD

“Slower than expected”

Cause: Too many workers for available cores
Fix: Set N_WORKERS to your actual core count (not more)

“Can’t find joblib”

Cause: Dependency not installed
Fix: pip install joblib or re-run setup.exe/setup.sh

Test Parallel Setup

Verify parallel processing works:

python main.py
→ [8] Advanced Options
→ [2] Test Parallel Processing

Should show 6 tests completing successfully:

TEST 1: Import settings.py... [OK]
TEST 2: Configuration Variables... [OK]
TEST 3: Required Imports... [OK]
TEST 4: Worker Optimization Logic... [OK]
TEST 5: Script Compatibility... [OK]
TEST 6: Parallel Threshold Logic... [OK]
✅ ALL TESTS COMPLETED SUCCESSFULLY!