Cutting distributed processing costs by 90%

Problem

The distributed photogrammetry pipeline handled workflows of 50,000+ high-resolution images, and cost grew in step with usage. Worse, the largest jobs, 50GB+ orthomosaic processing, were failing outright in production, which meant the biggest customers hit the worst reliability.

Constraints

• Existing pipelines depended on EFS as a caching layer, which dominated both cost and I/O contention.
• Rewrites had to land without interrupting customer processing already in flight.
• Throughput could not regress in exchange for the cost saving.

What I built

I redesigned the distributed synchronization pipelines around multiprocessing, threaded workers, and direct S3 streaming, eliminating the EFS caching layer entirely rather than trying to tune it. Data moves straight between S3 and the workers, so the intermediate storage tier and its cost disappear together.

• Direct S3 streaming replacing EFS-backed caching.
• Multiprocessing and threaded worker orchestration for parallel throughput.
• Resolved the production failures affecting 50GB+ orthomosaic processing and terrain generation.

Result

AWS terrain-processing costs dropped by 90%, the 50GB+ job class stopped failing, and throughput, scalability, and operational reliability all improved across the production system.