Speeding Up Downloads Data with Multithreading

Generating outbound reports in a Warehouse Management System can become extremely heavy—especially when dealing with thousands to tens of thousands of IMEIs, serial numbers, or phone numbers within a single week.
If you’ve ever experienced slow report downloads due to complex data structures, this article is for you.

🧱 Outbound Data Structure

Before jumping into the solution, let’s first understand the data structure:

• 1 outbound contains many shipments / packages
• 1 shipment contains many products
• 1 product can have multiple IMEIs / serial numbers / phone numbers

When a single week contains hundreds of shipments and thousands of products, the total number of IMEIs can easily reach tens of thousands.
No wonder the report generation becomes slow—especially if everything is processed linearly (single-threaded).

🧠 Solution: Fetch IMEIs in Parallel

To overcome this bottleneck, we can use a multithreading approach in Spring Boot, specifically by combining:

• CompletableFuture.supplyAsync(...)
• ExecutorService with a fixed thread pool

With this approach, we can fetch data per shipment (resi) in parallel.
The result? A significantly faster report generation process—without changing the database structure.

💻 Step-by-Step Implementation

1. Prepare the shipment awb list

JAVA
copy
List<String> awbList = List.of("ABC", "DEF", "GHI", "JKL", "MNO");

2. Create a fixed thread pool

JAVA
copy
ExecutorService executor = Executors.newFixedThreadPool(awbList.size());

Or limit the maximum number of threads:

JAVA
copy
int maxThread = Math.min(5, awbList.size());
ExecutorService executor = Executors.newFixedThreadPool(maxThread);

3. Run parallel fetch using CompletableFuture

JAVA
copy
List<CompletableFuture<Map.Entry<String, List<String>>>> futures = awbList
    .stream()
    .map(awb -> CompletableFuture.supplyAsync(() -> {
        List<String> imeis = fetchImeiByAwb(awb);
        return Map.entry(awb, imeis);
    }, executor))
    .collect(Collectors.toList());

4. Wait for all tasks to complete

JAVA
copy
CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();

5. Merge all results

JAVA
copy
Map<String, List<String>> result = new HashMap<>();
for (CompletableFuture<Map.Entry<String, List<String>>> future : futures) {
    Map.Entry<String, List<String>> entry = future.join();
    result.put(entry.getKey(), entry.getValue());
}

6. Shut down the thread pool

JAVA
copy
executor.shutdown();

✨ Final Result

With this approach:

• Report downloads are significantly faster
• No need for complex database configuration or heavy queries
• Reusable for other parallel fetching use cases (products, invoices, deliveries, etc.)

⚠️ Implementation Tips

• 🔄 Pass IDs, not entities
  Never pass JPA entities into threads—only pass IDs or codes
• 🧵 Thread count
  Use a thread pool of 5–10 threads for stability
• 🐢 Avoid overload
  Don’t fetch all shipments at once—consider batching
• 💥 Handle errors
  Use .exceptionally() or proper error logging in supplyAsync
• 📄 Large output?
  Consider generating CSV/Excel files first, then provide a download link

📌 Example fetchImeiByAwb(...) Method

JAVA
copy
public List<String> fetchImeiByAwb(String awb) {
    return imeiRepository.findByAwb(awb)
                         .stream()
                         .map(Imei::getNumber)
                         .toList();
}

🔚 Conclusion

With this simple multithreading strategy, outbound report generation—previously slow and heavy—can now be executed faster and more efficiently.
This approach is flexible, scalable, and well-suited for warehouse systems with large and complex data structures like WMS.

If you’re building a similar system, don’t hesitate to apply this pattern in your report services.

Happy coding 🚀