fromStreamHacker
19 hours agoPython Async Gather in Batches
Python's asyncio.gather function is great for I/O bound parallel processing. There's a simple utility function I like to use that I call gather_in_batches: async def gather_in_batches(tasks, batch_size=100, return_exceptions=False): for i in range(0, len(tasks), batch_size): batch = tasks[i:i+batch_size] for result in await asyncio.gather(*batch, return_exceptions=return_exceptions): yield result
Python