Streaming DynamoDB Data with Scanamo, Cats Effect, and FS2-The Memory-Safe Way

"2. ⚙️ Setting Up the DynamoDB Client Let's start with a simple, safe client setup using Cats Effect's Resource: import cats.effect.{IO, Resource}import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClientimport software.amazon.awssdk.regions.Regionimport org.scanamo.{ScanamoCats, Table}import org.scanamo.generic.auto._ val clientResource: Resource[IO, DynamoDbAsyncClient] =def createScanamoClient(client: DynamoDbAsyncClient): ScanamoCats[IO] = Resource.fromAutoCloseable( ScanamoCats[IO](client) IO( DynamoDbAsyncClient.builder() .region(Region.US_EAST_1) .build() ) ) Resource ensures the DynamoDB client is safely acquired and released - no leaks, no dangling threads. 📋 Example Setup For these examples, we'll use a simple DynamoDB table with the following structure: case class User(id: String, name: String, age: Int) DynamoDB Table: "users""

"DynamoDB offers two primary ways to retrieve data: Query for targeted lookups using partition keys or GSIs, and Scan for processing entire tables. Scanamo provides queryPaginatedM and scanPaginatedM functions that support pagination, allowing us to stream large datasets efficiently without loading everything into memory. 3.1 Streaming with Query Operations Let's stream users by tenant using queryPaginatedM. This method lets us fetch results page by page, lazily."

Use Cats Effect Resource to create and manage a DynamoDbAsyncClient safely, ensuring proper acquisition and release of resources. Define a simple User case class and a DynamoDB table keyed by tenant id to store millions of users partitioned by tenant. Use Scanamo's queryPaginatedM for paginated queries and scanPaginatedM for paginated scans to stream large datasets without loading everything into memory. Query operations require the queried attribute to be the partition key or a GSI. Use FS2 Streams to flatten pages into individual records or to process page-by-page as needed for backpressure and memory efficiency.

Read at Medium