AWS Lambda has a maximum execution time of 15 minutes , which makes batch-style processing unsuitable as workloads scale. AWS best practices recommend using event-driven architectures to process individual objects independently and in parallel.

Amazon S3 event notifications invoke Lambda automatically whenever a new object is uploaded. This allows each file to be processed in its own execution, eliminating long-running batch jobs and enabling horizontal scaling. AWS Lambda automatically scales concurrency based on event volume, ensuring faster processing as uploads increase.

To prevent duplicate processing, AWS documentation recommends using an idempotency mechanism . Storing processed file identifiers or transaction IDs in Amazon DynamoDB provides a highly available, low-latency way to track processed files. DynamoDB’s conditional writes can enforce exactly-once processing semantics.

Options B, C, and D either increase operational overhead, reintroduce batching issues, or rely on unreliable polling mechanisms that can miss or reprocess files.

Thus, using S3 events with DynamoDB tracking is the correct and scalable solution.