Building on AWS, Part 2: Event-driven processing

In part 1, I got a working presigned upload flow: a client hits an API Gateway endpoint, gets back a presigned S3 URL, and uploads a file directly to the bucket. The full chain was rebuilt with CloudFormation. Here’s the full pipeline I’m working towards:

This time, I’m making the pipeline reactive. When a file lands in S3, something should happen. That “something” is: S3 sends a notification to an SQS queue, a Lambda picks up the message, and the job gets recorded in DynamoDB.

What I built

I wasn’t able to do the teardown-and-rebuild-with-CloudFormation step this session. Schedule got tight and I wanted to keep moving forward with the new services. Everything here was built manually through the console, but I’m doing the CloudFormation rebuild before heading into Session 3.

Presigned uploads (revisited)

The presigned upload flow from Part 1 needed a couple of fixes before I could move on. The endpoint was working, but actually uploading files through it was a different story. I spent way more time here than I expected, which I’ll get into in the gotchas section below.

I also added a manual content type mapping for audio formats, since Python’s mimetypes module doesn’t handle all of them reliably. More on that later too.

SQS: the message queue

S3 can send event notifications to a few destinations: Lambda, SNS, or SQS. I went with SQS because that’s what the pipeline uses at work, and it gives you built-in retry and dead-letter queue support.

I created two queues: a main queue (MeetingNotesQueue) and a dead-letter queue (MeetingNotesDLQ). The DLQ catches messages that fail processing repeatedly. If the processing Lambda crashes three times on the same message, SQS moves it to the DLQ instead of retrying forever. You don’t want a corrupted audio file burning Lambda invocations in an infinite loop.

Setting up the S3 event notification required a queue policy. Same pattern as AWS::Lambda::Permission from Part 1: the resource itself has to explicitly say “I accept messages from this source.” S3 can’t just write to any queue.

{
  "Effect": "Allow",
  "Principal": { "Service": "s3.amazonaws.com" },
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:ap-southeast-1:ACCOUNT_ID:MeetingNotesQueue",
  "Condition": {
    "ArnEquals": {
      "aws:SourceArn": "arn:aws:s3:::your-upload-bucket"
    }
  }
}

Without this policy, S3 refuses to save the event notification configuration entirely. It validates permissions upfront, which is actually nice compared to things that fail silently at runtime.

DynamoDB: the job tracker

I needed somewhere to record job status. The obvious choice was Postgres (I know it, I love it), but DynamoDB is the better fit here for one reason: the connection model.

With RDS Postgres, your Lambda needs VPC configuration, private subnets, NAT gateways, and connection pooling (Lambda spins up many concurrent instances, each opening its own connection, and Postgres has a connection limit). That’s a lot of infrastructure just to write a row.

DynamoDB is an HTTP API call. boto3.resource('dynamodb').Table('MyTable').put_item(...) and you’re done. No connections, no VPC, no cold start penalty.

The tradeoff is the data model. DynamoDB is a key-value store. You need to know your access patterns upfront. “Get job status by job ID” is perfect. “Show me all jobs from last week sorted by adviser name” is painful. For job status tracking, key-value is the right tool.

I created a simple table with job_id as the partition key, no sort key. On-demand capacity mode so I don’t have to think about provisioned throughput. Getting this wired up was straightforward, though I did run into some IAM and API quirks that I’ll cover below.

The processing Lambda

This Lambda gets triggered by SQS. When a message arrives, it parses out the S3 bucket and key from the event, then writes a job record to DynamoDB with an initial status of UPLOADED. Simple for now, but this is the function that will eventually kick off the Step Functions state machine in Session 3.

What tripped me up

This is where most of my time actually went. If you’re building something similar, these might save you a few hours.

The presigned URL saga

I had the presigned URL endpoint working. I knew it worked because I built it in Part 1. But when I tried to actually upload a file using the URL, I kept getting AccessDenied from S3.

I went down a deep rabbit hole:

• Checked IAM policies (they were fine)
• Switched from Signature V2 to V4 (not the issue)
• Added AmazonS3FullAccess to the role (still failed)
• Tested put_object directly from the Lambda (it worked)
• Set endpoint_url explicitly on the boto3 client (made things worse)
• Enabled S3 server access logging (didn’t need it in the end)

So the Lambda could write to S3 directly, but the presigned URL it generated didn’t work. That made zero sense. The URL is signed with the same credentials.

Turns out it was two things, both outside AWS:

Gotcha #1: zsh auto-escapes pasted URLs. I use Oh My Zsh, and when I pasted a presigned URL into the terminal, zsh silently added backslashes before ?, &, and = characters. Inside double quotes, those backslashes are preserved literally and sent as part of the URL. The signature breaks because the URL S3 receives doesn’t match what was signed.

The fix is adding DISABLE_MAGIC_FUNCTIONS=true to your .zshrc.

Gotcha #2: Bruno double-encodes presigned URLs. Even after fixing the terminal issue, uploads from Bruno (the API client I use) still failed with AuthorizationQueryParametersError. Presigned URLs already contain percent-encoded values like %2F. Bruno re-encodes those into %252F, which completely breaks the signature.

This is a known open issue. The fix is to disable URL encoding in Bruno’s request settings for presigned URL requests. There’s a toggle under the request’s settings tab that says “URL Encoding.” Turn it off.

I lost a solid chunk of time on this. The Lambda, IAM role, and S3 configuration had been correct the entire time. The lesson: when your server-side code seems right but things still fail, check your client tooling before going deeper into AWS debugging.

Content type detection

My handler was deriving the content type from the filename using Python’s mimetypes module:

import mimetypes

content_type, _ = mimetypes.guess_type(filename)

This works for most formats, but mimetypes doesn’t know about .m4a files, especially on Lambda’s minimal Linux environment. It returned None, fell through to application/octet-stream, and the presigned URL was signed with the wrong content type.

The fix is a manual mapping for audio formats:

CONTENT_TYPES = {
    ".webm": "audio/webm",
    ".m4a": "audio/mp4",
    ".mp3": "audio/mpeg",
    ".wav": "audio/wav",
    ".ogg": "audio/ogg",
    ".flac": "audio/flac",
}

ext = os.path.splitext(filename)[1].lower()
content_type = CONTENT_TYPES.get(ext, "application/octet-stream")

It’s kind of hacky, but it was the quickest way around something that wasn’t supposed to be in my problem scope. I just needed content types to work so I could move on to the actual pipeline stuff.

The double Records

When I first looked at the event my processing Lambda received, I was confused. There’s a Records array inside a Records array:

def lambda_handler(event, context):
    for sqs_record in event["Records"]:           # SQS messages
        s3_event = json.loads(sqs_record["body"])
        for s3_record in s3_event["Records"]:      # S3 events
            bucket = s3_record["s3"]["bucket"]["name"]
            key = s3_record["s3"]["object"]["key"]

The outer Records comes from SQS (it can batch up to 10 messages per Lambda invocation). The inner Records comes from S3 (it batches event notifications, though in practice it’s almost always just one). Both services independently decided to call their top-level array Records. Classic AWS.

DynamoDB’s low-level client

DynamoDB’s low-level client (boto3.client('dynamodb')) requires you to annotate every value with its type:

Item={
    "job_id": {"S": "some-id"},
    "status": {"S": "UPLOADED"},
}

"S" for string, "N" for number, and so on. This is tedious. The higher-level resource interface (boto3.resource('dynamodb')) handles the type annotations for you and feels more like a normal ORM. Use that.

CloudWatch logging quirks

Two small things about CloudWatch that aren’t obvious:

Lambda doesn’t create a new log stream per invocation. It creates one per execution environment. When Lambda reuses a warm container (which it does most of the time), new logs get appended to the same stream. So don’t look at log stream creation timestamps to find your latest logs. Open the most recent stream by last event time and scroll to the bottom.

Testing from the Lambda console vs API Gateway produces different events. When you test from the console, you’re sending a raw JSON event directly to the handler. When a request comes through API Gateway with Lambda Proxy Integration enabled, the event has a completely different structure (with httpMethod, body as a JSON string, headers, etc.). Early on I was confused why my handler worked in the console but not through the API. It was because the console test event didn’t match what API Gateway actually sends.

IAM credential caching

A recurring theme with IAM: the error messages tell you what was denied, but not why. I got AccessDeniedException on dynamodb:PutItem even though my inline policy clearly had dynamodb:PutItem in the actions list. After staring at it for a while, I realized the policy’s resource ARN pointed to table/MeetingNotesQueue instead of table/MeetingNotesJobs. I had copy-pasted the wrong name.

After fixing the ARN, the error persisted. This was another instance of Lambda caching credentials from a warm execution environment. IAM policy changes take effect immediately, but the Lambda container holds onto the old credentials until it gets recycled. Adding a dummy environment variable to the Lambda forces a cold start and picks up the new credentials.

This came up multiple times during Session 2. If you change an IAM policy and the Lambda still gets denied, force a cold start before assuming something else is wrong.

What I learned

Debug your tools before debugging your infrastructure. I spent hours investigating IAM policies, signature versions, and S3 bucket configurations when the problem was my terminal and API client mangling URLs. If the server-side logic works in isolation (like put_object succeeding directly), the issue is probably on the client side.

The “resource needs to allow access” pattern is everywhere. S3 bucket policies, SQS queue policies, Lambda resource-based policies. It’s not enough for the caller to have permission; the target often has to explicitly allow the caller too. This is consistent across AWS, and once you internalize it, you stop being surprised by AccessDenied errors.

Lambda credential caching is a real debugging trap. When you update an IAM policy and the Lambda still fails, your instinct is to keep investigating the policy. But it might already be correct. Force a cold start.

DynamoDB is great when you know your access patterns. For job status tracking, it’s the right tool. Zero infrastructure overhead, no connections to manage. The moment you need ad-hoc queries or relationships, reach for Postgres.

What’s next

Session 3 is where the pipeline gets a brain. Right now, uploading a file creates a job record and that’s it. Next, I’m adding Step Functions to orchestrate the full processing pipeline: call AWS Transcribe, send the transcript to an LLM for summarization, format the result into a structured note, and notify the user via SNS. Each step gets retry logic and failure handling.

For the summarization step, I’m tentatively planning to use Claude, but I’m still looking into cheaper options. Running an LLM call on every meeting upload could add up fast, so I want to figure out the right balance of cost and quality before committing to a provider.