The State of Serverless: Choosing the Right Storage Solution

While this stateless nature makes serverless functions incredibly scalable and cost-effective, it creates a fundamental challenge: where do you store your data?

I recently dove deep into this question for a university research project, exploring different storage strategies for serverless architectures. Here's what I learned about managing state in a stateless world.

The Serverless Storage Dilemma

Traditional applications can store session data in memory, cache frequently accessed data locally, or maintain database connections across requests. Serverless functions? Not so much. Each invocation is isolated, ephemeral, and can disappear at any moment.

This means every piece of data your application needs to remember—user sessions, processing state, temporary files—must live somewhere external to your function. The question is: where?

Real-World Scenarios: When State Matters

Let me walk you through two common scenarios where storage decisions can make or break your serverless application.

Scenario 1: HTTP User Sessions

Imagine you're building a web application with serverless functions. Users log in, browse around, maybe add items to a shopping cart. In a traditional setup, you'd store this session data in memory or local files. But in serverless? You need a different approach.

Here are your main options:

Relational Database Storage Store session data in your existing MySQL or PostgreSQL database. It's straightforward—if you're already using a database, why not put sessions there too? But watch out for connection limits. Traditional databases weren't designed for the burst traffic patterns of serverless functions.

Pro tip: If you're on AWS, RDS Proxy can help manage those connections. Or consider DynamoDB, which scales automatically without connection limits.

Encrypted Cookies (JWT/JWE) Why store session data on the server at all? Encrypt it and send it to the client as a cookie. The client sends it back with every request, and your function decrypts it. No server storage needed!

The catch? Cookies have a 4KB size limit. This works great for storing a user ID or basic preferences, but forget about storing a complex shopping cart.

Cache (Redis) Redis is purpose-built for this. Fast key-value lookups, support for thousands of concurrent connections, and built-in expiration for sessions. It's probably your best bet for most session storage needs.

Scenario 2: Image Processing Pipeline

Here's another common use case: users upload images, and you want to process them through multiple steps—resize, optimize, generate thumbnails, maybe apply some AI magic. Each step might be a separate function.

How do you pass image data between functions?

S3/Object Storage The obvious choice. Upload the original image to S3, have each function read from and write back to S3. It's reliable, scalable, and designed for this exact purpose.

Database BLOBs You could store images as binary data in your database, but this is usually a bad idea. Databases aren't optimized for large binary files, and you'll hit size and performance limits quickly.

AWS Step Functions AWS's solution for orchestrating multi-step workflows. Instead of storing the actual image data, Step Functions can pass metadata between functions—file paths, processing parameters, status information. It's elegant but ties you to AWS.

The Trade-offs: Performance vs. Flexibility

Every storage solution comes with trade-offs. Here's what I found matters most:

Performance Considerations

• Latency: How long does it take to read/write your data? Redis is blazing fast, databases can vary, and S3 has network overhead.
• Throughput: How many concurrent operations can your storage handle? This varies wildly between solutions.
• Cold starts: If your storage requires connection setup, it can add significant latency to cold function starts.

Flexibility Factors

• Data types: What kind of data can you store? Key-value stores are simple but limited. Databases offer complex queries. Object storage handles any file type.
• Size limits: Cookies are tiny (4KB), databases have row size limits, object storage can handle massive files.
• Query capabilities: Sometimes you need to search or aggregate your data, not just retrieve it by key.

What I Wish I'd Known Earlier

If I were starting a new serverless project today, here's my practical advice:

Start with the simplest solution that works. Don't over-engineer from day one. If encrypted cookies can handle your session needs, start there. You can always migrate later.

Consider your scaling patterns. Will you have steady traffic or sudden bursts? Different storage solutions handle these patterns differently.

Think about vendor lock-in. AWS Step Functions is powerful but ties you to AWS. Redis can run anywhere.

Budget for storage costs. Serverless functions might be cheap, but storage costs can add up, especially for frequently accessed data.

The Reality Check

Here's the honest truth: my research project didn't go as planned. Time constraints meant I couldn't implement and test all these solutions like I wanted to. But sometimes the journey teaches you as much as the destination.

What I learned is that storage decisions in serverless aren't just technical—they're strategic. The "best" solution depends on your team's expertise, your scalability needs, your budget, and how much vendor lock-in you're comfortable with.

Looking Forward

Serverless is still evolving rapidly. New storage solutions appear regularly, and existing ones keep getting better. The fundamental challenge—managing state in stateless functions—isn't going away, but our tools for handling it are improving.

The key is to understand your options, prototype early, and be prepared to adapt as your application grows. Don't let perfect be the enemy of good—pick a solution that works for your current needs and iterate from there.

Key Takeaways

1. Serverless functions are stateless by design—embrace it, don't fight it
2. Choose storage based on your specific use case—sessions, files, and structured data have different optimal solutions
3. Performance and flexibility often trade off—know which matters more for your application
4. Start simple and evolve—you can always migrate to more sophisticated solutions later
5. Consider the total cost—not just storage, but development time and operational complexity

The serverless revolution is real, but it requires thinking differently about data persistence. Once you master that mindset shift, you'll find serverless applications can be incredibly powerful and cost-effective.

What storage challenges have you faced in your serverless projects? I'd love to hear about your experiences in the comments below.

---

This post is based on research conducted for a university project on serverless architectures. While the full experimental evaluation wasn't completed, the framework and insights should be valuable for anyone working with serverless storage decisions.