The AWS Data Ecosystem Grand Tour - Introduction
Written by Alex Rasmussen on December 2, 2019
This article is part of a series. Here are the rest of the articles in that series:
- Introduction
- Where Your AWS Data Lives
- Block Storage
- Object Storage
- Relational Databases
- Data Warehouses
- Data Lakes
- Key/Value and "NoSQL" Stores
- Graph Databases
- Time-Series Databases
- Ledger Databases
- SQL on S3 and Federated Queries
- Search
- Streaming Data
- File Systems
- Data Ingestion
- ETL
- Processing
- Data Interfaces
- Training Data for Machine Learning
- Data Security
- Business Intelligence
The software world is focused heavily on cloud computing, and Amazon Web Services (AWS) is the clear public cloud leader. Many of the software products that you use everyday are running on infrastructure inside of one of AWS's massive network of data centers.
When AWS publicly launched in 2006, it had just two services - Elastic Compute Cloud (EC2) for running virtual machines, and Simple Storage Service (S3) for large-scale storage. Today, AWS has more than 160 services. That number is certain to grow with re:Invent, AWS's flagship conference, happening in Las Vegas this week.
It's really hard to keep track of 160+ services. If you're building a software system in the AWS cloud today, it's also hard to know which services to use or what the trade-offs between the various services are. As a consultant, I'm asked to weigh in on these kinds of decisions a lot, so I need to have as broad of an understanding of the AWS ecosystem as I can.
When researching various AWS offerings as part of my work with clients, I found that there were a lot of how-to guides, tutorials and marketing materials for individual services, but little that tied those services together and anchored them in a broader shared context in a relatively succinct way. I decided to try to fix that.
Writing a guide that covers every AWS service would be an enormous undertaking. It would be hard for me to write, harder for you to read, and would be outdated the second I finished writing it. However, focusing on one facet of their portfolio - systems for storing and analyzing data - seemed both achievable and solidly in my wheelhouse, so that's what we're going to be doing here.
I hope that you'll come away from this series with a broader understanding of AWS's existing data offerings, and a framework for how to think about where new services fit in.
What We'll Be Covering
This series of articles is going to focus on AWS services whose primary responsibility is the storage and analysis of data. That's still a pretty broad umbrella, but it reduces the list of services under consideration from 160+ down to about 50.
Since 50 services is still quite a lot, I won't dive incredibly deeply into any of them in this series. My hope is that you'll come away from each article with a broad understanding of what the service is, what it's used for, how it differs from other similar services, how it relates to other parts of the AWS ecosystem.
Another big thing we'll be talking about for each service is how it's priced. However, we won't be talking about pricing in concrete terms. Concrete pricing in AWS depends on a lot of different factors, and focusing on how they're defined now would make this series age rather poorly as the particulars of pricing change. Instead, we'll look broadly at what you're being charged for. This is useful to help understand how the service itself is meant to be used and what kind of user it's meant for, and will help you understand if a given service is right for your use case.
If you're looking for detailed analysis of AWS pricing across their entire service portfolio, or you need help with getting your AWS costs down, check out The Duckbill Group. They specialize in AWS cost reduction, which requires that they know an awful lot about AWS. They also have a great newsletter and podcast summarizing the AWS news of the week that has been a valuable resource for me.
Let's Get Started
In the next article, we'll take a look at the globally distributed network of data centers where your data lives in AWS, and how the layout of that network impacts how data services are designed, used, and priced.
If you'd like to get notified when new articles in this series get written, please subscribe to the newsletter by entering your e-mail address in the form below. You can also subscribe to the blog's RSS feed. If you have any questions, comments, or corrections relating to any article this series, please contact me.