Cloud Storage: Bandwidth, Storage and BIG ZIPS

Hey.

Yeah. I've been dipping into this a little bit lately. I've I've hosted stuff on cloud providers for probably over 10 years now.

But recently, I was looking at moving some of my stuff to a cloud provider, and I was like, oh, let me revisit this. I've been using Backblaze b two for quite a while, but let's see if there's, anything else different. And I've been noticing the Cloudflare one a lot lately as well, Cloudflare r two.

So I I dipped into it, and I learned a lot. So here to explain it all to you.

Alright. Let's talk all about it. Storing data can be confusing and expensive.

You have to consider a couple things. You have to consider the cost of storing that data. You have to consider the bandwidth of people downloading it, and you also have to consider how quickly do you you need it. So let's say you have large files. So all of the ones we're gonna be talking about today are what are called object storage or blob storage. And, essentially, what that means is it's a spot to put files. It can be any type of file. It can be any hunk of a file. It could be part of a file, but you can essentially think of them as a place to store large files. So things like big zips, software downloads, log files, images, videos, user data backups, all of those types of things are often not part of your application, but you put them on a cloud storage provider because they it would be too expensive. I'll talk about that in just a second. So let's first talk about a couple different ways that we use cloud storage. I just went through all of mine, and I I took a look at, like, what am I actually storing? So Mhmm. My biggest one is I use backup backups, meaning that I have a Synology NAS locally to in my house, and I have that entire thing. Not the entire thing, actually. I have the important parts, which is my recording files, courses that I don't want on my laptop, stuff like that. I have that mirror to Backblaze b 2, and that's about 2 and a half terabytes of of data. And anytime I throw something onto my NAS, it it just auto auto syncs and uploads uploads it to Backblaze b 2 so that I have a backup backup because one backup of a file is not a backup. That's where your file lives.

And, like, maybe in, like, the developer community, this is a thing of the past. But my sister's a photographer. I talked to so many of them. They have dozens of these little drives all around their house. They're probably moving around. Every at least once a year, a year and a half, I get a Sanity message from a friend of a friend.

My hard drives are clicking, and I can't turn it on. And it has wedding photos from this couple. Like, please.

Especially just like a little thing that is so easily easily dropped. This is not the backup show. Well, maybe we'll do that again, that show as Wes. But, man, what else do we put there? I put the original recordings on. So when I'm recording a course or this podcast, the raw files can be very, very large. And sometimes you wanna keep those raw files, so it's nice to have a spot to put those. I put caption files on there so I can download them and put them into my course player.

These big zips for all of my courses is another one that I do. So, like, 99% of people stream the course. Right? You got a nice experience. You get the notes along with it. You get you track your progress, but some people just want the the m p four files so they can download it and watch it locally. So in order to to allow that, I I throw them up on, one of these cloud providers and allow them to download it. Older code Bos. I have, like, basically, like, 15 years of web development of old websites that I've built, and, like, I don't delete them. I just put them up. You could put them on GitHub as well, but I just drag and drop, put the whole sucker up on a cloud provider.

And then I've been recently this is why I got into this is I've been trying to find a solution for going off of cloud app, which is like my screenshot kinda quick file hosting. You need to send somebody a file. You need to send them a video. And somebody on Twitter recommended drop share, which is really cool. It's like a essentially, a screenshot tool, also file hosting, but you pay for the app once, and then you connect it to any of these cloud providers.

And it's that seems ideal to me. So I'm I'm looking at figuring out how to move all my 15 years of screenshots and Mhmm. File uploads over to that. It does seem like a a good

You gotta fix that tonight. You're not gonna be able to sleep. What happens you the last podcast, you just talked about a tree falling on your your office. What happens if a tree goes right through your hard drive? What if someone breaks in and steals it?

So let's talk about some things that we're not talking about really quickly is your your Vercel, and, like, why would you want larger storage? So usually, when you host your application, you get a server, and that server will have space that you can upload files to that will run your application. Right? And they're often limited in terms of both how much space you're allowed to upload things to. So they'll give you, I don't know, like, 20 gigs or or or 10 gigs, something like that. You can upload your whole application there, and often you can put files and and images on there if you want, but the bandwidth for those things are very expensive because the people who supply you with compute are not necessarily the same people that supply you with, like, large amount of storage and bandwidth.

Often, people will just stick a CDN in front of those files that are on the same server. That's a really good solution. This is not a show about CDNs, not a show about image compression, video streaming, things like that because that's a whole another level. And often those services that are image resizing, compression, CDN distribution around the world, those things sit on top of just your raw hosted object structure, which is what we're talking about today, which is basically just a place to put

Exactly. So big players in this space are the they're like I think the big first one was Amazon Wes three. They kinda came out and said, hey. You got wanna put stuff somewhere, and you'll probably have have clicked on files at some point. And you see in the URL, it says, like, bucket or s 3 Scott or, US East. It's like in the URL of the Node, it often will say these words, and that's kinda how you can tell that, oh, these files are actually being hosted on Amazon Wes 3. And then along with Amazon s 3, there's a whole bunch of different options out there. Cloudflare r two, which I thought was kind of funny. They took the letter in the number before Wes 3 and named it. But then as I made this list, I realized Backblaze b 2 is also like, did they do the 2 because of Amazon Wes 3? And then What's up with that? Yeah. I I I'm assuming they did that.

And then Synology has 1, which I didn't realize, c 2. And then all the usual suspects, Google Cloud Storage, Azure, DigitalOcean, Oracle, they all have this. And often, they are all s three compatible, meaning that if you if you just take a library, like Scott was talking to me about pocket Bos, and you can you can hook it up so that it'll it'll upload files to s 3. But you can just take your Cloudflare r two credentials or your Backblaze b two credentials and pop them in there because a lot of these APIs are s 3 compatible, so they'll work with with anything.

You're you're gonna you're gonna hear us talk a lot about Backblaze and Cloudflare both because like, I've I've used Backblaze, Amazon Pnpm, Cloudflare, Google Cloud Storage Mhmm. And Azure Store. I've used all of those, but I like Cloudflare and Backblaze the Bos, and I'll I'll tell you why in just a sec. But also this, like, bunny.net keeps popping up as well. They seem to have both, like, streaming products, CDN products, but they also have a storage product as Wes. And the prices on it seem very good. So

Yeah. They That's interesting.

Every time I talk about it, somebody comes out and says, hey. Check out Bunny because they have a lot of interesting products out there.

The I was thinking the other day I was logged into my Backblaze, and I use Backblaze both to mirror my computers, and I use b two to host a lot of files. And their entire their entire product offering, this is a massive, massive company, is, like, 15 links in the sidebar. And, like, there's there's that's it. Organization. Come on.

Yes. Oh, yeah. I've been probably for 10 years now. It basically is just always and that's one thing I'm doing is I'm always pausing it before we record this episode, because it will continually back up my entire computer's hard drive to backlaze b 2. And if it were to, if I would ever lose my computer or drop it in the ocean, something like that, I'd be able to download my files. I've used it a couple times just to roll back and get older versions of files as well, which is something most of these providers do as well. If you if you upload the same file again, it keeps the older versions, which you can roll back to. Yeah. Yeah.

This is a fictional thing. I'm not doing it. Fictional. Yeah. It's like an office space type of thing. Yes. Totally. Alright. So let's talk about we're we're gonna break it down into storage fees, bandwidth fees, retrieval fees, and operation fees or or retrieval speed. Sorry. So the first one is storage fees, and you can kind of break down the cost of of these things into how much does it cost to hold data in there month to month, and how much does it cost to get the data out if people are downloading it? Because some of these providers are cheap to hold data but more expensive to download it, and some of them are a little bit more expensive to hold data, and they go as far as being free to download them. So storage, it costs money just to hold on to data. Cloudflare is a 1 and a half cents per gig per month, whereas Backblaze is 0.6¢ per gig per month. So I'm taking a look at mine. I have 2 and a half terabytes of data being stored in Backblaze b two right Node. And because that is purely not purely, but almost entirely backup and downloads for my video courses, I actually don't incur any bandwidth, and it's just storage fees. So for 2 and a half terabytes of data is $15 per month for me to sort of hold it on there, which is that's cheap, but then you start thinking about, like like, this podcast Node. After we're done recording it, saving the high quality versions, editing it, and exporting it, every episode's probably wet. Like, we should get Randy to tell us, but it's probably 30 gigs. Right? Yeah. It's not small. If you think about 30 let's do the math right now. 30 gigs times, 150 episodes.

That's 4.5 terabytes a year for this podcast.

So that would probably cost $40 a year to host or to sorry. To hold on to GitHub even downloading it. Just to hold on to it. And, $40, not not too much, but then you start to think about like, I often think about Riverside.

You know? How much are they paying in in order to host all these files and hold on to them for forever? Because Riverside JS not it's not expensive. It's like $250 a Yarn, and we have probably

And I was just like, what

200 gigs, 1 and a half cents each, That's $3 a year to hold on to that file.

That's that's kind of expensive, don't you think? Expensive.

other ones, I won't go through all the prices. Amazon Wes three is actually quite a bit more expensive, 2 and a half cents per gig. And then at least when I did it, this was many years ago, it was also where you store it, it costs different amounts, like, because you have to pick where you're gonna store it. Right? Do you wanna put this in Virginia, or do you wanna put this in Australia? And then when when I was we'll talk about Bandwidth in just a sec, but I off I was looking at the prices, and and I have a lot of people from India download my courses.

And India is the most expensive, at least it was at the time, for people to to download from. Maybe that was their CDN product if I'm thinking about that again because, yeah. That that doesn't make sense that it would cost more depending on where they're coming from. It's just the Internet. Right? But it it would cost more if you need to put them in different different locations.

Yeah. There was a a website I was on, and they were trying to explain, like, this concept called temperature, which is, like, the faster it is to retrieve your data because some some of it does need to be retrieved extremely fast. Mhmm. The faster it is to retrieve your data, the more expensive it is. And you can even go 1 step further. And if you put a CDN in front of one of these, it's even more expensive because they're distributing copies around the world, so it's faster. And the slower it is, the the cheaper it it gets because, like, Amazon has a product. This is one of the features things, but I'll just talk about it now. Amazon has a thing called Amazon Glacier, which is offline cold storage. I think Azure has their own version of it as well where it will take longer. Like, I went to download some of our old episodes, and it takes 8 hours. You have to request it, and then 8 hours later, you get an email and says, hey. It's ready. Or 24 hours later, you get an email. This is ready for download, and that's because they are putting less frequently accessed episodes on cold storage so that it is much cheaper to hold on to, and then there's a process for getting it off of tape. You have to go into Jeff Bezos's basement and pull the VHS off the off the shelf, and they some poor person has, like, a FireWire 800 on a old Mac, and they pnpm it in and and put it on a USB stick and then run it over to you. Wow. Something like that. Yeah. That's the whole that's quite the story. If we had an animator, I would love to have somebody say that.

Let's talk about bandwidth now. So storage is one thing to think about, but the other Node is to think about bandwidth or often is referred to as egress. So when your data now needs to leave the server because it is being requested by somebody to download that file, then you also will incur some sort of costs.

Now the kind of interesting thing that has been happening lately is a lot of these cloud storage providers are giving you free egress, which is wild to think. So Cloudflare r two, I just told you, it's 3 times more expensive than Backblaze b two, but it has free egress. So if you're hosting files that need to be maybe they're they're a little bit smaller, but they're downloaded a ton. Like, you think about these websites that offer, like, NES ROMs or like, I downloaded a couple Wii games when I hacked my Wii. Yeah. I was like, who's paying to let me download this, like, 6 gig file of Wii bowling? Or I I don't know how big it was, but it was many gigs. You know? So I often think about that. I'm like, that's I know how much this Scott. That's kinda expensive

Why is it not called outgress? We don't know. Yeah. That's actually a good idea. Yeah.

So Cloudflare, free bandwidth, free egress. Backblaze b two, it it kinda has this really interesting thing Wes they'll give you 3 times your storage in free bandwidth.

So if you I'm hosting 2 and a half terabytes, so that means I have 7 and a half terabytes of free bandwidth. So, like, honestly, I'm gonna say right Node, it's very hard to beat back Blaze b 2.

And they're, like, just like a wicked all around company just because, like, there's no weirdness around them. There's no, like, marketing fluff or anything like that. It just it's a really good company. So if you have lots and if you use lots and lots and lots of bandwidth, probably Cloudflare r two is is for you. But if you're storing stuff long term and you have, like for example, my because I store so much backups, my actual course downloads never incur any bandwidth because I'm sort of offsetting them, in that regard.

sure. Amazon Wes is 9¢ a gig for bandwidth, which is very expensive. I'm surprised a lot of people. That's the other thing about it is there's a lot of people on Amazon Wes 3, but they can't necessarily get off of it because if you have 70 terabytes of user images, how much is it gonna cost to move those all off of Amazon? You're gonna incur the bandwidth for everything and and then moving it off. So sometimes a lot of people say it's cheaper just to stay on this than it would be to to use Amazon Wes three and and to move it off and pay the egress fees for that. It's also a pain in the ass to

Put a custom domain name in front of them.

Because if you do ever change them, at least you can write like a worker or something like that to jump in between and and rewrite those paths. Still pain, though. Right? It's huge pain. That's a lot of work, and it's likely not worth it for a lot of people. Totally. There's also this thing called the Bandwidth Alliance, which gives you either free or reduced cost of egress between service providers.

So talk about fat pipes.

A lot of these cloud providers got fat pipes between each other. So, for example, Backblaze and Cloudflare, they have, like, a a little string behind the offices where they can call each other directly, little tin can.

And they say, alright. Well, if somebody asks me for your data or if I ask for your data, it's gonna be free.

And that's really cool because if you are doing if you have a worker on Cloudflare, but your files are stored on on Backblaze and you need to download the file in order to to work with it in a worker, that's free because it's it's just going data center to data center, and it's not really costing them that much because they got a fat pnpm,

Thank you. Thank you. Who else is part of this? Azure and Google Cloud are discounted, they say. Mhmm. And then there's free I'm trying to see any of the big ones out there. There's a list of automatic is a big one, which is kinda interesting. So I guess, like, if you've got a massive WordPress site and you want to move your WordPress files to to Cloudflare, you could or vice Vercel, DreamHost, Backblaze, Oracle, and then there's a bunch I have never heard of here. Last thing we have here is just operation fees. These also cost money. So every time that you create a what's called a bucket, which is kinda like a folder in these cloud providers, that's that's an action. Right? It's it's an API hit. Create a bucket. Right? Mhmm. Every time you upload a chunk to a file, that's an action. Right? Every time you see if a file exists, that's an action. Every time you, delete a file, that's an action. Most of those actions, except for, like, deleting files, they charge you by the million, for those actions. So there's not a a a ton of them, but every single request of a URL, it has to pull up that file that lives behind that URL, and that is one action.

And the cost for those are are relatively minimal. What do we got here? Cloudflare puts them into class a and class b operations, which is class a is like upload, list a file, list files in a bucket, list all of my buckets.

And then class b is, like, check if a file is there or or or get the file size, but not download the file.

So those are 1,000,000 for $4.50 a month, and then 10,000,000 of the class b is 36¢ a month. So if you have a 1,000,000 images that are downloaded, that will cost you $4.50 for the initial request, but then the the bandwidth would be free. Yeah. Most people Scott something that they're worried too much. I I think and, also, like, if this if if you have such a scale that big, then you're going to basketball games and having dinners with the the sales reps

games. Yeah. You're you're negotiating, like, your own terms on these things. You're not taking the pricing page carte blanche.

So that's that's all we got today for cloud storage, providers, Bandwidths, and big zips.

Hopefully, you learned a thing or two, and we will catch you later.