Spooky Web Dev Horror Stories - PART 2

Welcome to syntax.

Woo hoo.

This is the 2nd part of our 2024 spooky stories. These are stories of web development gone bad, dropping a table, letting something sneak into production, totally shipping. We had 1 a couple Yarn ago, shipping, like, $80,000 worth of toilet paper to the wrong people and just awful stories of web development gone awry.

And, thankfully, we can laugh at them and also learn a thing or 2 of them. We would love to hear your spooky story.

If you want us to read it next year, go to syntax.fmforward/spooky and pop it in the box, and, we'll put it on the list for next year.

Yes. How's it going, Scott? Oh, yeah. I'm doing good, man. If your code is looking, gone bad if your code's gone bad yeah. There we go. If code's gone bad, head on over to century.i0forward/ syntax. Sign up, and it's a perfect place to make sure that you are on top of any errors, bugs, or creepy crawlies in your code base. Last thing we want is a bunch of big old spiders running around in our code base. Hey, Wes.

We were at the, what JS it? The the butterfly pavilion in Colorado, and my daughter who's 5 was like, I really wanna hold the tarantula.

Oh my gosh. She just stepped up and held that tarantula all day. So you don't want any tarantulas in your Node. You don't wanna hold that thing. Make sure that you solve those bugs with Sentry.

Oh, alright. The first Node we have here, I literally laughed so hard reading this for the 1st time because it's a 2 parter.

And it's just like, come on.

Alright. So the first one here we have here is called monkey business.

My first job as a nineteen year old and the only engineer on an entire product, I accidentally pushed some test code that added, oh my Node, I'm a monkey to the header of every single page.

It was immediately it was immediately noticed by leadership team during the team meeting where we were browsing the site on a projector.

I oh my gosh.

I never patched a bug faster. I believe this was also the first month or 2 of starting there.

Seems like everybody who's young in their career has done that at some point where you just put something ridiculous to make it show up on the page.

Never do that, especially for ESLint facing stuff. So that was the first one. And then and then he goes on to say, at the same company, I decided to do a late night planned deploy from a bar using my friend's BlackBerry as a network connection.

Of course, the deploy went bad, and I lost connection.

I was like, how long ago was this that your BlackBerry was being used to tether? You know? Like, that that's back 3 g days, probably.

Of course, the deployment went bad, and I lost a connection. I was able to run back to my apartment and finish it from there, but I certainly almost got fired the next day. Dude.

Don't don't deploy at a bar with a blackberry.

Yeah. I give this one about 3 rotten bananas. Oh. Next Node is the spooky integration bug.

My story is from the time I was working at a big Scott up. We were preparing to launch an integration with some third party software. This was a big deal. It was announced during a live stream with the CEO demonstrating everything.

Due to time zone differences, I went to sleep right after the announcement.

I woke up an hour later to dozens of messages asking me why the product is not working for anyone.

Turns out when I was working on the integration, I was accessing certain properties of the account object that would only be available if the integration was enabled.

This never came up in testing because of all of our test accounts had that integration enabled.

Oh, yeah. The fix took 15 seconds and all I did was add a question mark, but finding it took a while.

I later heard back that I was almost fired for that, but my manager argued that since my code was removed by the tech lead,

I shouldn't be blamed for it. Probably reviewed by the tech lead. Yeah.

Yeah. Yeah. So someone else has gotta be able to to at least try it. You know?

Yeah. I get that, though. I mean, if you have all the accounts everybody's working on, everybody has that role, everybody has their access, and then you push up and you just man. But, again, you gotta have these thing Wes you're when you're starting something like this. You gotta have that 4th forethought to just be like Those, like, edge cases. I often think about that where people have very complex navigations

where, oh, if they are this, then then add this thing to the navigation. You know? And if they have this feature and as you get more and more flags in your your application, it gets very complex. We had to,

put in, like, in a lot of, like, what Wes you impersonate, masquerade as a user and stuff like that to Wes. Or you had different classes of users. So or you you ran Wes as different types of users to check those access levels, maybe ESLint integration tests. Alright. And the 2nd part of this is I I did investigate why TypeScript didn't catch that. And it turns out someone casted the entire Redux store as any, which meant nothing was typed. I kept telling myself that it wasn't my fault because of that. Man, if you type everything JS any yeah. TypeScript not even doing anything. That is Oh. That's wild.

Yeah. As any just but as any Anytime, though, like, even the autocomplete stops working, that raises a red flag in my head being like, oh, something's not typed properly here. Let's let's chase it back.

Yeah. I give this one about, you know, I I don't know if this is a total 5 out of 5, but I do think this is yeah. Definitely 4 4 out of 5 in terms of, what what can we assign this 1? Bugs? Creepy crawlies? 4 to 5 creepy crawlies? Yeah. I think so. I'm sorry. I'm getting choked out by my popcorn here. It's

killing me. I'm, like, so hot right now.

This might be the the the episode I laugh at the most that we record all year.

Next one's called worst case wake up. My 1st big web success was designing a CMS for a customer service department of my company. It was the only source of information for our customer service agents, and without it, they could not process orders or answer many questions.

One time around midnight, I was emptying the tables of my local environment before syncing with production, and the query was taking a few seconds, when it was usually ESLint. Very similar. That's that's the same thing with the GitHub one where it's like, something's a bit odd. I knew right then I was still connected to production.

Instantly, the entire site was gone. 20,000 support articles, all gone.

Millions of edits. Thousands of user profiles, all gone. By the way, this was not a small community college. It was a very well known corporation with 4,000 frontline support agents who would be working in a few hours. The site got millions of hits per hour. I woke up the database admin, and he spent 3 hours restoring from a tape backup, tape drive. Like, this is like, not a CD, not a floppy, like, tape.

This was 25 years ago. Although, like, tape still is is used today, especially Amazon Glacier JS tape, and that was about a day old. I did not sleep that night, and nobody but me and my new DB admin friend ever fell out. What a cool database admin guy. He just just helped him out.

Didn't say a thing. You know? It happens.

Yeah. Yeah. Back restoring a backup from tape, I'm sure, is something that most people who back up to tape would say they would never want want to have to do that. It's just there for situations like this. Yeah. Even, like, backing up, if you have the backups, they can take a long time. Like, the databases can be tape.

Many gigs. You know? Wes, 30 gigs, and having to restore from that could take a while. Yeah.

Yeah. Yeah. This, this Node in particular, man, that

the only heads up at a like, like, 4000 people's jobs are on the you know Wes you get, like, you get a call or you're trying to get some customer service, and they're like, I'm sorry. The system is not working right now. Yeah. It's probably something like that.

Yeah. This, man, this help desk is gonna need a help desk. That is scary stuff. This one, I I I would give 5 out of 5. That one's Oh, yeah. Yeah. Five pumpkins out of 5 pumpkins. Yeah. Yeah. Yeah. 5 pumpkins for sure. Alright. Next 1, severed trunk and missing backups. When I was an intern at a small marketing agency, I ran r m hyphen r, which is recursive. So it's removing recursively Node directory too high while deleting that agency's staging site on the web server so I could reupload it, which is a pretty routine operation, believe it or not. Oh, come on. This is why we use, this is why we have our processes like that we do now, folks. You wonder why it's so complicated. We used to, what, SSH in. You'd remove things. You'd upload things. You drag and drop.

Oh, man. Which is, perfect for an opportunity like this.

And then deleted the overarching site folder, not the whole Vercel. Thank goodness. Oh, yes. Thank goodness. Which also contained the live site, the QA, and production databases, and the real kicker. All of that site's backups. Oh, no. They kept their backups.

They kept their back They kept their backups on the same server as their website. So if the website Right next to away, so does their backups.

That that probably hurts you specifically, Wes, because I know you have a thing about making sure off-site backups and backups Yarn away and stuff like that. Yes. Yes. I had a local development copy of the agency site, but the database was several months old and numerous copy tweaks have been made by the partners that were not recoverable.

Management improved the backup strategy. Thank Node.

Thank their lucky stars that this happened to their website and not a client's.

However, they fixed the backup strategy by backing up to a dedicated backup directory on the same server. No. They didn't fix it. That's not fixing it.

Oh,

a couple years later, the sole trunk line into the data center housing our server was severed.

So all of the websites went down for several days, and there was absolutely nothing we could do about it unless we had a recent local enough development copy. Man.

So you can't even access your server at the data center because their entire trunk was severed.

And if the only place where your website is is on the live server and you can't get to it, you're screwed.

This one's spooky, but I think it's somewhat lower stakes. I'll give this 3 skeleton hands coming out of the ground. Yeah. That's fair.

Next one's called brute force too brutal. 10 years ago, I was working at a Silicon Valley Scott up, and I'd only been there about a month ago or so.

Our cloud hosted Microsoft SQL Server was acting up, so I remote desktop into it and discovered someone was trying to brute force log into it.

So without thinking it through, I decided to disable the public network adapter to kill the attacker's connection. Yeah. If you can't there's no security issue if you can't access it. Right? That's when my mouse stopped moving and my heart sank.

I had just completely cut us off from accessing the database server that powered our store and had all the customer records in it.

Dude.

Luckily, our hosting provider had a utility where you could do a virtual login. But in the 15 minutes it took for another coworker to figure that out, I thought my career was over.

Oh, I'm gonna give that a 7 and a half, toffee candies out of 7a half toffee candies.

Yeah. Yes. Toffee candies.

Next Node is a dorm room b movie. It all started with a simple housing management app I was building as part of a program that I had dropped out of. Little did I know that this project would soon turn into a wild ride of lessons learned along the way.

I was just 17 teaching myself to code with the help of Google and YouTube when our class was tasked with building projects daily.

I was determined to create something cool, a shared housing app for us dorm dwellers on California Street.

Things were going well until 1 fateful night when someone decided to upload the entire b movie script in into a text input field.

I think they generated a buzz with that one, Wes.

Yeah.

Suddenly, my carefully crafted app was overrun with lines of dialogue from the classic comedy film. I scrambled to figure out how to address this issue, realizing the hard way the importance of pnpm, sanitization.

That night, my dorm room turned into a battleground as my classmates started dumping movie TypeScript left and right, testing the limits of the app. I was in over my head, but luckily, our instructor seasoned industry veterans stepped in to turn this disaster to a valuable learning experience.

Through their guidance, I not only learned how to properly secure my application, but also the power of collaboration and problem solving.

What could have been a humiliating experience became a testament to the resilience and camaraderie of our tight knit group.

Looking back at that fateful night in the dorm room taught me more than any textbook ever could. It was a baptism by fire, a reminder that in a world of software engineering, you never know what challenges may arise. But with the right mindset and support, you can turn even the most chaotic situations into opportunities for learning and growth. Yes. This one is low stakes. Obviously, I'm gonna give it I it you know, it it's funny. I'll give it, oh, you know, Node giant Vercel hornet coming to sting you.

I I actually got that story, from the guy because I was building the form for people to submit spooky stories.

And Mhmm. I had I had been scaffolding out the database for it. And I was like, like, how big should a story be? Because you have to choose the in MySQL, to choose the type of column where it will go and how much text it can possibly hold.

And a MySQL text column can hold 60,000 characters. And I was like, what's bigger than 60,000 characters? And and the the joke in the industry is the b movie script. Wes. So if you ever find an input, you can sometimes crash their server or at least throw an error on my SQL because you they're not expecting somebody to send the B movie script, especially on smaller pnpm, like, input type and name, or you can just remove with dev tools the max length on it. And, so I was using Beamovie to test. So Beamovie will not fit into a, MySQL text field.

It will fit into a I had 203 copies will fit into a medium text field, though. So if you if you do need that, that's there for you.

The this next one is the most hilarious. I not the most hilarious, but just I was laughing really hard at it. The title here is no goats. At my previous company, we added a no goats chat box to the definition of done checklist of pull requests.

We usually used Unsplash to have some images to work with while developing. This accidentally ended up on production for a customer. It was a fashion brand, and we had only been using pictures of goats.

Oh, it's you think you could be silly doing silly stuff, but when push comes to shove, things need to get done. You often let swears or, oh my Node. It's a monkey ESLint through. In this case, you gotta have a checkbox.

You Node, block merges until no goats is confirmed.

No goats. Yeah. Oh my gosh. Yeah. That's, yeah. We can give that we can give that, you know, 2 2 zombie goats out of 5. Yeah. Wow. Yeah. No goats.

Next Node, Pokemon problems.

Not as extreme as the others. But when I was on the production support team for my current company that deals with practitioner and patient interactions, I was looking into a customer escalation and was reviewing their account details.

At the same time, I was testing a bug around user diploma upload in a testing environment when I got my browsers mixed up and accidentally uploaded the test image to the real user's account.

To make things worse, I was using exclusively Pokemon images for testing.

I noticed my error and fixed the issue, but there was about 20 minutes where a person would have seen a friendly Pokemon instead of their diploma when review when viewing their account settings.

Oh, yeah.

That's a good one. So funny. That's often when I'm doing, like, an admin dashboard, I'll often do color it differently.

Like, in my own admin dashboard Wes I'm refunding money or deleting things or editing things, I have a huge green or red bar that says production or testing, because it's it's kind of scary Wes, oh, you have you have the wrong tab open and accidentally, you delete the wrong thing.

Yeah. I give that 1, 1 Gengar out of 5.

Next one is called late night with Safari. Back in 2016, I was a sole front end developer. So many of these stories start with, I was the only developer at a at a job in charge of a Fintech b two b application.

It was my job in tech even after doing this stuff for 15 years before as a hobby.

It was my 1st job. That's what he said. One day, I pushed an update to our filter experience. I say it was sized effort and things were looking good for a few few hours, but at some point, we got customer feedback that the web page would crash for some user.

We were able to narrow it down to this specific Safari. Oddly, the office was primarily Windows based machine except for 1 person, so I had to kidnap her Mac for the day. And what a day. I was able to reproduce a problem, which was good. I figured it must be something in JavaScript that is the problem. So I started commenting things out at the root of the cause. After hours of searching, rewriting, deploying, I discovered the problem. It was CSS.

At the time, it seemed that Safari couldn't handle a transition for the font size property, maybe with other size properties too. So if a font size transitions, Safari would crash.

Yeah.

Oh, I couldn't believe it. After I removed it, everything was fine, and I left the office at 9 Npm. And so kept my life in keeping motion limited had begun.

Yeah. You know what, Wes? I have found you know, people might rip on Safari for this, but I have found Safari to be way less permissible about things like that that you shouldn't be doing anyways. Like, Safari for me was also, like like, really slowing down my app like crazy for a bad transition as well. And and you could rip on Safari for that, but to be honest, I think Chrome might be a little too permissible. Yeah.

Yeah. That's also like Safari doesn't even let you do position fix still. You know? That's What? They don't let you do position fix? We can't position fix in in Safari.

What are you talking about? No.

You can't. Get look it up.

You can't position fixed?

No.

You can't you also can't do background position fixed.

Why can't you do position fixed? Because it

it's too intensive because you have to recompute with position fixed. It's stuck on the viewport.

So every single scroll, you have to repaint

everything. So what do you do in almost I use position fixed all the time. How do you do, like,

a bottom footer nav? Maybe I'm wrong. You know what? It's background position fixed. It's Okay. For the longest time, you couldn't do position fixed on on an iPhone, but background position fix is not not a thing. Okay. That that Wes, like, that was, like, 10 years ago. You couldn't do that. Okay.

Yeah. Okay. I was gonna say, I am, like yeah. I I do that. I'm concerned.

Yeah. Okay. Cool.

What do you use position fix for? I haven't used that in a in a while. I guess, like, if you want, like, a tab to over overlay the entire website.

Like, what about a footer navigation? Like, when you have, like, an app, the little Yeah. I would do, like, a layout, and then I would do, like, a grid layout and make the footer, make the the the body part scrollable instead of the entire thing. Put a you put an overflow scroll instead of a position fixed. Interesting. I think.

Wes should try. Always reaching for a position fixed for that. I I guess, you know, that that works. The Yeah. But it's because that with position fixed, then you you give up the flow of the document, and then you gotta account for the height of the nav. With with grid, you the footer just takes up as much space as it needs.

Yeah. But you just Bos it down there and put a padding.

I don't know. Yeah. How much padding do you do, folks? How much padding? What happens if they variable. Piece of size? What happens if it's a German word and it's multiple lines?

It's not because it's the little it's the same little nav you have in Instagram. It's just little icons down there.

My hair just fell off. Alright.

Oh, Wes.

Oh. Maybe we should we'll do a YouTube video of coding an app layout. Let's stop talking about it. So the timing I think that would be good. Yes. Yes. Okay. Next 1, boot camp bungle.

I was a few weeks into my 1st developer job after completing a coding boot camp in 2016 and I had joined a small real estate company with 2 other developers along with a bunch of product marketing people and a bunch of realtors.

At the time, the website was a rails app running on a custom front end framework that had been built by the previous team, which had been completely laid off after the company had been sold. Yes. This is the classic you use their framework or you, make Node yourself that is not as good.

My first task was to add some new Google Analytics tracking events to a few form submissions.

Seems simple enough. Right? Well, being fresh out of a boot camp and pretty unfamiliar with front end frameworks at all, especially custom built in house ones, I couldn't figure out how to do it. Rather than ask for help, I discovered that if I loaded the Google Analytics script in some file, I could access it in the files I needed to wire up the event handlers of the form.

I added the handlers, tested it out, looks good, l g t m, shipped to prod and moved on. Stoked to have made my 1st ever professional push to production.

A week a week or 2 later, my boss comes to me and asked me to take a look at the Google Analytics dashboards with him. Overnight, it looked like our traffic had increased dramatically starting on the day that my first deployment.

I'm sure you can all guess what happened, but, yes, loading the Google Analytics script twice on every page with the same credentials will indeed result in the page view being counted twice.

I had accidentally caused Google to double count all of our page visits in my 1st week on the job. Well, at least, Wes, you can just slash those by half. Right? If you were like Yeah. By some magnitude of unknownness. Yeah. For a small business that was heavily relying on SEO and Google rankings to compete with larger real estate firms, this felt like a big deal. I was convinced I was going to be fired. I reverted the change, asked for help this time, and added the events the proper way and prepared myself for the worst.

Every time my boss came into the little office where the dev sat, I felt my stomach drop. Oh. Convinced that it this was it. This was the day I was going to be fired.

Wes I Sanity imagine being fired for this personally, but you never know.

Of course, I didn't get fired. Once we figured out the problem, I worked with 1 of the other much more senior devs to revert the change and add the events properly.

Yeah. I messed up our Google Analytics stats for a week or two, but in the grand scheme of things, not a huge deal and a big learning experience.

8 years later, I'm still happily employed as a developer.

Moral of the story, never be afraid to reach out for help when you don't know what to do and accept the fact that everyone is going to make a mistake at one point or another. And Deno.

You most likely will not be fired for it. Yes. That is such good advice that I love that this one, it kinda ended with a little tail of, like, a morality tale. Like, hey.

Ask for help. Especially in a situation where you have a custom built, you you you end up in these situations where you're like, I don't know. I'll figure it out.

If you don't have a good review process, which it seems like they they didn't, ask for help. There's no harm in asking for help. Yeah. It should never be 1 person's

entire problem. You know? Someone else should have caught it and or at least have reviewed it and said looks good, and then it falls both on on both of you. But, like, what this guy's not fired because of how he handled the situation. You know? If some people would have done this and then reacted negatively or passed blame or simply just ignored it. You know? And if that's the case, then I think that's grounds for being fired.

Yes. This is a I I give this the alternate title of boot camp Crystal Lake, and I give this, 2 Jasons out of 5.

Next one is a film school f up. Years ago, I was in film school. I made an app to help my classmates, and I automate some paperwork on set.

When you're out in the forest, it can be incredibly annoying to have to handwrite camera reports and shooting summaries as well as prevent them from getting wet. I've been coding since I was a kid, but at that point, I had never worked professionally as a programmer. I didn't even use Git. I hacked together an Android app that could handle it all and the different paperwork our school required.

All the form data was stored locally in SQLite and synced to my MySQL database I hosted. So no matter what, there would be a backup of our paperwork either locally or on my server. Makes sense. You do it locally. You sync it to the remote server. Right? That's that's ideal.

Being self taught and never working professionally, I had to make my own syncing system, between the phone and my server. I handled this on the server side. Whenever the app synced, it would turn all the paperwork data into JSON and send it to the server. The server would then combine the query, merge it with the local data, and finally respond with the new data. The app would then overwrite its local data with the new merged data from the server. You may have noticed that overwrite was in quotes. Somehow, I figured the easiest day way to do this would be by deleting the app's local data and then inserting new rows into the database with the response from the server. Otherwise, I would have to rewrite all the complicated merging code I had already written on the server side. Right? Yeah. That that makes sense. You know? Delete it, send it off, and then and import it. I tested a 100 of times. It worked flawlessly. The problem happened when we were in the forest. That this is terrifying, though. This guy's just running around the forest as well.

When we were in the forest. What a great sentence. When you're in the middle of nowhere without good cell service, the app would try sync with the server, and then a post request would fail. Unfortunately, the function that returned the response from the server wouldn't raise an exception.

It just returned a null value. The sync function would then delete the local data and insert null into the local database. This is just screaming for a a syncing engine, like one of our local first episodes.

My teenage brain never fathom that a post request could possibly fail and never considered in that situation my crappy code would delete all of the data locally.

I had made an app that let you fill out the paperwork when you're filling in the woods and then delete it. There were no backups.

I was filming my own short form when I noticed the bug. My data disappeared. When I got home, I spent hours trying to figure out what happened before I realized it and quickly fixed the bug, but it was too late. My class had been filming our films for weeks.

I called for my classmates and begged them to immediately download the update. Since this bug only happened when you tried to sync without cell service, it didn't affect everybody, but there were a handful of people, lost a good portion of the required school port paperwork, and I was mortified.

Nobody ever used the app again. Some of those spooky dev stories involve companies losing 1,000,000 of dollars, but mine involved having to spend the next couple years sitting in the class next to all of the people that I had screwed over with my app.

It's like the opposite of the friends he made along the way. Fortunately, the school decided to give those people a pass since the paperwork losing the paperwork was out of their control.

Awful story. Awful story.

Oh, poor guy. Yeah. When you when it Wes it affects other people's lives, you know, like, it's not just money, but it's also, like, people's, like, people's livelihood. You know? This is their school mark. That sucks. Yeah.

It yeah. Which when we by the way, I wanna say, when I give these things spooky scores because a lot of these are like, oh, they're not as bad as whatever. Yeah. That's just how scared I would be if this happened to me.

That that's not how good the story is. These are all incredible. In fact, man, I I have no doubts that that that would suck. That would feel Yes. Scott that it's a will feel awful. It's a social problem as well. Social problem. Yeah. You feel it in the pit of your stomach type.

Next one here is AWS cluster f. I love that. I wanted to update a production cluster for several 100 customers. This was a self hosted ranger on AWS I had built.

Tested on everything in a staging environment. Was super proud of seamless upgrade there. Uses wrong token and password for upgrading.

Locked us out of the cluster for good.

No access to it for a week.

Applications were scaling up and down happily, but no one could access.

I had to rebuild the whole infrastructure again and deployed containers back for data. We luckily used AWS EFS and RDS.

On migration day, we just moved everything to a new VPC.

Done.

Looked like it never happened except a small maintenance went down.

Yeah.

Oh, man. That that's so funny. It's like an audit. It's just doing its thing, but you just can't get into it. You're outside looking in. Yeah.

Next 1 we have here, limitless coupon disaster. I was part of an incident review at one time. So this didn't happen to this guy directly. But when things happen, they often bring in engineers and people from management to sort of review what had happened and figure out how you can fix it.

Once upon a time, someone configured a promo code without any limits for $500.

The next morning, we woke up $3,000,000 lighter. The total loss added up to be about two point 8,000,000.

Those who responded at 3 AM were able to prevent any other $5,000,000 from evaporating.

The response time wasn't too bad once we are aware of the issue, tops 20 minutes. It became much harder to create a limitless promo Node after that. A hidden lesson is to not let your internal admin tools Scott as your business grows.

Oh, I I know I think it was Abercrombie had this issue. I don't know if this is Abercrombie or not, but Abercrombie had a similar issue where there was some weird combo of coupons that you could use, and it got you, like, 98% off. And I think it was, like, $20,000,000 or something like that. And I often wonder, was that on purpose? You know, was that an advertising campaign that it went just absolutely viral? Oh, yeah. Or did somebody really goof up their, yeah, the internal tooling, man. Just a process. Someone must approve a promo code. You know? It's not just simply like, check the database for this promo code, but coupon codes are do I think we even did we do a whole show on coupon codes at some point?

I feel like we must have because we we did that whole voucher site, which was like a whole thing on coupon codes. So I would imagine it was around that what we did. Scott complex coupon codes.

We should do another one.

Building a coupon engine Node number 4 fifty Node. Yep. That was the coupon. That was actually for our course sites. So that was not even the voucher one. We also did an episode on the the, giveaway voucher site site. Oh, yeah. Coupons. Those are both very good. The Syntax giveaway site episode 609, both are probably relevant in this scenario here. Yeah. Coupons agreed.

And, also, like, if you if the if the spooky story contains the word 1,000,000 of dollars or 1,000,000 of dollars, it automatically gets a 5 out of 5 for me. What what would be a good 5 out of 5?

Black holes.

Yes. I give this 1 5 out of 5, black holes. Woah. Alright. Next 1, ruined vacation. I well, terrible. No. No. Cove Bugs ruining a vacation. Your vacation is supposed to be your your chill time. My wife and I were leaving that morning for a trip to Hawaii.

We were going to be gone 7 days. This was before the time of just packing a laptop for the trip. So once I was out of the door, I'm basically offline.

The boss called and said, we have a group of users whose statuses are wrong. Can you update them before you head out? Last minute update.

A quick search shows me that all of the users who signed up for a specific date weren't being set to the proper status. They had access to way more than they should. No problem, I said, ESLint proceeded to run the following command, user table set status equals bronze.

The taxi showed up, and we're almost at the airport when I get a call on my brick sized cell phone.

Now there's no users who can access anything.

250,000 plus users just got reset back to new user status.

Remember to use those Wes clauses. Folks, man, this is the 3rd Wes clause.

Man. Werewolves. The yes. Werewolves. I I do give this 1 250. That's 4 werewolves out of 5 for me. Oh, that's a lot of people.

And then you gotta figure out how to roll that back. Unfortunately, he didn't say how he fixed it. But

woof. Yeah. Because That sucks. Especially if, like like, I know systems where it's, like, webhook based. You sign up for something. That webhook comes back. You give the user the role. They get access. Right? So you possibly have all that information somewhere.

But chances are you don't have the code to just resync those statuses on hand, so then you're probably having to write a whole bit to resync statuses. Yeah. Maybe you do already have that code, but maybe you should have that code if you don't.

Next Node is console dot swear. I was working for a nonprofit in 2011, and we had a big boost to our mailing ESLint, and we are building a new campaign. I was working on the JS error handling for the position signing form and was struggling with a regex. Oh, I already know this is gonna be awful.

If I remember correctly, it but got it sorted out and ready for production. One of my habits at the time was throwing alert boxes up with increasingly aggressive messages for logging, and this is why something like a 1000 people saw an alert dialogue that says f this when we ran ran the campaign, and I forgot to pull out the error handling out. Folks, stop doing this.

Stop. And it it it makes it even better. It's a nonprofit. It's probably like, would you like to save a starving puppy? Yeah. Oh my god. I keep logging messages g rated and keep them to the console now. The founders and my boss were very chill about it, but we could have gotten a lot more spooky if not.

Yes.

G rated.

If you're logging anything, keep it gee, I learn I I I learned this the hard way by using Wu Tang Ipsum in an SVG.

I had somebody be like, there's some language in an SVG on your website. And I was just like, oh, yeah. It's a Wu Tang Ipsum. I didn't I didn't remember that if it's text in an SVG, then it would be listed as text

in the code base.

Yeah. I think it was Triumph by Wu Tang, which you if you've heard of that song before, you know what that song's about.

Another coupon disaster. Oh, coupon disasters. These ones are good. I did tons of email campaigns and once accidentally sent an exclusive one time offer meant for 500 people to 80,000 people.

The client wasn't happy because I believed by law they had to honor it.

But they also knew that they were flooding me and jumping between 2, 3 campaigns at once. It was my 1st and only major screw up within 10 years of working with them, so they were forgiving.

Jeez. I'm sure. You Node? To 80,000 people? Yeah.

I then adjusted the process between myself and then to assure it would never happen again. Additionally, I got them their highest opening rate on a campaign by accidentally sending out GitHub a subject. You wonder if that, like that only hurts. I mean, it hurts. But that hurts, like, if they're selling at a loss with the coupon, you know, and the coupons are just there to drive. And then, oh, man, that could, like, really sink a business. Right? Oh, yeah. If you were selling at a loss, just trying to gain some hype as a promotion.

I once emailed, I think, about 250,000 people with a broken link. And the problem is that I noticed it before it was sent out with my email sent out. It it takes, like, I don't know, 5, 10 minutes before they actually start getting going, but there was no way to stop it because it it had, like, hit the no return. So I, like, messaged someone from support, and they're like, we'll see what we can do. But, unfortunately, the broken link went out to that many people, and I don't have a no reply. I have my own email address as my reply, and I got hammered. And then so I quickly sent a, sorry. The link was broken, but then everyone thought it was like, that was a tactic because that is a tactic to send multiple emails. Oops. Wrong link. You know? Oh, that was that was a bad day. You just get flooded with emails for the next 8 hours.

Yeah.

And there's nothing quite like being flooded with emails. You Node, Wes have one of those situations in college where everybody's on the email ESLint, and they make the email list, like, public or not, like, yeah. The c c? And I'm I'm in, like it's for an entire dorm. It's an entire dorm full of kids at at the University of Michigan, so it's a lot of people. And then everybody you know, they're they're college kids. And I I'm not gonna lie. I got in on this. The reply. Replying to all and reside. Take me off this ESLint. And and everybody's stop replying. Take me off this list. So then it really encourages you to reply, I'll take me off this list again. Those reply all

things like, I don't know if there's any way to stop them.

It just goes and goes and goes. I was part of 1 with, what was it, Microsoft MVP. They accidentally cc'd, I think, probably a 1000 people or something like that.

And, it it went on for a while, but luckily, most of the people on there were, like, pretty technical.

So it wasn't like a bunch of people being like, who? What's going on? Please stop this.

Yeah. I I I was technical enough to know what was going on, and I just thought it was fun. So honestly think I should've. Because, like, how

Node do you get to tell people you're a part of a reply all email storm? Yeah. Yes. I love it.

Alright. This next one just came in on Twitter, like, 10 minutes ago, and it's it's written in, like, a 4 chan dotted lines point. So the doctor was working on an amazing project.

Identify top 5,000,000 websites by rank, classify them, enumerate over 500,000 DNS names to identify all subdomains and validate all steps for rate limit. So I I think, like, yeah, they're doing some SEO ranking. It seems slightly illegal.

I'm not exactly sure, but this seems in a gray area of of being able to do this. So what you do is you can run a DNS lookup command on every single domain name, and that will give you some information back, and that will allow you to sometimes allow you to to find subdomains on that same domain.

And you can run them through various DNS providers, Google and Cloudflare being the the big ones.

So JS this temp banned for 15 Node.

ISP contacted us.

I I actually had to do this for 80,000 domain names for my burner email list. So I ran a I looped through every single domain name, and I did a DNS lookup to see if there were MX records sat on that domain name. And if there's no MX records, it can't receive email.

So, therefore, I removed it from the list. It must have been a dead domain.

And 80,000, I got banned very quickly. Mhmm. So I had to I had to really cut it back and do do rate limiting. I think it ended up taking, like, a whole like, a day, a day and a half to actually run through the whole list.

Because if you're doing too much DNS lookup, they think you're spamming. So I asked them, like, what happened? He said the ISP contacted us because they noticed unusual reverse DNS lookups being routed at a high rate before they routed it to Google or Cloudflare.

They thought we Yarn potentially compromised.

No. It was just them doing the the shady work, and they asked us not to use the their DNS as a backup. It was a fun week.

Yeah.

Yeah. DNS anything, man. Put just DNS on this. And, I mean, whether you're changing DNS, looking up, man, put it all on here.

Last 1 here, and this Node is, extra spooky because it's not dev related.

Not dev related, but funny anyways. A marketing person I know was creating and scheduling a post event email stating how successful the event in conference was. Breaking visitor attendance records was a huge success.

They accidentally sent it to the entire database 2 days before the event took place.

Oh my god.

Oh, Wes, you know what this reminds me of? There was, man, it was something very recently with the, Pittsburgh Penguins.

They sent out a newsletter, and it was very clearly, like, their test email that they meant to save for later in a draft. And it was, like, declaring that some contract had been signed when it hadn't. People were, like, really, like, waiting this contract to be signed. You know, all the fans are on edge about it. And they get an email saying so and so signed the contract.

And then everybody's like, what they did? Let's celebrate. And then they had to immediately send an email. Like, by the way, this player did not actually sign the contract. We were just we Scott send it, which has Scott put the person who had to sign the contract in a uncomfortable place. Right? Like, because then now they're like, I do I have to sign now? Like, do I Scott? Yeah. Like,

that that always creeps me out. Those, like, when somebody famous dies, a lot of these newspapers will already have the the thing written so that they can go live as soon as they pass away, which is an odd job, you think, to have to write all these, like like, somebody has been tasked with writing the Elton John has passed away article for the time that they and it's probably just sitting as a draft in WordPress somewhere.

And every now and then, these things get accidentally published when they shouldn't have, and then it just ripples through the entire thing, and and everybody starts freaking out. I just I would feel so bad if I did that. And, honestly, that's maybe we should move on to the next section, which is, like, what did we learn? Like, you need What did we learn? Processes for things that could potentially goof up.

You need processes, whether it's like, even with the database schema. The other day, I I messaged Scott, and I was like, hey. Like, wouldn't this goof up our database schema if this were to happen? And he's like, no.

We do not allow rights directly to the production database schema. You have to commit a, like, a pull request on the schema, and then that has to be approved and then merged. And I was like, okay. Good. Because I was just sitting there being like, that could possibly be an issue for us at some point. So having those processes in place so that you can't accidentally goof this type of thing up because we are all just human at the end of the day, and these things happen.

Yeah. And, actually, shout out to PlanetScale because, Wes, you might not know this, but there's a feature of PlanetScale.

And I forget exactly which feature it is specifically, but they don't allow you to enable this feature unless you turn on the frozen database schema feature for PlanetScale.

And I I did not want to turn on the frozen database schema feature because it's a giant pain, which, of course, it's only because I'm being, you know, lazy.

And and I I wanted this feature enabled. So I said, alright. It's worth it to enable this, you know, 10 step process Node to make changes to our our schema if I can have access to this feature, which, of course, is a good thing we have access to this feature. Because I probably would have accidentally deployed a change to the database schema at some point. So shout out to PlanetScale for making us make good choices there. Yeah.

I I was thinking about that last night as well. Do you remember, Firebase? When they when Firebase was announced, and for many, many years, Firebase's default read and write permissions were wide open. And that led to huge adoption of Firebase because you just had this object you could update, and everything would be mirrored, and it'd all be real time, and everything works great.

The downside to that is many, many people built applications on top of Firebase and never introduced any sort of authentication or you who can read and write, access control. And there there's tons of stories out there, so much to a point where Firebase got has a bit of a bad rap for security, which is they have fine Sanity, but it's just that by default, everything is wide open, and people don't realize that.

Yeah. And, also, that it's, it's a even Meteor themselves, they had, like, by default to get up and running really easily. They just had you connect to an unsecured MongoDB with, like, no creds. Right? It was just a completely open MongoDB, and it was local. And it was intended so that you could just install Meteor and start writing code right away.

And then in their docs, it's like, oh, yeah. Before you deploy to production, you have to lock down your database. And it turns out just people don't read. So people were just deploying their to production with an unsecured open database.

And, for a long time, I believe, Sasha no. Who who was it? Man, somebody from somebody from Meteor. I I forget. It was a while ago. It was like, my biggest regret with Meteor was shipping with an unsecured open database because Oh, man. That that was yeah. That absolutely.

Security might seem like extra work,

but it's necessary work. Don't deploy this to production, but Yeah. People do. People do.

Yeah. Next one here is ask for help. Nobody you know, here here's the big thing is you will be judged, much harsher by your teammates for shipping bad code or broken code or code that leads to something like this then you will be for asking a Wes, no one's gonna fault you for asking a question. If anything, they're gonna look at it as a positive. Like, this person takes us seriously, and they wanna make sure they do it right.

So ask questions. Ask for help.

Do things the right way.

And if you don't know what those are, which let's face it, I I think every single one of us has been a new job or there's new processes and, like, who who who's really good at documenting things? I don't know if everybody is. Like, we tried to document things really well, but there's still a handful of things that Yarn, like, just take a guess. And, it's always important to ask questions. And you're good about that, Wes. Whenever you have any major concerns, you ask me in Slack right away.

Yeah. Yeah. I don't I don't wanna end up on this podcast. You know? Dry run queries, soft deletes.

If you are doing a major major query where you could you're modifying data, it is often worth switching your update or delete clause to simply a select, or a lot of query languages will have, like, an explain or, like, an auto commit turned off. Mhmm. Basically, you want to run some sort of dry run, or you wanna be able to mark documents as deleted, which is a Scott delete. It's not actually deleted. There's a whole regulation there too because what happens with with, the Europe thing why am I forgetting the name for a Europe thing? GDPR.

You know? But so The Europe thing. Yeah. There's a lot to that as well. Yeah. For sure. That soft delete thing, I think some people might see that as a negative, but you could have soft delete and then a process that deletes those soft deletes after Yeah. A month or something. Or something like that. Yeah. Yeah. Totally. It that's not always a bad thing to have those soft deletes.

In fact, I learned that soft deletes were kind of necessary for doing some local first stuff when you're dealing with patch messages.

And you're let's say I have a button. Like, my Sanity tracker's toggling back and forth really quickly. Yeah. It made much more sense on the local database local sync to do a soft delete on the local sync database specifically.

So that way, if the message came back in, it was just, like, toggling that that value. But yeah. Oh, yeah. Yeah.

Next one is code reviews, which this should be a part of your process.

We had a really super good episode on code reviews in episode number 830.

We have, Sarah Vessels from GitHub give her professional take on code reviews and how best to do them. Code reviews are are fantastic. They might slow you down just a hair, but they are important because I think we all push stuff that could, you know, break. It's funny, Wes. We did a CJ was doing a live stream of the syntax production assistant, and I YOLO'd, like, just 5 or 6 package updates. And they were, like, all small patch Vercel.

And I tested it. It worked fine. Yeah.

CJ pulls it up on stream, and people on stream in the comments were like, this is too many pat this is too many updates for 1 PR. And I'm in the chat being like, they're all fine. They're all patch. And sure enough, something was broken in there. CJ's in there. And I was like, code review found this. Thank you, CJ.

Got you. Yeah. And then use version control to deploy. I think a lot of these stories are from before when version control was a thing, but certainly just have it there. I think there's still quite a few WordPresses that are only living on the Bos that is the live site, especially because, like, WordPress is weird where you upload a a photo. By default, it sticks it on the Vercel. Mhmm. So then that photo doesn't exist locally. Right? And if you don't have a backup process to to sync those down or to put them onto an external bucket, you're you're pooched.

Yeah. Man, that whole version control flow for any CMS based site is tough. I remember there Wes a Drupal plug in that simply just output your configuration because it's storing a lot of configuration in the database. It simply was just, like, backing that up to a file and then restoring it from a file. And so that Wes, like, had to be part of your version control Wes to whenever you made a change, you'd back up your configuration, commit that to version control on deployment. You would restore from version control. It was just like, man, it's kind of a pain. That's one of the reasons why I'm stoked that we have so many of these really great new deploy platforms.

Right? Anything, whether it is like CloudFlare or any of the stuff Wes you commit, push up, everything just works. It makes things so much easier. You roll back. If something fails, you don't have to worry about having those files on hand or something like that. If you don't have a process in place for making sure that you Node, deploy quickly, roll back quickly, data and site, all those things, off-site backups, all that stuff, you need to make sure you have that implemented because you will end up on this Node. Or, you know, you might not, but, you know, have that feeling in the pit of your stomach. Like, oof, you goofed up. Yeah. For sure.

Alright. Let's move into some sick picks. I'm going to sick pick truffle hot sauce. I've been I've been having this for a number of years because my sister had been give it gifting it to me for Christmas. She's from the States, but they now sell it in Canada, which I've been pretty excited about. So truff hot sauce is like, truffle I love truffle flavored hot sauces. And Yeah. The original, the black one JS really, really good. But now they have a a buffalo one, which is super good. They have a jalapeno lime, which I've not tried just yet. They've got some mayo. They haven't tried any of those yet. But if if you have someone in your life, you're, like, looking for a Christmas gift for them, and they, like, like Scott sauce or they like, like, nice food condiments, like, they're a condiment enthusiast, certainly grab them something made by Truff.

They sell this at my Costco. I always look at it and wonder if it's any good. So, I will have to pick this one up. Yeah. It's it's really good. I kinda wish it was in, like, a squeeze bottle, but it's in this, like, really high end glass bottle. It's kind of expensive, but, yeah, it just a a dab will do you.

I think if the, the word truffle is in the the description of it, it's probably you can count on it being somewhat more costly than the alternative.

I love truffle flavoring myself. So for me, this looks like a we're not you know, I love a good, like, truffle salt or something like that. Yeah.

I'm gonna sick pick a show on Netflix, which JS a surprise to me. It's, nobody wants this. And at at first, you see a trailer for this show and you think this is a rom Scott. This is not for me. Yeah. And so because of that, Courtney started watching it by Vercel.

And she was like, you gotta watch the show. It's very, very funny. And sure enough, I started watching it, and I was I was cackling at at many parts of the show. It is super funny.

It is not a show for kids. It's not a show you wanna watch with kids around. There JS adult themes and stuff. Follows the unexpected relationship between a rogue rabbi and an oh, man. What is that word? Man, I've never seen that word before.

Irshable, loud agnostic woman. And she's a, you'll get like this. She's a podcast host, and she hosts a podcast with her sister.

And, yeah, it's it's it's very, very funny. It's a sweet show. It's it's cute in all these ways, but, artistically, it's really good. The music's really good. It's super funny.

And my wife absolutely she's watched it twice now. She's gone through the show twice in, like, a week. So, yeah, if you're a fan of, like, movies like, I don't know, The Wedding Singer, Forgetting Sarah Marshall, those types of movies, it it's in that same kind of vein. And it's it's a cute cute romantic show, but at the same time, super funny. Yeah. I'll have to check it out.

Irascible having or showing the a tendency to be easily angered. I have never in my life heard that word before.

I have a big vocabulary.

I have a large yeah. I have a behemoth a behemoth of a vocabulary.

And I never big vocabulary?

Never heard it. Yes. Big very, very big vocabulary. I've never heard this word before in my entire life. It is Horrassable. Going on the list. Yeah. Man, I like that. Cool.

Shameless plugs, check out syntax.fem.

Go listen to all the past this is the 6th year in a row, so this makes 12 episodes of spooky stories. We have many, many maybe at one point, we need to do a best of spooky stories over the years because there's some some good ones.

But check it Scott on out, syntax.fm, forward slash spooky. I've got a list of all of the past spooky stories as well.

Yes.

Every single one of these spooky stories episodes is hilarious. They're they're my favorite. They're the funniest ones of the year, so, check them out. You may find some good stuff if you like this one. That's all I got.

Beautiful. Alright. Peace.

Have a good Halloween.