The ritual nobody questioned

For months, our MySQL master had a thing it did. RAM would creep up past 90%. CloudWatch alarm would fire. Someone would SSH in, restart MySQL, memory would drop back down, and everyone would get on with their day. No questions, no investigation, no "hey maybe we should figure out why this keeps happening."

It was just... the thing the server did. Like a car that needs a jumpstart every few weeks. You stop asking why and just keep jumper cables in the boot.

At some point this became my problem to actually fix. Not restart — fix.

I ran free -m on the master. 28GB of 32GB gone. Swap sitting at 5.5GB. This wasn't a traffic spike. This was just Tuesday.

So I started digging.

Two problems, one very unhappy server

Turns out the memory situation had two causes stacked on top of each other.

The first one was straightforward: the InnoDB buffer pool — basically MySQL's in-memory cache for data — was set to 20GB on a 32GB machine. That's 64% of total RAM, before MySQL has even allocated anything else. Before the OS gets its share. Before per-connection buffers. Before anything. There was just no headroom, and under any real load, stuff was spilling into swap.

The second one was weirder and took longer to understand. Every MySQL connection gets its own set of memory buffers — for sorting, joining, reading. When the connection closes, the memory allocator (glibc, the default on Linux) is supposed to return those buffers to the OS. The thing is, it often doesn't. It keeps them in fragmented chunks — technically free, practically useless, never given back. Our Max_used_connections had hit 201 at peak. That's 201 connections worth of accumulated buffer fragments just floating around in memory doing nothing, growing week over week, the only cure being a full MySQL restart.

This phenomenon has a name: memory fragmentation. And the fix has a name too: jemalloc — a smarter allocator that's way better at returning memory to the OS. You don't recompile MySQL, you don't change any application code. You just preload jemalloc before MySQL starts, and it quietly takes over. MySQL never even knows.

So the plan was: move the master to a 64GB instance (no more buffer pool math that only works on paper), set the pool to 40GB, and inject jemalloc to handle fragmentation going forward.

Simple enough. Except the master is a production write server. Every single write the platform makes goes through it. You can't just stop it and spin up a new one. You need a strategy.

The strategy — replace the master without the master knowing it's being replaced

The approach was a zero-downtime promotion. Take an AMI from one of the existing slaves — it already has all the data, just slightly behind on recent writes. Spin up a new instance from that AMI (call it Slave 4), configure it with jemalloc and the new settings, let it replicate until it's fully caught up with the master. Then during a maintenance window: freeze the old master, wait for Slave 4 to reach the exact same point in the binlog, promote it, flip the DNS record, done. Old master stays on standby for a couple of days in case something goes wrong.

Before:
Master (32GB, dying) ──► Slave 1, Slave 2, Slave 3

After:
New Master = Slave 4 (64GB, jemalloc) ──► Slave 1, Slave 2, Slave 3
Old Master (standby, watching nervously)

Clean on paper. I was not going near production with this until I'd actually done it in staging first — because the gap between "I understand this conceptually" and "I can execute this without breaking replication on a live system" is enormous, and I'd rather find that out somewhere that doesn't matter.

Staging, act one: building master-slave from scratch

Beta environment. No existing replication setup — I had to wire everything up myself. Which was honestly the point. I didn't want to just run commands from a doc; I wanted to understand what each piece was actually doing.

So: configured binlog on the master, created a replication user, pointed a slave at it with the right binlog coordinates. Watched Seconds_Behind_Master count down from 300-something to zero for the first time.

Two separate databases. One following the other in real time. There's something genuinely satisfying about that moment when it clicks — not because you ran the right command, but because you actually understand why it works.

Then I needed a second slave — the one that would eventually be promoted to master in staging, mirroring exactly what Slave 4 would do in production. And this is where things got interesting.

Staging, act two: the AMI that lied to me

To create Slave 1, I had to clone from somewhere — and in staging, the only thing I had was the master. So I took an AMI of it while it was running. No lock, no flush, just — right-click, create image, wait.

Launched the instance from the AMI, pointed it at the master with the binlog coordinates I'd noted, started replication.

Last_SQL_Error: Could not execute Write_rows event

Replication errors. The slave was choking on inconsistent data.

What had happened: the master kept accepting writes while the snapshot was being taken. By the time the AMI actually captured the disk state, the binlog position I'd noted was stale. The new instance had data from slightly-before-snapshot, but was told to start replicating from slightly-after — and the gap in between was just missing. MySQL was trying to apply writes to rows that didn't exist yet.

The fix was obvious in hindsight: lock the master first so it stops accepting writes, then snapshot.

FLUSH TABLES WITH READ LOCK;

Trigger the AMI, wait for it to hit pending state in EC2, then unlock. Slave 1 came up clean. Lesson one learned.

Staging, act three: locked, flushed, still broken

Now for Slave 2 — the future master. This time I wasn't going near the master at all. Snapshotting a write-accepting master is messy; snapshotting an existing slave is much cleaner. So the plan was: wait for Slave 1 to fully catch up with the master, stop its replication to freeze the binlog position, lock writes on it as a safety layer, then snapshot it.

I watched SHOW SLAVE STATUS until Seconds_Behind_Master hit zero and — important detail I'd learned by now — Exec_Master_Log_Pos and Read_Master_Log_Pos were identical. That means everything fetched from the master had actually been applied. Nothing sitting half-executed in the relay log.

Stopped replication. Locked writes. Triggered the snapshot. Launched the instance. Configured replication.

Still getting errors.

Not the same explosive crash as before — subtler, but wrong. I sat with this for a while. The position was frozen. The lock was on. What was I missing?

The issue is that STOP SLAVE and FLUSH TABLES WITH READ LOCK both operate at the MySQL level. They stop MySQL from doing new things. But there's a layer below MySQL — the OS's page cache — where writes that MySQL has already acknowledged can still be sitting in memory, not yet flushed to the actual disk. The snapshot grabbed the physical disk. The physical disk was slightly behind what MySQL's logs said was committed. Small gap, real inconsistency.

The fix: before locking, explicitly drain everything to disk.

STOP SLAVE;
FLUSH LOGS;

sync  # tells the kernel: stop procrastinating, write everything to disk now

Then lock, then snapshot. This time Slave 2 came up clean, synced to the master without a single error, and Seconds_Behind_Master hit zero and stayed there.

Two slaves. Two different failure modes. Both found in staging. And the exact same sequence — STOP SLAVE → FLUSH LOGS → sync → lock → snapshot — is what we'd use for Slave 4 in production.

The lock blocks new writes. FLUSH LOGS and sync drain what's already in the pipes. Without the drain, your AMI is physically inconsistent no matter how careful you are at the MySQL level.

jemalloc, and an unexpected education in Linux init systems

With the staging replication working, the next thing to figure out was actually getting jemalloc installed and loading correctly — because this was going to be a required step on Slave 4 before it could be promoted.

First problem: apt-get install libjemalloc1 failed. Our MySQL servers are old, long-lived EC2 instances — the kind that predate everyone treating servers as disposable. cloud-init was installed but broken, its hooks were firing during apt operations and causing Python version conflicts that blew the whole thing up.

Workaround: hold cloud-init so it stops interfering, download the .deb directly, install it manually with dpkg. Library installed.

Now configure MySQL to actually use it. Every Stack Overflow answer, every blog post, every piece of documentation said the same thing: add malloc-lib to [mysqld_safe] in my.cnf. So I did. Restarted MySQL. Checked if jemalloc had loaded:

sudo lsof -p $(pidof mysqld) | grep jemalloc

Nothing. Empty. Not loaded.

Okay — maybe a permissions issue on the library file? Found the path, checked the permissions, they looked off. chmod 755. Restarted MySQL. Ran the check again.

Still nothing.

Tried a few more restarts. Still nothing. At this point I was genuinely questioning whether I understood how Linux worked at a basic level.

Then I actually stopped and looked at how MySQL was being managed on this machine. Ran service mysql status. The output said mysql start/running — that specific format, that phrasing. That's upstart. Not systemd. Not SysV init. Ubuntu 14.04 uses upstart, which has its own config file living at /etc/init/mysql.conf — completely separate from my.cnf, completely separate from /etc/init.d/.

The malloc-lib setting in my.cnf only works when MySQL is started through mysqld_safe. Upstart wasn't going through mysqld_safe. It was launching mysqld directly. So my.cnf's [mysqld_safe] section was being read and then politely ignored.

Tried adding it to /etc/init.d/mysql. Also didn't work — upstart knows that file exists, nods at it respectfully, and then does whatever it was going to do anyway.

The actual fix was opening /etc/init/mysql.conf and adding one line above the exec command:

env LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1

Restarted. Ran the check. jemalloc showed up in the process's loaded libraries.

An hour, maybe more, spent on this. The config I needed was correct — LD_PRELOAD is exactly the right mechanism. The problem was putting it in a file that the thing starting MySQL actually reads. my.cnf, init.d — both ignored. The upstart job config — the one that matters.

If you're configuring how a process starts, figure out what's actually starting it first. The right answer is completely different on systemd vs upstart vs SysV. Copy-pasting config from a blog that was written for a different init system gets you nowhere.

The production window, which was supposed to be anticlimactic

Slave 4 had been live for two days before the actual window — just sitting there, replicating, catching up, while I kept half an eye on read queries hitting it to make sure nothing was weird under real-ish load. Two days of "is this thing actually fine or does it just look fine" paranoia.

The night of the actual cutover, it was me and a colleague from the team running through the checklist together — which, by this point, was less a checklist and more a sequence I could recite in my sleep. TTL dropped to 60 seconds, Slave 4's weight zeroed out in the weighted DNS record, maintenance page live on the load balancer. Celery stopped. Rhaegal — the short-form video processing pipeline — gracefully drained and confirmed dead, no pending tasks left hanging.

Checked CloudWatch and ran a quick SHOW PROCESSLIST on the master to make sure literally nothing was still hitting it. Watched SHOW SLAVE STATUS until Slave 4 was bang on zero seconds behind. Locked the master, flushed logs, ran sync, checked the binlog position twice ten seconds apart to make sure nothing had snuck in. Checked it against all four slaves — yes, four, because at this point in the night we were tracking Slave 4 alongside the three existing ones until the handoff was actually complete. Everything matched.

Stopped Slave 4's replication, took one last SHOW SLAVE STATUS screenshot for the record, reset its slave config, flipped its server-id to 1, uncommented the bits that made it a master instead of a replica, restarted MySQL. Noted its fresh binlog position. Repointed the three old slaves at it. Updated the Route 53 record for the internal master DNS, waited the couple of minutes for propagation, SSH'd in from an app server just to be paranoid and confirmed it was actually resolving to the new IP.

This is the part where, on paper, you remove the maintenance page and go to sleep.

That's not what happened.

The 3 AM plot twist

Traffic came back, but something was off. Not catastrophic — just off in a way that made my stomach drop a little. Some requests were behaving like the cutover hadn't actually happened.

Spent a good chunk of time chasing this before finding it: one of our background workers — Worker-2 — had its own auto-restart logic, and it had restarted itself sometime during the window using a cached connection that completely bypassed the DNS cutover we'd just done. It was still happily talking to the old master like nothing had changed. A process going "I'll just reconnect myself, thanks" right when you need every single thing pointed at the new source of truth is exactly the kind of edge case you don't think to check until it bites you.

And as if that wasn't enough — buried in the load balancer config was an old ALB rule nobody had flagged, still actively routing a slice of requests straight to MySQL instead of through the application layer the way it was supposed to. Easy to miss, very not-easy to debug at 3 AM with half-working traffic and a half-asleep brain.

Fixed both. Restarted Worker-2 properly so it picked up the new connection, corrected the ALB rule, watched traffic settle back into a shape that actually made sense.

After

Verified replication sync across all three slaves — clean, zero lag. Validated the read query queue was processing normally. Tested the Jodo webhook integration end to end, since that's one of the things that quietly breaks if the DB layer is even slightly inconsistent, and retriggered the handful of Jodo tasks that had queued up during the maintenance window.

Sat there watching dashboards for a while longer than strictly necessary, mostly out of nerves at this point. Signed off at 5:30 AM once everything had been stable long enough that I trusted it.

The numbers, once things settled over the following days: master memory usage dropped from a near-constant 95%+ to a steady ~70%. Swap usage, which used to sit around 5.5GB, is now essentially zero — it occasionally blips to a few hundred MB under heavy batch jobs and comes right back down. Buffer pool hit rate climbed into the high 90s within the first day as it warmed up, which is roughly what you'd expect once fragmentation stops eating into usable memory. Query latency on the master, which used to have these unexplained slow spikes every few hours (almost certainly GC-adjacent fragmentation pauses), has been flat ever since.

The CloudWatch alarms that used to fire every couple of weeks like clockwork? Quiet ever since.

The honest version of what happened

The actual cutover — locking the old master to bringing the new one live — was 5 to 10 minutes. The two days before it watching Slave 4 sync and the weeks before that spent breaking staging in increasingly specific ways were the real work. And even with all of that, production still found a way to throw something at us that staging never could have — a worker with its own auto-restart logic and a forgotten ALB rule from who-knows-when.

That's the part nobody tells you about these migrations. You can rehearse the database side until you can do it half-asleep — and I basically could, by the end. But production has more moving parts than the database. It has every cron job, every worker, every load balancer rule someone added two years ago and forgot about. The DB cutover went exactly to plan. The 3 AM debugging session happened because of everything around the database that wasn't part of the plan at all.

If you're doing one of these: rehearse the database part until it's boring. Then stay awake anyway, because something adjacent to it is going to surprise you.

Doing something similar and hit a wall? The jemalloc + upstart thing especially has basically zero documentation. Happy to help.

I said yes before I fully understood what I was signing up for. Classic fresher move confidence before comprehension. But I was deep in an infra rabbit hole at the time and this felt like exactly the kind of thing I wanted to figure out.

Context: our Django API and React SSR frontend both lived on the same EC2 instance. Fine for a fast-moving startup. But at some point you outgrow it and the real reason for this task, which I figured out later, was that the team wanted to measure and scale each service independently. You literally can't do that when two completely different apps share the same CPU and memory budget on one box.

First things first — VPCs, subnets, and the security group lesson everyone learns the hard way

Before touching any server, I had to understand how everything was connected. All our services Django, React, MySQL, MongoDB, Redis, the jobs server lived inside the same VPC. Think of a VPC as your own private slice of AWS's network. Nothing gets in or out unless you explicitly allow it.

Within the VPC, services talk over private IPs. Security groups are the firewalls at the instance level rules for what traffic each instance accepts and from where. When I spun up new instances for Django and React, I had to update SG rules so they could actually reach MySQL, Redis, and MongoDB.

💡 Rule of thumb I learned the hard way: if your app connects fine locally but times out in prod, check security groups before you check your code. Nine times out of ten, it's the SG. I absolutely forgot to update them the first time and spent an embarrassing amount of time wondering why everything was timing out.

Getting into the servers — SSH keys, bastion hosts, and feeling like a sysadmin for the first time

Most of our instances lived in private subnets no direct public access. Which is correct for security, but means you can't just SSH in from your laptop directly. The solution was a bastion host one instance sitting in a public subnet that acts as a gateway into everything else.

We stored all private keys on the bastion and set up SSH configs with identity files so you're not juggling key paths manually every time:

# ~/.ssh/config on the bastion
Host django-server
    HostName 10.0.2.45
    User ec2-user
    IdentityFile ~/.ssh/django-key.pem

Host react-server
    HostName 10.0.3.12
    User ec2-user
    IdentityFile ~/.ssh/react-key.pem

Flow: SSH into bastion → hop into any private instance you need. Clean, auditable, no scattered keys across everyone's laptops. Once I had this set up, hopping between servers became second nature.

Also made sure the bastion's SG only allowed inbound SSH from our office IPs. Please don't open port 22 to the world.

How apps actually get served — the bit nobody tells you

The full request journey:

User request → Route 53 → ALB → Target Group → EC2 → Nginx → Unix Socket → Gunicorn / PM2 → App

For Django: Gunicorn runs the WSGI app as a process, bound to a Unix socket (faster than TCP for same-machine comms). Nginx proxies to it. A systemd service file keeps it alive on restarts and crashes.

For React: PM2 manages the Node process the same way - keeps it alive, restarts on crash, handles logs. Nginx proxies incoming traffic to it.

Writing these configs from scratch - Nginx server blocks, Gunicorn socket files, PM2 ecosystem configs - is tedious but incredibly educational. You stop thinking of your app as just "code that runs" and start seeing it as a process with a lifecycle, wired to the OS through sockets and service files.

💡 The moment I wrote my first Nginx config and watched traffic actually route through it — proper "oh, THIS is how it works" moment. No tutorial gave me that. Doing it live did.

Once both services were stable, I set up:

• Separate Auto Scaling Groups and launch templates per service

• Individual target groups behind the ALB

• CloudWatch alarms per service — CPU and memory for React (spiky during peak traffic), latency and error rate for Django

• Route 53 private hosted zones so services talk over hostnames, not hardcoded IPs

Then they said "do it again" — non-prod from scratch in a new AWS account

Right after prod was stable, the next ask landed: set up the entire non-prod environment — staging, internal testing servers — in a fresh AWS account with its own VPC. Completely isolated from prod. Which makes total sense — you don't want a misconfigured staging box anywhere near your production network.

Starting from a blank account meant redoing everything: VPC, subnets, SGs, bastion, instances, Nginx, Gunicorn, PM2, ASGs — the whole stack.

Except this time there was one extra challenge that took way longer than expected: the ALB listener rules.

In prod, the routing rules were already figured out — which paths go to which target group, how Django vs React traffic splits at the load balancer. For non-prod, there was zero documentation on any of it. No runbook, no comment in a config, nothing.

I had to reverse-engineer the routing logic by reading through the codebase, watching how requests actually behaved, and trial-and-erroring through the ALB config until it matched prod behaviour. Not fun in the moment. But I probably understand our traffic routing better than anyone now because of it.

⚠️ Document your ALB rules. Even a basic table of "this path pattern → this target group → why" would've saved hours. If you're setting up infra and there's no doc, write one as you go. Future you will be grateful. Future teammates definitely will.

What came out of all of this

The real takeaway — what a fresher actually learns doing infra work

Feature work teaches you to build. Infra work teaches you how things actually run.

Before this project, "the app lives on a server somewhere" was basically my entire mental model. After it, I had the full picture — from a DNS lookup all the way down to a Unix socket on a Linux box.

The broken SG rules, the silent Gunicorn crashes, the ALB rules I had to reverse-engineer with zero docs — all of that was more educational than any course. Systems reveal themselves when they break in front of you. That's not a bad thing. It's genuinely the best way to learn.

And the meta-lesson: when a task comes with no brief and no documentation, that's not a gap — it's an invitation. Figure out the why yourself. That's where the interesting stuff lives.

If you're an early-career dev who's been handed an infra task with zero context — feel free to reach out. I was exactly there not too long ago.