My wife started a new job Tuesday. To make sure she wakes up on time, we set the alarm on both our phones to 6 am, just in case.
At 3 am, I woke up to feed one of our children. As she finished her bottle and went back to sleep, I noticed a white light coming from my wife’s iPhone. My first thought: Not good. Most of her passwords and work-related information is accessible via her phone.
After a few of minutes of checking it and reading online, it became clear the device is stuck in a boot loop. I knew I have a few hours to make it right, and decided to stay up and hack at it.
The rest of this article details the list of issues I encountered along the path to a full fix.
Part 1: iTunes-based OS upgrade
My Google search immediately pointed me to the fact Apple has just released iOS 14.3 a bit earlier, which explained the probably source of the issue.
Incompetence #1: there was no upgrade notification presented prior to performing the actual update . Before pointing to the fact this is controllable via the iPhone’s settings, remember that most people just keep their default settings.
I was happy to learn iTunes can “fix” a broken update without requiring a restore from backup, and immediately started the process. iTunes asked whether I want to Update (fix the iOS update, keep the data) or Recover (fresh install, requires restore from backup later on); I chose “Update”, hoping this will all finish before my wife is ready to leave for work.
While iTunes did not provide any clear message saying the update is not successful, it ended up doing a full factory reset. That meant a few days of lost data (will touch on that soon), and a much longer recovery time (full download of all data to the device).
Incompetence #2: no clear indication that update failed.
Part 2: no iCloud backup
“OK, I’ll recover from iCloud.”
Nope; nothing there. There was not a single successful backup for this phone, ever. That seemed crazy to me, but after verifying it’s the actual case, I went on to recover from our computer: by a stroke of pure luck, we backed this device up 4 days earlier as we felt the phone was slower than usual and did a full wipe and restore. Not perfect, but pretty good.
Incompetence #3: we have a 2TB iCloud account (thank you Apple for monopolistically not allowing 3rd-party backup solutions). A phone does not complete a full backup for ~3 years, and the user does not have a huge red icon yelling at them? This is a super-basic feature of backup clients.
Part 3: restore fails
At this point, my wife was already awake and concerned. The restore would take about 3 hours – she would need to take the computer with her, but it would complete the restore after her first hour at work, so we were still good.
30 minutes pass, and we get an error message: iTunes failed reading from or writing to the device, and so restore failed. Pretty obvious what to do, right?
Luckily, I’ve debugged my fair share of problems in the past – anything from low-level assembly debug of failing OS boots to networking SDK-s passing 10s of terabits per second, and so I followed a hunch: maybe I’m running out of disk space? Bingo.
I have a 1TB SSD drive, so that seemed a bit weird, but I found the culprit pretty soon; iTunes kept two backups, each ~200GB, on my drive. I deleted one that seemed to be irrelevant (more on that soon) and continued with restore.
Incompetence #4: an incomprehensible error message that would’ve kept 9 out of 10 people waiting for professional help.
Incompetence #5: are you seriously unable to ensure there’s enough disk space for restore to be successful? Would’ve saved tons of time.
Incompetence #6: turns out the 2nd backup (the one I deleted) was actually being generated during the restore operation. So, the restore I did a few days earlier just kept wasting another ~200GB of space on my drive for no useful reason.
Is it really so hard to clean up after yourself?
Part 4: a complete recovery
My wife left by that time; I let the restore process go to ~20 minutes remaining, then put both laptop and iPhone in the car and started driving. It actually took more than 5 minutes to finish from the point iTunes said “5 seconds remaining”, but hey – it is what it is.
The iPhone didn’t initially download apps since it was connected to a mobile network only; I pressed one of the apps to have it installed, only to be asked if I’d like to let “(null)” install over a cellular network.
Incompetence #7: really? (null)?
Part 5: aftermath
I ended up handing my wife the phone ~3 hours after her day started; overall, this took ~7 straight hours of my time (Apple: you owe me for that).
There was still one issue nagging me that I had to figure out: why did iCloud backup not work?
Reading through the docs, I learned that iCloud backup only happens when your phone is charging, locked, and connected to WiFi, which is what we do every night. Also, our network at home is a 200Mb/5Mb (down/up) link; at this rate I’d expect even an initial drop of 200GB to be backed up in ~30 days; we have that phone for several years already.
When my wife came home, I took the phone and triggered a manual backup; it ran a bit and failed, even though I repeated that a few times.
The day after, I asked her for the phone; turned off the automatic screen lock; connected it to power; and triggered a manual iCloud backup. A few hours later, we had our first successful iCloud backup, and I’ll be tracking it for the next few days to ensure backups continue successfully.
Incompetence #8: there’s no visibility into why this failed, and no simple way to enable backup over cellular (which would’ve had a much faster uplink).
In the end, we were able to contain this incident almost entirely.
Like many other software products, Apple software works very well for the common case, but not always for the less common cases.
We were lucky to have a working backup, and (humility aside) a talented engineer that cares enough to start working on it non-stop at 3 am.
If you have an Apple product, don’t assume your backup works just because “Apple knows how to do it”, “Apple devices have no viruses” etc. Set a monthly reminder and check that your backups are alive and well.
If you’re working for Apple, please send this to the right person(s) so they can get that fixed. No need for anyone else to suffer through this.