Technology – Beyond good and evil

My wife started a new job Tuesday. To make sure she wakes up on time, we set the alarm on both our phones to 6 am, just in case.

At 3 am, I woke up to feed one of our children. As she finished her bottle and went back to sleep, I noticed a white light coming from my wife’s iPhone. My first thought: Not good. Most of her passwords and work-related information is accessible via her phone.

After a few of minutes of checking it and reading online, it became clear the device is stuck in a boot loop. I knew I have a few hours to make it right, and decided to stay up and hack at it.

The rest of this article details the list of issues I encountered along the path to a full fix.

Part 1: iTunes-based OS upgrade

My Google search immediately pointed me to the fact Apple has just released iOS 14.3 a bit earlier, which explained the probably source of the issue.

Incompetence #1: there was no upgrade notification presented prior to performing the actual update . Before pointing to the fact this is controllable via the iPhone’s settings, remember that most people just keep their default settings.

I was happy to learn iTunes can “fix” a broken update without requiring a restore from backup, and immediately started the process. iTunes asked whether I want to Update (fix the iOS update, keep the data) or Recover (fresh install, requires restore from backup later on); I chose “Update”, hoping this will all finish before my wife is ready to leave for work.

While iTunes did not provide any clear message saying the update is not successful, it ended up doing a full factory reset. That meant a few days of lost data (will touch on that soon), and a much longer recovery time (full download of all data to the device).

Incompetence #2: no clear indication that update failed.

Part 2: no iCloud backup

“OK, I’ll recover from iCloud.”

Nope; nothing there. There was not a single successful backup for this phone, ever. That seemed crazy to me, but after verifying it’s the actual case, I went on to recover from our computer: by a stroke of pure luck, we backed this device up 4 days earlier as we felt the phone was slower than usual and did a full wipe and restore. Not perfect, but pretty good.

Incompetence #3: we have a 2TB iCloud account (thank you Apple for monopolistically not allowing 3rd-party backup solutions). A phone does not complete a full backup for ~3 years, and the user does not have a huge red icon yelling at them? This is a super-basic feature of backup clients.

Part 3: restore fails

At this point, my wife was already awake and concerned. The restore would take about 3 hours – she would need to take the computer with her, but it would complete the restore after her first hour at work, so we were still good.

30 minutes pass, and we get an error message: iTunes failed reading from or writing to the device, and so restore failed. Pretty obvious what to do, right?

Luckily, I’ve debugged my fair share of problems in the past – anything from low-level assembly debug of failing OS boots to networking SDK-s passing 10s of terabits per second, and so I followed a hunch: maybe I’m running out of disk space? Bingo.

I have a 1TB SSD drive, so that seemed a bit weird, but I found the culprit pretty soon; iTunes kept two backups, each ~200GB, on my drive. I deleted one that seemed to be irrelevant (more on that soon) and continued with restore.

Incompetence #4: an incomprehensible error message that would’ve kept 9 out of 10 people waiting for professional help.

Incompetence #5: are you seriously unable to ensure there’s enough disk space for restore to be successful? Would’ve saved tons of time.

Incompetence #6: turns out the 2nd backup (the one I deleted) was actually being generated during the restore operation. So, the restore I did a few days earlier just kept wasting another ~200GB of space on my drive for no useful reason.
Is it really so hard to clean up after yourself?

Part 4: a complete recovery

My wife left by that time; I let the restore process go to ~20 minutes remaining, then put both laptop and iPhone in the car and started driving. It actually took more than 5 minutes to finish from the point iTunes said “5 seconds remaining”, but hey – it is what it is.

The iPhone didn’t initially download apps since it was connected to a mobile network only; I pressed one of the apps to have it installed, only to be asked if I’d like to let “(null)” install over a cellular network.

Incompetence #7: really? (null)?

Part 5: aftermath

I ended up handing my wife the phone ~3 hours after her day started; overall, this took ~7 straight hours of my time (Apple: you owe me for that).

There was still one issue nagging me that I had to figure out: why did iCloud backup not work?

Reading through the docs, I learned that iCloud backup only happens when your phone is charging, locked, and connected to WiFi, which is what we do every night. Also, our network at home is a 200Mb/5Mb (down/up) link; at this rate I’d expect even an initial drop of 200GB to be backed up in ~30 days; we have that phone for several years already.

When my wife came home, I took the phone and triggered a manual backup; it ran a bit and failed, even though I repeated that a few times.

The day after, I asked her for the phone; turned off the automatic screen lock; connected it to power; and triggered a manual iCloud backup. A few hours later, we had our first successful iCloud backup, and I’ll be tracking it for the next few days to ensure backups continue successfully.

Incompetence #8: there’s no visibility into why this failed, and no simple way to enable backup over cellular (which would’ve had a much faster uplink).

Epilogue

In the end, we were able to contain this incident almost entirely.

Like many other software products, Apple software works very well for the common case, but not always for the less common cases.

We were lucky to have a working backup, and (humility aside) a talented engineer that cares enough to start working on it non-stop at 3 am.

If you have an Apple product, don’t assume your backup works just because “Apple knows how to do it”, “Apple devices have no viruses” etc. Set a monthly reminder and check that your backups are alive and well.

If you’re working for Apple, please send this to the right person(s) so they can get that fixed. No need for anyone else to suffer through this.

Cheers,

Yair

This post helps set-up a VPN service on a home router such that different devices can use different VPN tunnels. This allows setups such as having gaming go through a local tunnel (or no tunnel), while having your streaming go through a tunnel leading to a different geography.

I recently decided to start using a paid VPN service for my personal connectivity: mobile, home computing and streaming.

VPN-s are great in providing you basic privacy and some anonymity, masking your IP, and allowing you to access geo-specific content. You can read more about VPN-s here.

After quickly enabling ExpressVPN on our family’s mobile devices, I went about installing it for our home network. While many VPN services offer up to N parallel connections at once (5 in ExpressVPN’s case, circa October 2019), I encountered two issues:

We have >5 devices (several phones, personal and work laptops, streamers etc).
Some of our devices (read: Roku Streaming Stick) do not support a built-in VPN connection.

To handle these, I decided to install the VPN client on my home router instead of each local device.
During the process, a new requirement came in: different devices should connect to different VPN tunnels.
This is important since streaming services are insensitive to latency, whereas gaming and work-related remote connectivity is.
I therefore wanted to have both a US tunnel (for streaming) and a local one (for most of my other work).

Multiple searches didn’t come up with a trivial way to do it that is easy for an inexperienced person – one that’s not regularly familiar with Linux, routing and towels. Moreover, while you can get routers with a pre-installed VPN service, these were much more expensive than stock routers and still do not support the multiple tunnels requirement.

Therefore, I’m including a very short version here (you still have to work for it) of the setup I ended up with, and how to achieve it. Overall, this should take you less than an hour with the right router.

Step 1: Get an Asuswrt-Merlin compatible router

Asuswrt-Merlin is a custom firmware for some Asus routers (read: it’s special software that makes the router more customizable). Any router on the supported devices list should be fine.

Due to personal preference, I was not initially enthusiastic about getting an Asus router (do not like their looks). After a lot of searching online, I ended up deciding this is the fastest, simplest path forward.

Step 2: Get a VPN subscription

I chose to go with ExpressVPN. Now that I’m using the service for a few months, I can say their support is excellent, and the service is definitely worth a few extra dollars a month.

Step 3: Install Asuswrt-Merlin on your router

First, download the firmware here. Make sure to choose the latest release, not latest beta.

Now, open your router’s administration console at http://router.asus.com. If you haven’t changed the username/password, it’ll be the default (typically admin/admin).
Exact instructions to upload the new firmware are here.

When you’re done, you will need to connect to your router using its IP; that will typically be at http://192.168.50.1.

Step 4: Configure your VPN tunnels in the router

In the router administration console, go to ‘Advanced Settings’, ‘VPN’, ‘VPN client’, ‘OpenVPN’. You have up to 5 configurable tunnels.

For each tunnel you want to set up, just give it a name (the Description field), and choose ‘Automatic start at boot time’.
Finally, you’ll need an OpenVPN (*.ovpn) file with the actual VPN settings.

If you went with ExpressVPN, log in to your account; choose ‘My account’, ‘Set up ExpressVPN’; ‘Router’; ‘Asus (including Merlin)’; ‘Configure OpenVPN’.

Now, just copy the username and password (and place them in the router’s tunnel configuration; these are not the username and password you used to register with your provider; the VPN provider’s page you’re currently on will have these; copy them from that page).

Next, select the exact VPN location you want for this tunnel; you will then download the relevant OpenVPN file, and that file can be uploaded to your router through the ‘Import .ovpn file’ option in the screen you’re currently at.

After you’re ready, change the ‘Service State’ from Off to On. If it works, you’ll see a ‘Connected’ note near it.

Note: do not forget to click ‘Apply’ when done on each screen, otherwise your settings may not get stored.

Step 5: choose the right tunnel for each device

To associate a device with the right tunnel, simply go to the tunnel’s settings (same way you did in step 4).

At the bottom of the screen, click ‘Source IP’ and choose your device from the list. For ‘Iface’ (interface), choose VPN if you want traffic from this device to go through this tunnel, or WAN if you do not want it to go through the tunnel. No value is needed in the ‘Destination IP’ field; just press the + button on the right when ready.

Again, do not forget to apply the settings before leaving this screen.

Step 6: Testing

For mobile devices, go to your VPN provider’s page and use their IP check. This is the one for ExpressVPN.

Step 7: Bender is great!

Relax and enjoy your new setup.

Cheers.

Category: Technology

iPhone backup: a case of software incompetence