Content Migration to WPEngine the Easy Way with LFTP

It was time to handoff a project and the perfect answer to my hosting question came back: WPEngine. WPEngine’s hosting is worlds ahead of everyone else’s, especially their eCommerce hosting (that is an affiliate link and I’ll receive a small commission if you sign up with them after clicking it). I always found the WPEngine migration process to be a headache though because they block shell access (no longer accurate as of 2021). To WPEngine’s credit, they’ve developed an automatic migration tool. To my dismay, the tool is slow and doesn’t play well with Trellis and Bedrock (it’s gotten faster, but still doesn’t do so well with Trellis migrations—SSH/scp/rsync transfers are your best bet these days).

After I manually migrated my theme, plugins, and database to WPEngine, it was time to migrate the content. The project that spawned this post sat on a staging server for a while and the client added a lot of content they wanted included in the migration… over 2 GB of image uploads.

The prevailing wisdom for manually migrating content to WPEngine is to download the content from your staging server to your local machine, then upload it via SFTP to WPEngine. SFTP is really your only option. With a media library larger than 2 GB and a very slow upstream thanks to my shoddy ISP’s infrastructure, I was facing a potential multi-day migration. I knew there had to be a better way. My beautiful DigitalOcean droplet was sitting there with outstanding untapped upstream. My 2 GB media library could transfer over in less than a minute even at 300Mbps/37.5MBps upstream. Man, I really needed to bypass the download and re-upload step. Enter LFTP.

LFTP is a sophisticated file transfer program with command line interface and it supports many transfer protocols. The one we obviously care about is SFTP. I actually found LFTP on Samuel Reed’s tutorial, which is a great starting point. I’d used it to transfer local files to a remote, but I never tried transferring files from one remote host to another and using his tutorial wasn’t perfectly smooth.

My Trellis server was running on a DigitalOcean droplet with Ubuntu 16.04.3 and here are the steps I took to transfer over 2 GB of content to WPEngine in about 45 seconds(!!):

1. Install LFTP on the remote/staging server $ sudo apt-get install lftp
2. If ~/.ssh/config exists then make sure PasswordAuthentication yes is in it; if the file doesn’t exist then create it with that one line
3. cd into the content directory that contains the dirs and files you need to transfer over
4. Run $ lftp -e "set sftp:auto-confirm yes;" sftp://<USERNAME>@yourdomain.sftp.wpengine.com:2222
5. You should be prompted for your password; enter it
6. cd into the destination directory for your file transfer—if this fails then see troubleshooting notes below
7. Run mirror -R --parallel=4
8. Your files should transfer from one remote host to WPEngine with lightning speed!
9. Depending on your security needs you may want to reset ~/.ssh/config back to PasswordAuthentication no

Some troubleshooting notes. If you run into any issues at all you can enable LFTP’s debugging by including it in your extra arguments like this: `$ lftp -e "debug; set sftp:auto-confirm yes;" sftp://<USERNAME>@yourdomain.sftp.wpengine.com:2222. It seems to only connect when you actually give it a command, so I like to use ls just as a quick test. If that doesn’t work then you can troubleshoot by looking at the debug output. Mine says something like:

---- Running connect program (ssh -a -x -s -l <USERNAME> -p 2222 yourdomain.sftp.wpengine.com sftp)

So I know that if I can’t complete that SSH (really SSH2) command successfully then LFTP itself will fail. I attempted to make that connection directly via $ ssh -vvv -a -x -s -l <USERNAME> -p 2222 yourdomain.sftp.wpengine.com sftp (I added -vvv for more details) and after some chin scratching and searching that led me to the SSH config file issue. I knew my issues were SSH related because I’d received error messages such as:

• Fatal error: Host key verification failed (this is why I added set sftp:auto-confirm yes; up above in #4)
• Permission denied (publickey,password)

You may encounter different issues and I suggest you go about troubleshooting the same way: debug LFTP to see exactly what it’s running to connect to WPEngine and run that on its own to figure out why that fails. Once you can connect via that SSH command to WPEngine then you can go back to LFTP and you should be ready to rock and migrate your content to WPEngine the easy way.

So much for FileZilla!