The 502 Bad Gateway error isn't coming from Moodle, it's almost certainly from the server probably the Nginx web server. When Nginx encounters some PHP it passes it to the PHP-FPM process. PHP-FPM runs the PHP then sends the results to Nginx. But if the PHP takes a long time – for example with a large course backup – then Nginx may give up waiting and show a system error. In which case Nginx would need configuring to wait longer, this blog post suggests how.
Presumably Enable asynchronous backups (under Site Administration ▸ Courses ▸ Backups ▸ Asynchronous backup/restore) isn't enabled? I would try enabling this and trying the backup again. With asynchronous backups instead of Nginx having to wait, the backup is instead run by the site cron which isn't subject to the timeouts above. Of course the cron must be running for this to work.