Hi - The RE650 has impressive credentials. I am not familiar with it from an experience point of view. A few suggestions for your consideration.
The Optus Modem. Latest Firmware SG4Y1T001009
When the F@ST5366TN was released it showed two separate broadcast names (SSID) for the 2.4 Ghz and 5 Ghz bands and I understand the latest firmware '001009' introduced Band Steering which allows one common SSID for both bands. I don't have that version, if you have the latest firmware, is it an option that you can toggle it ON/OFF, if so, can you Disable it so that both bands are broadcasting their SSID. The previous model F@ST3864V3HP had configurable Beam Forming settings, but I couldn't see them on my firmware version '000105' - suspect it may ON and hidden as the Optus modem is Wave 2 802.11ac compliant.
Beam Forming - 802.11ac
The RE650 is a modern Wave 2 802.11ac device providing high speed functionality to both bands. In theory, if all devices in the coverage zone of the Range Extender support the same form of Beam Forming method as the RE650 , the RE650 can focus its beams rather than generating a signal in a standard spherical pattern. Within the beam zone only, a maximum of 4 Wi-Fi devices can be served (Standard defined). I would expect it to service multiple beam zones given its other credentials. I don't think that is the issue with your unit as there is no option to put it ON/OFF for the backhaul link to the Optus modem. Up until Wave 2 802.11ac, there was no Standard for Beam Forming - two types exist 'Explicit' - where the Router and the WiFi Adaptor in a wireless device support the same Standard and 'Implicit' where the device Wi-Fi Adapter doesn't support the same version ( has its own proprietary version) or if an old legacy device have does not have the function - in Implicit arrangement, the former scenario the Router does its best guess to focus on the location of remote devices.
The TP-Link RE650
The last entry "Check server connection - not ok" points to the RE650 not seeing the Optus Gateway address. The RE650 defaults to 192.168.0.254 for its own IP address and the same for its Gateway; Page 16 of User Guide. This needs to show the Optus Gateway address, but first some other changes need to be implemented. The RE650 comes with its DHCP Server ON using its IP address as 192.168.0.254 and range 192.168.0.100 - 192.168.199. The 192.168.0.254 can be left as is if it is not used by another device on the Optus modem - if so, it is good practice to set that address as a Static fixed address within the Optus modem.
Can you try these settings for both screens shown on Page 16 to see if connection to Internet works.
IP Address: 192.168.0.254
Subnet Mask: 255:255:255:0
Default Gateway: 192.168.0.1 (Optus modem) and SAVE.
DHCP Server - change to OFF - all devices will be assigned their IP address from the Optus Modem.
Default Gateway : change to 192.168.0.1 (Optus Modem)
Hope this helps.