It does seem like sooner or later, if someone is able to build a reliable AI model of my face and voice, they could even phish my own relatives by video call.
Seems like a Philip K. Dick novel—objective reality is something you could only see around you, while the machine would be completely untrustworthy.
An attack that would be protected against with a yubikey, webauthn, or passkey.