Researchers found that large language models (LLMs) yield concerning results when identifying login URLs, delivering incorrect information one-third of the time. Specifically, 30% directed users to unregistered or inactive domains, increasing risks of phishing attacks, while 5% referred them to unrelated organizations. These findings demonstrate that AI can easily mislead users without proper URL validation mechanisms. The potential for threat actors to exploit AI-suggested links emphasizes the near necessity for protective measures to ensure users access legitimate sites and mitigate risks associated with phishing and malware deployment.
If AI suggests unregistered or inactive domains, threat actors can register those domains and set up phishing sites. As long as users trust AI-provided links, attackers gain a powerful vector to harvest credentials or distribute malware at scale.
Without guardrails enforcing URL correctness, AI responses can mislead users. Guardrails should validate domain ownership before recommending login, and any request/response containing a URL can be vetted using common practices.
Collection
[
|
...
]