The social media network X has sparked widespread anger after individuals used its artificial intelligence assistant Grok to manipulate photos of females by stripping their garments.
Multiple instances have been seen where the system has been used to remove clothing from female subjects to render them in swimwear non-consensually, as well as to insert them in sexualized scenarios.
Developer Reaction and Legal Stance
XAI, the firm that created Grok, offered no comment to questions, other than an automatically-generated message stating "legacy media lies".
A government spokesperson stated that new laws is being brought forward to ban nudification software. Under a proposed law, anyone who distributed such tools would "receive a prison sentence and heavy penalties".
"X or Grok has the ability to stop these forms of abuse if they wanted to," said a law professor, adding they "operate with a lack of consequences."
Oversight Body Position and Tool Functionality
The communications watchdog Ofcom said tech firms must "consider the danger" of people accessing illegal content on their services, but did not confirm whether it was presently investigating X or Grok in regarding artificial intelligence images.
Grok is a no-cost chatbot—with some paid-for features—which responds to X users' prompts when they mention it in a post.
It is often used to offer reactions or further context to other posters' remarks, but people on X are also can edit an posted photo through its AI image editing tool.
History of Criticism and Rules
The system has been criticised for allowing users to create photos and footage with explicit content and sexually suggestive material, and it was previously accused of generating graphic material of celebrities.
"The platform has been allowing the generation and distribution of these pictures for months without taking any action and we have not seen any intervention by oversight bodies," Professor McGlynn added.
XAI's own acceptable use policy bans "showing images of persons in a graphic manner".
In a statement, Ofcom said it was illegal to "generate or distribute deepfakes or illegal content" and confirmed this covered sexual deepfakes generated with AI.
It said networks such as X were required to take "suitable measures" to "reduce the danger" of users coming across illegal content on their services and to remove it swiftly when they become aware of it.
