This raises important questions about consent and autonomy. The impact isn’t always dramatic or immediately visible, but accumulates gradually across the everyday infrastructure of modern AI. It appears in decisions about who collects data and under what conditions, as well as in the less visible choices shaping what these systems are designed, and quietly optimized, to do.
When an AI developer assembles a training dataset by scraping the web, the people whose content ends up in that dataset are typically neither consulted nor notified, let alone asked for permission. Office of the Australian Information Commissioner has made clear that publicly accessible information is not the same as information freely available for any commercial use, especially when personal data is involved.
For consent to carry real legal and ethical weight, it must be specific to a defined purpose, genuinely informed, and freely given. A platform’s privacy policy cannot simply extend that consent to a separate third party building a commercial AI model. As a result, the individuals whose words and photographs now help train billion-parameter systems were never given a meaningful opportunity to understand the implications or to opt out.
The scale of the privacy breach in AI training datasets has become concrete through recent audits. Researchers who examined DataComp CommonPool, a successor to LAION-5B and a training source for major image generation models, audited just 0.1% of its contents and found thousands of identifiable photographs, along with driver's licenses, credit cards, passports, birth certificates, and job application documents linked to real people. Because that figure came from a fractional sample, the researchers estimated that the full dataset contains hundreds of millions of images with personally identifiable information. A cognitive scientist and AI ethicist at Trinity College Dublin noted that any large-scale web-scraped dataset is likely to contain content never meant for commercial use, including children's personal records and identity documents.2
The copyright dimension of AI training data collection reinforces how systematically consent has been bypassed. In its 2025 report, the U.S. Copyright Office determined that multiple steps in building and deploying a generative AI system can constitute prima facie copyright infringement, including downloading works, converting file formats, copying data to high-performance storage, and exposing works to the model during training.
The Office further concluded that training AI systems to generate expressive content that competes with original works, particularly in commercial contexts, is less likely to qualify as fair use. Creators who discover that their novels, musical compositions, and visual art have been absorbed into model weights receive no notification, no attribution, and no compensation. This structural exclusion reflects a deeper, systemic disregard for intellectual consent.3
Non-Consensual Synthetic Media
Generative AI can produce highly realistic images and videos of people appearing to say or do things they never actually did. Non-consensual deepfakes represent a significant violation of personal privacy and autonomy, extending the impact of these systems from abstract concerns about data use into direct, tangible harm for individuals.
Recent research published in the International Review of Law, Computers & Technology found a 464% increase in such deepfakes, with women and gender-diverse individuals facing most of the negative impact. The technology removes individuals' control over their own likenesses, producing versions of their identities that circulate without participation.4
The harm operates on multiple levels simultaneously. At the individual level, victims lose the ability to determine how they are portrayed and perceived. At the societal level, the existence of convincing synthetic media degrades the reliability of visual and audio evidence, which people have historically relied upon to maintain accountability. The UK National Cyber Security Centre has warned that AI-generated deepfakes of political candidates can reach millions of viewers before corrections can spread, materially distorting democratic participation. When synthetic representations reshape what people believe they witnessed, individual and collective autonomy both erode.4
Private individuals, not just public figures, have had their likenesses used to generate content they never authorized. The disparity in capability is notable because the technical cost of producing a convincing deepfake has fallen sharply, while legal and social mechanisms for obtaining consent and providing redress remain underdeveloped. Researchers and advocates working on gender-based violence have documented that nonconsensual AI-generated intimate imagery targeting ordinary people has grown common, constituting a form of harm that existing consent frameworks have not yet adequately addressed.5
AI-Driven Manipulation of Decisions
Generative AI’s challenge to human autonomy extends well beyond synthetic media and into the mechanics of everyday decision-making. Researchers have examined how AI agents configured with manipulative objectives can shape human choices in both financial and emotional contexts. Even relatively simple manipulative goals, without advanced psychological strategies, proved effective at steering participants toward decisions that served the AI’s hidden incentives rather than their own interests.
Participants interacting with these manipulative agents exhibited significantly higher rates of harmful decision-making compared to those engaging with neutral systems. Over time, their ability, and inclination, to select genuinely optimal choices declined, suggesting that the influence of such systems can compound across repeated interactions rather than remaining isolated or short-lived.6,7
This finding carries significant implications for how autonomy operates in human-AI interactions. Autonomy requires that decisions reflect a person's own values and reasoning rather than be subject to covert external interference. When a generative AI system is optimized for an objective that diverges from user welfare without the user knowing this, choices that feel freely made are compromised. Research involving AI chatbot interactions has argued that false beliefs generated by AI systems sever the connection between a person's actions and their authentic intentions, effectively making those actions unreliable expressions of the person's genuine values.7,8
Epistemic Autonomy and Collective Belief
Generative AI reshapes not just individual decisions but the broader informational environments in which people form their beliefs. A recent ACM Digital study on epistemic injustice in generative AI identified four dimensions through which these systems undermine collective knowledge, namely amplified testimonial injustice, manipulative testimonial injustice, hermeneutical ignorance, and access injustice. The authors argue that generative models reproduce unjust narratives and generate new interpretive frameworks for events, embedding bias and misinformation into a shared epistemic ecosystem that individuals cannot easily audit or correct. Outputs carry the appearance of fluency and authority, which makes them harder to challenge even when inaccurate.9
Research published in the Harvard Misinformation Review has noted that generative AI enables the creation of personalized misinformation tailored to individual belief profiles, making false content more persuasive and harder to identify. This personalization means that autonomy risks are not evenly distributed. People whose existing beliefs make them susceptible to particular forms of persuasion face informational environments strategically shaped to reinforce those vulnerabilities. Genuine autonomous reasoning depends on access to information that is both accurate and not optimized to guide the reader toward predetermined conclusions.10
Rebuilding Consent as a Design Requirement
Regulators have begun articulating what meaningful consent for AI systems requires in practice. Australia's information commissioner has specified that consent for training generative AI models must be voluntary, current, and specific, and that a broad privacy policy does not meet that standard. Individuals must receive accessible information about how their data will be used across all stages of model development, and opt-out mechanisms must be substantive rather than nominal. These requirements shift the burden toward developers, obliging them to restructure data practices rather than rely on passive acquiescence obtained under terms most users never read.1
The violations documented across data collection, synthetic media, manipulative agents, and epistemic distortion share a common origin. Generative AI systems were built and scaled without sufficient regard for the consent and autonomy of the people whose data, likenesses, and cognitive attention those systems depend on. Treating consent as a fundamental technical and ethical requirement at every stage of development, from dataset curation through interface design and deployment, is the necessary starting point for correcting that deficit.7-9
Download the PDF of this article here
References and Further Reading
- Guidance on privacy and developing and training generative AI models. (2024). Australian Information Commissioner. https://www.oaic.gov.au/privacy/privacy-guidance-for-organisations-and-government-agencies/guidance-on-privacy-and-developing-and-training-generative-ai-models
- Massive AI Dataset Breach: DataComp CommonPool Reveals Widespread Personal Data Exposure. (2025). Captain Compliance. https://captaincompliance.com/education/massive-ai-dataset-breach-datacomp-commonpool-reveals-widespread-personal-data-exposure/
- Levi, S. D. et al. (2025). Copyright Office Weighs In on AI Training and Fair Use. Skadden Publication. https://www.skadden.com/insights/publications/2025/05/copyright-office-report
- Romero Moreno, F. (2024). Generative AI and deepfakes: a human rights approach to tackling harmful content. International Review of Law, Computers & Technology, 38(3), 297–326. DOI:10.1080/13600869.2024.2324540. https://www.tandfonline.com/doi/full/10.1080/13600869.2024.2324540
- Non-consensual deepfakes, consent, and power in synthetic media. (2026). Geneva Internet Platform DigWatch. https://v45.diplomacy.edu/updates/non-consensual-deepfakes-consent-and-power-in-synthetic-media
- Sheriff Y. Ahmed, & Jamshid Pardaev. (2025). Human-AI Decision Dynamics: How Risk Propensity and Trust Impact Choices Through Decision Fatigue, Conditional on AI Understanding. Decision Making: Applications in Management and Engineering, 8(2), 96–113. DOI:10.31181/dmame8220251484. https://dmame-journal.org/index.php/dmame/article/view/1484
- Sabour, S. et al. (2025). Human Decision-making is Susceptible to AI-driven Manipulation. arXiv e-prints. DOI:10.48550/arXiv.2502.07663. https://ui.adsabs.harvard.edu/abs/2025arXiv250207663S/abstract
- Lu, W., & Hu, Z. (2025). Addressing Autonomy Risks in Generative Chatbots with the Socratic Method. Science and Engineering Ethics, 31(6), 41. DOI:10.1007/s11948-025-00567-8. https://link.springer.com/article/10.1007/s11948-025-00567-8
- Kay, J. et al. (2025). Epistemic Injustice in Generative AI. AIES '24: Proceedings of the 2024 AAAI/ACM Conference on AI, Ethics, and Society. DOI:10.5555/3716662.3716722. https://dl.acm.org/doi/10.5555/3716662.3716722
- Simon, F. M. et al. (2023). Misinformation reloaded? Fears about the impact of generative AI on misinformation are overblown. Harvard Kennedy School Misinformation Review. https://misinforeview.hks.harvard.edu/article/misinformation-reloaded-fears-about-the-impact-of-generative-ai-on-misinformation-are-overblown/
Disclaimer: The views expressed here are those of the author expressed in their private capacity and do not necessarily represent the views of AZoM.com Limited T/A AZoNetwork the owner and operator of this website. This disclaimer forms part of the Terms and conditions of use of this website.