Hija Kamran

In Pakistan, online gendered violence is often framed around visible harms like harassment, blackmail, and the non-consensual sharing of intimate images. But the violence rarely begins with a viral post or a threatening message. Instead, it begins with data, like phone numbers circulated without consent, women coerced into befriending strangers, photographs scraped from social media profiles, identity related information exposed through data breaches, and location data shared through apps – all of this form the foundation of what is now widely recognised as technology-facilitated gender-based violence (TFGBV). These are all the patterns rooted in weak data protection, poor privacy safeguards, and limited accountability for perpetrators of violence and of law enforcement authorities.

The Digital Rights Foundation’s Digital Security  Helpline has consistently documented complaints involving blackmail, non-consensual intimate image sharing, hacking, deepfakes, and breaches of privacy and trust. These cases demonstrate that gendered online violence is enabled by an ecosystem that allows personal data to be extracted and circulated with minimal oversight. Platforms collect and monetise user data at scale, while regulatory frameworks remain fragmented and under-enforced. In this environment, privacy is treated as an afterthought compared to growth and engagement, and survivors are left navigating legal and technical systems that offer little meaningful redress.

The country’s governance landscape further complicates this reality. Debates around data regulation have often prioritised state’s access to information and control over digital spaces rather than centring user rights and protections. Without a robust, rights-based data protection regime and clear accountability obligations for platforms, the burden of safety continues to fall on individuals, particularly women and marginalised communities who navigate digital spaces and civil liberties distinctively differently than others. Additionally, in a country of over 220 million people, rapid digital adoption presents significant commercial opportunities for technology companies, without essential rights-based safeguards, to advance their services and products in exchange for a lucrative business revenue.

Furthermore, with the lack of a data protection law, sensitive data including biometric records, contact information, private media, and real-time location details, all of which is liberally collected by the public and private sectors, moves across digital ecosystems with limited transparency and weak institutional oversight. Civil society organisations and independent researchers have repeatedly raised concerns about recurring data breaches and inadequate safeguards, including reported leaks allegedly linked to national databases and telecom records.

At the same time, research by digital rights groups has documented how women face threats involving the non-consensual sharing of private photos that are facilitated by social media spaces, often through cross-platform sharing features built into social media infrastructure. These features allow personal content to travel rapidly beyond its original context, often without meaningful consent or traceability. In the absence of clear liability standards and robust enforcement mechanisms, both state and corporate actors operate within a framework where accountability for data misuse remains absent, and accessing justice for affected individuals remains difficult.

When biometric identity systems intersect with weak privacy protections and platform data harvesting, the risks for women and marginalised communities deepen. The convergence of national ID databases, telecom registration requirements, and social media profiling means that personal identifiers can be linked and exposed in ways that make individuals more traceable and more vulnerable. As a result, doxxing, impersonation, harassment, and AI-generated intimate content are operationalised as easily as the data is shared across digital networks where rights are only stifled with no legal recourse. For example, Amnesty International has documented how online abuse against women often involves coordinated harassment campaigns that rely on the circulation of personal information and manipulated images.

As a result, the violence weaponises personal data for perpetuating harm as well as boosting engagement-driven ecosystems that profit off the activities that violent content generates.

Generative AI further intensifies these privacy harms. Globally, researchers have documented how AI tools are being used to create deepfake sexual imagery, overwhelmingly targeting women. One study estimates that up to 95% of all deepfakes are non-consensual pornographic images, with around 90% depicting women. In Pakistan, journalists and activists have reported rising concerns about AI-generated intimate images being used to target them as an additional layer to existing methods of violence that they have been facing. The speed, scale, and ease with which such material can now be produced and circulated is closely tied to platform recommendation systems that privilege virality and emotional intensity.

This dynamic emerges as a structural and infrastructural problem of privacy and power. Platforms collect vast amounts of behavioural and personal data to optimise engagement; recommendation algorithms, ranking systems, and monetisation features then amplify content that provokes strong reactions, including outrage and sexualised harm. A 2016 internal Facebook study revealed that the company’s own research found its algorithms “exploit the human brain’s attraction to divisiveness,” and that content eliciting anger received disproportionately higher distribution because it drove engagement. The same disclosures showed that 64% of extremist group joins on Facebook were due to its recommendation tools. These findings demonstrate that algorithmic design choices, instrumentalised to maximise engagement, can systematically amplify polarising and harmful content because it performs well commercially.

Violent, hateful, and misinformed content is widely recognised to generate high levels of engagement because it provokes strong emotional reactions, prompting users to comment, share, or respond. These interactions, while experienced as anger, fear, or outrage by individuals, are processed by platform algorithms as signals of interest. Most engagement-based systems do not distinguish between positive and negative interaction, so each click, share, or reply is treated as an indicator of relevance. As a result, the algorithm is more likely to recommend similar content to the same user and to others within their network, expanding the visibility and impact of material that would otherwise warrant restriction.

Business of hate

The amplification of hate speech and gendered abuse on social media is not an accidental byproduct of scale. It is closely tied to business models that prioritise engagement, growth, and advertising revenue. Most major platforms operate on an attention economy logic, i.e. the longer users stay, react, comment, and share, the more data is generated and the more targeted advertising can be sold. Recommendation systems and algorithmic ranking are especially optimised to surface content that provokes strong emotional responses.

Independent research corroborates this dynamic, as seen in a 2021 study published in Science Advances analysing posts on Facebook containing language aligning with moral outrage, and how they were more likely to be shared and spread widely, reinforcing the platform’s tendency to amplify emotionally charged and polarising speech. Platforms monetise this engagement through targeted advertising, which is fuelled by behavioural data collected from user interactions. As I report in my analysis for the Center for the Study of Organized Hate (CSOH), social media platforms’ revenues are directly linked to user attention and data extraction, creating structural incentives to prioritise engagement over safety.

In South Asia, these structural incentives intersect with fragile political contexts and deeply embedded gender hierarchies. During periods of political tension between India and Pakistan in 2025, it was widely documented how inflammatory content spreads rapidly across platforms, often promoted by recommendation systems that reward virality and emotional intensity. A report by The Guardian highlighted how social media fuelled disinformation during the war between the two countries.

While such campaigns rely on organised actors, their scale and reach depend on algorithmic amplification that privileges content already generating high engagement. In gendered contexts, the same infrastructure facilitates the rapid circulation of non-consensual intimate images and targeted harassment. The UN Special Rapporteur on violence against women has recognised TFGBV as a systemic issue, noting that platform design and algorithmic amplification intensify abuse by increasing visibility and persistence of harmful content.

Monetisation tools further strengthen these dynamics, as features like ad revenue sharing as offered on X, YouTube’s monetisation models, creator funds, and monetised live streams directly and indirectly incentivise sensational or abusive content if it drives traffic and interaction. Although platforms maintain community standards prohibiting hate speech and harassment, enforcement remains uneven, while algorithmic systems continue to prioritise engagement metrics. The Mozilla Foundation has repeatedly highlighted that recommender systems on major platforms like YouTube amplify harmful content because their primary objective is to maximise time spent and clicks, not to safeguard users. So when hateful or misogynistic content generates high engagement, it becomes profitable within this ecosystem, regardless of its social cost, and hence it is not in the financial interest of the company to moderate this content.

This commercial logic complicates the narrative that harmful speech spreads solely because of individual bad actors. Instead, hate speech and gendered abuse are embedded within a technical and economic architecture that lowers the cost of targeting and increases the rewards for virality. Cross-platform sharing tools enable content to travel seamlessly between Facebook, Instagram, WhatsApp, X, and YouTube, multiplying exposure. Algorithmic ranking systems prioritise posts with high interaction rates, creating feedback loops in which outrage begets visibility, which in turn generates more engagement and advertising revenue, and the cycle goes on. In this sense, amplification is structurally aligned with profit incentives. Addressing hate speech and gendered violence online therefore requires scrutiny not only of moderation policies but of the underlying revenue models, algorithmic optimisation strategies, and platforms’ infrastructures that make such content commercially advantageous.

This model of prioritising engagement over safety has become even more prominent with the rise of generative AI tools linked directly into social platforms. A recent troubling example is Grok, the AI chatbot deployed on X. Between late December 2025 and early January 2026, analysis estimated that Grok generated millions of sexualised deepfake images, many of them depicting women in sexualised settings. According to the Centre for Countering Digital Hate (CCDH), Grok produced about 3 million images in a matter of days, with 65% of the sample data appearing to contain sexualised imagery. Users were coaxing the system with prompts that turned ordinary photos of women into images wearing bikinis or lingerie, sometimes with offensive or demeaning edits, which were then shared widely across public timelines and feeds.

These incidents reveal how algorithmic systems don’t merely respond to user behaviour, instead they increase the visibility of certain kinds of abuse because those kinds of posts generate attention that can be monetised directly or indirectly. AI-generated sexual content is precisely the sort of ‘engaging’ material that keeps people interacting with the platform by generating shares, replies, views, and subscriptions tied to premium features and ad impressions. When Grok was criticised for producing this sexualised content, X’s response was not to entirely disable the problematic feature or fundamentally retrain the model, it instead restricted access and in some contexts linked features to paid subscriptions. Users were told that certain generative functions were available only to paying customers, or geoblocked where local laws prohibited them, essentially enforcing a pay-to-play model rather than safety-first design.

The fact that Grok could generate thousands of sexualised images without robust safeguards in place reflects an incentive structure that places profitability and user engagement ahead of harm prevention. Platforms like X have seen major advertisers leave or reduce spending in the past, but still the underlying systems continue to emphasise engagement-first design because more interactions translate into more ad inventory and proof of higher revenues for both the advertisers and the platforms. Algorithms that pull users deeper into networks of sensational or provocative content regardless of whether that content is harassment, deepfakes, harmful, or hateful, function as infrastructure for amplification. These same dynamics apply beyond Grok as psychological triggers like happiness, sadness, anxiety, fear and anger are drivers of engagement, and engagement is the commodity sold to advertisers at scale.

In the context of gendered violence online, this intersection of algorithmic design and commercial incentive is especially concerning. Non-consensual image creation, harassment campaigns, and coordinated abuse become amplified not just by users who intend to harm others, but by platforms whose business models reward the engagement such content generates. As a result, gendered violence and harmful content thrive not because platforms lack rules, but because the architecture and optimisation logic of these systems embeds incentives for visibility, interaction, and retention that are antithetical to safety. Addressing this requires more than reactive moderation of individual posts, and calls for systemic reform of the engagement-driven infrastructure that forms the foundation of modern social media and AI platforms.