Wen Li, a Shanghai marketer in the hospitality industry, first suspected that an algorithm was messing with her when she and a friend used the same ride-hailing app one evening.
Wen’s friend, who less frequently ordered rides in luxury cars, saw a lower price for the same ride. Wen blamed the company’s algorithms, saying they wanted to squeeze more money from her.
Chinese ride-hailing companies say prices vary because of fluctuations in traffic. But some studies and news reports claim the apps may offer different prices based on factors including ride history and the phone a person is using. “I mean, come on—just admit you are an internet company and this is what you do to make extra profit,” Wen says.
On March 1, China will outlaw this kind of algorithmic discrimination as part of what may be the world’s most ambitious effort to regulate artificial intelligence. Under the rules, companies will be prohibited from using personal information to offer users different prices for a product or service.
The sweeping rules cover algorithms that set prices, control search results, recommend videos, and filter content. They will impose new curbs on major ride-hailing, ecommerce, streaming, and social media companies.
The regulations will extend an extraordinary crackdown on China’s most popular and most valuable tech companies, which has included big fines and sidetracked stock offerings. “Some unhealthy and disorderly signals and trends have occurred in the rapid development of our country’s digital economy,” President Xi Jinping said in a speech in October, according to a translation by the newsletter Pekingnology.
Policymakers elsewhere are taking note. “I was just this morning in a meeting with the Dutch Ministry of Foreign Affairs, and I got asked if China is doing things we are not,” says Rogier Creemers, an expert on Chinese law and governance at Leiden University in the Netherlands. “They are moving extremely fast.”
The regulations, known as the Internet Information Service Algorithmic Recommendation Management Provisions, were drafted by the Cyberspace Administration of China, a powerful body that enforces cybersecurity, internet censorship, and ecommerce rules. Among other things, they prohibit fake accounts, manipulating traffic numbers, and promoting addictive content. They also provide protections for delivery workers, ride-hail drivers, and other gig workers.
Some provisions aim to address complaints about online services. Under the rules, for instance, companies will be prohibited from using personal characteristics to offer users different prices for a product; they also will be required to notify users, and allow them to opt out, when algorithms are used to make recommendations.
Companies that violate the rules could face fines, be barred from enrolling new users, have their business licenses pulled, or see their websites or apps shut down.
Some elements of the new regulations may prove difficult or impossible to enforce. It can be technically challenging to police the behavior of an algorithm that is continually changing due to new input, for instance.
But the Chinese public seems largely supportive of measures aimed at curbing the power of the big tech platforms, whose services they use every day. Lillian Li, founder of the Chinese tech publication Chinese Characteristics, compares public sentiment about the tech companies to the ambivalence of Westerners toward Facebook or Google, which offer great convenience at the cost of a user’s personal data.
And the past 18 months have shown that Chinese regulators are not shy about punishing prominent companies. The Didi ride-hailing app was pulled from Chinese app stores soon after its American IPO due to government concerns about its data practices. Ecommerce giant Alibaba has been forced to pay millions in fines over antitrust violations.
Other elements of the regulations are hazy or open to interpretation, for example provisions that order companies to “uphold mainstream value orientations,” “vigorously disseminate positive energy,” and “prevent or reduce controversies or disputes,” according to a translation by Stanford’s DigiChina project.
Another section of the proposal requires Chinese companies to avoid policies that lead users to “addiction or excessive consumption.” Similar concerns led to last year’s crackdown on celebrity culture, and strict limits on how much time minors can spend playing video games.
Even before the rules take effect, Chinese companies are making changes. ByteDance, the Beijing-based company behind the popular short-video app Douyin, in October began showing five-second videos urging users to log off after they’ve watched for a long time. The move was designed to curb addiction to the algorithmically curated feed. The company has also introduced features to Douyin and other apps that let users opt out of having personal information fed into recommendation algorithms. An investigation by one Chinese media outlet found that 26 out of 28 popular apps introduced ways for users to opt out of personalized recommendations last year, in response to another law concerning the protection of private data.
Chinese tech firms already closely monitor content to comply with rules around sharing information that the government deems harmful. Earlier this month, the social media platform Weibo removed 71,000 posts attacking Olympic athletes—including American-born figure skater Zhu Yi, who competes for China. Tencent Video posted a version of the 1999 movie Fight Club with an ending altered by the Chinese distributor (the original was quickly reinstated). And scenes depicting LGBT themes in Friends, a sitcom that is wildly popular with Chinese viewers, have reportedly been excised or altered on Chinese streaming platforms.
A separate, but related, proposal aims to address “synthetic content,” an umbrella term encompassing fake news, synthetic audio, and deepfake images and videos, in which a person’s face is stitched onto someone else using AI. Among the provisions, makers of deepfake software would be required to verify the real names of creators and “conspicuously label” any deepfaked content. Deepfake apps, popular in China, have stirred public debate over privacy and ownership of personal data.
China’s AI rules are not the world’s first. The European Union has proposed regulations that would restrict use of facial recognition, prohibit algorithmic manipulation, and regulate AI-enabled products like chatbots and computer games. But the EU rules, drafted in April 2021, could be debated for several more years if previous regulations are any guide.
Silvia De Conca, a legal scholar at Vrije University in the Netherlands, says there are parallels between China’s regulations and the proposed EU legislation—particularly a focus on informing users and letting them opt out of targeting. But, she says, European lawmakers tend to think in terms of fostering markets and individuals’ rights; in China, by contrast, there’s an emphasis on societal wellbeing. Authorities “see these issues as collective issues,” she says, pointing to rules that address morality and “mainstream values.”
The Chinese regulations are far more ambitious than anything being considered on a national level in the US, which has generally taken a hands-off approach to AI. Police use of facial recognition is restricted only in certain cities. An Algorithmic Accountability Act, proposed in the US Senate, aims to control algorithms that discriminate by gender or race, but has not progressed beyond legislative hearings. Senators Amy Klobuchar (D-Minnesota) and Cynthia Lummis (R-Wyoming) recently introduced legislation that would require platforms to implement tools to “to reduce addiction and the amplification of harmful content.” Even if such bills were to pass, however, they likely would face legal challenges from technology companies.
“The US is woefully behind in this space, and it might need to take a better degree of leadership, and help set the norms for how the technology can be used,” says Russell Wald, director of policy for Stanford’s Institute for Human-Centered Artificial Intelligence.
Wald argues that China is well-placed to influence other countries because it will have both the first working policies and home-grown companies with wide reach. He notes that the rules will not apply to the government’s use of algorithms or data, and he says that some new rules, such as those promoting particular values, are about maintaining government control. “We are talking about regulation that's geared towards benefiting the regime,” he says.
Efforts to regulate tricky new technologies in one country or region have sometimes had a discernible impact elsewhere. The EU’s General Data Protection Regulation, which took effect in 2018 and restricts companies’ use of private data, has influenced company policies worldwide, and legislation elsewhere, including in California.
Bipartisan hostility toward China means US lawmakers are unlikely to cite Chinese regulations as inspiration. But Beijing’s maneuvers could perhaps have a subtle effect. In the UK, some lawmakers have called for online companies to shield young people from harmful content in an approach that some have likened to China’s proposals.
“These ideas could ripple out,” says Matt Sheehan, a fellow at the Carnegie Endowment for International Peace who researches China’s AI ecosystem. “What's interesting in China is that they're going to be able to run experiments at a very large scale on what it actually means to implement these ideas.”
Other countries, including Pakistan, the United Arab Emirates, and Japan, are exploring strategies to manage AI. China’s new regulations are likely to be most influential in countries that already have strong ties to China; one recent academic analysis finds the reach of Chinese tech companies has helped shape policies in countries such as Pakistan. Proposed regulations in Pakistan could give the government a greater hand in deciding what sort of online content is appropriate.
“What we've done with the digital realm [in the West] is to abdicate the sense that the government has a role to play,” says Creemers, the Dutch student of Chinese law. “It's really interesting China is going after its own. In a way one has to admire that.”
- 📩 The latest on tech, science, and more: Get our newsletters!
- They were “calling to help.” Then they stole thousands
- This is how humans are messing with evolution
- Battery-powered trains are picking up speed
- How did the bored ape yacht club get so popular?
- Cybercrime will cause more real-world harm
- 👁️ Explore AI like never before with our new database
- 📱 Torn between the latest phones? Never fear—check out our iPhone buying guide and favorite Android phones
