Why Cloudflare wants AI companies to pay for content
On TechCrunch's Equity podcast, hosts Kirsten Korosec and Maxwell Zeff discuss Cloudflare's new experiment to make AI companies pay for content. Cloudflare, which powers around 20% of the web, is launching Pay per Crawl, which would let publishers charge AI firms each time their bots scrape a site. The episode frames this as a natural step after a year of bot-blocking work and debates whether the plan is genius or wishful thinking. The show also touches on other tech stories of the week.
Key Takeaways
- Cloudflare is launching Pay per Crawl, an experiment that would let publishers charge AI firms each time their bots scrape a site.
- Cloudflare powers around 20% of the web.
- The hosts call it a natural next step after a year of laying groundwork for bot-blocking tools.
- The episode debates whether sitting at the center of a pay-for-content protocol is genius or wishful thinking.
- The discussion is part of TechCrunch's Equity podcast, which posts every Wednesday and Friday.
Stats & Key Facts
- #Cloudflare powers around 20% of the web
- #Figma reported 48% revenue growth
- #ICEBlock is one of the most-downloaded free iPhone apps in the U.S.
What Pay per Crawl Is
Cloudflare's experiment centers on charging for content access by bots.
- ›Pay per Crawl would let publishers charge AI firms every time their bots scrape a site.
- ›Cloudflare is the cloud infrastructure provider that powers around 20% of the web.
- ›The plan could reshape how content is accessed and monetized online.
The hosts describe Pay per Crawl as a natural next step after a year of Cloudflare laying groundwork for bot-blocking tools.
Genius or Wishful Thinking
The episode weighs whether Cloudflare's bet will work.
- ›The plan would put Cloudflare at the center of a pay-for-content protocol.
- ›Hosts debate whether that position is genius or just wishful thinking.
- ›The discussion digs into Cloudflare's big swing and its prior bot-blocking work.
Other Stories in the Episode
The podcast also covers several other tech topics.
- ›ICEBlock, an app for anonymously reporting ICE sightings, went viral after backlash from former prosecutor Pam Bondi and is now one of the most-downloaded free iPhone apps in the U.S.
- ›Figma's S-1 filing could set the stage for a blockbuster IPO, with 48% revenue growth cited as a signal of demand for design tools.
- ›Grammarly's acquisition of Superhuman signals its vision for an agentic future of productivity.
- ›Tesla co-founder JB Straubel's new venture might challenge Tesla's own storage business.
About the Equity Podcast
Equity is TechCrunch's flagship podcast.
- ›It is produced by Theresa Loconsolo and posts every Wednesday and Friday.
- ›This episode was hosted by Kirsten Korosec and Maxwell Zeff.
- ›Listeners can subscribe on Apple Podcasts, Overcast, Spotify, and other platforms.
Kirsten Korosec is TechCrunch's transportation editor and co-host of Equity, while Maxwell Zeff was a senior reporter specializing in AI.
Frequently Asked Questions
What is Cloudflare's Pay per Crawl?
It is an experiment that would let publishers charge AI firms every time their bots scrape a site, potentially reshaping how content is accessed and monetized online.
How much of the web does Cloudflare power?
Cloudflare powers around 20% of the web.
Why is Pay per Crawl seen as a natural move for Cloudflare?
The hosts say it follows a year of Cloudflare laying groundwork for bot-blocking tools.
What other topics does the episode cover?
It discusses ICEBlock going viral, Figma's S-1 filing and 48% revenue growth, Grammarly's acquisition of Superhuman, and JB Straubel's new venture.
Where can listeners find the Equity podcast?
Equity posts every Wednesday and Friday and is available on Apple Podcasts, Overcast, Spotify, and other platforms.
Cloudflare's Pay per Crawl puts the company at the center of a debate over whether AI firms should pay publishers to scrape their content.
Continue Learning
Comments
Sign in to join the conversation