Meta is scraping the web at a frenzied pace to build its own AI search index — and indie developer Pieter Levels says the company’s crawlers have already tripped load alerts on one of his VPS servers multiple times this week.
DataDome’s metrics for the second quarter of this year show Meta’s two AI crawlers sent out a combined 9.1 billion requests. The indexing bot, Meta-WebIndexer, grew 163% quarter over quarter, and in June its request volume surpassed the training crawler for the first time.
Meta was reported to be building its own search engine as far back as 2024, aiming to make Meta AI less dependent on Google and Bing. That search index now appears to be expanding fast — but for all that aggressive scraping, it’s bringing almost no real users back to websites.
One Meta employee said in a direct message that the company wants to build its own “Google” — rather than handing AI search queries over to Google, only to have Google use them as training data.
https://twitter.com/levelsio/status/2085467097405247945?s=46