This data is useless, total pages online are meaningless, you have to count on the total alive domains with at least some pages, also other very useful data are how many domains with alive traffic, Top 1 million, 10 millions domains...
Some websites can find these data, include ahrefs.com
I think this site was trying to measure the size of the Google index, not necessarily the size of the entire web, despite the site name. The page talks about "Indexed Web" and "Indexed Dutch Web".
But their numbers still look wrong. Yep / Ahrefs claim to have indexed 100 Billion pages and reindex 8 Billion every day. While OpenWebSearch EU claim to have indexed over 9 Billion pages:
From my own attempt at a small personal search engine of the web, which doesn't yet use vector embeddings, each page averages to 10kB of indexed storage.
That would make the final index size of 4 Billion pages about 40 Terabytes. Those charts seem to suggest that's just the size of the Bing index though, and that Google is actually 10x larger at 40 Billion pages. So that would be 400 Terabytes.
My little engine doesn't index the full HTML. If I did, each page averages to 170KB in size, and your index storage just grew 17x.
On a tiny scale - single digit millions - you can get ridiculously far with just SQLite.
So, I clearly need to write a blog post about this one day - there are so many answers and they won't all fit in an HN reply. So I'll give some higher level concepts that I found helpful. Not to dodge your questions, but it gives a way to think about things.
* Do Things That Don't Scale. There's a Paul Graham essay on this, but the general idea I took is that a naive, brute force solution will get you quite a long way, and you'll learn lots from it. PHP & SQLite on shared web hosting should never work, but, it goes SO much further than you'd think.
* Wait Until Something Is A Problem. You'll eventually feel the parts that need scaling. Worry about them then. For a long time I didn't even have an inverted index, didn't even have FTS enabled on SQLite. Things still worked. (But you really do want an inverted index & full text search.)
* Do You Actually Need This? You probably don't need to index the entire internet. You only want the parts of the internet that are relevant to you. You can use your browser's history to get a list of the websites / pages that are actually important to you. It is not that many. This is roughly the idea behind asciimoo's Hister project, which makes a local index of the internet while you browse the web.
On headers & proxies & javascript etc... so far, I don't bother. If a website doesn't want me indexing them, I respect that. There's still large parts of the web that do want to be indexed. Read their robots.txt files so you can learn what websites would like you to do, especially Crawl-Delay.
I have never used AWS, so I can't help there. My stuff is all just Apache, PHP & SQLite, and a bunch of self-made Go command-line programs.
For further research, read all of the Marginalia Search blog, you'll learn lots from that. Remember Marginalia is run by just one person. Also look at Seirdy's list of Search Engines With Their Own Indexes.
> Since January 15, 2025 we cannot receive any data from Google. They make it very difficult to automatically retrieve data from the search engine
Google became truly Evil. The sooner this evil company is gone the better.
Meanwhile I think we need a new variant of the world wide web. The old one is too easily broken, e. g. by Google and its pathetic attempt to install a private version here. Also aided by AI slop. Plans are inter-connected here.
This data is useless, total pages online are meaningless, you have to count on the total alive domains with at least some pages, also other very useful data are how many domains with alive traffic, Top 1 million, 10 millions domains...
Some websites can find these data, include ahrefs.com
I think this site was trying to measure the size of the Google index, not necessarily the size of the entire web, despite the site name. The page talks about "Indexed Web" and "Indexed Dutch Web".
But their numbers still look wrong. Yep / Ahrefs claim to have indexed 100 Billion pages and reindex 8 Billion every day. While OpenWebSearch EU claim to have indexed over 9 Billion pages:
https://openwebsearch.eu/
- very stupid question here
- let us say i want to index all 4 billion pages to generate vector embeddings and make a new search engine
- what does my AWS infra look like?
- how much storage on s3?
- what size of an ec2 instance?
From my own attempt at a small personal search engine of the web, which doesn't yet use vector embeddings, each page averages to 10kB of indexed storage.
That would make the final index size of 4 Billion pages about 40 Terabytes. Those charts seem to suggest that's just the size of the Bing index though, and that Google is actually 10x larger at 40 Billion pages. So that would be 400 Terabytes.
My little engine doesn't index the full HTML. If I did, each page averages to 170KB in size, and your index storage just grew 17x.
On a tiny scale - single digit millions - you can get ridiculously far with just SQLite.
- thank you for sharing that
- another stupid question: how do you about loading data from millions of pages simultaneously
- here is my silly thought process for this:
- get multiprocessing library in python
- combine that with asyncio and aiohttp
- send a whole bunch of requests and save raw html to file storage?
- some big questions...
- how often do you scan the same website
- what headers do you need to add in order to make it not look like some bot or should you actually reveal that you are a search engine bot
- do you need rotating proxies? is something like brightdata or residential proxies used or am I overthinking this?
- I am thinking of taking a small subset of 400 billion pages (like maybe just every blog ever listed on HN) and vector embedding all the text
- what do you think the cloud infra side specifically on AWS (since I am highly familiar with it) would look like?
So, I clearly need to write a blog post about this one day - there are so many answers and they won't all fit in an HN reply. So I'll give some higher level concepts that I found helpful. Not to dodge your questions, but it gives a way to think about things.
* Do Things That Don't Scale. There's a Paul Graham essay on this, but the general idea I took is that a naive, brute force solution will get you quite a long way, and you'll learn lots from it. PHP & SQLite on shared web hosting should never work, but, it goes SO much further than you'd think.
* Wait Until Something Is A Problem. You'll eventually feel the parts that need scaling. Worry about them then. For a long time I didn't even have an inverted index, didn't even have FTS enabled on SQLite. Things still worked. (But you really do want an inverted index & full text search.)
* Do You Actually Need This? You probably don't need to index the entire internet. You only want the parts of the internet that are relevant to you. You can use your browser's history to get a list of the websites / pages that are actually important to you. It is not that many. This is roughly the idea behind asciimoo's Hister project, which makes a local index of the internet while you browse the web.
On headers & proxies & javascript etc... so far, I don't bother. If a website doesn't want me indexing them, I respect that. There's still large parts of the web that do want to be indexed. Read their robots.txt files so you can learn what websites would like you to do, especially Crawl-Delay.
I have never used AWS, so I can't help there. My stuff is all just Apache, PHP & SQLite, and a bunch of self-made Go command-line programs.
For further research, read all of the Marginalia Search blog, you'll learn lots from that. Remember Marginalia is run by just one person. Also look at Seirdy's list of Search Engines With Their Own Indexes.
https://www.marginalia.nu/log/
https://seirdy.one/posts/2021/03/10/search-engines-with-own-...
99% of the web is just scraper fodder now.
It indeed has changed. I wonder if we can change it back.
Someone ask Alan Kay how to fix today's web.
So what's with the increased variance since 2021? Better tracking of pages maybe...
> Since January 15, 2025 we cannot receive any data from Google. They make it very difficult to automatically retrieve data from the search engine
Google became truly Evil. The sooner this evil company is gone the better.
Meanwhile I think we need a new variant of the world wide web. The old one is too easily broken, e. g. by Google and its pathetic attempt to install a private version here. Also aided by AI slop. Plans are inter-connected here.
> "size of the World Wide Web (The Internet)"
The WWW is not the internet.
Yah but the average person (layman) reading this won't know or care much, they'll understand what the website tries to show.