No wonder it's so hard to get ranked well

Google just hit a big milestone - they crossed  1,000,000,000,000 URLs in their crawl.  Not all are indexed of course, but that's still a lot of web pages.  They say that the web is growing by several billion pages a day, and I guess Google would know. 

http://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html

I believe it too.  It's just that a billion a day seems like a number almost too big to imagine.  Although, just think about forums.  Each new topic is a new page.  When a topic goes over 10 posts it becomes a new page. 

The example they gave is a calendar one a web page with a view next month link. In theory they could keep following that link infinitely, but after a certain point the information on the page becomes worthless so they don't worry about keeping that page in their index.  Blogs are another good example.  The page that lists all of the posts usually has the same content as the post itself so one gets flagged as duplicate and may not be indexed. 

My guess on the majority of that is spam sites that just throw up random pages using scripts.