Build Your Own Search Engine
One of the topics covered in The Search is the idea of vertical search engines. These are topic related search engines that search a specific part of the web (i.e. engineering, science, etc.). The book went on to suppose that the future might be in many high quality vertical search engines linked together in a metasearch.
Well today it appears Alexa has opened the doors to anyone (with a little bit of cash) to take advantage of their web crawl to build what you want. You can even host your search on their platform (for a little extra cash). You can download the data for research or create whatever you want (feeds search). The data store is 100 terabytes and they have 3 snapshots available at a time (300 Terabytes).
The example they give is a person who has a site devoted to cameras. He creates a subset of the crawl that has images. He then processes all the images for the EXIF information such as camera model and settings. People on his site can they use the engine to compare pictures taken by various cameras. Of course, you can actually use see and use this example.
The Alexa Web Search Platform provides public access to the vast web crawl collected by Alexa Internet. Users can search and process billions of documents -- even create their own search engines -- using Alexa's search and publication tools. Alexa provides compute and storage resources that allow users to quickly process and store large amounts of web data. Users can view the results of their processes interactively, transfer the results to their home machine, or publish them as a new web service.