Amazon Interview Question

design of a web crawler