Search results

2 results for “crawling”.

• RSS Feeds Are Still the Best Way to Build a News Tab
  Before ranking news, you have to ingest it — and plain RSS over HTTPS remains the fastest, most honest pipeline from publisher to search results.
• My First Crawler: The Spider That Collected the World
  Building a web crawler of my own: from blindly following links to respecting robots.txt, handling heavy pages, and avoiding becoming a burden to others.