Hacker News
Google no longer provides direct URLs in search results
Google Search now rewrites organic result links to google.com/goto?url=... and hides the destination URL in the HTML. The url parameter is a proprietary encoding that requires reading the Location header of the redirect response. This change forces an extra request per result, slowing bulk scraping and exposing automated SERP harvesting to Google. Autom has updated its pipeline to resolve these goto links via HEAD requests and return the final destination URLs in its API responses.