The Odosano crawler

Odosano is a used-car search. Its crawler, odosano-bot, reads the public inventory pages of car dealers' websites so people can search them in one place. We aim to make finding a used car easier, and to send shoppers to the dealer's own page.

Who we are

Odosano is under development and is not released yet. We are still perfecting our data and features. Questions, complaints and requests to stop go to bots@odosano.com. We read that address and reply.

How to recognise us

Every request carries this user agent:

odosano-bot/0.1 (+https://odosano.com/bot)

We never pose as a browser or as any other crawler.

What we fetch

• A dealer's robots.txt, and then its sitemaps.
• The vehicle detail pages the sitemaps list, to read the vehicle's identification number (VIN) and basic facts about the car.

We don't log in, submit forms or fetch anything a visitor couldn't open without an account. We will link every listing back to the dealer's own page.

We review each site's terms of use and don't crawl sites whose terms prohibit automated collection.

How gently we do it

• We follow robots.txt. Anything it disallows for us, or for all crawlers, we don't request.
• We wait at least 4 seconds between requests. The wait is shared by every dealer that sits on the same website platform, so a platform hosting many dealers still sees one slow stream from us.
• We crawl each dealer's inventory at most once a day, and we don't fetch a detail page more than once a day. If a sitemap says its pages change less often, we crawl less often.
• We follow a site's Crawl-delay if it sets one.
• We don't try to work around bot-protection challenges.

Asking us to stop or slow down

To block us entirely, add this to your robots.txt:

User-agent: odosano-bot
Disallow: /

You can also email bots@odosano.com with your domain and we'll stop, or change the rate, straight away. If you run a bot-protection service and would like us to identify ourselves in a way you can verify, write to us and we'll work with you.