AutoScout24 Scraper

AutoScout24 Scraper

Give it a make and model with the filters you'd set on the site, or paste a search from any AutoScout24 country site, and it returns the cars as rows: price, mileage, first registration, fuel, gearbox, power, where the car is, whether a dealer or a private person is selling it, the dealer's name, the link and the photos.

It reads AutoScout24's results pages, the same ones you scroll through in a browser, and never opens the individual listings. Germany, Austria, Italy, France, Spain, the Netherlands, Belgium and Luxembourg are all on it.

Two things to know before you start. AutoScout24 shows at most 200 pages of any search, however many cars it says it has. At 17 to 20 cars a page, which is what our tests saw, that's 3,400 to 4,000 cars, and the run tells you when a search runs into it. And the sellers' phone numbers stay out of the rows. If someone types a phone number or an email address into a listing's text, it's replaced with a marker. A number written in an unusual way, broken up with spaces and missing both a country code and a leading zero, can still get through.

What it doesn't do

• No sellers' phone numbers, contact names or street addresses. The results pages carry dealers' phone numbers, and for some private sellers a phone number too. They're left out, along with the names of the people to contact and the street part of the address. You get the dealer's company name, the postcode and the town. Some sellers also type a phone number or an email address into the car's title, or even into the town. Wherever it turns up in a row, it comes out as [phone removed] or [email removed]. The kind that can still get past is the unusual one described above.
• Nothing that's only on the car's own page. The seller's description, the full equipment list and the finance and leasing terms live there, and this actor doesn't open it. You can filter by equipment or by a monthly rate; the rows just don't list them.
• Not past page 200. That's AutoScout24's own ceiling. For a search with more cars, split it by price or by year and run the parts.
• No searches without a make. A search link where no make was chosen is skipped. AutoScout24's robots.txt asks automated visitors to stay off those pages, and this actor keeps to it.
• Three filters from autoscout24.de aren't in the form. The manufacturers' certification seals change with the make. Smyle online buying only works on the German site. SuperDeal does run on the English site, but there it brought back mostly cars without the badge. Set any of them on autoscout24.de and paste the link: the link keeps it.
• Cars only. Motorbike searches are skipped, and autoscout24.ch isn't read.

Input

You can search two ways, and use both in one run.

Make and model. Type them the way autoscout24.com writes them: BMW and 3 Series, Volkswagen and Golf, Mercedes-Benz and C-Class. The model is optional; leave it empty for every model of that make. If the site doesn't recognise a name you typed, you get no rows rather than the wrong cars, and the run report lists the names it does use for that make.

Filters. The fields right under the make and model are the ones most people use: words in the listing title (the site's "Model version" box), country, new or used, price, first registration year, mileage, fuel, gearbox, and dealer or private seller. The rest of AutoScout24's search form sits in the sections below them, grouped the way the site groups it:

• Place. A postcode or town and a radius. Pick one country first: with all eight, AutoScout24 doesn't place a postcode at all, so the run asks for one. A postcode is the surest. A town is read the site's way, and "Munich" once came back as Münichham, a village about 85 km east, so the status and the run report say which place was searched. With no radius the site uses 200 km.
• Condition and history. Offer type (new, used, employee's car, classic, demonstration, pre-registered), accident vehicles (left out unless you ask, as on the site), the most previous owners, warranty, full service history, non-smoking, and a fresh German inspection (HU/AU).
• Engine and body. Power in kW or hp, electric range, body type, seats and doors.
• Colour and interior. Colour, metallic paint, interior colour and upholstery.
• Environment. Euro emission class and the German emission sticker, both "at least" as on the site, and a particulate filter.
• Equipment. AutoScout24's own list, 136 items long.
• Price and listing. The site's price rating, SuperDeal, VAT a business can reclaim, and how recently the car went up.
• Finance and leasing. The monthly finance rate, and the German site's leasing filters: leasing offers only, rate, term, yearly kilometres, available now, trade-in and the environmental bonus. The last two found no cars at all in our tests.

Picking several values in one list means any of them: black or grey. Equipment works the other way round, as on the site: every item you pick has to be on the car.

Order decides which cars come first. With a limit smaller than the search, it also decides which cars you get, so "newest listings first" gives you the latest arrivals and "price, low to high" the bottom of the market. "Nearest first" needs a place.

Search links. Set up a search on any AutoScout24 country site (.de, .com, .at, .it, .fr, .es, .nl, .be or .lu) with whatever filters you like, copy the address from the browser and paste it. Up to 10 links a run. Each link keeps its own filters and its own country site. The form fields don't touch links.

Maximum listings. 1,000 unless you change it, and up to 40,000 a run across all searches. With several searches, each gets an equal part of it, and what one doesn't use passes to the next. A car that turns up twice counts once.

{
  "make": "BMW",
  "model": "3 Series",
  "country": "DE",
  "condition": "used",
  "priceTo": 25000,
  "yearFrom": 2019,
  "fuel": "D",
  "sort": "newest",
  "maxItems": 200
}

With a place and some of the newer filters:

{
  "make": "Volkswagen",
  "model": "Golf",
  "country": "DE",
  "condition": "used",
  "location": "10115",
  "radiusKm": "100",
  "powerFrom": 85,
  "powerUnit": "kw",
  "bodyTypes": ["station-wagon"],
  "equipment": ["apple-carplay"],
  "fullServiceHistory": true,
  "sort": "nearest",
  "maxItems": 300
}

Output

One row per car.

Prices are what the seller typed, so a car listed at 1 euro comes through as 1.

Every car is checked against the search before it's kept, on everything the results page shows about it: make, model, price, year, mileage, power, fuel, gearbox, seller type, country, new, used or accident, price rating and distance. The filters the results page doesn't show per car, like body type, colour, owners, emission class or equipment, are checked against the site's own record of the search it ran. That matters because AutoScout24 sometimes answers a search it can't match with a broader one: a model it doesn't know can come back as every car of that make, and a place it can't find as every car in the country, both on a normal-looking page. The actor returns nothing rather than the wrong cars, and charges nothing for them.

AutoScout24 draws a radius its own way, so a car a few kilometres past the edge can turn up. Up to 10% or 5 km past it, whichever is more, the car is kept, and anything further is left out.

The run also writes a report, RUN_REPORT in the key-value store: each search, what AutoScout24 said it has, the place it searched around, the pages read, the repeats skipped, cars left out because they didn't fit and why, and why the run stopped.

What you pay

You're charged per car returned. Repeats, cars left out because they didn't fit your search, and searches that come back empty add nothing to it. If you set a maximum charge for the run, it stops when that's reached, and every row you get has been paid for. The price is on the Pricing tab.

Limits

• 200 pages per search, which came to 3,400 to 4,000 cars in our tests. That ceiling is AutoScout24's.
• 10 links plus the make-and-model search per run, and 40,000 cars per run.
• One page at a time, with a short pause between pages. A thousand cars took a minute and a half in our tests, and a search read to its last page just under six minutes.
• If a page doesn't come through, it's tried again a few times. Two pages in a row that fail end that search, and the status says so. Nothing is charged for cars that never arrived.
• Listings come and go while a long search is being read. If the same car turns up twice, the second copy is dropped.

Questions

Why do I get fewer cars than the site says it has? Usually the 200-page ceiling. Next come cars left out because they didn't fit the search. RUN_REPORT counts both. The site also shows some cars twice, once in a slot at the top and once in the list; those copies are dropped and were never in its count.

Can I get the sellers' phone numbers? No. See the top of this page.

A demonstration car says Used. That's how AutoScout24's results page lists employee's cars, classics, demonstration and pre-registered cars. Pick them under Offer type and you get only those.

My town search came back around somewhere else. AutoScout24 picked a different place with that name. The status says which one it used. Try the postcode.

Why does it say Gasoline, not petrol? That's autoscout24.com's word, and the rows use the English site's words whichever country the search came from, so a German and an Italian search line up in one sheet.

A model I know exists came back empty. The site probably names it differently. RUN_REPORT lists the names AutoScout24 uses for that make. Or set the search up on the site and paste the link instead.