Keyword Autocomplete Scraper

Keyword Autocomplete Scraper

Start typing "coffee grinder" into Google in the US and it offers "coffee grinder for espresso", "coffee grinder walmart", "coffee grinder near me". Those guesses are real searches, ranked by the engine. This actor collects them for your seed keywords from Google, YouTube, Amazon and Bing, for the country and language you pick. You get one row per suggestion, with the rank the engine gave it and the exact text that was typed.

It asks each engine's own suggestion service, the one the search box calls while you type. No API key, no account.

What it doesn't do

• No search volume, cost per click or competition. Suggestions tell you what people type, not how many of them do. If you need volumes, this is the wrong tool, however many suggestions it returns.
• It's a snapshot. Engines reshuffle their suggestions during the day. Run it again tomorrow and some will have changed.
• Nothing personal. The engines see a visitor with no history and no account, so you get what a signed-out searcher in that country is shown.
• Google and Bing add a few local suggestions. Both mix in suggestions for the area a search comes from, whatever country you set. In US runs that shows up as a handful of place names around Washington, D.C.: "guitar lessons alexandria va", "car insurance maryland". When we checked a German run against the same searches sent straight to each engine at the same moment, 252 of its 254 suggestions matched.
• Amazon goes by the marketplace, not the language. amazon.com gave the same list with its English and Spanish settings when we tried, and so did amazon.ca and amazon.ae with theirs.
• Amazon sometimes corrects your spelling and answers that instead. Asked for "does coffee grinder", it answered for "dose coffee grinder". Those answers are left out and not charged, and the status says how many there were.
• Bing only answers for language and country pairs it has a market for. English in Germany isn't one of them. When your pair isn't a market, Bing is asked in that country's own market and the status says so. For a country Bing has no market for at all, Bing is skipped.

Input

Seed keywords. One per line, written the way you'd start a search. Up to 1,000 per run.

Engines. Google, YouTube, Amazon and Bing, in any combination. Google alone unless you choose otherwise.

Country and Language. Google and YouTube lean their suggestions toward the country and answer in the language. Asked "bank" in English, Google offered Bank of America for the US, Bank of Scotland for the UK and Bank of Baroda for India. Bing reads country and language together as one market.

Amazon marketplace. Match the country uses the Amazon of the country you picked (amazon.de for Germany), or amazon.com when that country has no Amazon of its own. Or pick one of the 23 marketplaces yourself.

Amazon department. Empty means all of Amazon. To stay inside one department, give Amazon's own short name for it, as it appears in its search links: stripbooks for books, electronics, grocery. A name Amazon doesn't know gets no suggestions, and the status tells you.

Google suggestions from. Web is the Google search box. Shopping leans toward products: for "coffee" it led with coffee table and coffee maker, where web search led with coffee near me. Images and News come out close to Web.

Variations. Each one adds lookups for every seed and engine:

• *Seed followed by a space* asks "coffee grinder " with the space typed. Engines often finish it differently.
• *a to z* asks "coffee grinder a" through "coffee grinder z". Most of the long phrases come from here.
• *0 to 9* is good for model numbers, sizes and years.
• *Question words* go in front: "how coffee grinder", "why coffee grinder". Change the list to suit you, or add words such as best or cheap.

With all four on, "coffee grinder" came back as 1,776 different suggestions from 204 lookups across the four engines, in about 15 seconds.

If an engine has no suggestions at all for a seed, its space, letter and number variations aren't asked. They only make the same text longer.

Most suggestions per lookup. Keep only the top of each list. Empty keeps all of it: up to 15 from Google, 14 from YouTube, 10 from Amazon and 25 from Bing.

Maximum suggestions. 5,000 unless you change it, and up to 50,000 per run, all engines together.

{
  "seeds": ["coffee grinder", "french press"],
  "engines": ["google", "youtube", "amazon"],
  "country": "GB",
  "language": "en",
  "expandAlphabet": true,
  "expandQuestions": true,
  "maxSuggestions": 3000
}

Output

One row per suggestion, seed by seed. Within a seed, the seed's own lookup comes first, then the variations in the order above.

A phrase comes back once per engine. If "coffee grinder electric" shows up under the seed, then again under "coffee grinder e", you get it once, from the first lookup that found it. Two engines suggesting the same phrase give you two rows, one each.

The run also leaves RUN_REPORT in its key-value store: how many lookups each engine was asked and answered, how many came back empty, repeats skipped, what each seed brought back, and why the run stopped.

What you pay

Each suggestion in your dataset is one charge, at the price the Pricing tab shows for its engine. A lookup that comes back empty costs nothing, and neither does a phrase an engine already gave you in the same run. If you set a maximum charge for the run, it stops when that's reached, and every row you get has been paid for.

Limits

• 1,000 seeds, 50,000 suggestions and 25,000 lookups per run.
• Each engine is asked a couple of lookups at a time, so a big run with every variation on takes several minutes.
• When an engine stops answering for a while, the actor waits and carries on. If it still doesn't answer, the status names the engine and says how many lookups were left out, and nothing is charged for them.

Questions

Why fewer rows than lookups times 15? Repeats. Many variations bring back phrases an earlier lookup already found, and those aren't returned twice. Some lookups return short lists too: on amazon.com, "how coffee" got two suggestions.

Can I track suggestions over time? Schedule the actor with the same input and compare the datasets. scrapedAt and rank are there for that.

Does it work in other alphabets? Yes. Japanese, Korean, Russian and Hebrew seeds came back in their own scripts. The a to z variation adds Latin letters, which suits some languages better than others.