What is a Google Search Result Scraper, and How Does It Work?
The Google SERP data scraping tool on our platform crawls search results on the largest search engine in the world and extracts data from crawled web pages in usable formats like CSV, JSON, Excel, or XML. Using Google SERP Scraper, you can scrape the following data from the Google search engine.
- Organic results
- People also ask
- Related queries
- Product ads
- Paid results
- Ratings and reviews
- Other customized attributes
Why Use Google Search Result Scraper for Data Extraction?
For the last 13 years, Google hasn't had any official search result API to show you how your competitors and you perform on Google. Therefore, you must find an alternative tool to monitor your and your competitor's SERP. Web scraping tools like Google SERP Scraper for SERP analysis help fulfill your needs.
Our Google Search Result Scraper gives you a customizable Google scraper with RESTful SERP API that gives you optimized outputs from Google search in real-time that you can download and in a usable format. After collecting and exporting the data from Google search results, you can import it into your projects or integrate it with other platforms like Airbyte, Google Sheets, Google Drive, etc.
How to use collected data from Google search results?
- Track how often people search a specific search term on Google and its comparison with the total tentative search volume.
- Track the performance of your website on the Google search engine for specific keywords and use the scraped data to optimize your website on the search engine.
- Study display advertisements for selected keyword sets.
- Study Google algorithms and discover essential trends.
- Track competition for your website in paid and organic search results.
- Compile a list of URLs for specific search queries. You can use it to find starting points while scraping required search result pages from Google.
How to use Google Search Result Scraper?
In a stepwise tutorial, we have shared the process to set up and execute this scraper. Explore it to understand various aspects of Google SERP data collection API.
What is the cost of using Google Search Result Scraper?
The cost of using this scraper varies depending on your requirements for Google search result data. Using the trial plan of 5 USD, you will get up to five thousand results monthly. If you want more data often, you can use our starter plan and scrape over one and a half million results monthly. You can contact us for custom requirements if you still need more data.
To check your platform credits and balance, visit the pricing page.
Is it mandatory to use proxy servers to scrape search results from Google?
You must use a proxy server to set up enough to run the scraper successfully. You have the option to use our proxies from your console account.
But the scraper consumes one SERP proxy server for each request. Using the trial plan, you can use five hundred proxy requests from our platform regardless of the results you extract using the scraper. You can optimize the proxy usage by customizing the per-page outputs of the scraper.
Input to Google Search Result Scraper
The SERP API allows you to customize it for specific outputs. For this, you can mention the following input settings:
- Search domain or country
- Search language
- Raw URL for Google search or keyword phrases
- The resulting count for each page
- Device version outputs
- The exact location of the user.
Check out the input tab for a detailed explanation of every input setting.
How to extract Google SERP using keywords or URLs?
There are two methods to scrape Google SERP: search keyword and URL.
- Crawling the Google SERP will help you get data from the result pages. You can add unlimited search queries according to your requirements.
- If you crawl the SERP using a URL to collect the data, the scraper will give you data from any domain or copy-pasted link. You can add unlimited URLs according to your needs.
Both alternatives to scraping the Google search engine are suitable for this scraper.
How to extract search results from Google using URLs?
It is straightforward to scrape Google search engines using URLs. You only need to copy and paste the URL for the required data and mention the pages from the platform you want to extract.
Check out the below JSON example to scrape Google SERP using URLs.
{
"countryCode": "uz",
"csvFriendlyOutput": false,
"customDataFunction": async ({ input, $, request, response, html }) => {//n return {//n pageTitle: $('title').text(),//n };//n};",,
"languageCode": "en",
"maxPagesPerQuery": 2,
"mobileResults": false,
"queries": "",,
"resultsPerPage": 100,
"maxConcurrency": 10,
"saveHtml": false,
"saveHtmlToKeyValueStore": false
}
How to extract Google SERP using search keywords?
Likewise, scraping Google search engine pages using search keywords is also very simple. You only needed to mention the search queries and required pages from SERP you want to extract. You have multiple options to use keywords for scraping Google.
Those options are:
- Mention domain or country location and language as a location parameter.
- Mention the result count you want to collect for each page of Google.
- Extract the SERP using a single search query.
- Extract the SERP using multiple search queries parallelly.
Check out the below JSON example to scrape Google SERP using search queries.
{
"countryCode": "us",
"customDataFunction": "async ({ input, $, request, response, html }) => {//n return {//n pageTitle: $('title').text(),//n };//n};",
"includeUnfilteredResults": false,
"languageCode": "en",
"maxPagesPerQuery": 2,
"mobileResults": false,
"queries": "hotels in Seattle /n hotels in New York",
"resultsPerPage": 100,
"saveHtml": false,
"saveHtmlToKeyValueStore": false,
"maxConcurrency": 10
}
Output Example of Google Search Result Scraper
It saves its output in a default output dataset related to the scraper execution. The scraper lets you download the data from a dataset in multiple formats like CSV, JSON, Excel, or XML.
You can export these outputs directly from the API endpoint from Get dataset details.
< https://api.RealdataAPI.com/v2/datasets/[DATASET_ID]/items?format=[FORMAT] >
The datasets formats are available as [FORMAT], like xlsx, xml, html, json, or rss with [DATASET_ID] as a dataset ID.
Search Query-based Output Example
The output dataset will have a single record for every Google search engine result page in the following JSON format. Remember that you will find sample values in a few fields.
{
"searchQuery": {
"term": "hotels in seattle",
"url": "http://www.google.com/search?num=100&q=hotels%20in%20seattle",
"device": "DESKTOP",
"page": 1,
"type": "SEARCH",
"domain": "google.com",
"countryCode": "US",
"languageCode": null,
"locationUule": null,
"resultsPerPage": "100"
},
"url": "http://www.google.com/search?num=100&q=hotels%20in%20seattle",
"hasNextPage": true,
"serpProviderCode": "N",
"resultsTotal": 54000000,
"relatedQueries": [
{
"title": "Feedback",
"url": "https://www.google.com/#"
},
{
"title": "Hyatt Regency Seattle",
"url": "https://www.google.com/search?num=100&q=Hyatt+Regency+Seattle&stick=H4sIAAAAAAAAAOOQUeLSz9U3MDHOSTEwNRJNzsnMTSxJVSjISU1JT1XIyC9JzSmOEoTQCpl5CsWpiSUlOamnGJG0QTlGVWbpBkVQTm5ZXnFG8ilGHv10fcPKXAujksKUNKhceVp6RVI5lGOcYZicnnSKkRuksCSl2CwlJ_cXo6gz1CEByA5pYGFcxCrqUQl0gkJQanpqXnKlQjDEQbfYJBlKnl2b9VUhNGan8tzMrdvyU0SSttVPXDDVAAAvau1f5QAAAA&sa=X&ved=2ahUKEwiRxJ_e2J__AhUVZTABHRosAFcQs9oBKAB6BAgXEAI"
}
// ... and many more
],
"paidResults": [
{
"title": "Hotels in Seattle, WA - Lowest Price Guarantee.",
"url": "https://www.google.com/aclk?sa=l&ai=DChcSEwiMzqXe2J__AhW_gVoFHbh6BGYYABABGgJ2dQ&sig=AOD64_3PbWI_s3u1fE0Dh4Dobi56jDDCfw&q&adurl",
"displayedUrl": "",
"description": "",
"emphasizedKeywords": [],
"siteLinks": [
{
"title": "",
"url": "https://www.google.com/aclk?sa=l&ai=DChcSEwiMzqXe2J__AhW_gVoFHbh6BGYYABAHGgJ2dQ&sig=AOD64_3x8pu1mywduFwZgi9cwut7L9Nrvg&q=&ved=2ahUKEwiRxJ_e2J__AhUVZTABHRosAFcQh78CegQIDhAB&adurl=",
"description": ""
},
{
"title": "Hotels at Great Prices",
"url": "https://www.google.com/aclk?sa=l&ai=DChcSEwiMzqXe2J__AhW_gVoFHbh6BGYYABADGgJ2dQ&sig=AOD64_1Bl0Ju75dgDPwLJAb53UYrV1xyMQ&q&adurl",
"description": ""
}
// ... and many more
],
"type": "paid",
"adPosition": 1
},
{
"title": "Cheap Hotels in Seattle - Best Deals in Seattle",
"url": "https://www.google.com/aclk?sa=l&ai=DChcSEwiMzqXe2J__AhW_gVoFHbh6BGYYABAAGgJ2dQ&sig=AOD64_27qRewPYeehK82DHa3ck7RlyilRg&q&adurl",
"displayedUrl": "",
"description": "",
"emphasizedKeywords": [],
"siteLinks": [
{
"title": "",
"url": "https://www.google.com/aclk?sa=l&ai=DChcSEwiMzqXe2J__AhW_gVoFHbh6BGYYABACGgJ2dQ&sig=AOD64_0sufwpcy9IZTR-ZX82HaFX1Xqc-w&q=&ved=2ahUKEwiRxJ_e2J__AhUVZTABHRosAFcQh78CegQIBxAB&adurl=",
"description": ""
}
],
"type": "paid",
"adPosition": 2
}
],
"paidProducts": [],
"organicResults": [
{
"title": "The Best Hotels in Seattle, WA",
"url": "https://www.hotels.com/de1481165/hotels-seattle-washington/",
"displayedUrl": "https://www.hotels.com › ... › Hotels in Washington",
"description": "The Inn at Virginia Mason · The Belltown Inn · Hyatt Regency Seattle · Mediterranean Inn · Hotel Max · Crowne Plaza Seattle - Downtown, an IHG Hotel · Mayflower Park ...",
"emphasizedKeywords": [
"Seattle",
"Hotel",
"Seattle",
"Hotel"
],
"siteLinks": [],
"productInfo": {},
"type": "organic",
"position": 1
},
{
"title": "THE 10 BEST Hotels in Seattle, WA 2023 (from $87)",
"url": "https://www.tripadvisor.com/Hotels-g60878-Seattle_Washington-Hotels.html",
"displayedUrl": "https://www.tripadvisor.com › ... › Seattle Hotels",
"description": "Popular hotels in Seattle right now · 1. Embassy Suites by Hilton Seattle Downtown Pioneer Square · 1,293 reviews · 2. Hyatt Regency Seattle · 1,917 reviews · 3.",
"emphasizedKeywords": [
"hotels in Seattle"
],
"siteLinks": [],
"productInfo": {},
"type": "organic",
"position": 2
}
// ... and many more
],
"peopleAlsoAsk": [
{
"question": "Is it better to stay in downtown Seattle or?",
"answer": "For most visitors, the best area to stay is downtown Seattle (and near Pike Place Market). Downtown is where most of Seattle's best hotels, top restaurants, attractions, and shopping are located – all within an area dense, compact, and walkable.",
"url": "https://santorinidave.com/best-places-seattle#:~:text=For%20most%20visitors%2C%20the%20best,dense%2C%20compact%2C%20and%20walkable.",
"title": "WHERE TO STAY in SEATTLE - Best Areas & Neighborhoods",
"date": "May 9, 2023"
},
{
"question": "Is it easy to get around Seattle without a car?",
"answer": "You can easily get around Seattle without a car. Places like Pike Place Market, the Seattle Center, the waterfront, and a lot of neighborhoods are walkable.",
"url": "https://travellemming.com/getting-around-seattle/#:~:text=You%20can%20easily%20get%20around,lot%20of%20neighborhoods%20are%20walkable.",
"title": "Getting Around Seattle (A Local's Transportation Guide)",
"date": "Jan 13, 2023"
},
{
"question": "Where not to stay in downtown Seattle?",
"answer": "Avoid downtown Seattle's 3rd Avenue, particularly between Pike and Pine, and James and Yesler. Pioneer Square, SoDo, the International District, First Hill, and certain parts of Belltown can be a bit sketchy. That said, visitors should exercise caution when exploring at night.",
"url": "https://travellemming.com/is-seattle-safe/#:~:text=Avoid%20downtown%20Seattle's%203rd%20Avenue,caution%20when%20exploring%20at%20night.",
"title": "Is Seattle Safe in 2023? (Honest Info From a Local) - Travel Lemming",
"date": "Feb 16, 2023"
},
{
"question": "Should I stay in West Seattle?",
"answer": "West Seattle Don't get me wrong: West Seattle is a great place. In fact, I highly recommend taking a half-day or day to visit the area if you have the time during your Seattle trip. Especially along Alki Beach, you can enjoy sweeping views of Puget Sound and the Seattle skyline, plus enjoy local food and drink.",
"url": "https://www.valisemag.com/where-to-stay-seattle/#:~:text=5.-,West%20Seattle,enjoy%20local%20food%20and%20drink.",
"title": "The 10 Best Areas to Stay in Seattle (& Where Not To!) - Valerie & Valise",
"date": "Sep 29, 2020"
}
],
"customData": {
"pageTitle": "hotels in seattle - Google Search"
}
}
How to collect a single search output in a row?
We have a toggle button for CSV-based output for this condition. However, if you want to do it using code, follow the below guide.
Built-in Approach
If you need output for Google searches, with different results for paid and organic results for each row, you have to set the input field csvFriendlyOutput to true. It is switched off by default, ignoring each extra field besides paidResults, organicResults, and searchQuery to store in a CSV format. It also stringifies the array emphasizedKeywords and deletes the siteLinks data array from paid and organic outputs. You'll see the output datasets in paid and organic output arrays.
You'll see the organic search result output representation in the below format:
{
"searchQuery": {
"term": "laptop",
"device": "DESKTOP",
"page": 1,
"type": "SEARCH",
"domain": "google.com",
"countryCode": "US",
"languageCode": "en",
"locationUule": null,
"resultsPerPage": 10
},
"type": "organic",
"position": 1,
"title": "Laptops & Notebook Computers - Best Buy",
"url": "<https://www.bestbuy.com/site/computers-pcs/laptop-computers/abcat0502000.c?id=abcat0502000>",
"displayedUrl": "<https://www.bestbuy.com> › Computers & Tablets",
"description": "Shop Best Buy for laptops. Work & play from anywhere with a notebook computer. We can help you find the best laptop for your specific needs in store and online.",
"emphasizedKeywords": "laptops | laptop",
"productInfo": {}
}
Here is the comparison of paid result sample with the organic one:
{
"searchQuery": {
"term": "hotels in seattle",
"url": "http://www.google.com/search?num=100&q=hotels%20in%20seattle",
"device": "DESKTOP",
"page": 1,
"type": "SEARCH",
"domain": "google.com",
"countryCode": "US",
"languageCode": null,
"locationUule": null,
"resultsPerPage": "100"
},
"url": "http://www.google.com/search?num=100&q=hotels%20in%20seattle",
"hasNextPage": true,
"serpProviderCode": "N",
"resultsTotal": 54000000,
"relatedQueries": [
{
"title": "Feedback",
"url": "https://www.google.com/#"
},
{
"title": "Hyatt Regency Seattle",
"url": "https://www.google.com/search?num=100&q=Hyatt+Regency+Seattle&stick=H4sIAAAAAAAAAOOQUeLSz9U3MDHOSTEwNRJNzsnMTSxJVSjISU1JT1XIyC9JzSmOEoTQCpl5CsWpiSUlOamnGJG0QTlGVWbpBkVQTm5ZXnFG8ilGHv10fcPKXAujksKUNKhceVp6RVI5lGOcYZicnnSKkRuksCSl2CwlJ_cXo6gz1CEByA5pYGFcxCrqUQl0gkJQanpqXnKlQjDEQbfYJBlKnl2b9VUhNGan8tzMrdvyU0SSttVPXDDVAAAvau1f5QAAAA&sa=X&ved=2ahUKEwiRxJ_e2J__AhUVZTABHRosAFcQs9oBKAB6BAgXEAI"
}
// ... and many more
],
"paidResults": [
{
"title": "Hotels in Seattle, WA - Lowest Price Guarantee.",
"url": "https://www.google.com/aclk?sa=l&ai=DChcSEwiMzqXe2J__AhW_gVoFHbh6BGYYABABGgJ2dQ&sig=AOD64_3PbWI_s3u1fE0Dh4Dobi56jDDCfw&q&adurl",
"displayedUrl": "",
"description": "",
"emphasizedKeywords": [],
"siteLinks": [
{
"title": "",
"url": "https://www.google.com/aclk?sa=l&ai=DChcSEwiMzqXe2J__AhW_gVoFHbh6BGYYABAHGgJ2dQ&sig=AOD64_3x8pu1mywduFwZgi9cwut7L9Nrvg&q=&ved=2ahUKEwiRxJ_e2J__AhUVZTABHRosAFcQh78CegQIDhAB&adurl=",
"description": ""
},
{
"title": "Hotels at Great Prices",
"url": "https://www.google.com/aclk?sa=l&ai=DChcSEwiMzqXe2J__AhW_gVoFHbh6BGYYABADGgJ2dQ&sig=AOD64_1Bl0Ju75dgDPwLJAb53UYrV1xyMQ&q&adurl",
"description": ""
}
// ... and many more
],
"type": "paid",
"adPosition": 1
},
{
"title": "Cheap Hotels in Seattle - Best Deals in Seattle",
"url": "https://www.google.com/aclk?sa=l&ai=DChcSEwiMzqXe2J__AhW_gVoFHbh6BGYYABAAGgJ2dQ&sig=AOD64_27qRewPYeehK82DHa3ck7RlyilRg&q&adurl",
"displayedUrl": "",
"description": "",
"emphasizedKeywords": [],
"siteLinks": [
{
"title": "",
"url": "https://www.google.com/aclk?sa=l&ai=DChcSEwiMzqXe2J__AhW_gVoFHbh6BGYYABACGgJ2dQ&sig=AOD64_0sufwpcy9IZTR-ZX82HaFX1Xqc-w&q=&ved=2ahUKEwiRxJ_e2J__AhUVZTABHRosAFcQh78CegQIBxAB&adurl=",
"description": ""
}
],
"type": "paid",
"adPosition": 2
}
],
"paidProducts": [],
"organicResults": [
{
"title": "The Best Hotels in Seattle, WA",
"url": "https://www.hotels.com/de1481165/hotels-seattle-washington/",
"displayedUrl": "https://www.hotels.com › ... › Hotels in Washington",
"description": "The Inn at Virginia Mason · The Belltown Inn · Hyatt Regency Seattle · Mediterranean Inn · Hotel Max · Crowne Plaza Seattle - Downtown, an IHG Hotel · Mayflower Park ...",
"emphasizedKeywords": [
"Seattle",
"Hotel",
"Seattle",
"Hotel"
],
"siteLinks": [],
"productInfo": {},
"type": "organic",
"position": 1
},
{
"title": "THE 10 BEST Hotels in Seattle, WA 2023 (from $87)",
"url": "https://www.tripadvisor.com/Hotels-g60878-Seattle_Washington-Hotels.html",
"displayedUrl": "https://www.tripadvisor.com › ... › Seattle Hotels",
"description": "Popular hotels in Seattle right now · 1. Embassy Suites by Hilton Seattle Downtown Pioneer Square · 1,293 reviews · 2. Hyatt Regency Seattle · 1,917 reviews · 3.",
"emphasizedKeywords": [
"hotels in Seattle"
],
"siteLinks": [],
"productInfo": {},
"type": "organic",
"position": 2
}
// ... and many more
],
"peopleAlsoAsk": [
{
"question": "Is it better to stay in downtown Seattle or?",
"answer": "For most visitors, the best area to stay is downtown Seattle (and near Pike Place Market). Downtown is where most of Seattle's best hotels, top restaurants, attractions, and shopping are located – all within an area dense, compact, and walkable.",
"url": "https://santorinidave.com/best-places-seattle#:~:text=For%20most%20visitors%2C%20the%20best,dense%2C%20compact%2C%20and%20walkable.",
"title": "WHERE TO STAY in SEATTLE - Best Areas & Neighborhoods",
"date": "May 9, 2023"
},
{
"question": "Is it easy to get around Seattle without a car?",
"answer": "You can easily get around Seattle without a car. Places like Pike Place Market, the Seattle Center, the waterfront, and a lot of neighborhoods are walkable.",
"url": "https://travellemming.com/getting-around-seattle/#:~:text=You%20can%20easily%20get%20around,lot%20of%20neighborhoods%20are%20walkable.",
"title": "Getting Around Seattle (A Local's Transportation Guide)",
"date": "Jan 13, 2023"
},
{
"question": "Where not to stay in downtown Seattle?",
"answer": "Avoid downtown Seattle's 3rd Avenue, particularly between Pike and Pine, and James and Yesler. Pioneer Square, SoDo, the International District, First Hill, and certain parts of Belltown can be a bit sketchy. That said, visitors should exercise caution when exploring at night.",
"url": "https://travellemming.com/is-seattle-safe/#:~:text=Avoid%20downtown%20Seattle's%203rd%20Avenue,caution%20when%20exploring%20at%20night.",
"title": "Is Seattle Safe in 2023? (Honest Info From a Local) - Travel Lemming",
"date": "Feb 16, 2023"
},
{
"question": "Should I stay in West Seattle?",
"answer": "West Seattle Don't get me wrong: West Seattle is a great place. In fact, I highly recommend taking a half-day or day to visit the area if you have the time during your Seattle trip. Especially along Alki Beach, you can enjoy sweeping views of Puget Sound and the Seattle skyline, plus enjoy local food and drink.",
"url": "https://www.valisemag.com/where-to-stay-seattle/#:~:text=5.-,West%20Seattle,enjoy%20local%20food%20and%20drink.",
"title": "The 10 Best Areas to Stay in Seattle (& Where Not To!) - Valerie & Valise",
"date": "Sep 29, 2020"
}
],
"customData": {
"pageTitle": "hotels in seattle - Google Search"
}
}
Check the paid result to note the difference in adPosition fields with the position and value of the type field. The scraper computes the position of paid outputs compared to organic results separately and stores it in the adPosition field.
Optional Approach Using API Call
You can pass search term parameters unwind=organicResults and fields=searchQuery,organicResults to the link of the API endpoint:
< https://api.RealdataAPI.com/v2/datasets/[DATASET_ID]/items?format=[FORMAT]&fields=searchQuery,organicResults&unwind=organicResults >
The API will reflect the output in the following JSON data format:
While using CSV, Excel, or other tabular data formats, you will get a table with a single organic output in every row. Explore the documentation to learn more about formatting and exporting the recorded datasets.
Bits of Advice
You can extract many results effectively using a single search term with a set of a hundred search outputs for every page to get a hundred outputs on each page instead of getting ten search results on ten Google pages.
If you need to clarify the quality and efficiency of search results, the scraper stores the complete HTML page in the key-value store for every execution. You can read it and compare outputs. Our dedicated team continuously tracks the resulting quality. However, we are happy to resolve your issues.
Disclaimers
How to extract Google search results in millions?
Please remember that, though the largest search engine displays millions of search results available for specific keywords, it will not show you over a hundred or thousand results for a search query. If you want more results, you can create similar search queries combining various locations and parameters and get results.
How to extract data from Google ads?
Scraping Google search results is the option to collect required data for specific search terms using search results and paid advertising results from Google SERP. These results depend on browser history and the location of users. Further, it relies on Google algorithms on which ads it wants to display to which users. You may not get the expected results due to Google's accurate and ever-changing algorithms.
Can I Legally Use Google Search Result Scraper?
If you scrape public data from Google, web data collection is legal. However, you should check personal data and property regulations. Remember to extract only public data. Luckily, you can extract private data if you have a legitimate purpose and follow Google's terms and conditions effectively. You can consult your advocate to clarify whether your reason for scraping Google search results is genuine.
Google SERP Scraper with Integrations
Lastly, you can integrate Google SERP scraping tool into any web application or cloud service with the help of integrations on our platform. Connecting the scraper with Airbyte, Zapier, Make, Google Drive, Google Sheets, Slack, and other platforms is possible. Further, use webhooks to set up if any event occurs, like a successful run of the scraper.
Using Google SERP Scraping Tool with Real Data API Actor
Our actor allows you to access Real Data API programmatically. We have organized it around RESTful HTTP endpoints to allow you to schedule, manage, and run our actors. It also allows you to track performance, extract results, access scraped datasets, update and create scraper versions, and more.
You can use our client PyPl and client NPM package to access the API Using Python and Node.js, respectively.
What are the other Google data scrapers?
- Google Maps Scraper
- Google Trends Scraper
- Google Play Reviews Scraper
- Google Shopping Scraper
- Google Datasets Translator
- Google Maps Reviews Scraper
- Google Trending Searches
- Google Maps with Contact Details
