Total Wine Scraper - Web Scraping Total Wine Liquor Data
RealdataAPI / total-wine-scraper
Scrape wine product data, like brand name, images, descriptions, prices, stock information, and more, from the Total Wine website using Total Wine & More Scraper. The scraper is available in countries like the USA, UK, UAE, Canada, Australia, France, Germany, Spain, Italy, Mexico, Singapore, etc.
Since the Total Wine & More platform has no official data API, this scraper will help you scrape the required product data.
The Total Wine & More scraper has the below features.
Extract lists: gather data for any list from the Total Wine & More platform.
Search from any query: you can use this scraper to search for any query to get the required results.
Updates, Bugs, Fixes & Changelogs
Our team is actively working on developing this scraper. If you want to suggest any feature, you can request us by creating an issue.
Setup and Usages To Scrape Total Wine & More Data
Enter the start URL or search query to use the Total Wine scraper. Presently, the scraper only supports the URL list as the start URL. Further, it would help to use a US-based proxy server to run the scraper. You can set the location or store using the ShippingInfo input field and collect the exact product listing.
Using Search
You can search the single keyword and set endPage, or maxItems if you want to.
The scraper goes through every search result page till it satisfies every condition.
Check out the output for this scraper example here.
Enter any start URL counts with endPage or maxItems according to your needs.
The scraper parses listed products for all URLs and go through the given start URL.
URL Query Parameters
The scraper uses the aty query parameter of the selected link if you provide it. If you don't, the tool uses the query parameter aty=1,1,0,1 to retrieve every existing product. Here is the sample URL:
If you provide it, the scraper keeps the query parameter pageSize for the selected pages. If you don't, it uses the parameter pageSize=24 to retrieve twenty-four products on each page. Here is the sample URL:
You must provide JSON Formatted input to this scraper with the list of Total Wine & More pages to visit. Here are the required input fields:
Field
Type
Description
search
String
It is an optional field for search queries you wish to search on the Total Wine platform.
startUrls
Array
It is an optional field for Total Wine & More web page URLs. The scraper only accepts product list links.
endPage
Integer
It is an optional input field to set the number of last web pages you wish to crawl with an infinite default. You can apply it to every start URL and search request separately.
shippingInfo
String
The field will set the location or store choice for the Total Wine website. The source website displays products for selected locations or stores based on these preferences. If you don't provide choices, the scraper uses the address of the first proxy server as the location to scrape the product data.
maxItems
Integer
You can restrict the collected products using this optional field. The field is valuable when you search big subcategories.
proxy
Object
Set proxy servers to execute the scraper successfully.
extendOutputFunction
String
It is an optional function to take the argument based on the JQuery handle ($) and give you the object with the required data.
customMapFunction
String
It is an optional function to take arguments based on every object and reflects objects with the function execution.
You've two options to set up proxies, like choosing them from our platform or using external proxies.
Tip
To scrape a particular listing link, only copy and paste the URL in the startURL field.
If you wish to extract only the first web page from the list, enter the link for that page and choose endPage as one.
As per the above explanation, you can retrieve any page. If you ask to scrape the 7th page and define the endPage parameter to 8, the scraper will only crawl the 7th and 8th pages.
If you want to extract all the products from the selected section, like white wine, red wine, and spirits, you must use the query parameter pageSize, which can increase the product counts for every request. The scraper may fail if the pageSize has a value greater than 200.
Use the EditThisCookie Chrome extension to scrape cookie value for the location or store choice. Check out the details to use the Chrome extension:
Visit any initial search or listing page to crawl.
Find the Pickup At section near the search bar, and edit it.
Use the EditThisCookie Chrome extension to find the twm-userStoreInformation cookie after selecting the location/store.
Copy the critical value, and paste it into the Shipping Info data field.
While executing, the scraper will send the message alert with the live scraping update. Every message includes a short label mentioning the current page it is scraping from the list. You have to read the message for the event with the total item count and loaded item count for every page after loading the items from the page.
The scraper will only run if you give the correct input. It will display the result with the failure reason in detail.
Export Total Wine & More Data
The Total Wine scraper saves the output in a customized dataset, with every item having a separate value and location.
You can handle the outputs in any coding language, like PHP, Python, NPM, or Node.js.
Properties of Scraped Total Wine & More Data
Check out the format of every item in the Total Wine & More:
Detailed Item Information
{"bay":"","brand":{"id":"Mascota Vineyards","name":"Mascota Vineyards"},"categories":[{"id":"000284","name":"Mendoza","type":"REGION","url":"https://www.totalwine.com/wine/argentina/mendoza/c/000284","storefrontUrl":"/wine/argentina/mendoza/c/000284"},{"id":"000009","name":"Red Wine","type":"PRODUCT_TYPE","url":"https://www.totalwine.com/wine/red-wine/c/000009","storefrontUrl":"/wine/red-wine/c/000009"},{"id":"000011","name":"Cabernet Sauvignon","type":"VARIETAL_TYPE","url":"https://www.totalwine.com/wine/red-wine/cabernet-sauvignon/c/000011","storefrontUrl":"/wine/red-wine/cabernet-sauvignon/c/000011"}],"containerType":"Bottle","customerAverageRating":4.4,"customerReviewsCount":998,"department":"c0020","directType":"Winery Direct","id":"130007750","images":[{"imageType":"DEFAULT","mobileOptimizedUrl":"https://www.totalwine.com/media/sys_master/twmmedia/h72/h2e/15949509165086.png","thumbnailUrl":"https://www.totalwine.com/media/sys_master/twmmedia/h72/h2e/15949509165086.png","url":"https://www.totalwine.com/media/sys_master/twmmedia/h72/h2e/15949509165086.png","zoomImageUrl":"https://www.totalwine.com/media/sys_master/twmmedia/h72/h2e/15949509165086.png","altText":"Mascota Vineyards Unanime, 2017"}],"location":"Aisle 02, Left","name":"Mascota Vineyards Unanime, 2017","packageDescription":"750ml","packageValue":"Single","price":[{"price":25.99,"type":"EDLP"},{"price":19.99,"type":"LTSP"}],"productUrl":"/wine/gift-center/deals/red-wine/cabernet-sauvignon/mascota-vineyards-unanime/p/130007750","review":"James Suckling-Mendoza, Argentina - /"This is a rich, dense red with blackberry and blueberry aromas and flavors. Extremely well-crafted tannins. Hints of vanilla to the ripe fruit at the end. Incredible value...already beautiful to taste./"","rating":95,"ratingSource":"James Suckling","salesStrategy":{"name":"Winery Direct"},"shoppingOptions":[{"eligible":false,"location":"US-TX","message":["Unavailable"],"type":"SHIPPING","selected":false},{"eligible":true,"location":"Oak Lawn, TX","type":"INSTORE_PICKUP","selected":true},{"eligible":true,"location":"Oak Lawn, TX","type":"DELIVERY","selected":false}],"skuId":"130007750-1","stockLevel":[{"purchaseLimit":6,"stock":211}],"storeDistance":0.0001798902,"storeId":"531","storeName":"Oak Lawn","itemTasteProfile":"Blackberry, Chocolate","itemStyle":"Elegant","itemBody":"Full-bodied","transactional":true,"type":"PRODUCT","volume":"","stockMessages":{"messages":[{"shoppingMethod":"INSTORE_PICKUP","stockMessage":"In stock","addToCartMessage":"Add to cart","addToCartStatus":true},{"shoppingMethod":"DELIVERY","stockMessage":"Available","addToCartMessage":"Add to cart","addToCartStatus":true},{"shoppingMethod":"SHIPPING","stockMessage":"Unavailable","addToCartMessage":"View Product","addToCartStatus":false}],"digitalTransactional":true,"digitalInventoryQuantity":211,"digitalSpecialOrder":false,"digitalLongTermOOS":false,"digitalLimitedStock":false,"digitalInStock":true,"digitalStoreQuantity":213,"shippingTransactional":false,"shippingInventoryQuantity":211,"shippingSpecialOrder":false,"shippingLongTermOOS":false,"shippingLimitedStock":false,"shippingInStock":true,"shippingStoreQuantity":213,"digitalDeliveryEligible":true}}
Industries
Check out how industries use Total Wine & More Scraper worldwide.
You should have a Real Data API account to execute the program examples. Replace
< YOUR_API_TOKEN>
in the program using the token of your scraper. Read about the live APIs with Real Data API docs for more explanation.
import{ RealdataAPIClient }from'RealdataAPI-Client';// Initialize the RealdataAPIClient with API tokenconst client =newRealdataAPIClient({token:'<YOUR_API_TOKEN>',});// Prepare actor inputconst input ={"maxItems":20,"endPage":1,"startUrls":["https://www.totalwine.com/wine/red-wine/c/000009"],"extendOutputFunction":($)=>{return{}},"customMapFunction":(object)=>{return{...object}},"proxy":{"useRealdataAPIProxy":true,"RealdataAPIProxyCountry":"US"}};(async()=>{// Run the actor and wait for it to finishconst run =await client.actor("oyildirim/totalwine-actor").call(input);// Fetch and print actor results from the run's dataset (if any)
console.log('Results from dataset');const{ items }=await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item)=>{
console.dir(item);});})();
from RealdataAPI_client import RealdataAPIClient
# Initialize the RealdataAPIClient with your API token
client = RealdataAPIClient("<YOUR_API_TOKEN>")# Prepare the actor input
run_input ={"maxItems":20,"endPage":1,"startUrls":["https://www.totalwine.com/wine/red-wine/c/000009"],"extendOutputFunction":"($) => { return {} }","customMapFunction":"(object) => { return {...object} }","proxy":{"useRealdataAPIProxy":True,"RealdataAPIProxyCountry":"US",},}# Run the actor and wait for it to finish
run = client.actor("oyildirim/totalwine-actor").call(run_input=run_input)# Fetch and print actor results from the run's dataset (if there are any)for item in client.dataset(run["defaultDatasetId"]).iterate_items():print(item)
# Set API tokenAPI_TOKEN=<YOUR_API_TOKEN># Prepare actor inputcat> input.json <<'EOF'
{
"maxItems": 20,
"endPage": 1,
"startUrls": [
"https://www.totalwine.com/wine/red-wine/c/000009"
],
"extendOutputFunction": "($) => { return {} }",
"customMapFunction": "(object) => { return {...object} }",
"proxy": {
"useRealdataAPIProxy": true,
"RealdataAPIProxyCountry": "US"
}
}
EOF# Run the actorcurl"https://api.RealdataAPI.com/v2/acts/oyildirim~totalwine-actor/runs?token=$API_TOKEN"/-X POST /-d @input.json /-H'Content-Type: application/json'
Disclaimer : Real Data API only extracts publicly available data while maintaining a strict policy against collecting any personal or identity-related information.
