Parser of competitor prices is automated price collection from the websites predefined by you. These may be both individual online shops and price aggregators. To parse open data from websites, developers write web applications adjusted to scan data with a required frequency.
Thus, each parser (web scraping) quickly collects prices from hundreds and thousands of item pages, which significantly saves the time of workers. Or when the assortment of items is immense, it solves the impossible for a human task at all.
As a rule, eventually, you receive data in Excel or CSV formats (if that is a parser) or analyzed and structured data in the account of a SaaS service with a convenient interface. The latter option saves you days or even weeks whenever you monitor prices; also, it vastly increases the efficiency of the use of received results.
Plan
1. Methods for competitor price parsing
2. Using a ready-made SaaS service for competitor price monitoring
3. Competitor pricing parsing with the help of uXprice
5. Independent development of a parser for competitor pricing monitoring
7. Tools for competitor prices’ analysis
8. The cost and terms of price-parser development
1. Methods for competitor price parsing
Like any other task, competitor pricing parsing may be problematic and time-consuming, especially if it is not automated. The starting point in this question is the manual price collection when it is done by special staff and when it is their main responsibility. It looks absurd, but mobile phones did not exist in the past, too. Due to the fact this method is outdated, we are not going to consider it. If you decided to parse the prices of competitors, today there are three options for that:
1) To use a ready-made SaaS service for competitor pricing monitoring.
This is the most optimal way to set up effective pricing and quickly increase sales. That is why the majority of business owners opt for this very option.
Depending on the size of the shop and the number of employees, the choice of the most appropriate service may lay both on the owner, purchasing managers, or marketers.
The main criteria of choice are the simplicity of integration, usability as well as the availability of the required functionality.
2) To use a ready-made SaaS service for competitor pricing monitoring.
This solution was used 7-10 years ago because there were hardly any other ways to do that. At that moment, it was the only way to obtain competitor prices. It is hard to believe but even though this method has got many problems, some businesses still monitor competitor pricing with the help of desktop apps even today.
3) To write a parser independently and place it on a web hosting.
This is one of the most difficult and expensive ways for tracking and collecting competitor prices. Nevertheless, it is rarely used by large retail chains that, as it may seem, have enough opportunities for them. Primarily because they do not want to take risks of launching a new project and cannot guarantee cost recovery. A ready-made solution is always easier to evaluate in this regard.
Online shops of a smaller scale also want to do their thing – the sales and don’t want to step into a completely new and unfamiliar field of programming. The chances of losing time and money are too big.
Those brave ones who decide to try this option do not understand the extent of future tasks. Having believed the promises of programmers, who have just completed the courses, to receive a complete product for just $1000, they try to save money. In fact, 99% of such users feel disappointed and return to choose the ready-made products.
In this article, we are going to review each of the above options in maximum detail to tell you about all their pros and cons.
2. Using a ready-made SaaS service for competitor price monitoring
To create such products, a whole team of professionals that is attentive to the smallest details and constantly improves the service for the customers is engaged.
The advantages of SaaS services for competitor pricing monitoring:
- The simplicity of use.
You do not need to deal with the purchase of proxy, technical support as well as configurations: everything is clear, simple, and convenient.
- Presence of thought-out tools to analyze the received data.
- Many services of that kind have got API.
Due to the availability of API, it is possible to transmit data from the price monitoring service right into the administrative panel of your website. Or you may do that to the services of data visualization (such as Google Data Studio, ChartBlocks, Datawrapper, etc.).
- Price scanning takes place in the background on the servers of a SaaS service.
For instance, there is a solid number of products, and scanning will take 4 hours. You may plan this activity any time of the day, and your PC may be off during this time, while an Internet connection will be required only to check and download the final results.
- Several users can work in the same service simultaneously.
- Having enough capacity for various tasks.
The service will not be ‘down’ if you have got a lot of products, or if there is an unplanned necessity to increase the volume of items for further scanning.
If we sum up, the main advantages of SaaS solutions are the actual information with visualization as well as the presence of pricing recommendations, depending on the results received.
3. Competitor pricing parsing with the help of uXprice
A major advantage of cloud services is their free trial option. In uXprice, this period is 7 days. To create an account, a simple registration is fully enough:
After that, on the starting page, choose a country for monitoring and a method for uploading the products for which you want to monitor prices. The mass upload is executed via a link to the product feed of Google Merchant, Facebook, or Yandex feed. Another approach is to add the items one by one by mentioning the links to the required product cards:
Click the button ‘Upload items’ and you will get into a personal account to the page ‘Reports’. Here you can see the products that we have uploaded on the starting page; all you need is to choose the competitors for price parsing. This should be done for every item you have, that is why there is an appropriate button called ‘To choose competitors’ near every product:
After you have clicked the button, a new page ‘The choice of competitors’ will open in a new tab. On the left, you can see the information about your product: image, name, and price. On the right, there are potential competitors that the system has identified for you independently. In the future, the automatic search of competitors can be turned on according to schedule; in a trial mode, it is activated by default. Also, use filters to find necessary sellers or models in a convenient way:
If among the sellers suggested by the system you find those that you would like to parse regularly, simply click the ‘Track’ button under them:
After that, the items chosen for parsing will appear on a separate tab ‘Tracked’. Each product has got it so that you always could see based on which competitor prices you receive the reports. Let us take as an example another product having a smaller number of shops for tracking:
If the number of sellers found by the system is not enough, you may add links to product cards of specific competitors by clicking the button ‘Add competitors’:
In an opened new tab, it is necessary to type in each link from the new line. After the process of uploading items is over, click the button ‘Save’:
There is an opportunity to monitor the prices of completely different sellers on different items. There are no limitations to the number of shops. However, the number of tracked competitors for each of your products may differ as well: for one item it will be 30 sellers, while for another item — only 5.
Adding competitors for tracking is one-time work. During the next parsing, their prices will be collected automatically. But we would advise you to use the automatic search of competitors from time to time. This will help you evaluate the competition on each trade item and define new sellers on time.
Report on the parsing results is a chart that includes a competitive analysis with the following metrics:
– average price
– minimum price
– maximum price
– competitor price range
– median price
– price position and others.
This is how the report for the items added by us in the trial mode looks like:
One more important metrics is Recommended price. It allows automating revaluation of your items based on competitor pricing. The service offers an optimal price depending on the median price by default, but you may choose another strategy and set up your own rules. This may be done on the page called ‘Pricing strategy’:
During the trial period, all registered users will receive 100 credits daily. After the 7-day period is over, free credits will reset; that is why you should necessarily use this opportunity to assess the work of the service and make the right decision.
This option may seem more expensive in terms of costs than the next two. It is so in a way, but quality always costs money. But if we count up for what you are paying and take the case with creating your own professional SaaS service as an example, then you lose time and money with no guarantee of the result. And in case with the use of cheap parsers and desktop apps, you receive information that cannot be efficiently used. So, here the proverb ‘measure thrice and cut once’ is more than true.
4. Specialized desktop app
The use of a special program for competitor price monitoring does not require development but the lack of a good interface and the necessity to make the adjustments also takes a lot of time. By the way, in this case, you will have to purchase a proxy for the work of the program independently and control the workability. Not to mention the fact that the work of such apps is often related to failures when working with huge data volume, Internet issues, and other non-standard situations.
Talking about its positive sides, we should mention that this option is faster to implement than the first variant but worse than the third one. Here are some more advantages:
- this is the cheapest way of parsing;
- a possibility to monitor prices by a set schedule;
- you can get the results comparatively fast in contrast to independent development;
- usually, there is no correlation between the price and the number of competitors being tracked.
But there are disadvantages as well:
- a little choice of such programs;
- no functionality for analysis of received data;
- no recommendations on price;
- independent parser configuration for every separate competitor website;
- connection by API is absent;
- requires individual configuration and maintenance of proxy workability;
- parser needs to be readjusted after the changes on the competitor website have happened;
- PC must be turned on during parsing;
- requires uninterrupted access to the Internet with high speed.
Clearly, the main disadvantage is the absence of functionality to analyze the received data. Similar applications have no tools for that, so comparison of information for thousands of products (that is the number of products in a middle-sized online store) will need to be done manually again. In doing so, the worker engaged in this activity must not only understand the assortment well but also define the price of the product on his own. And that is a very demanding and responsible task, especially, taking into account the volume of analyzed data.
As you can see, desktop programs for competitor pricing monitoring do optimize the processes of necessary data collection, but their use does not simplify further data processing and changing your prices (as a result of such an analysis).
Anyway, parsing prices with the help of such programs may be a working method for online stores with a limited assortment of goods, a list of which is also permanent. Particularly, in fields with few competitors and without abrupt price fluctuations.
5. Independent development of a parser for competitor pricing monitoring
This option will work best for you if:
— you can create a precise technical task and understand well what kind of information you need from a parser and how it should look like;
— you have proven developers who can maintain and develop the project in the long run.
At this stage, we should mention that many sellers have learned how to protect their websites from parsing. If you do not want the website to block your IP address during a big number of requests during data collection, it is required to buy a proxy. When parsing the data, it will interchange your IP address with other IP addresses, which will allow you to collect the necessary information comparatively safely.
But there are many other, more tricky ways to protect a website from parsers. For example, when the security system identifies the collection of prices on the website and automatically changes them into other (wrong) ones. In this case, competitors may ‘let’ you continue parsing not only intentionally false prices but those that will give them a competitive advantage (for instance, overpriced prices).
One more measure to prevent parsing is fixating cursor movements, targeting of elements, playing videos, page scrolling, etc. All of that interferes with the algorithms to define the structure of HTML elements correctly and parse the data.
It is also common when developers of an online store intentionally create the code of the website deviated from the classical structure or ‘dynamic’. Talking about creative methods, we can distinguish creating the pages that regular users will never bump into but parsers will definitely ‘have a look’ at them. The IP addresses of ‘curious users’ are later added to blacklists.
But serious large-scale companies usually use a multi-level approach for their websites’ protection when several ways of security are used at the same time. New methods are created regularly, so be on the lookout to find and neutralize them in time (and you may need the whole team of specialists for that purpose).
Thus, there are so many factors that require constant technical support and making corrections to the code of the parser:
- changes in the code of a source website from which data parsing is executed;
- issues with a proxy that may break down any moment, so that its work must always be controlled;
- changes in the website policy prices from which you parse information. For example, CAPTCHAs will appear or the type of CAPTCHA will change, and the parser will not work;
- the hosting does not fit you because the parsing task is too resource-intensive. If you plan to monitor the prices for at least 3000 items, a dedicated server may be required. And if you have your own server, then you will surely need a system administrator to maintain the server workability.
6. Analysis of received data
Let us assume that everything goes smoothly and you received parsed information in Excel or CSV format. What is next? Now we should do what the data was collected for – the analysis of the information. That is why the second important part of the task during the independent development of a parser is to think in advance about all necessary tools for further processing of received information. Sure, one can work with charts, but that is inconvenient and extremely time-consuming.
All programs that we use on a daily basis have got a well-thought web interface. This is not only about intuitively clear visual content but also about the convenient implementation of a number of functions. The functions that we got used to so that sometimes we do not even notice them.
And to be able to work with parsed data in the web interface independently, the first thing to do is to develop it. And that is a tough task. First and foremost is to come up with a detailed technical task as well as to find a designer and a front-end developer who can handle these tasks. Besides, you need to have a clear vision of how and what to do in terms of a visual interface.
7. Tools for competitor prices’ analysis
If you are of the opinion that the interface is not important for you, then you just forget about the working time of your employees. Without having the necessary tools for data analysis, it will take weeks and months to process the results of parsing, while the actuality of the data will be lost.
For example, you have 1000 items prices of which you need to check for actuality and, if necessary, replace them with competitive ones. The following points should be met, so you need to:
- compare the prices of several competitors with your own ones;
- check your procurement price;
- consider the desired markup;
- define a new optimal price for each item;
- replace all prices on the website.
Also, remember that the procurement price is often indicated in another currency different from the price currency on the website, which also complicates the task. Plus, the competitors for each item will both appear or disappear; that is why if your parser does not show and consider them, the price will not be competitive, too.
Is it a real task to complete all of that quickly if each of the points is located in different documents (tabs), and the number of items is 1000? Saying objectively — no. And taking into account the speed of appearing new sellers and the frequency with which competitor prices change, this work is senseless at all.
8. The cost and terms of price-parser development
In general, the whole development of a parser may take from several weeks to several months and even years. The terms are greatly dependent on the possibilities that the parser should possess as well as the difficulty of a website it will be able to parse.
You might have seen the cost of creating a parser within $1000-$5000. It seems to be affordable. But you can expect prices like that only from beginner specialists with low-level skills. Having contacted them, you will receive the ‘product’ that only a programmer himself will be able to use eventually. Due to a lack of experience, he will not be able to realize many tasks that you set for him.
Also, bear in mind that the information received from parsing is going to be too much, and you will have to deal with optimization of the database additionally. Otherwise, everything will work slowly with huge amounts of information. Especially, if the architecture of the base will be planned poorly. There is a chance you will need to find a separate specialist working in database optimization for this task.
The hourly rate of specialists who will deal with database optimization according to your needs may cost $400 per hour. That is assuming that you will create proper technical tasks and will answer the possible questions on time. Under other circumstances, this task may drag on greatly. In this situation, you need to know for sure what you need in order not to spend money in vain having received no result.
For example, one of the issues you may face is the slow loading of pages in the web interface that has just been created. Imagine that the web page with data on each item will load for 30 seconds. Low speed is caused by a huge amount of data and poor code optimization. Even if there are only 1000 products, loading of pages will take 500 minutes — that is about 8 hours that you or your employee will waste.
As a result, writing the parser itself is only 20% of the whole work, while creating the interface in which you can work efficiently and in a convenient way is 80% of the time and 90% of the money. Also, it is most likely that the same person will not be able to write a parser independently and set it up visually (as well as prepare pages, develop a convenient interface taking into account UX, etc.). That is a rare case when the same specialist is also familiar with databases and writes PHP code, etc. Respectively, you will need to engage other specialists and reveal the details of the project to them separately. Most likely, you will need a front-end developer, administrator of web services, and, probably, a back-end developer.
Thus, a well-written and well-thought parser of competitor prices is an expensive option that requires constant financial investments and modernization in the future. A ‘simple’ option will not be enough to deal with the tasks in a full volume because it will not simplify the process of revaluation at all. Eventually, you may spend money on it and do not use it after that.
Afterword
The number of online stores is growing immensely. That is why monitoring competitor prices is one of the most actual tasks for e-commerce projects. To solve it, it is possible to create a parser independently, use ready-made software or a specialized SaaS service.
If first or second options are chosen, there are hundreds of things that must be thought out independently and without which the parser will be a useless program that does not answer the main question: ‘what price should I set?’. Developing your own parser for competitor pricing monitoring is long and expensive, while a ready-made option is a lot of present issues and a lack of any functionality.
A middle ground is the use of a SaaS service. In such a case, you will receive precise, structured, and analyzed by algorithms data about the prices of your competitors. Not to mention recommendations on best pricing to achieve your business goals: increasing the sales, income, and cost-effectiveness of investments into advertisements. The proof of the efficiency of using SaaS services is successful cases.
