Data Quality
A scenario can finish without an error and still return disappointing data. A supplier may change its website: products are collected, but some have no price. Or a catalog that usually contains a thousand products returns only twenty. Quality helps you notice this.
A rule describes the result you expect. A finding records a specific completed collection that did not meet those expectations. Findings appear automatically; you do not create them manually.
Where to start
Suppose you collect car-part prices every day. Consider three rules:
- Products without a price: report when more than 5% of products have no price.
- Catch a shrinking catalog: report when the product count falls by at least 40%.
- The same SKU on different cards: find repeated SKUs that may prevent reliable price-list matching.
You do not need all three. Pick what can actually affect your work. Phone numbers and dates may matter most for listings; a storefront needs images; matching a supplier's price list may depend on reliable SKUs.
Choose a template and save a rule
Open Quality and the Templates tab. Cards describe familiar business tasks. Suggestions use the names and types of fields on your accessible sites. Treat a match as a hint: check that the selected field is your current price, not an old price before a discount.
- Choose a card or open All templates.
- Select a site. The available fields follow that site's schema.
- Choose a field and suitable limits. Labels explain what each number means.
- Read the preview: when the rule triggers and what happens next.
- Keep Notify only while learning how the source behaves.
- Select Create rule. For an existing rule, select Save changes.
Saving a rule does not run a scenario. It checks the next successful complete collection for that site. Existing records are not changed, and historical collections are not checked again. A trial that visits a few pages is not a complete collection.
If none of the examples fits, select Create your own rule. Choose a check type and configure it in the same way. You do not need code or a regular expression.
The interface keeps a saved rule's site and check type fixed. Create another rule for a different site or check type. You can change its name, limits, action and enabled state.
Cards show a placeholder while their illustrations load. If an image fails to load, you can still set up the rule. Try loading again reloads the illustrations without changing your rules.
Add more suggestions, hide them, or bring them back
Saving your first rule does not remove the other options. More useful checks stays at the bottom of the section, with the list scrolling above it. Add another rule, select All templates, or return later. The same template can be used on other sites or with different limits.
To remove the suggestions from view, use the Hide suggestions close button. The choice is remembered in this browser. Saved rules continue working. Use Show suggestions at the bottom of the section to bring the cards back.
Platform-wide rules may already be enabled before you create your own. They also appear in the Rules tab. Your own rules do not disable them; both groups are evaluated. Only a platform operator can change platform-wide rules.
Understand percentages and limits
More than 5% empty means five missing prices out of 100 checked products are allowed, but six produce a finding. A zero price is populated. To report zero prices, add a value check too.
40% fewer products means the earlier collection had 100 products and the new one has 60 or fewer. Comparison needs a previous successful complete collection from the same scenario. A first collection is not bad merely because there is nothing to compare it with.
Price from 0.01 to 100,000 includes both ends. Clear a bound in the form if that side should be unlimited. At least one bound must remain. A minimum of 0.01 is unsuitable if prices of 0.001 are valid for your task; choose your own minimum.
Offer age of 48 hours uses the selected date field. It measures age at collection completion, not when you open the page. Exactly 48 hours is allowed. Invalid dates and dates more than five minutes in the future also need review.
All 24 templates with examples
These numbers are starting points. Review the fields and conditions before saving. A template never changes the source itself.
| Template | When it helps and what to inspect |
|---|---|
| Catch a shrinking catalog | Yesterday there were 1,000 products; today there are 500. Some catalog pages may have been missed. |
| Enough data to work with | Your price list needs at least 200 products. A collection of 199 is too small. |
| Unexpectedly many records | A catalog of 500 products suddenly returns 1,500 cards. Check extra pages or variants. |
| Watch rejected records | The scenario submitted 100 products, but field validation prevented 20 from being saved. |
| Products without a price | Product names remain, but prices are missing. Select the current-price field. |
| Keep product names in view | A product without a name is difficult to find or show to a buyer. |
| SKUs ready for matching | Missing SKUs make reliable matching against your price list difficult. |
| Links back to source cards | A sales manager needs the supplier's product link to inspect the item quickly. |
| Product cards with images | A storefront needs images. Select the main image or gallery field. |
| Descriptions are still present | A website change can prevent descriptions from being collected. |
| Prices without zero values | Prices should be at least 0.01. Adjust this if free products are valid. |
| Stock quantities that are not negative | A stock quantity of −3 may be an error if the supplier does not use negative stock. |
| Discounts between 0 and 100% | A discount of 150% needs review. Check that the source stores percentages, not fractions from 0 to 1. |
| Links in a complete format | The field contains “/product/123”, but you need “https://example.com/product/123”. Availability is not checked. |
| Email addresses without obvious format errors | “sales@” has no domain. No message is sent and mailbox existence is not confirmed. |
| Phone numbers without stray text | The field contains “Call seller” rather than a number. A phone number needs 10 to 15 digits. |
| Russian tax IDs without code errors | Checks 10 or 12 digits and checksum digits. It does not search a business register. |
| Product barcodes without format errors | Checks GTIN, EAN, or UPC length and checksum. It does not confirm product authenticity. |
| Availability in familiar statuses | Allow only your source's actual statuses, such as “In stock”, “Preorder”, and “Unavailable”. Replace the example list. |
| Names with enough text | A one-letter name may be too short for your catalog. You can also set a maximum length. |
| The same SKU on different cards | Two separate products share one SKU. A, A, B contains one extra repeat among three values. |
| Repeated product-page links | Different records point to one supplier card. Check whether these are expected product variants. |
| Prices change across many products | Most prices changed in one collection. This may be a price-list update or an extraction error. New records are not counted as changed. |
| Offers within your age limit | A listing's update date should be no older than two days. |
Format, range, and length checks do not replace completeness checks. Use both if you
need to catch missing prices and prices of −100. Strings containing ordinary or
non-breaking spaces count as empty; 0 and false remain populated.
Read a finding and investigate
Open Findings and select a row. The right panel shows:
- The site, rule, and collection involved.
- What differed from the expected result.
- The measured result and limit saved at the time of the check.
- Saved conditions, such as price bounds and the allowed share of errors.
- How many records were checked and affected.
- A suggested first step for investigation.
For a shrinking catalog, compare the two collections and check that all pages were processed. For missing prices, open several source cards and inspect the price-field collection settings. For an unknown status, inspect the actual source values: the supplier may simply have introduced a status you want to allow.
A rule does not fix scenario code. First resolve the cause or deliberately adjust the conditions, then run a new complete collection and review its result. The latest check may still refer to old settings after you edit a rule. Historical findings remain when limits change. If a rule is deleted, its findings remain as history without a live rule link.
The Conditions at the time of the check section shows settings saved with the result. Editing a rule does not change them. Older findings may lack this snapshot. The panel says so and shows only available measurements and any recorded limit, without using the rule's current settings.
Notify only or flag for review
Notify only creates a finding and leaves automatic delivery after the collection enabled. It is a useful starting point for new rules and expected changes.
Flag for review creates a finding, flags the whole collection, and pauses automatic delivery after it. Choose this when the result needs review before reaching your system. It is not a promise to block every manual export or HTTP data request.
A strict rule affects delivery only when that rule actually triggers. A strict rule that passed does not strengthen a separate warning.
No findings does not always mean everything was checked
A rule can have three outcomes:
- Passed: checked data met the conditions.
- Breached: a violation produced a finding.
- Skipped: a reliable check could not be completed; a reason is shown.
For example, a first collection has no comparison baseline, all values may be empty, or the field may have been removed from the site's schema. A collection continued after an interruption may contain only part of the data and cannot be compared with a whole catalog.
Count checks use the complete collection totals. Field checks normally read the first 5,000 records in internal order. A limited-sample label means more records exist beyond the sample. The outcome applies to the checked portion.
There is also a safe memory budget: up to 8 MiB of data and 100,000 sampled values, with no more than 32 distinct fields in one collection. Values larger than 8 KB are excluded separately. Reaching the overall budget produces a skipped check with partial measurements, not a successful check. Original records remain saved.
Large lists and deeply nested data can consume substantial memory even in a short JSON document. The service estimates their structure before decoding. If it cannot read them safely, a separate skip reason is shown. This also applies to the first record: the message will not incorrectly claim that there are no records.
If a rule has no latest result, wait for a new complete collection and check that it is enabled. An empty Findings list does not replace that step.
Team permissions and your own agent
You see only sites and findings accessible in the selected workspace. Creating or editing a rule requires permission to edit that site. If controls are unavailable, ask the workspace owner or a member with suitable permissions.
On your own agent, rules are viewed locally and definitions are edited in the cloud. Synchronize the agent after changes. Checks run alongside the collected data, so a missing cloud copy must not be mistaken for a successful empty-data check. A compatible agent and cloud version are required.
A notice that reports are still being delivered does not mean there are no findings. Wait for delivery or inspect the result on the agent itself. An unsupported-version notice requires updating the agent and cloud, not creating a local rule outside sync.
For integrations, see the kind vocabulary, API templates and suggestions, and rule creation.