Business Rules

Data Quality Rules - Identify non-compliant rows in your data ..

It’s not simply the quality and timeliness of data that matters, it’s what you can do with it that really matters. It’s all about showing more value to business people from having higher-quality, more up-to-date data.

Business Rules translate business requirements into logic-based rules you can use to tag your data. You can define business rules to manage your data and track its quality by designating whether or not that data is compliant.

You can use the Business Glossary page to define the compliant and non-compliant data and data formats.

Using these definitions, you can use business rules to apply SQL commands (called Data Quality Rules) that identify non-compliant rows in your data. You can add any number of data quality rules to a Business Rule.

If rows are returned it means some of our data is non-compliant (this data is of poor data quality) and we should further investigate and remedy the data at source. Once that is complete, the number of non-compliant rows will hopefully reduce to zero, thus increasing the overall data quality.

Business Rules can be assigned any number of Data Quality (DQ):

• You can choose whether or not to enable data quality rules.

• You can also decide if a rule requires supervisor approval before being deployed.

• Use custom tags to track and group business rules according to your needs.

• For data quality type rules, you can further define one of the 7 standard dimensions of DQ.

Accessing Your Catalog

To access your catalog, please follow these steps:

Open Google Chrome web browser. and click on the bookmark, or
Navigate to: https://pdc.pentaho.example/
Enter the following email and password, then click Sign In.

Username

[email protected]

Password

Welcome123!

For enhanced security, it is strongly recommended that users avoid saving their login details directly in web browsers. Browsers may inadvertently autofill these credentials in unrelated fields, posing a security risk.

Best Practice

• Disable Autofill: To mitigate potential risks, users should disable the autofill functionality for login credentials in their browser settings. This preventive measure ensures that sensitive information is not unintentionally exposed or misused.

From the Business Rules card click Add New and select: Add Business Rule.

Business rules relate to data quality in at least two fundamental ways. First, they can automate the decisions that the company makes in its day-to-day operations, Second, they can be used to audit data produced by existing processes for compliance with external regulation as well as internal business policies and goals.

Click Management in the left navigation menu.
In the 'Business Rules', click: 'Add Business Rule'.
In the Create Business Rule page, enter the following information.

Field

Description

Business Rule Name (Required)

Enter the unique name of the rule that your users will recognize. Names must start with a letter, and contain only letters, digits, hyphens, or underscores. White spaces are supported, but trailing spaces are not allowed in names.

Created by

Select the username of the owner of the rule. The default value of this field is the logged-in user.

Description

Enter a description for this rule. For example, you may want to indicate the purpose of the rule to assist other users.

Note

Enter additional comments for the rule. For example, you may want to describe the workflow or use case of the rule.

Custom Tag

Add Tag - enables you to group the rules.

Rule Enabled

By default, a new rule is enabled.

Clear the check box to disable the rule. When a Rules Execution job is run, disabled rules are skipped and are not evaluated.

Rule Approved

Select to approve the rule. This option is only available to users with the Data Quality Administrator role.

If you've taken a look at the synthea:patients table you will have noticed that not all the patient passport numbers have been entered.

Let's create a Business Rule, a Data Quality Rule, checking for the passport number.

When the Data Quality Rule is applied (non-compliant passport numbers >= 10% of total count) the corresponding Business Rule status = FAILED.

Field

Setting

Business Rule Name

synthea:patients:passport

Created by

System Operator

Description

DQ - passport number

Note

Applies to all records

Custom Tag

Rule Enabled

check

Rule Approved

uncheck - if you 'approve' the rule you will not be able to configure the rule.

Click Create Business Rule to save your rule.

In the confirmation window, click Configure to Configure a rule or click Close to configure it later.

After you create a Business Rule, you can configure it in the Configuration view of the Business Rule page.

This task assumes you have completed Create a rule and are on the Business Rules page.

Locate the business rule you want to configure in the table of rules and select the View Details button (>) in its row.

In this example the Business Rule has already been previously executed & modified ..

If you have a large number of rules, select Show Filters to help you find the rule you want to edit.
Click Configuration tab and enter the following details:

Business Rule Type

Internal DQ

Data Quality Dimension

Completeness

Schedule

None

Update Row Count

Enable

Select the Data Quality Dimension - to differentiate between different quality rules that you create.

Dimension

Description

Accuracy

Uniqueness

Consistency

Timeliness

Conformity

Completeness

Validity

Set the schedule to run the business rule - None

As the Business Rule has been previously executed the Last Run detials are dispalyed.

Set the rule scope and condition:
• Select resources on which you want the rule to be evaluated and applied. Select the target table or column.
• Define the rule's condition for evaluation using the SQL query. Write an SQL query to identify non-compliant data by returning rows that do not match compliant rows and the scope count.

In the following SQL example patients table is selected to identify patients with missing (NULL) passport numbers.

SELECT 
	count(*) total_count,
	count(passport) scopeCount,
	SUM(CASE 
		WHEN passport isnull THEN 1
		ELSE 0
	END) nonCompliant
FROM synthea.patients

count(*) total_count

The total number of rows in the passport column in the patients table.

count(passport) scopeCount

The number of patients with a passport number value.

SUM(CASE 
		WHEN passport isnull THEN 1
		ELSE 0
	END) nonCompliant

When the value of the passport number is null then set as 1 = nonCompliant

Finally .. configure some rule actions:

Set the rule: IF NON COMPLIANT ITEMS >= 10% then set the Business Rule Status to: FAILED

Run the rule ..

Check the status of the Job

And take a look at the Details ..

PreviousData Identification Methods NextMetadata Rules

Last updated 1 year ago

Accessing Your Catalog

Security Advisory: Handling Login Credentials