Subsetting
Reduce records to create a smaller, representative subset of a relational database while maintaining referential integrity
Introduction Subsetting
What is subsetting?
Decrease the number of records to create a smaller representative subset of a relational database with preserved referential integrity
Why do organizations use subsetting?
Many organizations have production environments with massive amounts of data and do not want massive amounts of data in non-production test environments. Hence, database subsetting is used to create a smaller, representative subset of a larger relational database with preserved referential integrity. Organizations utilize sub-setting for test data to reduce costs, to make it manageable and for faster setup and maintenance.
Reduce infrastructure and computational costs
Excessive data volumes can lead to high infrastructure and computation costs, which are unnecessary for test data in non-production environments. With subsetting capabilities, you can easily create smaller subsets of your data to reduce your costs.
Manageable test data by testers and developers
Managing huge data volumes in non-production environments poses challenges for testers and developers. Smaller and thereby more manageable test data, significantly streamlining testing and development processes, ultimately optimizing the entire cycle in terms of time and resources.
Faster test data setup and maintenance
Smaller data volumes facilitate faster and more straightforward setup and maintenance of non-production test environments. This is particularly relevant in complex IT landscapes and when frequent changes in data structures require regular updates and refreshes to ensure the representativeness of test data.
What is referential integrity and why is it important?
Referential integrity is a concept in database management that ensures consistency and accuracy between tables in a relational database. Referential integrity would ensure that every value that corresponds to “Person 1” of “Table 1” corresponds to the correct value of “person 1” in “Table 2” and any other linked table.
Enforcing referential integrity is crucial for maintaining the reliability of test data in a relational database as part of non-production environments. It prevents data inconsistencies and ensures that relationships between tables are meaningful and reliable for proper testing and software development.
Test data in a relational database environment should preserve referential integrity to be usable. Maintaining referential integrity in non-production environments, such as those used for testing and software development, is important for several reasons:
Subsetting, not as simple as “just deleting data”
Subsetting is not as easy as simply deleting data, as all downstream and upstream related linked tables should be subsetting proportionally to preserve referential integrity. Subsetting ensures that not only data in a target table is deleted, but also that any data in any other linked table related to the deleted data from the target table is deleted. This ensures that referential integrity across tables, databases and systems is preserved as part of data deletion.
Reducing the data volume by removing “Person X” from “Table Y”, all records related to “Person X” in “Table Y” should be deleted, but also all records related to “Person X” in any other upstream or downstream related table (table A, B, C etc.) should also be deleted.
Reducing the data volume by removing “Richard” from the “Customers” table, all records related to “Richard” in the “Customer” table should be deleted, but also all records related to “Richard” in any other upstream or downstream related table (Payment table, Incidents table, Insurance Coverage Table etc.) should also be deleted.
Acros tables
Subsetting works across tables
Acros databases
Subsetting works across databases
Acros Systems
Subsetting works across systems
Do you have any questions?
Talk to one of our experts
How can I utilize sub-setting?
Proportional subsetting
You can configure the Syntho Engine to subset a relational database and to ensure that all “linked tables” are subsetted based on the “Target Table”.
- Target table: Users can define the target table as starting point for subsetting.
- Users can for example define to subset the “Patient Table” to 5% or to 500k records instead of 10.000k records.
- Linked tables: These are all directly or indirectly connected tables to the “Target Table”. Links between tables may be direct, such as a target table listing allergies that references a patients table through a foreign key relationship, or indirect, such as a target table referencing a patients table, which in turn references a hospital’s table.
- Subsetting ensures that all records related to the deleted data in the “Patient Table” will also be deleted. In the example, subsetting ensures that in any “Linked Table” there is only data that is related to the 5% (500k recors) and that all other data related to the 95% (10.000k – 500k = 9.500k records) are deleted to create a smaller representative subset of a relational database with preserved referential integrity
Subsetting based on business rules
In addition to proportional subsetting, where you specify a percentage for data extraction, our advanced capabilities allow you to precisely define the target group for subsetting. For instance, you can specify criteria to include or exclude specific subsets, providing greater flexibility and control over the data extraction process
- Customers younger than 60 years and older than 30 years and
- Als Male customers
Save your synthetic data guide now!
- What is synthetic data?
- Why do organizations use it?
- Value adding synthetic data client cases
- How to start
