Rules

The classes in synthesized.metadata.rules allow the user to constrain the synthetic dataset, ensuring it confirms to pre-defined business logic, or a custom scenario. They can be used with the ConditionalSampler to generate custom synthetic data:

from synthesized import ConditionalSampler, MetaExtractor, HighDimSynthesizer

df = ...
df_meta = MetaExtractor.extract(df)
synth = HighDimSynthesizer(df_meta=df_meta)
synth.learn(df)

synth.df_meta.associations.append(...)  (1)
sampler = ConditionalSampler(synthesizer=synth)
df_synth = sampler.synthesize(
    num_rows=...,
    expression_rules=[...],  (2)
    generic_rules=[...]  (3)
)

1	Strict associations between categorical columns can be enforced.
2	Expressions can be used to describe one column as a mathematical transformation of other columns.
3	Prescribing generic rules and relationships for one or more columns is also possible.

Associations

Association rules can only be declared between columns containing categorical variables.

Often, datasets may contain two columns with an important well-defined relationship. For example:

Table 1. cars-original.csv
Make	Model	Total
"Ford"	"Fiesta"	372013
"BMW"	"M3"	10342
"BMW"	"X5"	39753
"Volkswagen"	"Polo"	87421
"Ferrari"	"California"	632

In the above dataset, "Make" has a one-to-many association with "Model". In other words, certain categories in "Model" only appear with certain categories in "Make". The HighDimSynthesizer captures highly detailed dataset-wide information, but as it also attempts to generalize specific row-level information, a case such as "Polo" always appearing with "Volkswagen" isn’t strictly followed. A possible output of the synthesizer could be:

Table 2. cars-synthetic.csv
Make	Model	Total
"BMW"	"X6"	36382
"Ford"	"Fiesta"	401877
"BMW"	"Polo"	67862

In this example, the HighDimSynthesizer has generated a row with a "BMW Polo", which is an unrealistic combination. If capturing strict column associations such as this is important, the synthesizer can be configured to do so by defining an Association rule.

from synthesized import HighdimSynthesizer, MetaExtractor
from synthesized.metadata.rules import Association

df_meta = MetaExtractor.extract(df)
synth = HighDimSynthesizer(df_meta=df_meta)
synth.learn(df)

rule = Association(associations=["car_manufacturer", "car_model"])
rule.extract(df, df_meta=df_meta)
synth.df_meta.associations.append(rule)

synth.synthesize(10)

If you create the association class prior to creating the DataFrameMeta object, a convenient way to extract all the needed information and include it in the DataFrameMeta object is to use the associations argument of the MetaExtractor.extract method:

rule = Association(associations=..., nan_associations=...,allocated_memory=...)
df_meta = MetaExtractor.extract(df, associations=[rule])  (1)

1	This will automatically extract the associations.

In addition, sometimes empty values are correlated, e.g: if one column specifies the number of children in a family, we would expect that the names of these children to be empty if they don’t exist:

rule = Association(associations=["NumberOfChildren"], nan_associations=["Child1Name", "Child2Name", ...], allocated_memory='1gb')
rule.extract(df, df_meta=synth.df_meta)

The association object must call the method .extract(df, df_meta) in order to learn which values in one column appear with which values in another column. If any particular values of a column never coexist with particular values of another column then it will association rule the Synthesizer to never output those values together.

The method .extract(df, df_meta) assesses the approximate amount of memory needed by the association and raises a memory error if this exceeds the parameter allocated_memory, which should be a string comprising a number and a unit 'b','kb', mb','gb','tb'.

There are some constraints on what rules you can define: the HighDimSynthesizer requires requires that every column appears in no more than one association object. In addition, for each association object, a column cannot appear in both the associations and nan_associations arguments.

Columns and Values

When using rules, we sometimes want to refer to specific columns in a table or to a specific string or numerical value. To make this clear to the ConditionalSampler, we use the classes Column and Value.

from synthesized.metadata.rules import Column, Value
column = Column("A")  (1)
value = Value(10)  (2)

1	Refers to column A.
2	Refers to the value 10.

These can then be used in expression rules and generic rules.

Expressions

When it is known a priori that a field in a dataset is related to others through a mathematical transformation, this can be enforced with an Expression rule. This takes a string expression that can be parsed by pandas.eval

from synthesized import ConditionalSampler, HighdimSynthesizer, MetaExtractor
from synthesized.metadata.rules import Column, Expression

df_meta = MetaExtractor.extract(df)
synth = HighDimSynthesizer(df_meta=df_meta)
synth.learn(df)
sampler = ConditionalSampler(synth)

column = Column("A")
rule = Expression(column=column, expr="a+b+c")
rule.set_meta(df_meta)

sampler.synthesize(num_rows=10, expression_rules=[rule])

Generic

A GenericRule is a special type of rule that can be enforced by conditional sampling of ConditionalSampler.

As these rules are enforced by iterative conditional sampling, it may not be possible to fully generate the desired number of rows if the rules cannot be fulfilled, or represent a very small proportion of the original data. In this case, it will return the data it was able to generate. Increasing the max_trials parameter may resolve this issue.

Equals

Equals enforces the field of a dataset to be strictly equal to a specified value. Unlike the Expression rule Equals can refer to either numeric or categorical.

from synthesized import ConditionalSampler, HighdimSynthesizer, MetaExtractor
from synthesized.metadata.rules import Column, Equals

df_meta = MetaExtractor.extract(df)
synth = HighDimSynthesizer(df_meta=df_meta)
synth.learn(df)
sampler = ConditionalSampler(synth)

column_x = Column("x")
column_A = Column("A")
rule = Equals(column_x, column_A)
rule.set_meta(df_meta)

sampler.synthesize(num_rows=10, generic_rules=[rule])

IsIn

IsIn is similar to Equals, but specifies a list of allowed values.

from synthesized import ConditionalSampler, HighdimSynthesizer, MetaExtractor
from synthesized.metadata.rules import Column, IsIn, Value

df_meta = MetaExtractor.extract(df)
synth = HighDimSynthesizer(df_meta=df_meta)
synth.learn(df)
sampler = ConditionalSampler(synth)

column = Column("x")
values = [Value("A"), Value("B")]
rule = IsIn(column, values)
rule.set_meta(df_meta)

sampler.synthesize(num_rows=10, generic_rules=[rule])

ValueRange

ValueRange can be used to constrain synthesized data to a user-defined range, either to improve the quality of the synthetic data or to generate custom scenarios. The upper and lower bounds of the range can be numeric, e.g., "0 < x < 10".

from synthesized import ConditionalSampler, HighdimSynthesizer, MetaExtractor
from synthesized.metadata.rules import Column, Value, ValueRange

df_meta = MetaExtractor.extract(df)
synth = HighDimSynthesizer(df_meta=df_meta)
synth.learn(df)
sampler = ConditionalSampler(synth)

column = Column("x")
values = [Value(0), Value(10)]
rule = ValueRange(v1=column, v2=values)
rule.set_meta(df_meta)

sampler.synthesize(num_rows=10, generic_rules=[rule])

or they can be defined by another field of the dataset, e.g., "z < x < y".

from synthesized import ConditionalSampler, HighdimSynthesizer, MetaExtractor
from synthesized.metadata.rules import Column, ValueRange

df_meta = MetaExtractor.extract(df)
synth = HighDimSynthesizer(df_meta=df_meta)
synth.learn(df)
sampler = ConditionalSampler(synth)

column_x = Column("x")
bounds = [Column("z"), Column("y")]
rule = ValueRange(v1=column_x, v2=bounds)
rule.set_meta(df_meta)

sampler.synthesize(num_rows=10, generic_rules=[rule])