YAML (YAML ain’t markdown language) is a human-readable serialization language. It can be used to configure the SDK
from the Command Line Interface with a user defined configuration using the
synthesize -c config.yaml raw_input.csv
config.yaml is the configuration file and
raw_input.csv is the input data.
The YAML configuration can be used to specify nearly all of the publicly available functionalities of the SDK for tabular data accessible through the Python API. These functionalities can be roughly divided into three categories:
Each of the listed functionalities is specified with an associated keyword, as listed in the table below:
If a YAML configuration is specified during command line synthesis it is automatically validated against a default schema, ensuring that the appropriate keywords and values dtypes have been provided.
It’s also possible to generate a default config for any given dataset using the
A config file, in this case
Single column actions can be specified by following a very similar trend. The action/functionality (e.g.
masking) is specified
as a top level keyword in the config YAML file. The set of transformations to be implemented are then specified as nested
keys beneath. Beneath each of these keys are a list of dictionaries containing the names of the columns to be acted upon
by the particular transformation, as well as any required or optional arguments. The name of the column is always specified
Below are examples of this for each single column action.
Masks are specified using the
masking property. In order to apply a mask to a given column
or set of columns, the mask in question is provided as a key. The masks available for use in command line synthesis and
their associated keys used in the YAML configuration are given in the below table:
The syntax for applying the four masks is given in the example below. For each column to be masked, a dictionary is
supplied where the
name key specifies the name of the column. Additional keyword arguments can also be specified in each
column dictionary. While the
bins property for the
RoundingMask and the
key property for the
pattern must be specified for each column to be masked using the
- name: column_to_nan
- name: column_to_round_0
- name: column_to_round_1
- name: string_column
- name: id_column
meta keyword can be used to control and override the inferred data types
(internally referred to as "metas") used during training of the model and subsequent synthesis. Using a YAML
configuration in command line synthesis it is possible to specify the meta of any column in the input dataset. The table
below details the possible meta types and their associated keys that can be used in the YAML configuration:
The below example demonstrates how to specify the meta overrides for the named columns. Meta types are nested keys,
and the columns are lists beneath these keys. If a column in the dataframe
is not specified in the
meta section of the YAML configuration the column will still be used to train the
HighDimSynthesizer, but where the meta of the column is determined by an automatic inference procedure.
- name: float_column_0
- name: float_column_1
- name: int_column_0
- name: int_column_1
- name: string_column
Entity Annotation can be used to group together a set of columns in order to treat them as a single entity
during training of the model and subsequent synthesis. To annotate a set of columns to be treated as a single entity,
annotations keyword can be used. In order to identify a group of columns as single entity, the desired annotation
should be given as a key where the values are a list of dictionaries specifying each unique entity. These dictionaries
consist of a
name property, representing the name that will be assigned to this unique entity, and a
labels property, giving
details of the columns that are grouped to form this unique entity. Refer to Entity Annotation for details
labels for each specific annotation type. Note that each entity should have a unique name.
The table below details the possible annotations that can be used in command line synthesis and their associated keywords:
The example below demonstrates how columns within a dataset may be treated as an instance of a
Person annotation. The
last_name_0 are used to specify the first and last name, respectively, of the entity known as
while the columns
last_name_1 are used for
person_1. An additional
is also specified, describing the columns
country as a single entity.
- name: person_0
- name: person_1
- name: company_0
model keyword can be used to override the default method the
HighDimSynthesizer uses to model any column in an
input dataset. To use a particular model on a column or set of columns the model type in question is given as a key,
the values of which are a list of dictionaries specifying the details of the column to be modelled. The column is specified
using as the value of the
name key, while any optional arguments that can be used when creating a given model can be
specified as additional key value pairs.
The table below details the possible model types and their associated YAML key:
The example below demonstrates the syntax required to specify columns as each of the three model types. It is required to specify the
stop keywords for
- name: enumeration_column
- name: categorical_column_0
- name: categorical_column_1
- name: categorical_column_2
- name: continuous_column_0
- name: continuous_column_1
For more information on models and overriding the default behaviour of the SDK see Overrides.
Rules are the only multi-column transformation that can be applied through YAML configuration. Additional
multi-column transformations will be added.
To specify Rules for synthesizing data, the
is used. Currently, of the rules supported by the SDK, only Associations are currently
supported by YAML configuration. To specify an association, the
association keyword is used. A list of lists, specifying
the groups of columns to be associated then follows.
In the example below, the columns
car_model are to form one association, while the columns
city are to form another.
- - car_brand
- - country
Model configuration concerns the configuration of the
HighDimSynthesizer during training, the number of training steps
to use and the synthetic data to be output from a trained model.
Any values set using the
HighDimConfig can also be tuned from the command line by using the
learn property in the
YAML configuration. For instance, the
latent_size can be specified as shown below:
Note that setting the parameters from the command line is only possible with licences where
The number of training steps for training the
HighDimSynthesizer can also be configured using the
learn property in the YAML configuration.
The number of rows to synthesize and whether to synthesize
NaN s can be configured using the
synthesize property in the YAML configuration.
Data Rebalancing can be configured using the YAML config with the
synthesize section. The columns to rebalance are then specified in a list of dictionaries. The
keyword of this dictionary is used to specify the name of the column to be rebalanced.
marginals keyword can then be used to specify the desired marginal distributions of the values within these columns.
The value of
marginals is a dictionary where the values present in the column are given as keys, the values of which
are the desired proportions they should appear in, in the synthetic data.
The below example demonstrates how to rebalance two columns,
- name: fraud_column
- name: sex_column