26 January 2024
It is now possible to automatically generate YAML config files for datasets.
A YAML schema for YAML config files can now be set in IDEs to enable YAML config file type hinting for improved and
easier writing of YAML config files. I.e. users can now hit the
tab button when writing YAML config files and see
the available configuration options for the SDK.
Native support to train and synthesize Spark
DateType columns was added (in addition to the
TimestampNTZType data types already supported).
2x faster extraction of Spark dataset meta information was achieved by implementing various performance optimisations.
Automatic detection of very high cardinality columns was added, with such columns
now automatically modelled with the
SamplingModel model, matching the behaviour of SDK 2.9 for minimal code-conversion
Automatic detection of enumerated columns (i.e. columns with predictable increases in values, like ID columns) was
added, with such columns now automatically modelled with the
EnumerationModel model, matching the behaviour of
SDK 2.9 for minimal code-conversion impact.
01 December 2023