# Configuration reference

:::callout{theme="warning" title="Sunset"}
HyperAuto V1 is in the [sunset](../../platform-overview/development-life-cycle/index.md) phase of development and will be deprecated at a future date. Full support remains available. The creation of new V1 pipelines is discouraged, and users should migrate from HyperAuto V1 to V2 as detailed in the [migration documentation](../v1-to-v2-differences/index.md#migrating-existing-hyperauto-v1-pipelines-to-hyperauto-v2).
:::

SDDI's pipeline is generated by a fully automated code repository. To review the steps involved, read about [pipeline generation](../v1-overview/index.md#pipeline-generation). [Cockpit](../v1-cockpit/index.md) is the default place to interact with those configurations, but you may have to manually amend the configuration files for advanced parameters or to configure non-standard source types.

Configurations are performed within two main files located in the `transforms-bellhop/src/config/` folder:

* [SourceConfig.yaml](#sourceconfigyaml)
* [PipelineConfig.yaml](#pipelineconfigyaml)

:::callout{theme="warning"}
The advanced manual settings documented on this page can bring your SDDI pipeline into a broken state if not applied correctly. Always verify changes on a branch before deploying to production.
:::

## SourceConfig.yaml

The following is a notional example of a fully-defined SourceConfig file.

```yaml
sourceName: MY_SOURCE
sourceRid: ri.magritte..source.abcdefgh-1234-5678-910a-zyxwvut
sapContext:
  type: direct
rawFolderStructure:
  raw: /HyperAuto/source/raw
  dataDictionary: /HyperAuto/source/metadata
cleaningLibraries:
  - convert_all_columns_to_clean_types
deploymentSemanticVersion: 2
metadataSparkProfiles:
  - DRIVER_MEMORY_MEDIUM
languageKey: 'E'
tables:
  - tableName: ABCD
    datasetTransformsConfig:
      datasetName: ABCD
      deduplicationComparisonColumns: []
      batchUnionComponents: []
      tableCleaningLibraries: []
  - tableName: WXYZ
    datasetTransformsConfig:
      datasetName: WXYZ
      deduplicationComparisonColumns:
        - /PALANTIR/TIMESTAMP
        - /PALANTIR/ROWNO
      batchUnionComponents:
        - WXYZ_historical
        - WXYZ_incremental
      tableCleaningLibraries:
        - parse_timestamp_column
      sparkProfiles:
        profiles:
          - EXECUTOR_MEMORY_MEDIUM
          - NUM_EXECUTORS_4
```

### Parameters description

| Parameter | Description|
|---|---|
| `sourceName` | The name to identify a source system. Used to prefix primary and foreign keys. |
| `sourceRid` | The RID of the source attached to this SDDI instance. |
| [`sapContext`](#sapcontext) | (Optional) Details of the SAP context. |
| [`rawFolderStructure`](#rawfolderstructure) | Defines the folders in which the raw data and metadata reside. |
| [`cleaningLibraries`](#cleaninglibraries) | List of cleaning libraries to apply to all tables. |
| [`deduplicationConfig`](#deduplicationconfig) | (Optional, default: None) Config used to specify which columns to use for the deduplication logic. |
| [`metadataSparkProfiles`](#metadatasparkprofiles) | (Optional, default: None) List of Spark profiles to apply to metadata generation. |
| `languageKey` | (Optional, default: 'E') Language to use in enrichments. |
| [`deploymentSemanticVersion`](#deploymentsemanticversion) | (Optional, default: 0) Semantic version of the pipeline; incrementing it will force a snapshot. |
| [`tables`](#tables) | List of tables from that source to be processed by SDDI. |

#### `sapContext`

(Optional) Details of the SAP context. SAP Explorer will use this to pre-select the context. Each context will need to have its own SourceConfig file.

#### `rawFolderStructure`

Defines the folders in which the raw data and metadata reside.

Fields:

* **`raw`:** Path of folder where raw tables are ingested.
* **`dataDictionary`:** (Optional, default:`raw`) Path of folder where metadata tables are ingested.

#### `cleaningLibraries`

List of cleaning libraries to apply to all tables. Cleaning functions are defined in `transforms-bellhop/src/software_defined_integrations/transforms/cleaned/function_libraries`.

Adding or removing a function requires incrementing the [`deploymentSemanticVersion`](#deploymentsemanticversion).

#### `deduplicationConfig`

(Optional, default: None) Config used to specify which columns to use for the deduplication logic. Configuration defined here is applied across all tables.

Fields:

* **`comparisonColumns`:** Columns for which the max value will be taken to determine the uniqueness of primary keys.
* **`changeModeColumn`:** (Optional) If specified, rows having value `D` in this column will be deleted.

#### `deploymentSemanticVersion`

(Optional, default: 0) Semantic version of the pipeline; incrementing it will force a snapshot.

See [Incremental Transforms](../../transforms-python/incremental-usage/index.md) for the effects of `deploymentSemanticVersion` on incremental and snapshot transforms.

#### `metadataSparkProfiles`

(Optional, default: None) List of Spark profiles to apply to metadata dataset generation (`objects`, `fields`, `links` and `diffs`).

Be sure the profiles are [added to the repository](../../code-repositories/spark-profiles/index.md#importing-spark-profiles) before referencing them here.

#### `tables`

List of tables from defined source to be processed by SDDI.

Fields:

* **`tableName`:** Name of the table in metadata.
* **`datasetTransformsConfig`**
  * **`datasetName`:** Foundry dataset name of the raw data.
  * **`deduplicationComparisonColumns`:** Table-specific config used to deduplicate data and specify which columns to use for the deduplication logic. Applied after the global deduplication fields.
  * **`changeModeColumn`:** (Optional) If specified, rows having value `D` in this column will be deleted. Applied over the global change mode column.
  * **`batchUnionComponents`:** List of input dataset names that should be unioned before the cleaning step.
  * **`sparkProfiles`:** (Optional) Spark profiles to apply at different stages of the transforms.
    * **`profiles`:** Spark profiles; see [details for adding them to the repository](../../code-repositories/spark-profiles/index.md#importing-spark-profiles).
    * **`stages`:** (Optional, default: None) Transform stages the profiles should be applied to. Value should be in \[CLEANED, DERIVED, ENRICHED, FINAL, RENAMED, RENAMED\_CHANGELOG]. If None, profiles are applied at all stages.
  * **`tableCleaningLibraries`:** List of cleaning libraries to apply to this table. Cleaning functions are defined in `transforms-bellhop/src/software_defined_integrations/transforms/cleaned/function_libraries`. Adding or removing a function will require you to increment the [`deploymentSemanticVersion`](#deploymentsemanticversion).
  * **`enforceUniquePrimaryKeys`:** (Optional, default: False). If True and `deduplicationComparisonColumns` are defined, guarantees that only one record per primary key will be kept at the deduplication stage. This may result in non-deterministic behavior.

## PipelineConfig.yaml

Example of a notional fully-defined PipelineConfig file.

```yaml
sourceName: HyperAuto
sourceType: SAP_ERP
sourceConfigFileNames:
  - SourceConfig.yaml
outputFolder: /HyperAuto/source/output
workflows:
  my_workflow:
    variables:
      - name: my_variable_name
        value: my_variable_value
    enrichments:
      - my_enrichment_name
tables:
  ABCD:
    displayName: Header Table
    types:
      - OBJECT
  WXYZ:
    displayName: Item Table
    types:
      - OBJECT
      - METADATA
disableForeignKeyGeneration: False
disableEnrichedStage: False
disableRenamedStage: False
```

### Parameters description

| Parameter | Description|
|---|---|
| `projectName` | Project name. Serves as a prefix to Ontology objects. |
| `sourceType` | Type of sources supported by SDDI. Should be one of \[SAP\_ERP, SALESFORCE, ORACLE\_NETSUITE]. |
| `sourceConfigFileNames` | List of SourceConfig filenames to include in the pipeline. |
| `outputFolder` | Defines the folder in which output datasets will be written. |
| `workflows`| List of workflows to deploy, with configurations. |
| [`tables`](#tables) | List of tables processed in this SDDI pipeline. |
| `disableEnrichedStage` | (Optional, default: False) If enabled, no enriched datasets will be produced. Use with caution, as enabling will break workflows. |
| `disableRenamedStage` | (Optional, default: False) If enabled, no renamed\_changelog datasets will be produced. Use with caution, as enabling will break workflows. |
| `disableForeignKeyGeneration` | If enabled, no foreign key columns will be produced. Use with caution, as enabling will break workflows. |

#### `tables`

List of tables processed in this SDDI pipeline:

* **`displayName`:** Human-readable name of the table. Output dataset name will be constructed in the form `displayName (technicalName)`
* **`types`:** List of data types this table represents (can be many).
  * **OBJECT:** Master data table that constitutes an object in the ontology.
  * **METADATA:** Metadata table that contains information on objects and constructing primary keys.
  * **CUSTOMIZATION:** Enrichment table that is joined to master data tables at `enriched` step of SDDI pipeline.