# Transforms and pipelines

A `Transform` is a description of how to compute a dataset. It describes the following:

* The input and output datasets,
* The code used to transform the input datasets into the output dataset (referred to as the compute function), and
* Any additional configuration (such as the custom Transforms profiles to use at runtime).

All transformations currently default to transaction type `SNAPSHOT`.

The input and output datasets, as well as the transformation code, are specified in a `Transform` object and then registered to a `Pipeline`. How you can define a `Transform` depends on two factors:

* Whether you are defining [a high-level or low-level Transform](#transform-type), and
* Whether you are using automatic or manual registration to [add your Transform to your project’s Pipeline](#registration-type).

### Transform type

:::callout{theme="success" title="DataFrame means Spark Dataset in the Java API"}
Data transformations can be expressed in terms of `DataFrame` objects as well as files. These `DataFrame` objects just refer to regular Spark DataFrames. In the Spark Scala/Java API, a `DataFrame` is represented by a `Dataset`. Thus, as a user, you directly interact with `Dataset` objects in your data transformation code.\
For more information about working with Spark, you can refer to the [Java API for Spark documentation ↗](https://spark.apache.org/docs/latest/api/java/) that is available online.
:::

For transformations that rely on `DataFrame` objects, you can:

* Define a high-level Transform, which supports input and output of type `Dataset<Row>`, or
* Define a low-level Transform and explicitly call a method to access the `Dataset<Row>` containing your input dataset.

For transformations that rely on files, you must define a low-level Transform and then access files within your datasets.

Here is a summary of the key differences between the two types of Transforms:

|Description	|High-Level Transform	|Low-Level Transform	|
|---	|---	|---	|
|Allows for data transformations that depend on `DataFrame` objects	|✓ \*	|✓	|
|Allows for data transformations that depend on access to files	|	|✓	|
|Supports multiple input datasets	|✓	|✓	|
|Supports multiple output datasets	|	|✓	|
|Compute function must return `DataFrame` value	|✓	|	|
|Compute function writes to output, instead of returning a value	|	|✓	|

\* We recommend using high-level Transforms for data transformations that depend on `DataFrame` objects.

For both `Transform` types, you need to create a class that contains your compute function. Within this class, your compute function must be a public, non-static method that is annotated with `@Compute`. Without this annotation, your data transformation code will not get correctly registered.

## Registration Type

Each Transforms Java subproject within a repository exposes a single `Pipeline` object. This `Pipeline` object is used to:

1. Register datasets in Foundry with instructions for how to build them, and
2. Locate and execute the `Transform` object responsible for building a given dataset during a Foundry build.

### Entry Point

The runtime responsible for executing a Java transformation needs to be able to find the project’s `Pipeline`. Transforms Java uses the [standard Java facility for service loading ↗](https://docs.oracle.com/javase/7/docs/api/java/util/ServiceLoader.html).

In order to define a `Pipeline` object that is associated with your project, you must implement a `PipelineDefiner` object. In this `PipelineDefiner` object, you can add Transforms to your project’s Pipeline. Specifically, it is required that each Java subproject implements a single `PipelineDefiner` object:

```java
package myproject;

import com.palantir.transforms.lang.java.api.Pipeline;
import com.palantir.transforms.lang.java.api.PipelineDefiner;

public final class MyPipelineDefiner implements PipelineDefiner {

  @Override
  public void define(Pipeline pipeline) {
    // Code here to add Transforms to your project's Pipeline using
    // automatic or manual registration.
  }
}
```

Once you create a Java package and implement a `PipelineDefiner` object, you must update `resources/META-INF/services/com.palantir.transforms.lang.java.api.PipelineDefiner` to point to your `PipelineDefiner` implementation:

```java
// Replace this with the class name for your "PipelineDefiner" implementation.
// Since each Java subproject implements a single "PipelineDefiner", this file
// can only contain a single entry.
myproject.MyPipelineDefiner
```

`MyPipelineDefiner` refers to the class name for your `PipelineDefiner` implementation.

### Adding transforms to a pipeline

Once a `Transform` associated with your project’s Pipeline declares a dataset as an output, you can build this dataset in Foundry. The two recommended ways to add `Transform` objects to a `Pipeline` are [manual registration](#manual-registration) and [automatic registration](#automatic-registration).

:::callout{theme="success" title="Automatic registration is recommended for most workflows"}
If you have a more advanced workflow and/or want to explicitly add each `Transform` object to your project’s Pipeline, you can use manual registration. For instance, it is useful to use manual registration if you want to meta-programmatically apply the same data transformation logic to multiple input and output dataset combinations.

Otherwise, it is highly recommended to use automatic registration to ensure that your registration code is concise and contained. With automatic registration, the `Pipeline.autoBindFromPackage()` discovers any `Transform` definitions in a package (provided that these objects have the required `@Input` and `@Output` annotations).
:::

#### Automatic registration

As the complexity of a project grows, manually adding `Transform` objects to a `Pipeline` can become unwieldy. Thus, the `Pipeline` object provides the `autoBindFromPackage()` method to discover all `Transform` objects within a Java package. To use automatic registration, you must do the following:

* Define a class corresponding to your `Transform`. With automatic registration, you define a class that contains information about your input and output datasets as well as your compute function.
* Add the sufficient `@Input` and `@Output` annotations.
* Call the `Pipeline.autoBindFromPackage()` method to register any `Transform` definitions in your provided Java package. The autoBindFromPackage() method will only register Transform definitions that have the required annotations. Any Transforms that do not have the required annotations will not be added to your project’s `Pipeline`, even if these Transforms are in the Java package you provide to the `autoBindFromPackage()` method.

#### Manual Registration

`Transform` objects can manually be added to a `Pipeline` using the `Pipeline.register()` method. Each call to this method can register one `Transform`. In order to use manual registration with Transforms, you must do the following:

* Define a class containing the compute function for your `Transform` object. Unlike automatic registration, with manual registration, you provide information about your input and output datasets within your PipelineDefiner implementation
* Use the `HighLevelTransform.builder()` or the `LowLevelTransform.builder()` to specify which compute function to use as well as provide your input and output datasets.
* Call the `Pipeline.register()` method to explicitly add your `Transform` definitions to your project’s Pipeline.

:::callout{theme="warning" title="StopPropagating and StopRequiring require automatic registration"}
Annotations such as `@StopPropagating` and `@StopRequiring` are only supported for automatically registered Java transforms.
:::

## Transform Context

There may be cases when a data transformation depends on things other
than its input datasets. For instance, a transformation may be required
to access the current Spark session or access transforms parameters in
the jobSpec. In such cases, you can inject a `TransformContext` object
into the transformation. To do this, your compute function must accept a
parameter of type `TransformContext`. `TransformContext` contains the
Transforms authHeader, Spark session, transform parameters and a
`ServiceDiscovery` object. The `ServiceDiscovery` class exposes service URIs
of discovered Foundry services.

```java
package myproject.datasets;

import com.palantir.transforms.lang.java.api.Compute;
import com.palantir.transforms.lang.java.api.Input;
import com.palantir.transforms.lang.java.api.Output;
import com.palantir.transforms.lang.java.api.TransformContext;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;

/**
* This is an example high-level Transform that accesses the TransformContext
*/
@Compute
public Dataset<Row> myComputeFunction(Dataset<Row> myInput, TransformContext context) {
    int limit = (int) context.parameters().get("limit");
    return myInput.limit(limit);
}
```

```java
package myproject.datasets;

import com.palantir.transforms.lang.java.api.Compute;
import com.palantir.transforms.lang.java.api.Input;
import com.palantir.transforms.lang.java.api.Output;
import com.palantir.transforms.lang.java.api.TransformContext;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;

/**
* This is an example low-level Transform that accesses the TransformContext
*/
@Compute
public void compute(FoundryInput input, FoundryOutput output, TransformContext context) {
    int limit = (int) context.parameters().get("limit");
    output.getDataFrameWriter(input.asDataFrame().read().limit(limit)).write();
}
```