# Foundry

The Foundry connector enables data sharing from one instance of Foundry to another. This workflow requires access to both Foundry instances, and designates one instance as the "source" and the other as the "destination." Throughout the data connection process, users will perform most functions on the destination instance.

This connector is not currently compatible with [views](../../data-integration/views/index.md) as inputs, nor is it compatible with [restricted views](../../security/restricted-views/index.md). Datasets ingested by the destination instance must first be materialized within their source instance.

For example, if a use case requires the transfer of data from `red.palantirfoundry.com` to `blue.palantirfoundry.com`, most of the setup and subsequent interactions will take place in the destination instance `blue.palantirfoundry.com`, which is where the transferred data will ultimately land. The workflows discussed below read data via ingest, rather than write data via export.

## Supported capabilities

| Capability | Status |
| --- |--- |
| Bulk import | 🟢 Generally available |
| Streaming ingests | 🟢 Generally available |
| Incremental ingests | 🟢 Generally available |
| Exploration | Coming soon |
| Virtual tables | 🟡 Beta |
| Compute pushdown | Not available |
| Table exports | Not available |
| Export tasks | Not available |

## Setup

1. Open the [Data Connection](../../data-connection/overview/index.md) application and select **+ New Source** in the upper-right corner of the screen.

2. Select **Foundry** from the available connector types.

3. Choose to use a [direct connection](../../data-connection/set-up-direct-connection/index.md) over the Internet or to connect [through an intermediary agent](../../data-connection/set-up-agent/index.md). Agent-based connections using the Foundry connector require your agent to run Java 21, at a minimum, as they are not compatible with prior versions of Java.

4. Input the `hostname` of your source Foundry instance. In this case, `blue.palantirfoundry.com` will pull data from `red.palantirfoundry.com`, so `red.palantirfoundry.com` is the source instance.

5. Choose a means of authentication.

6. Create an egress policy for the source instance if you are using a direct connection. To ingest data from `red.palantirfoundry.com` to `blue.palantirfoundry.com`, create an egress policy for the URL `https://red.palantirfoundry.com` on **port 443**. Unlike traditional data connections, you must whitelist all IP addresses within the source instance. This is done through Control Panel by selecting the option to **Configure network ingress**

7. Follow the instructions below to configure ingress IP allowlisting for the source instance by adding the destination instance's IP addresses to the source instance's **Network ingress** extension in Control Panel:
   1. Navigate to the **Network egress** Control Panel extension in the destination instance to identify the appropriate IP addresses.
   2. Launch the **Network ingress** Control Panel extension in the source instance.
   3. Review the existing documentation to [configure network ingress](../../administration/configure-ingress/index.md).

If you are creating an [agent-based connection](../../data-connection/set-up-agent/index.md), you must instead provide the appropriate IP addresses based on your agent's host. [Learn more about identifying IPs when configuring network egress](../../administration/configure-egress/index.md#which-ips-do-connections-from-foundry-come-from).

If you are unable to access the **Network ingress** Control Panel extension, contact Palantir Support.

Learn more about [setting up a connector](../../data-connection/set-up-source/index.md) in Foundry.

## Authentication

The Foundry connector supports the following authentication methods:

**Client credentials (production):** For long-lived connections, we only allow client credentials. To create a client credential follow the below steps:

1. Navigate to Developer Console with the following link:

```url
https://<SOURCE_FOUNDRY_INSTANCE>.palantirfoundry.com/workspace/developer-console/
```

2. Select **+ New application** and provide a name.
3. Select **No, I will not use an Ontology SDK** and be sure to **enable** your application after selecting the organization it belongs to.
4. Select **Backend service**.
5. Provide your application with appropriate permissions. You can choose **Application permissions** or **User permissions**, but you should leave application permissions checked by default.
6. You will be shown a client secret that you must copy to your clipboard then paste into the destination Foundry instance.
7. After saving, navigate to **OAuth & restrictions** in the left navigation panel.
8. Copy your client ID.

This process will create a **Service user** for which you can provide or deny access to assets in Foundry. To check if this service user has access to a dataset or a project, you can use the **Check access** feature for the given asset.

**Personal access token (temporary):** For security purposes, we do not allow tokens to be used in production use cases. Ingests will fail if a sync is run while relying on a token with a life span greater than 36 hours.

Authentication credentials are input in the destination instance. In the source instance, you must create a token that will afford the destination instance the ability to read data. To do so, navigate to the following URL:

```url
https://<SOURCE_FOUNDRY_INSTANCE>.palantirfoundry.com/workspace/settings/tokens
```

Then, select **+ Create token** in the upper-right corner. At this step you can name your token and choose its lifespan. Then, copy your token and navigate to the destination Foundry instance.

The provided credentials must have the following necessary privileges:

* Browse and read datasets in the source Foundry instance
* Read from specific projects and datasets being synced

## Networking

The Foundry connector requires network access to the destination Foundry instance on port 443 (HTTPS). The destination instance needs an egress policy that corresponds to the URL of the source instance.

:::callout{theme="warning"}
The data flow is pull-based: the destination enrollment initiates the network connection to the source enrollment and pulls data from it. This means the destination instance requires an egress policy to reach the source instance, and the source instance must allowlist the destination instance's IP addresses for ingress.
:::

To enable direct connections from a Foundry instance to another Foundry instance, the appropriate [egress policies](../../administration/configure-egress/index.md) must be added when setting up the source in the [Data Connection application](../../data-connection/overview/index.md).

Egress policies are not needed for connection using an agent.

## Sync data from Foundry

To set up a Foundry-to-Foundry sync, select **Explore and create syncs** in the upper-right of the source **Overview** screen. Browse the available projects and datasets in the source Foundry instance, then select the datasets you want to sync. When ready, select **Create sync for x datasets**.

### Permissions and markings

The authenticating user (whether using client credentials or a personal access token) can sync any dataset for which they have **view permissions** on the source instance. The user does not need to be the owner of the dataset.

:::callout{theme="danger"}
Because markings do not transfer between Foundry instances, ensure you have a process in place to apply appropriate security controls to synced data on the destination instance, particularly when moving sensitive or classified data.
:::

### Incremental syncs

Incremental, or Append, syncs maintain state about the most recent sync and only ingest new or changed data from the target dataset. There are two ways of establishing these ingests with the Foundry Connector.

### Incremental option 1: Ingest all data, then updates only

The "initial incremental state" can be set to an arbitrarily distant date, like January 1, 1970. On the first run of the ingest, all data will be extracted. Starting from the second run onwards, each ingest will only extract the newest data available.

### Incremental option 2: Ingest all data after a specific date

The "initial incremental state" can be set to a date of your choosing, like January 1, 2024. Similar to the above option, the first run of the ingest will extract all data. Then subsequent ingests will only extract the newest data available. This is a more filtered option for use cases where the author of the ingest knows that they want to exclude data that was written in the external source system prior to a particular date.

## Create streaming syncs

You can ingest a stream from one Foundry enrollment to another. The dataset from the source enrollment must be a stream. A sync can be established with a dataset RID and a branch name. After specifying a schema and running for the first time, a new dataset will be created in the destination enrollment.

### Streaming offset options

The sync can be configured to ingest only newly created rows, or to start by ingesting all existing rows of the stream. The main trade-off to consider is that ingesting all historical rows can be expensive from a time and compute perspective if the streaming dataset is sufficiently large.

## Virtual tables

:::callout{theme="warning"}
Datasets are not supported with virtual tables. Only [managed](../../iceberg/storage/index.md) [Iceberg tables](../../data-integration/iceberg-tables/index.md) on the "source" Foundry instance can be virtualized.
:::

This section provides additional details around using [virtual tables](../../data-integration/virtual-tables/index.md) with a Foundry source. This section is not applicable when syncing to Foundry datasets.

The table below highlights the virtual table capabilities that are supported for Foundry.

| Capability | Status |
| --- | --- |
| Bulk registration | 🔴 Not available |
| Automatic registration | 🔴 Not available |
| Table inputs | 🟢 Generally available: tables in [Code Repositories](../../code-repositories/overview/index.md), [Pipeline Builder](../../pipeline-builder/overview/index.md) |
| Table outputs | 🟢 Generally available: tables in [Code Repositories](../../code-repositories/overview/index.md), [Pipeline Builder](../../pipeline-builder/overview/index.md) |
| Incremental pipelines | 🟢 Generally available |
| Compute pushdown | 🔴 Not available |

Review the [virtual tables documentation](../../data-integration/virtual-tables/index.md#supported-foundry-workflows) for details on the supported workflows where Foundry tables can be used as inputs or outputs.

Ensure that the "destination" Foundry instance has network access to the "source" Foundry instance as well as the location of the bucket backing the Iceberg table. Verify that this bucket allows ingress from the "destination" Foundry instance.

## Call the Foundry API from code

In addition to the Foundry-to-Foundry sync workflow above, a Python transform can call the Foundry API directly using the [OAuth2 client credentials grant](../../platform-security-third-party/writing-oauth2-clients/index.md#client-credentials-grant). Use this pattern when you need to invoke Foundry endpoints that are not exposed as syncs — for example, to enumerate project contents, trigger builds, or read from the Ontology API from within a transform.

The overall setup (REST API source configuration, storing `client_id`/`client_secret` as additional secrets, and the generic token-request and pagination scaffolding) is the same as any other OAuth2 client credentials flow. Review the [OAuth Client Credentials grant example](../rest-apis/index.md#oauth-client-credentials-grant) on the REST API connector page for the generic pattern.

The sections below cover the Foundry-specific details that differ from a third-party API.

### Foundry token endpoint

Foundry's OAuth2 token endpoint is:

```
POST /multipass/api/oauth2/token
```

The endpoint is hosted on the Foundry instance you are calling. If the transform runs on the same instance it is calling, the `hostname` on the REST API source is that same instance; if it is a different instance, the source's `hostname` is that of the target instance and you must configure [egress policies](../../administration/configure-egress/index.md) and [ingress allowlisting](../../administration/configure-ingress/index.md) as described in [Networking](#networking).

Learn more about the [token endpoint parameters](../../platform-security-third-party/writing-oauth2-clients/index.md#token-endpoint).

### Foundry scopes

Every Foundry API endpoint documents the scope it requires — see the per-endpoint reference in the [API documentation](../../api/general/overview/introduction/index.md). Some common examples:

| Scope                  | Grants access to                        |
|------------------------|-----------------------------------------|
| `api:datasets-read`    | Read datasets                           |
| `api:datasets-write`   | Write datasets                          |
| `api:ontologies-read`  | Read Ontology objects and link types    |

Request only the scopes your transform needs. Multiple scopes are separated by spaces, for example `api:datasets-read api:datasets-write`.

### Create a client

The `client_id` and `client_secret` used in the token request come from a [third-party application](../../platform-security-third-party/register-3pa/index.md) registered on the target Foundry instance. Follow the steps in [Authentication](#authentication) above to create a backend service application and obtain the `client_id` and `client_secret`, then store them as [additional secrets](../../data-connection/external-transforms/index.md#access-source-attributes-and-credentials) on your REST API source.

The service user created for the client must be granted permissions on every project, dataset, or Ontology resource the transform needs to access.

### Example: List project children

The following transform requests an access token against `/multipass/api/oauth2/token`, then uses the token to list the children of a project via the Foundry `/api/v2/filesystem/resources/{rid}/children` endpoint. For the generic token-request and pagination scaffolding this example reuses, see the [REST API OAuth Client Credentials grant example](../rest-apis/index.md#oauth-client-credentials-grant).

```python
import logging

import pandas as pd
from transforms.api import Output, transform_pandas
from transforms.external.systems import external_systems, Source, ResolvedSource

logger = logging.getLogger(__name__)


@external_systems(
    foundry_api_source=Source("<source_rid>")
)
@transform_pandas(
    Output("<output_dataset_rid>"),
)
def compute(foundry_api_source: ResolvedSource) -> pd.DataFrame:
    base_url = foundry_api_source.get_https_connection().url
    client = foundry_api_source.get_https_connection().get_client()

    client_id = foundry_api_source.get_secret("additionalSecretClientId")
    client_secret = foundry_api_source.get_secret("additionalSecretClientSecret")

    token_response = client.post(
        base_url + "/multipass/api/oauth2/token",
        data={
            "grant_type": "client_credentials",
            "client_id": client_id,
            "client_secret": client_secret,
            "scope": "api:datasets-read",
        },
        headers={"Content-Type": "application/x-www-form-urlencoded"},
    )
    token_response.raise_for_status()
    access_token = token_response.json()["access_token"]

    auth_headers = {"Authorization": f"Bearer {access_token}"}

    project_rid = "ri.compass.main.folder.xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    resources = []
    page_token = None
    while True:
        params = {"pageSize": 100}
        if page_token:
            params["pageToken"] = page_token

        response = client.get(
            base_url + f"/api/v2/filesystem/resources/{project_rid}/children",
            headers=auth_headers,
            params=params,
        )
        response.raise_for_status()
        body = response.json()

        for resource in body.get("data", []):
            resources.append({
                "rid": resource.get("rid"),
                "name": resource.get("displayName"),
                "type": resource.get("type"),
            })

        page_token = body.get("nextPageToken")
        if not page_token:
            break

        logger.info(f"Fetched {len(resources)} resources so far, continuing to next page.")

    return pd.DataFrame(resources)
```

## Troubleshooting

### Preview or sync fails with authentication errors

If you suspect a network connectivity issue, verify TLS connectivity to the source instance before investigating scope issues.

If preview or sync operations fail with authentication errors, check your authentication configuration:

**Personal access token issues:** Static tokens are intended only for testing. Any token with a lifespan greater than 36 hours is rejected at runtime. For production connections, use OAuth.

**Client credentials issues:** If preview fails with an `INVALID_ARGUMENT` error when using client credentials, the issue is likely related to the scopes configured on the application. Restrictive scopes may block access to the target dataset. To resolve this:

1. Navigate to the third-party application configuration in Developer Console on the source instance.
2. Review the scopes assigned to the client credentials.
3. Remove or relax restrictive scopes that may be preventing access to the target dataset.
4. Ensure the service user has appropriate permissions on the datasets being synced.

### Related documentation

* [OAuth Client Credentials grant example](../rest-apis/index.md#oauth-client-credentials-grant) — generic token-request and pagination pattern
* [Writing OAuth2 clients for Foundry](../../platform-security-third-party/writing-oauth2-clients/index.md) — OAuth2 protocol details and endpoint reference
* [External transforms](../../data-connection/external-transforms/index.md) — setup guide and full feature reference
* [Application restrictions](../../developer-console/application-restrictions/index.md) — configuring restrictions for third-party applications