# Streaming resource guide

This page lists the resources you may need to reference when implementing an end-to-end streaming workflow.

Data Connection supports syncing data from a wide variety of streaming platforms into Foundry streaming datasets, which can then be used in [streaming pipelines](../../building-pipelines/streaming-overview/index.md). Streaming syncs enable data to flow into Foundry with low latency and high throughput to support real time decision-making processes.

There are two ways to sync data from streams into Foundry:

* Data Connection supports pulling records from streaming platforms into Foundry. As with batch syncs, data is read from a stream and synced to Foundry using only unidirectional connections using the [agent architecture](../../data-connection/architecture/index.md).
* If desired, Foundry enables pushing records from a stream directly into a Foundry stream via the *stream proxy*.

Foundry can connect to many sources of streaming data. Sources with dedicated connectors include:

* **[Apache Kafka](../../available-connectors/kafka/index.md)**
* **[Amazon Kinesis](../../available-connectors/amazon-kinesis/index.md)**
* **Amazon SQS**
* **[Aveva PI](../../available-connectors/pi/index.md)**
* **[Google Pub/Sub](../../available-connectors/pubsub/index.md)**

For streaming sources without a dedicated connector, you can connect to them using [external transforms](../../data-connection/external-transforms/index.md). This includes sources such as:

* **ActiveMQ**
* **Amazon SNS**
* **IBM MQ**
* **RabbitMQ**
* **MQTT \[Beta]**
* **Solace**

This page lists the resources you may need to reference when implementing an end-to-end streaming workflow.

## 1. Core concepts

We recommend reviewing the following introductory concept page to understand what streams are, how they are stored, and how they are processed.

* [Streams](../streams/index.md)
* [Flink in Foundry streaming](../flink-streaming/index.md)

## 2. Overview

These pages will offer a broader scope of the various points to consider when determining if streaming is right for your use case or when deploying production streams.

* [Streaming pipelines overview](../../building-pipelines/streaming-overview/index.md)
* [Comparison: Streaming vs. batch](../../building-pipelines/stream-vs-batch/index.md)
* [Performance considerations](../../building-pipelines/streaming-performance-considerations/index.md)
* [Streaming compute usage](../../building-pipelines/streaming-compute-usage/index.md)
* [Streaming profiles](../streaming-profiles/index.md)
* [Stream monitoring](../stream-monitoring/index.md)
* [Streaming keys](../../building-pipelines/streaming-keys/index.md)
* [Streaming stateful transforms](../../building-pipelines/streaming-stateful-transforms/index.md)
* [Stream debugging](../../dataset-preview/overview/index.md#stream-only-for-streaming-datasets)

## 3. Connect to data sources

You will need to complete one of the following workflows to connect your external data sources to Foundry for streaming. We recommend reviewing both options to understand possible benefits and limitations for your use case.

* [Set up a streaming sync](../../data-connection/set-up-streaming-sync/index.md)
* [Push data into a stream](../../data-connection/push-based-ingestion/index.md)

## 4. Transform your streaming data

You can use [Pipeline Builder](../../pipeline-builder/overview/index.md) to transform your live data. Outputs of your Pipeline Builder transforms will still be streaming datasets that you can [use in real time throughout Foundry](../../building-pipelines/stream-vs-batch/index.md#front-end-tools).

* [Create a streaming pipeline with Pipeline Builder](../../building-pipelines/create-stream-pipeline-pb/index.md)
* [Integrate your stream with the Ontology](../../object-indexing/funnel-streaming-pipelines/index.md)

## 5. Monitor streaming pipelines \[Beta]

Set up alerting around your pipeline's health.

* [Stream monitoring](../stream-monitoring/index.md)

## 6. Development tools

Here, you can find tools to improve development of streaming pipelines.

* [Reset stream](../reset-stream/index.md)