A modular and extensible data pipeline framework for .NET that pulls data from various sources and pushes it to multiple targets using a plugin-based architecture.
- Plugin-based architecture — Easily extend functionality by creating custom data source and target providers
- Multiple data flows — Configure multiple independent data pipelines in a single instance
- Regex filtering — Include or exclude content using regular expression patterns
- State persistence — Automatically tracks processed items to avoid duplicates
| Provider | Description |
|---|---|
DvachThreadDataSourceProvider |
Fetches new threads from 2ch.hk boards |
DvachPostDataSourceProvider |
Fetches new posts from 2ch.hk threads |
VkFeedDataSourceProvider |
Fetches posts from VK (VKontakte) feed |
TelegramChannelDataSourceProvider |
Fetches messages from Telegram channels |
| Provider | Description |
|---|---|
FileTargetProvider |
Saves data to a local file |
TelegramTargetProvider |
Sends data to a Telegram chat/channel via bot |
The project uses System.Composition (MEF - Managed Extensibility Framework) to dynamically load plugins at runtime. Providers are automatically discovered by scanning assemblies with names matching Edp.DataSourceProvider.* or Edp.TargetProvider.*.
- .NET 10 SDK or later
-
Clone the repository:
git clone https://github.com/magicxor/ExtensibleDataPipeline.git cd ExtensibleDataPipeline -
Build the solution:
dotnet build
-
Configure
appsettings.json -
Run the CLI:
dotnet run --project Edp.Cli.NetCore