Skip to content

About

A scalable, container-ready file transfer and orchestration system. Supports UNC, FTP, and SFTP with Redis-backed coordination, OpenTelemetry logging/metrics, and Azure-integrated configuration.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Repository files navigation

FileHorizon

CI

Disclaimer: A significant portion of this project's code is intentionally authored with AI assistance (pair‑programming style) via pull requests that still pass through normal version control, code review, and CI quality gates. All generated contributions are curated, adjusted, and ultimately owned by the repository maintainer. If you spot something that can be improved, please open an issue or PR.

FileHorizon is an open-source, container-ready file transfer and orchestration system. Designed as a modern alternative to heavyweight integration platforms, it provides a lightweight yet reliable way to move files across local/UNC paths, FTP, and SFTP while ensuring observability and control. By leveraging Redis for distributed coordination, FileHorizon can scale out to multiple parallel containers without duplicate processing, making it suitable for both on-premises and hybrid cloud deployments.

Configuration is centralized through Azure App Configuration and Azure Key Vault, enabling secure, dynamic management of connections and destinations. With OpenTelemetry at its core, FileHorizon delivers unified logging, metrics, and tracing out of the box—no separate logging stack required. The system emphasizes safety and consistency, ensuring files are only picked up once they are fully written at the source.

FileHorizon is built for teams that need the reliability of managed file transfer (MFT) but want the flexibility, transparency, and scalability of modern open-source tooling.

Container Image

This repository includes a multi-stage Dockerfile for building a lean runtime image that runs as a non-root user.

WSL / containerd environments: If you're on WSL2 using Rancher Desktop / Lima / containerd, prefer nerdctl over the classic docker CLI. The examples below show both forms where it matters. Mixing docker (Moby) and nerdctl (containerd) against different runtimes in the same workspace can produce confusing state (images not found, networks missing, etc.). Pick one consistently—on WSL + containerd choose nerdctl.

Build

docker build -t filehorizon:dev .

Optional build args:

  • BUILD_CONFIGURATION (default Release)
  • UID / GID to align container user with host filesystem permissions
  • VERSION to stamp the build version (e.g. --build-arg VERSION=0.2.0). .git is not part of the build context, so without it the image reports 0.0.0-alpha.0; see Versioning

Run (example)

docker run --rm \
	-p 8080:8080 \
	-e ASPNETCORE_ENVIRONMENT=Development \
	-v C:/Temp/FileHorizon/InboxA:/data/inboxA:ro \
	-v C:/Temp/FileHorizon/OutboxA:/data/outboxA \
	filehorizon:dev

Health endpoint:

curl http://localhost:8080/health

Non-Root Execution

The image defines user appuser (UID 1001 by default). If you need to write into mounted volumes, ensure host directories grant appropriate permissions. Override UID/GID at build time if integrating with existing volume ownership models.

Configuration (High-Level)

Runtime configuration is provided via appsettings.json / environment variables. To override via environment variables, use the standard ASP.NET Core naming pattern, e.g.:

docker run --rm -p 8080:8080 \
	-e "Pipeline__Role=All" \
	filehorizon:dev

Running Without the HTTP Endpoint

By default FileHorizon serves /health and (when Prometheus is enabled) /metrics over HTTP. When several instances run directly on one server, outside containers, each one would need its own port, and a second instance with no URL configured fails at startup because the default port is already taken. Nothing in the pipeline needs HTTP, so the endpoint can be switched off:

Http__Enabled=false

With Http:Enabled=false (default true) FileHorizon runs as a plain worker process and binds no port at all; ASPNETCORE_URLS and urls are ignored. Polling, transfer, idempotency and telemetry export are unaffected. What you lose:

  • /health is not served, so use the process supervisor (Windows service, systemd) to watch the process instead.
  • /metrics is not served. Export metrics over OTLP instead (see Enabling OTLP Export). If Telemetry:EnablePrometheus is still true (the default), a warning is logged at startup; set it to false to silence it.

Pollers & Feature Flags

File discovery is handled by protocol-specific pollers composed by a multi-protocol coordinator. Each poller can be toggled independently via feature flags so you can roll out new protocols safely.

Feature flags (section Features):

Flag Default Purpose
EnableLocalPoller true Enable local/UNC directory polling sources configured under FileSources (legacy/local).
EnableFtpPoller false Enable FTP remote sources listed in RemoteFileSources:Sources.
EnableSftpPoller false Enable SFTP remote sources listed in RemoteFileSources:Sources.

There is no dry-run switch: any instance whose role processes events (All or Worker) transfers files for real, and deletes sources configured with DeleteAfterTransfer: true. An older Features:EnableFileTransfer key was never read and has been removed; if it is still in your configuration it is ignored.

Environment variable examples:

Features__EnableLocalPoller=true
Features__EnableFtpPoller=false
Features__EnableSftpPoller=true

If all three poller flags are disabled the composite poller runs with an empty set (harmless no-op). This is useful for staging environments while only processing already enqueued events.


Remote Source Configuration

Remote (FTP/SFTP) directories are defined under the RemoteFileSources options section. Each source entry specifies protocol, connection details, remote path, and readiness behavior.

Example appsettings.json excerpt:

{
  "RemoteFileSources": {
    "Ftp": [
      {
        "Name": "UpstreamFtp",
        "Host": "ftp.example.com",
        "Port": 21,
        "RemotePath": "/drop",
        "Username": "ftpuser",
        "PasswordSecretRef": "secrets:ftp-pass",
        "MinStableSeconds": 5,
        "Enabled": true
      }
    ],
    "Sftp": [
      {
        "Name": "PartnerSftp",
        "Host": "sftp.partner.net",
        "Port": 22,
        "RemotePath": "/inbound",
        "Username": "sftpuser",
        "PasswordSecretRef": "secrets:sftp-pass",
        "PrivateKeySecretRef": "secrets:sftp-key",
        "PrivateKeyPassphraseSecretRef": "secrets:sftp-key-pass",
        "HostKeyFingerprint": "SHA256:...base64...",
        "HostKeyFingerprints": ["SHA256:...ed25519...", "SHA256:...rsa..."],
        "StrictHostKey": true,
        "MinStableSeconds": 8,
        "OperationTimeoutSeconds": 60,
        "KeepAliveIntervalSeconds": 30,
        "ConnectTimeoutSeconds": 30,
        "Enabled": true
      }
    ]
  }
}

Environment variable form (first FTP source):

RemoteFileSources__Ftp__0__Name=UpstreamFtp
RemoteFileSources__Ftp__0__Host=ftp.example.com
RemoteFileSources__Ftp__0__Port=21
RemoteFileSources__Ftp__0__RemotePath=/drop
RemoteFileSources__Ftp__0__Username=ftpuser
RemoteFileSources__Ftp__0__PasswordSecretRef=secrets:ftp-pass
RemoteFileSources__Ftp__0__MinStableSeconds=5

Validation rules enforced at startup:

  • Name, Protocol, Host, RemotePath required.
  • Port must be > 0.
  • Appropriate credential secret(s) must be present (username/password or key for SFTP; username/password for FTP).
  • MinStableSeconds must be >= 0.
  • Every configured host key fingerprint must be a recognised format (see below).
  • StrictHostKey: true requires at least one fingerprint, otherwise every connection would be rejected.
  • SFTP OperationTimeoutSeconds and ConnectTimeoutSeconds must be between 1 and 2147483; KeepAliveIntervalSeconds must be between 0 and 2147483. There is no "infinite" setting. These are checked for disabled sources too.

SFTP Timeouts

Each SFTP source bounds how long it waits on the server. The same values apply to polling and to the download of files from that source.

Setting Default Meaning
OperationTimeoutSeconds 60 Maximum wait for the response to one SFTP request. A directory listing is made of many requests, so a large listing can take longer than this in total.
KeepAliveIntervalSeconds 30 Interval between SSH keep-alive messages, which keep idle connections open through NAT and firewalls. 0 disables them. Keep-alives expect no reply, so they do not detect a stalled server; OperationTimeoutSeconds does.
ConnectTimeoutSeconds 30 Maximum wait for the TCP connect and SSH handshake.

When a request times out (for example on a half-open connection, where the server stops answering without closing the socket) the poll for that source fails with an error log, the source goes into backoff, and the next poll opens a new connection. Other sources keep polling.

SFTP Host Key Verification

The server's host key is checked against the fingerprints pinned in configuration. Three formats are accepted:

Format Example
OpenSSH SHA256 SHA256:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU
Bare base64 SHA256 (padded or unpadded) 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU
Legacy MD5 colon hex 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48

Get a server's fingerprint with ssh-keyscan -t ed25519 sftp.partner.net | ssh-keygen -lf -.

Use HostKeyFingerprint for a single key. Use HostKeyFingerprints to pin several — the presented key is trusted when it matches any entry. Both properties may be set at once; the union is accepted, and formats can be mixed within a list. Pinning several keys covers two common cases:

  • Multiple host key algorithms. A server offering both ed25519 and RSA may present either, depending on what gets negotiated. Pinning only one means an algorithm change (server reconfiguration, library upgrade) breaks the connection.
  • Key rotation. Pin the old and new keys together for the duration of the cutover, then remove the old one.

Behaviour when a key does not match, or when none is configured, is governed by StrictHostKey:

StrictHostKey No fingerprints configured Fingerprints configured
false (default) Connect, log a warning with the presented fingerprint Connect only on match; reject otherwise
true Reject (and fail startup validation) Connect only on match; reject otherwise

A delimited list in the singular HostKeyFingerprint property (for example two fingerprints separated by a comma) is not supported and is rejected at startup — use HostKeyFingerprints.

Secrets

Secrets are referenced indirectly (PasswordSecretRef, PrivateKeySecretRef, etc.). The application resolves them through an ISecretResolver abstraction. In development a simple in-memory + environment variable resolver is used; production hosts should plug in Azure Key Vault (or alternative) without changing poller code.

Readiness (Size Stability)

Files are only enqueued once their size remains stable for MinStableSeconds. A per-file observation snapshot retains last size & timestamp; unstable files are skipped (counted in metrics) until stable. This minimizes partial file ingestion.

Backoff & Resilience

Each remote source tracks consecutive failures. An exponential backoff (base 5s doubling up to 5 minutes) delays subsequent attempts after errors (client creation including secret lookup, connection, auth, listing). A single success resets the backoff window.

A rejected enqueue is not a source failure, and queue.enqueue.failures (not poll.source.errors) counts it. What the remote pollers do depends on the error the queue returns:

  • Invalid (Validation.*, for example a listing without a parsable size): that version of the file is skipped for good and the rest of the listing continues.
  • Transient (Redis unreachable or timed out, queue full): the whole FTP or SFTP poller pauses with the same exponential delay, and the file is offered again afterwards. A cycle without such a rejection ends the streak.
  • Anything else: the file is skipped for this cycle and offered again on the next poll; other files and sources are not held up.

Retrying after a transient rejection makes dispatch at-least-once. If Redis stored the event but the reply timed out, the file is enqueued a second time. With Idempotency:Enabled=true the second event is skipped, unless two workers process both at the same moment. With the default (false) the file is transferred twice, and with DeleteAfterTransfer=true the second event fails as file-not-found. Enable idempotency if a duplicate delivery is a problem downstream.


Orchestrated Processing and Routing (New)

The orchestrated processor is enabled by default. It uses a modular orchestrator that selects a protocol-specific reader, routes the file via rules, and writes to a destination sink. This unlocks multi-destination routing and a clean separation of concerns.

Key pieces:

  • Readers: local, sftp (FTP reader pending). The orchestrator selects by FileReference.Protocol.
  • Router: Matches by protocol and path into a single destination (current implementation is 1:1).
  • Sinks: Currently local filesystem sink; remote sinks can be added later.
  • Idempotency: Prevents duplicate transfers across restarts. Files are tracked by identity (SourcePath|SizeBytes|LastModifiedUtc, prefixed fh:idemp:v2:), so a modified file counts as a new version and is transferred again. The marker is written only after a successful transfer (a failed or crashed transfer is retried, never permanently suppressed). Store selection: Redis (if Redis:Enabled) → file-backed JSONL (if Idempotency:DataDirectory is set) → in-memory (non-durable).

Source File Deletion (Event-Driven)

File deletion after a successful transfer is now controlled directly by each FileEvent through the DeleteAfterTransfer flag. This removes hidden, config-only coupling in the orchestrator and makes behavior explicit and testable.

How the flag is populated:

Source Type Config Flag Event Field Behavior
Local (FileSources) DeleteAfterTransfer DeleteAfterTransfer When true, the original source file is deleted after it has been successfully written to its destination(s).
SFTP (RemoteFileSources:Sftp) DeleteAfterTransfer DeleteAfterTransfer Remote file removed via SFTP client after successful sink write.
FTP (RemoteFileSources:Ftp) DeleteAfterTransfer DeleteAfterTransfer Remote file removed via FTP client after successful sink write.

Notes:

  • If a remote deletion fails (e.g., transient network error), the failure is logged at Warning level but processing still succeeds (idempotent by design — file may be re-polled if still present unless already deleted server-side later).
  • Local deletions are attempted only if the file still exists; a missing file (e.g., manual cleanup) does not cause failure.
  • The Redis-backed queue persists the flag (deleteAfterTransfer field) so horizontally scaled workers will honor the intent identically.
  • This design allows future protocols (e.g., Cloud object storage) to simply project their own deletion flag into the event without additional orchestrator changes.

Example environment variables:

FileSources__Sources__0__DeleteAfterTransfer=true

# Remote SFTP delete after transfer
RemoteFileSources__Sftp__0__DeleteAfterTransfer=true

If neither flag is set (false / omitted), no deletion occurs; files remain at the source.

Configuration: Destinations + Routing + Transfer

Example appsettings.json excerpt:

{
  "Destinations": {
    "Local": [
      {
        "Name": "OutboxA",
        "RootPath": "/data/outboxA"
      }
    ],
    "Sftp": [
      {
        "Name": "PartnerX",
        "Host": "sftp.partner.net",
        "Port": 22,
        "RootPath": "/outbound",
        "Username": "sftpuser",
        "PasswordSecretRef": "secrets:sftp-pass",
        "StrictHostKey": false
      }
    ]
  },
  "Routing": {
    "Rules": [
      {
        "Name": "local-txt",
        "Protocol": "local",
        "PathGlob": "**/*.txt",
        "Destinations": ["OutboxA"],
        "RenamePattern": "{fileName}",
        "Overwrite": true
      }
    ]
  },
  "Transfer": {
    "ChunkSizeBytes": 32768
  },
  "Idempotency": {
    "Enabled": true,
    "TtlSeconds": 0,
    "DataDirectory": "/var/lib/filehorizon"
  }
}

Environment variable form (Windows PowerShell examples):

Destinations__Local__0__Name=OutboxA
Destinations__Local__0__RootPath=/data/outboxA

Routing__Rules__0__Name=local-txt
Routing__Rules__0__Protocol=local
Routing__Rules__0__PathGlob=**/*.txt
Routing__Rules__0__Destinations__0=OutboxA
Routing__Rules__0__Overwrite=true

Transfer__ChunkSizeBytes=32768
Idempotency__Enabled=true
Idempotency__TtlSeconds=0
Idempotency__DataDirectory=/var/lib/filehorizon

Notes:

  • On Windows, paths are normalized internally; the router matches against a normalized forward-slash path.
  • Idempotency is a top-level section (not nested under Transfer). TtlSeconds=0 (the default) keeps processed-file markers forever — required for sources that keep files in place (DeleteAfterTransfer=false); a positive value expires markers after that many seconds.
  • Idempotency store selection: Redis when Redis__Enabled=true; otherwise a durable file-backed store (idempotency.jsonl under Idempotency__DataDirectory) when that directory is set; otherwise in-memory (markers do not survive restarts — a warning is logged if idempotency is enabled in this state).
  • The file-backed store grows a small line per transferred file, unbounded under indefinite retention. Deleting the file while the service is stopped resets tracking and causes at most one re-transfer per still-present source file.
  • Current sink support is local filesystem; remote sinks may be added in future.

For a deeper overview see docs/processing-architecture.md.

Adopting a source that already has a backlog

Pointing FileHorizon at a remote source that already holds files means the first two poll cycles enqueue every one of them (a file is never ready on first sighting — see MinStableSeconds). For a source holding thousands of files that is a long, unintended bulk transfer.

To adopt such a source without transferring its history, seed the idempotency store first:

FileHorizon.Host.exe --seed-idempotency --source PartnerSftp [--pattern "*.zip"] [--dry-run]

This lists the configured source, writes an idempotency marker for every matching file, and exits without starting the pipeline. Files already present are then suppressed; only files arriving afterwards are transferred. Fetch the historical files by whatever means suits you.

Because it runs inside the normal host, it uses the same appsettings.json, secret resolution, host key validation and remote client as the poller, so the keys it writes are the keys the pipeline will look up. Markers are written through IIdempotencyStore, so the Redis store works the same way.

  • --source is required and names an entry under RemoteFileSources:Sftp or RemoteFileSources:Ftp. The relevant Features:Enable*Poller flag must be on.
  • --pattern overrides the source's configured Pattern, for seeding a subset.
  • --dry-run reports the count and a few sample keys without writing anything.
  • Exit codes: 0 success, 1 seeding failed, 2 bad arguments or unusable configuration.

Notes:

  • Stop the service first. How this is enforced differs by platform, and only Windows enforces it at all: there, idempotency.jsonl is opened with FileShare.Read and a running instance blocks the seeder outright. On Linux .NET does not enforce FileShare between writers, so both processes open the file happily — and seeding a live instance is ineffective rather than merely untidy, because the store loads markers only at construction. The running instance never sees what the seeder appends and transfers the backlog anyway; the markers only take effect after a restart.
  • A durable store is required — seeding refuses to run against the in-memory store, since the markers would vanish when the process exits. The refusal distinguishes the two ways of ending up there: no durable store configured, or one configured that would not open (a locked file being the usual cause), in which case it prints the underlying error rather than pointing at configuration.
  • Seeding a file that is mid-upload is harmless: the completed file has a different size or mtime, hence a different key, and still transfers.
  • Marker keys have the form fh:idemp:v2:{identityKey}|{size}|{mtime}, e.g. fh:idemp:v2:sftp://host:22/upload/a.zip|1234|2026-07-14T09:13:22.0000000+00:00. Note the +00:00 offset rather than Z; prefer this command over hand-writing markers, since a mismatched key is skipped with only a warning and the file transfers anyway.

Destinations: Adding Azure Service Bus

Destinations now support a Service Bus variant alongside Local and Sftp. A Service Bus destination allows routing rules to direct a file's full content into a queue or topic after it has been read. The orchestrator identifies the destination kind and invokes the IFileContentPublisher abstraction instead of a file sink.

Configuration (Destinations:ServiceBus):

{
  "Destinations": {
    "ServiceBus": [
      {
        "Name": "Events",
        "EntityName": "files-events", // queue or topic name
        "IsTopic": false,
        "ContentType": "text/plain"
      }
    ]
  },
  "Routing": {
    "Rules": [
      {
        "Name": "TxtToEvents",
        "Protocol": "local",
        "PathGlob": "**/*.txt",
        "Destinations": ["Events"]
      }
    ]
  }
}

Environment variable equivalents (first Service Bus destination):

Destinations__ServiceBus__0__Name=Events
Destinations__ServiceBus__0__EntityName=files-events
Destinations__ServiceBus__0__IsTopic=false
Destinations__ServiceBus__0__ContentType=text/plain

Routing rule notes:

  • The Destinations array in each rule lists logical destination names (e.g., Events). The router resolves the kind (ServiceBus) via the Destinations options.
  • Current implementation still processes only the first matching destination; multi-destination fan-out is planned.

Processing flow for Service Bus destination:

  1. Poller emits FileEvent once file is stable.
  2. Router matches rule and yields a DestinationPlan with Kind=ServiceBus.
  3. Orchestrator reads the file content stream, converts to UTF-8 bytes, builds a FilePublishRequest.
  4. Publisher sends one message containing the entire file content.
  5. Optional source deletion executes if DeleteAfterTransfer is true.

Size considerations:

  • Ensure file sizes do not exceed Azure Service Bus message limits (Standard: ~256 KB, Premium: ~1 MB). Oversized files will require future chunking logic (not yet implemented).
  • Binary files are currently treated as UTF-8 text if routed to Service Bus; define ContentType appropriately or avoid routing binary blobs until chunking/streaming support is added.

Telemetry:

  • Publish operation creates an Activity (servicebus.publish) with tags: messaging.system=azure.servicebus, messaging.destination=<EntityName>.
  • Failures propagate as Result.Failure with categorized messaging errors.

Future enhancements under consideration:

  • Retry/backoff for transient publish failures.
  • Multi-destination fan-out (e.g., local + Service Bus).
  • Session / scheduled messages.
  • Line/record splitting with transformation stage before publish.

Destinations: Adding Azure Blob Storage

Destinations also support an Azure Blob Storage variant (Destinations:AzureBlob). A blob destination lets routing rules write files to a storage account container with a streaming upload (no full buffering), making it suitable for large files.

Configuration (Destinations:AzureBlob):

{
  "Destinations": {
    "AzureBlob": [
      {
        "Name": "AnalyticsBlob",
        "ContainerName": "incoming",
        "RootPathPrefix": "ingest/2025",
        "AccessTier": "Hot",
        "ContentTypeStrategy": "InferFromExtension",
        "OverwritePolicy": "FailIfExists",
        "BlobTechnical": {
          "AccountName": "myaccount"
        }
      }
    ]
  },
  "Routing": {
    "Rules": [
      {
        "Name": "CsvToBlob",
        "Protocol": "local",
        "PathGlob": "**/*.csv",
        "Destinations": ["AnalyticsBlob"]
      }
    ]
  }
}

Options:

  • ContainerName (required): target blob container (must already exist).
  • RootPathPrefix (optional): virtual folder prefix prepended to every blob path.
  • AccessTier (optional): Hot | Cool | Cold | Archive; applied per upload.
  • ContentTypeStrategy: InferFromExtension (default; falls back to application/octet-stream), Provided (requires ContentType), or None.
  • OverwritePolicy: FailIfExists (upload is rejected with Storage.AlreadyExists when the blob exists) or Overwrite. When omitted, the routing rule's Overwrite flag applies.
  • ClaimCheck (optional): publish a pointer to the uploaded blob on Service Bus after every successful write. See Claim-Check.
  • BlobTechnical: authentication and retry settings. Provide ConnectionString for shared-key/Azurite scenarios, or AccountName (optionally with ManagedIdentityClientId) for managed identity via DefaultAzureCredential. ServiceUri overrides AccountName for sovereign clouds or a custom Azurite endpoint. Transient failures (throttling, timeouts, 5xx) are retried by the Azure SDK per MaxRetries / RetryBaseDelayMs / RetryMaxDelayMs.

Local development against Azurite:

Destinations__AzureBlob__0__Name=AnalyticsBlob
Destinations__AzureBlob__0__ContainerName=incoming
Destinations__AzureBlob__0__BlobTechnical__ConnectionString=UseDevelopmentStorage=true

Telemetry:

  • Uploads run inside a sink.write activity tagged with sink.name=AzureBlob, blob.container, blob.path, blob.tier and blob.account.
  • Metrics on the shared FileHorizon meter: blob.upload.bytes, blob.upload.duration.ms, blob.upload.successes (tagged with tier), and blob.upload.failures (tagged with reason).
  • Failures are translated to domain errors (Storage.AlreadyExists, Storage.ContainerMissing, Storage.Authorization, Storage.UploadTransient, Storage.UploadError), never raw exceptions.

Destinations: Claim-Check (blob + Service Bus pointer)

A blob destination can notify Service Bus once the upload succeeds, sending a pointer to the blob instead of the file bytes. This is the claim-check pattern: the file goes to storage, the message carries only its address, and a consumer that recognises the envelope fetches the blob itself. It keeps arbitrarily large files off a Service Bus namespace whose message ceiling is 256 KB total on Standard.

Add a ClaimCheck block naming an existing Destinations:ServiceBus entry:

{
  "Destinations": {
    "AzureBlob": [
      {
        "Name": "DespatchAdviceInbox",
        "ContainerName": "inbox",
        "RootPathPrefix": "despatchadvice",
        "OverwritePolicy": "Overwrite",
        "ClaimCheck": {
          "ServiceBusDestination": "DespatchAdviceQueue"
        },
        "BlobTechnical": {
          "ConnectionString": "..."
        }
      }
    ],
    "ServiceBus": [
      {
        "Name": "DespatchAdviceQueue",
        "EntityName": "despatchadvice-sbq",
        "ServiceBusTechnical": {
          "ConnectionString": "..."
        }
      }
    ]
  },
  "Routing": {
    "Rules": [
      {
        "Name": "DespatchAdviceToInbox",
        "Protocol": "sftp",
        "PathGlob": "**/*.edi",
        "Destinations": ["DespatchAdviceInbox"]
      }
    ]
  }
}

Routing rules still name the blob destination. The Service Bus destination is referenced only by the ClaimCheck block, so it does not need a rule of its own.

The message. The body is the envelope, and only the envelope:

{ "blobUrl": "https://acct.blob.core.windows.net/inbox/despatchadvice/order.edi", "contentType": "application/edifact", "length": 51234 }
  • contentType is the blob's content type, resolved by the blob destination's ContentTypeStrategy. The Service Bus message advertises that same content type rather than application/json, so a consumer that resolves the pointer ends up with exactly what an inline message would have carried. When the strategy is None and no type is resolved, the property is omitted rather than serialized as null.
  • The application property claimCheck is set to "true". That marker, not the body's shape, is how a consumer tells a pointer from a payload; claimCheck is reserved and cannot be set through ApplicationProperties.
  • fh.fileId and fh.protocol are set as on any published message, and any ApplicationProperties configured on the Service Bus destination are applied. Keep setting the properties your subscriptions filter on — a claim-checked message that drops them matches no CorrelationFilter rule and disappears without an error.
  • A consumer that does not recognise the envelope must fail loudly rather than forward a pointer as if it were a payload.

No size threshold. If ClaimCheck is configured, every file routed to that destination is claim-checked, whatever its size. A producer and a consumer never have to agree on where a cutoff sits.

Ordering and failure. The blob write happens first and the publish only runs if it succeeded, so a pointer never names bytes that are not in the container. A failed publish fails the whole transfer: the file is not marked as processed and will be retried. The blob from the failed attempt stays behind, so a claim-check destination requires OverwritePolicy: Overwrite — under FailIfExists the retry would fail permanently on Storage.AlreadyExists and the file would never get through. The Service Bus target is also resolved before the upload, so a config reload that drops it fails the transfer with nothing written rather than orphaning a blob.

Validation rejects a ClaimCheck that names a Service Bus destination which does not exist, one with EnableGzipCompression enabled (gzipping a ~100-byte pointer only forces consumers to decompress before they can read a URL), or a blob destination left at OverwritePolicy: FailIfExists (or its default).

Telemetry: the publish runs inside a claimcheck.publish activity tagged with messaging.destination, blob.destination, blob.uri and messaging.content_type, and increments the same servicebus.publish.* metrics as an inline publish.


Service Bus Egress (File ➜ Azure Service Bus Queue/Topic)

The Azure Service Bus publisher lets FileHorizon emit the full contents of a processed file as a single message to a queue or topic. (Per‑line record splitting was intentionally deferred to keep v1 scope minimal.)

Configuration (Options)

Add the ServiceBusPublisher section to appsettings.json (or provide via environment variables):

{
  "ServiceBusPublisher": {
    "ConnectionString": "Endpoint=sb://<namespace>.servicebus.windows.net/;SharedAccessKeyName=FileHorizonPublish;SharedAccessKey=***",
    "MaxConcurrentPublishes": 4,
    "EnableTracing": true
  }
}

Environment variable equivalents:

ServiceBusPublisher__ConnectionString=Endpoint=sb://<namespace>.servicebus.windows.net/;SharedAccessKeyName=FileHorizonPublish;SharedAccessKey=***
ServiceBusPublisher__MaxConcurrentPublishes=4
ServiceBusPublisher__EnableTracing=true

Publishing a File Programmatically

Below is a simplified example showing how you can publish a file (once discovered and considered ready) using the injected IFileContentPublisher.

using FileHorizon.Application.Abstractions;
using FileHorizon.Application.Models;
using FileHorizon.Application.Common;

public sealed class SimplePublishExample
{
  private readonly IFileContentPublisher _publisher;
  private readonly IFileContentReader _reader;

  public SimplePublishExample(IFileContentPublisher publisher, IFileContentReader reader)
  {
    _publisher = publisher;
    _reader = reader;
  }

  public async Task<Result> PublishFileAsync(string fullPath, string destinationName, CancellationToken ct)
  {
    // Read bytes via the existing content reader abstraction.
    var bytes = await _reader.ReadAsync(fullPath, ct);

    var request = new FilePublishRequest(
      SourcePath: Path.GetDirectoryName(fullPath)!,
      FileName: Path.GetFileName(fullPath),
      Content: bytes,
      ContentType: "application/octet-stream", // or text/plain, etc.
      DestinationName: destinationName,          // Queue or topic name
      IsTopic: false,                            // Set true if sending to a Topic
      ApplicationProperties: new Dictionary<string,string>
      {
        ["source"] = "inboxA",
        ["env"] = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "dev"
      }
    );

    return await _publisher.PublishAsync(request, ct);
  }
}

Notes:

  • The current implementation sends one message per file (whole content). No line/record splitting.
  • IsTopic is reserved for topic semantics; publisher treats queue vs topic uniformly at this stage.
  • ApplicationProperties are promoted to Service Bus application properties for downstream consumers.
  • Ensure large files fit within Service Bus message size limits (256 KB standard / 1 MB premium) — otherwise introduce a pre‑processing/chunking layer (future enhancement).

Using Environment Variables (Compose / Container)

When running in containers you can supply:

ServiceBusPublisher__ConnectionString=Endpoint=sb://demo.servicebus.windows.net/;SharedAccessKeyName=FileHorizonPublish;SharedAccessKey=***

If publishing is conditional, you can wrap calls in a feature flag (future: may add Features__EnableServiceBusEgress). For now, simply avoid invoking the publisher when not desired.

Future Enhancements

Planned extensions (tracked in Issue #14):

  • Optional record/line splitting via a transformation stage.
  • Retry with exponential backoff for transient publish failures.
  • Topic filters & advanced metadata (session id, scheduled enqueue time).
  • Config-driven routing rule: source path ➜ queue/topic name mapping.

Identity & De‑Duplication

All files (local and remote) are assigned a normalized identity key:

<protocol>://<host>:<port>/<normalized/path>

Local paths use a normalized form (e.g. local://_/:0/drive/path/file.txt). This shared scheme powers duplicate suppression and telemetry tagging.


Additional Telemetry (Polling)

Poller metrics (Meter FileHorizon):

Metric Type Description Key Tags
filehorizon.poller.poll_cycles Counter Number of poll cycles executed per source protocol, source
filehorizon.poller.files.discovered Counter Files discovered (ready) protocol, source
filehorizon.poller.files.skipped.unstable Counter Files skipped due to instability protocol, source
filehorizon.poller.errors Counter Poll errors (enumeration failures) protocol, source, error.type

Traces include an Activity per remote source (poll.source) with tags: protocol, host, source.name, backoff.ms (if applied).


Using docker-compose (nerdctl compatible)

A docker-compose.yml is provided to spin up Redis + the FileHorizon app quickly. The file is compatible with nerdctl compose in containerd environments (Rancher Desktop, Lima, etc.).

A second sample, docker-compose.azurite.yml, runs an SFTP -> Azure Blob pipeline against the Azurite emulator: no Redis (in-memory queue, single process), durable file-backed idempotency, and a one-shot init container that creates the blob container. Start it with:

docker compose -f docker-compose.azurite.yml up -d --build

If you are on WSL + Rancher Desktop (containerd)

Use nerdctl compose (NOT docker compose). Example mapping:

Purpose Docker CLI nerdctl
Build & up (foreground) docker compose up --build nerdctl compose up --build
Detached docker compose up -d --build nerdctl compose up -d --build
Scale docker compose up -d --scale app=2 nerdctl compose up -d --scale app=2
Logs docker compose logs -f app nerdctl compose logs -f app
Stop docker compose down nerdctl compose down

Why: Rancher Desktop (containerd backend) manages images separately from Docker Desktop (Moby). If you run docker build then nerdctl compose up, the image may not exist in containerd and the deployment will fail. Always build with the same tool:

nerdctl build -t filehorizon:dev .
nerdctl compose up -d --build

Quick start (nerdctl):

# Create local data folders (Linux/macOS examples)
mkdir -p _data/inboxA _data/outboxA

# Or on PowerShell (Windows):
New-Item -ItemType Directory -Path _data/inboxA,_data/outboxA | Out-Null

# Build and start (foreground)
nerdctl compose up --build

# Or start detached
nerdctl compose up -d --build

# Check service status
nerdctl compose ps

# Tail logs
nerdctl compose logs -f app

Health check:

curl http://localhost:8080/health

Stop & remove:

nerdctl compose down

Environment Variables in Compose

The compose file sets sensible defaults:

  • Redis__Enabled=true enables Redis Streams queue (falls back to in-memory if false).
  • FileSources__Sources__0__* defines the first file source (InboxA). Add more sources incrementally:
    • FileSources__Sources__1__Name=InboxB
    • FileSources__Sources__1__Path=/data/inboxB
  • Pipeline__Role determines which background services run (see section below). Any role that processes events (All, Worker) performs real transfers.

You can override any value using an .env file placed next to docker-compose.yml:

# .env example
PIPELINE__ROLE=All
REDIS__ENABLED=true
POLLING__INTERVALMILLISECONDS=500

(Compose automatically loads .env; ensure variable names match exactly.)

Common Adjustments

  • Faster polling during development:
    • Polling__IntervalMilliseconds=500
  • Larger batch processing:
    • Polling__BatchReadLimit=25
  • Switch to local fallback queue:
    • Redis__Enabled=false
  • Separate stream per environment:
    • Redis__StreamName=filehorizon:dev:file-events

Scaling Out (Preview)

To test horizontal scaling (Redis-backed queue required):

nerdctl compose up -d --build --scale app=2

Each replica will create a unique consumer name derived from Redis__ConsumerNamePrefix ensuring cooperative consumption via the shared consumer group. For a clean separation:

Example: one poller + multiple workers

# poller (enqueue only, no processing)
Pipeline__Role=Poller

# worker (process only)
Pipeline__Role=Worker

Alternatively (common simpler pattern):

# single poller that also processes
Pipeline__Role=All

# additional workers (no polling - processing only)
Pipeline__Role=Worker

Reminder (WSL + containerd): If you previously built with docker build, rebuild with nerdctl build to ensure the image exists in the containerd image store before scaling.

Generating files

Use these commands (or similar) to generate lots of files quickly in the inboxes.

seq 1 1000 | xargs -I{} -P 8 sh -c 'echo "test" > inboxA/file{}.txt'
seq 1 1000 | xargs -I{} -P 8 sh -c 'echo "test" > inboxB/file{}.txt'

To count files i a folder you can use

find . -maxdepth 1 -type f | wc -l

Future Hardening Ideas

  • Switch to distroless or -alpine base (after validating native dependencies).
  • Add read-only root filesystem (--read-only) with tmpfs mounts for transient storage.
  • Introduce health/liveness/readiness probes in orchestration environments.

Roadmap (Excerpt)

Recently completed (this branch): FTP & SFTP pollers, feature flags, remote readiness/backoff, poller telemetry.

Upcoming / still planned:

  • Message claiming / retry hardening for Redis Streams (queue still ACKs on read)
  • Service Bus ingress / egress bridge
  • Extended telemetry (error categorization, size histograms)
  • Configurable secret resolver (production Key Vault implementation)
  • Optional archive / checksum verification stage

Observability (OpenTelemetry)

FileHorizon ships with unified tracing, metrics, and structured logging via OpenTelemetry. No other logging framework is used.

What Is Collected

  • Traces: file processing spans (file.process, file.orchestrate), reader/sink spans (reader.open, sink.write), queue enqueue/dequeue spans (queue.enqueue, queue.dequeue), lifecycle span (pipeline.lifetime).
  • Metrics (Meter FileHorizon):
    • files.processed (counter; transfers only, skipped files excluded)
    • files.failed (counter)
    • bytes.copied (counter)
    • queue.enqueued (counter)
    • queue.enqueue.failures (counter)
    • queue.dequeued (counter)
    • queue.dequeue.failures (counter)
    • processing.duration.ms (histogram; skipped files excluded)
    • poll.cycle.duration.ms (histogram)

Prometheus Endpoint

If enabled (default), metrics are exposed at GET /metrics using the Prometheus scrape format. Health remains at /health. Both require the HTTP endpoint; with Http:Enabled=false (see Running Without the HTTP Endpoint) neither is served and metrics should go out over OTLP.

Configuration

Telemetry section (appsettings or env variables):

Key Default Description
EnableTracing true Enable Activity/trace pipeline
EnableMetrics true Enable metrics collection
EnableLogging true Route structured logs through OTEL exporter
EnablePrometheus true Expose /metrics endpoint (requires Http:Enabled=true)
EnableOtlpExporter false Enable OTLP exporter for traces/metrics/logs
OtlpEndpoint null OTLP endpoint (e.g. http://otel-collector:4317 for gRPC)
OtlpProtocol Grpc Wire protocol: Grpc (port 4317) or HttpProtobuf (port 4318)
OtlpHeaders null Additional OTLP headers, comma separated (key=value,key2=value2)
TracesSampleRatio null null = sample all; 0..1 = parent-based ratio sampler
ServiceName FileHorizon Override service.name resource attribute
ServiceVersion Build version Override service.version (see Versioning)
DeploymentEnvironment ASPNETCORE_ENVIRONMENT Adds deployment.environment attribute

Environment variable mapping uses double underscores, e.g.:

Telemetry__EnableOtlpExporter=true
Telemetry__OtlpEndpoint=http://otel-collector:4317
Telemetry__DeploymentEnvironment=dev

Versioning

The build version comes from git tags via MinVer (configured in src/Directory.Build.props):

Commit Version reported in service.version
Tagged v0.2.0 0.2.0+<commit sha>
3 commits after v0.2.0 0.2.1-alpha.0.3+<commit sha>
No tag in history 0.0.0-alpha.0.<height>+<commit sha>

A release is cut by tagging main (git tag v0.2.0 && git push upstream v0.2.0). Builds need the full git history to see tags (CI checks out with fetch-depth: 0). Where there is no .git directory, such as the Docker build context, pass the version explicitly: dotnet publish -p:MinVerVersionOverride=0.2.0 or docker build --build-arg VERSION=0.2.0. Telemetry:ServiceVersion still overrides the reported value at runtime.

Enabling OTLP Export

Point to a collector (recommended) rather than vendors directly:

Telemetry__EnableOtlpExporter=true
Telemetry__OtlpEndpoint=http://otel-collector:4317

For the HTTP/Protobuf variant (collector listens on 4318), select the protocol and adjust the endpoint:

Telemetry__EnableOtlpExporter=true
Telemetry__OtlpProtocol=HttpProtobuf
Telemetry__OtlpEndpoint=http://otel-collector:4318

If headers are required (e.g. an authenticating gateway), supply them comma separated (the OTLP spec format — semicolons are not recognized as separators):

Telemetry__OtlpHeaders=api-key=XYZ123,tenant=abc

All three signals — traces, metrics and logs — are exported to the same endpoint. This suits a collector-based topology where the collector fans out to backends (e.g. exporting everything to Elasticsearch). Providers are flushed on graceful shutdown by the OpenTelemetry.Extensions.Hosting integration.

A plaintext (no TLS) http:// endpoint works out of the box for both protocols — on .NET 8 and later the runtime supports HTTP/2 cleartext (h2c) for gRPC without any opt-in switch.

Docker Compose Example (Prometheus + Collector)

Add a collector and Prometheus service (sketch):

	otel-collector:
		image: otel/opentelemetry-collector:latest
		command: ["--config=/etc/otel-collector-config.yaml"]
		volumes:
			- ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro
		ports:
			- "4317:4317" # OTLP gRPC

	prometheus:
		image: prom/prometheus:latest
		volumes:
			- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
		ports:
			- "9090:9090"

The application container only needs relevant environment variables; /metrics will be scraped by Prometheus.

Log Export

Logs go through the OpenTelemetry logging provider. When the OTLP exporter is enabled (EnableOtlpExporter=true with an endpoint) log records are exported to the collector over OTLP using the same endpoint/protocol/headers as traces and metrics. Without OTLP enabled they remain local (console, in Development or when LOG_CONSOLE_DEV=true).

File logging

A rolling file provider (NReco.Logging.File) is available alongside the console and OTLP providers. It is opt-in: nothing is written unless Logging:File:Path is configured.

"Logging": {
  "LogLevel": { "Default": "Information", "FileHorizon": "Debug" },
  "File": {
    "Path": "_data/logs/filehorizon.log",
    "Append": true,
    "FileSizeLimitBytes": 10485760,
    "MaxRollingFiles": 10,
    "MinLevel": "Debug"
  }
}
Key Default Description
Path (unset) Log file path; file logging stays off while this is empty
Append true Append to an existing file instead of truncating on start
FileSizeLimitBytes 0 (off) Roll to a new file once this size is exceeded
MaxRollingFiles 0 (all) Number of rolled files to keep (filehorizon1.log, …2.log, …)
MinLevel (unset) Minimum level for this provider only

Relative paths resolve against the process working directory. In containers, point Path at a mounted volume (e.g. /data/logs/filehorizon.log) so logs outlive the container. Level filters under Logging:LogLevel apply to every provider; Logging:File:LogLevel narrows the file provider alone.

Configuration announcement

At startup, and again when the configuration is reloaded, FileHorizon logs one Information event per enabled source and one per routing rule. Log consumers can then see what an instance polls and how it routes without reading its config:

Source configured {Source} protocol={Protocol} root={Root} pattern={Pattern} recursive={Recursive}
Rule configured {Rule} order={Order} protocol={Protocol} glob={PathGlob} regex={PathRegex} source={SourceName} -> {Destination} kind={Kind} entity={Entity}
  • root uses the same form as file paths in other log events (sftp://host:port/path for remote sources).
  • order is the rule's position. The router uses the first rule that matches.
  • entity is where the destination writes: the Service Bus queue or topic, the blob container (plus prefix), or the root path. The events carry no credentials or secret references.
  • Sources are announced only when Pipeline:Role polls (All, Poller), and rules only when it routes (All, Worker).

Warnings flag a rule that routes to a destination name that isn't configured (the router treats it as Local), a rule that lists more than one destination (only the first is used), a rule that sets SourceName (not enforced yet), and an empty rule list.

Each routed file also logs Router matched rule {Rule} -> {Destination} for file {FileId} path={Path} at Information, so a single event links a source file to its rule.

Tag / Attribute Conventions (Initial)

Span Key Example
file.process file.id f_123
file.process file.protocol local
file.process file.source_path /data/inbox/file1.txt
file.process file.size_bytes 2048
reader.open file.protocol sftp
sink.write sink.name OutboxA
queue.enqueue / dequeue messaging.system redis
queue.enqueue / dequeue messaging.destination filehorizon:file-events
queue.dequeue messaging.batch.message_count 10

These may evolve toward official semantic conventions as they stabilize.

Viewing Metrics Locally

curl http://localhost:8080/metrics | head -n 40

You should see counters like files_processed_total and processing_duration_ms_bucket (Prometheus histogram exposition).

Troubleshooting

Issue Likely Cause Remedy
No metrics at /metrics Prometheus disabled Set Telemetry__EnablePrometheus=true
No traces in collector OTLP exporter disabled Set Telemetry__EnableOtlpExporter=true and endpoint
Service name wrong Override provided Adjust Telemetry__ServiceName
High cardinality risk Dynamic paths tagged Consider trimming or hashing path tags in future

Future Telemetry Enhancements

Planned / candidate improvements:

  • File size distribution histogram
  • Error categorization with semantic convention attributes (e.g. error.type)
  • Redis pending / claim latency metrics
  • Service Bus ingress / egress spans and metrics
  • De-duplication cache hit/miss counters
  • Optional span events for validation / archive stages
  • Configurable sampling (probabilistic/parent-based) via Telemetry options

Contributions welcome—feel free to open issues or draft PRs as the architecture evolves.


Pipeline Roles (New Split Architecture)

The runtime now supports explicit role selection separating file discovery (polling) from event processing.

Roles are configured via Pipeline:Role (or environment variable Pipeline__Role).

Available values:

Role Hosted Services Started Typical Use Case
All Polling + Processing Local dev / simple single-node deploy
Poller Polling only Dedicated ingestion node
Worker Processing only Horizontal scale-out workers

Example environment overrides:

Pipeline__Role=Poller

Pipeline__Role=Worker

Pipeline__Role fully determines polling vs processing. There is no separate switch for transfer side effects: every processing role copies files and (when configured) deletes the source.

About

A scalable, container-ready file transfer and orchestration system. Supports UNC, FTP, and SFTP with Redis-backed coordination, OpenTelemetry logging/metrics, and Azure-integrated configuration.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages