Skip to content

fix: Tune NiFi startup defaults (election timeout, archive retention) #936

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lfrancke merged 3 commits into from
May 26, 2026
Merged

fix: Tune NiFi startup defaults (election timeout, archive retention) #936

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
27671a7
fix: Tune NiFi startup defaults (election timeout, archive retention)
lfrancke May 20, 2026
c8cb0b8
fix(smoke_v2): adapt snapshot to default changes
maltesander May 21, 2026
3dc11d8
fix(tests): override nifi.cluster.flow.election.max.wait.time to "10 …
maltesander May 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ All notable changes to this project will be documented in this file.
Previously, arbitrary keys were silently accepted but ignored ([#921]).
- Bump `stackable-operator` to 0.110.1 and `kube` to 3.1.0 ([#921]).
- Internal operator refactoring: introduce dereference() and validate() steps in the reconciler ([#935]).
- Default `nifi.cluster.flow.election.max.wait.time` to NiFi's upstream value (`5 mins`) instead of the operator's previous `1 mins`. The operator no longer sets this property explicitly; the previous shorter value was left over from a TODO marked as "for testing" and may have caused flow election to settle on incomplete vote sets in cold-start scenarios ([#936]).
- Set `nifi.content.repository.archive.max.retention.period` to `3 days` (previously empty, which NiFi interprets as `Long.MAX_VALUE` and effectively disables time-based archive purge). Without a time-based ceiling, the content archive can grow to half the content PVC and accumulate millions of files, which makes the synchronous startup directory scan in `FileSystemRepository.initializeRepository` very slow. Users requiring a longer content-replay window can extend via `configOverrides`. The provenance audit trail is independent of this setting and unaffected ([#936]).

### Fixed

Expand All @@ -29,6 +31,7 @@ All notable changes to this project will be documented in this file.
[#924]: https://github.com/stackabletech/nifi-operator/pull/924
[#928]: https://github.com/stackabletech/nifi-operator/pull/928
[#935]: https://github.com/stackabletech/nifi-operator/pull/935
[#936]: https://github.com/stackabletech/nifi-operator/pull/936

## [26.3.0] - 2026-03-16

Expand Down
4 changes: 4 additions & 0 deletions docs/modules/nifi/pages/usage_guide/operations/graceful-shutdown.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,7 @@ nifi 2023-11-08 10:52:15,139 INFO [Thread-0] org.apache.nifi.nar.NarAutoLoader N
nifi 2023-11-08 10:52:15,139 INFO [Thread-0] o.a.n.f.r.CompositeExternalResourceProviderService External Resource Provider Service is stopped
nifi 2023-11-08 10:52:15,139 INFO [Thread-0] org.apache.nifi.NiFi Application Server shutdown completed
----

NOTE: For high-throughput clusters (large numbers of in-flight FlowFiles or high provenance event rates), 5 minutes may not be enough time for NiFi to flush the FlowFile write-ahead log and finalize the provenance index.
If `SIGKILL` is consistently reached before NiFi can shut down cleanly, repositories are left in a partially-flushed state, which makes the next startup significantly slower (FlowFile WAL recovery and provenance reindex have more work to do).
Increase `gracefulShutdownTimeout` if you observe `SIGKILL` being issued, or if subsequent restarts take noticeably longer than expected.
8 changes: 8 additions & 0 deletions docs/modules/nifi/pages/usage_guide/resource-configuration.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,3 +46,11 @@ A minimal HA setup consisting of 2 NiFi instances has the following https://kube
* `18432Mi` persistent storage

You can find the default resource configuration values in the {crd-docs}/nifi.stackable.tech/NifiCluster/v1alpha1/#spec-nodes-config-resources[NifiCluster reference docs {external-link-icon}^].

== Content archive retention

When a FlowFile's content claim is no longer referenced, NiFi moves the file into an `archive/` subdirectory of the content repository rather than deleting it immediately.
Archived content is what powers *Replay* and *View Content* in the NiFi provenance UI.
The operator configures both a time-based and a size-based purge threshold; the provenance audit trail itself is stored in a separate repository and is not affected by content archive retention.

If the operator defaults don't fit your workload, for example you need a longer replay window, or you want to disable content archiving entirely, override the relevant `nifi.content.repository.archive.*` properties via xref:usage_guide/overrides.adoc[configOverrides].
17 changes: 11 additions & 6 deletions rust/operator-binary/src/config/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -344,9 +344,19 @@ pub fn build_nifi_properties(
"nifi.content.repository.directory.default".to_string(),
NifiRepository::Content.mount_path(),
);
// Cap archived content age so the archive directory stays bounded in
// file count. NiFi treats empty as Long.MAX_VALUE, leaving size-based
// purge as the only trigger; that lets the archive grow to whatever
// half the PVC holds, and the startup directory scan in
// FileSystemRepository.initializeRepository scales with file count.
// 3 days covers a Friday-incident-investigated-Monday window for
// content replay; users with longer requirements can extend via
// configOverrides. The percentage-based threshold below acts as a
// safety net if write rate outpaces time-based purge.
// Also see https://github.com/stackabletech/nifi-operator/issues/354
properties.insert(
"nifi.content.repository.archive.max.retention.period".to_string(),
"".to_string(),
"3 days".to_string(),
Comment thread
maltesander marked this conversation as resolved.
);
properties.insert(
"nifi.content.repository.archive.max.usage.percentage".to_string(),
Expand Down Expand Up @@ -599,11 +609,6 @@ pub fn build_nifi_properties(
"nifi.cluster.node.protocol.port".to_string(),
PROTOCOL_PORT.to_string(),
);
// TODO: set to 1 min for testing (default 5)
properties.insert(
"nifi.cluster.flow.election.max.wait.time".to_string(),
"1 mins".to_string(),
);
Comment on lines -603 to -606

@sbernauer sbernauer May 27, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I often lower this as it's a real blocker for development.
WDYT of setting this to a very low value in case only a single replica is configured?

@sbernauer sbernauer Jun 30, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gentle ping @lfrancke :)

@lfrancke lfrancke Jun 30, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For that usecase there's a better property nifi.cluster.flow.election.max.candidates which we can set to 1 if there is only a single replica. It looks like we don't have the replica count in this part of the code though.

@sbernauer sbernauer Jul 1, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is great news! It makes so much sense to just set nifi.cluster.flow.election.max.candidates to the number of NiFi nodes (in the 99% case they are static): #953

properties.insert(
"nifi.cluster.flow.election.max.candidates".to_string(),
"".to_string(),
Expand Down
2 changes: 2 additions & 0 deletions tests/templates/kuttl/cluster-sync-cs-bug/20-install-nifi.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,8 @@ spec:
nifi.properties:
nifi.web.https.sni.required: "false"
nifi.web.https.sni.host.check: "false"
# Quicker startup, and we only have a single node
nifi.cluster.flow.election.max.wait.time: "10 secs"
podOverrides:
spec:
initContainers:
Expand Down
4 changes: 4 additions & 0 deletions tests/templates/kuttl/cluster_operation/20-install-nifi.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,10 @@ spec:
gracefulShutdownTimeout: 1m
logging:
enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}
configOverrides:
"nifi.properties":
# Quicker startup, and we only have a single node
"nifi.cluster.flow.election.max.wait.time": "10 secs"
roleGroups:
default:
replicas: 2
4 changes: 4 additions & 0 deletions tests/templates/kuttl/custom-components-git-sync/30_install-nifi.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,10 @@ spec:
config:
logging:
enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}
configOverrides:
"nifi.properties":
# Quicker startup, and we only have a single node
"nifi.cluster.flow.election.max.wait.time": "10 secs"
roleConfig:
listenerClass: external-unstable
podOverrides:
Expand Down
4 changes: 4 additions & 0 deletions tests/templates/kuttl/external-access/30_nifi.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,10 @@ spec:
config:
logging:
enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}
configOverrides:
"nifi.properties":
# Quicker startup, and we only have a single node
"nifi.cluster.flow.election.max.wait.time": "10 secs"
roleConfig:
listenerClass: test-external-unstable-$NAMESPACE
roleGroups:
Expand Down
4 changes: 4 additions & 0 deletions tests/templates/kuttl/ldap/12-install-nifi.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@ spec:
gracefulShutdownTimeout: 1m
logging:
enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}
configOverrides:
"nifi.properties":
# Quicker startup, and we only have a single node
"nifi.cluster.flow.election.max.wait.time": "10 secs"
roleConfig:
listenerClass: external-unstable
roleGroups:
Expand Down
4 changes: 4 additions & 0 deletions tests/templates/kuttl/logging/04-install-nifi.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,10 @@ spec:
nodes:
config:
gracefulShutdownTimeout: 1m
configOverrides:
"nifi.properties":
# Quicker startup, and we only have a single node
"nifi.cluster.flow.election.max.wait.time": "10 secs"
roleGroups:
automatic-log-config:
replicas: 1
Expand Down
4 changes: 4 additions & 0 deletions tests/templates/kuttl/orphaned_resources/02-install-nifi.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,10 @@ spec:
gracefulShutdownTimeout: 1m
logging:
enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}
configOverrides:
"nifi.properties":
# Quicker startup, and we only have a single node
"nifi.cluster.flow.election.max.wait.time": "10 secs"
roleGroups:
default:
config: {}
Expand Down
4 changes: 4 additions & 0 deletions tests/templates/kuttl/resources/02-install-nifi.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,10 @@ spec:
capacity: 2Gi
stateRepo:
capacity: 2Gi
configOverrides:
"nifi.properties":
# Quicker startup, and we only have a single node
"nifi.cluster.flow.election.max.wait.time": "10 secs"
roleGroups:
resources-from-role:
replicas: 1
Expand Down
2 changes: 2 additions & 0 deletions tests/templates/kuttl/smoke_v1/30-install-nifi.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,8 @@ spec:
"nifi.properties":
"nifi.diagnostics.on.shutdown.enabled": "true"
"nifi.diagnostics.on.shutdown.verbose": "false"
# Quicker startup, and we only have a single node
"nifi.cluster.flow.election.max.wait.time": "10 secs"
config:
logging:
enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}
Expand Down
2 changes: 2 additions & 0 deletions tests/templates/kuttl/smoke_v2/30-install-nifi.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,8 @@ spec:
"nifi.properties":
"nifi.diagnostics.on.shutdown.enabled": "true"
"nifi.diagnostics.on.shutdown.verbose": "false"
# Quicker startup, and we only have a single node
"nifi.cluster.flow.election.max.wait.time": "10 secs"
config:
logging:
enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}
Expand Down
4 changes: 2 additions & 2 deletions tests/templates/kuttl/smoke_v2/35-assert.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ commands:
nifi.administrative.yield.duration=30 sec
nifi.authorizer.configuration.file=/stackable/nifi/conf/authorizers.xml
nifi.cluster.flow.election.max.candidates=
nifi.cluster.flow.election.max.wait.time=1 mins
nifi.cluster.flow.election.max.wait.time=10 secs
nifi.cluster.is.node=true
{% if test_scenario['values']['use-zookeeper-manager'] == 'true' %}
nifi.cluster.leader.election.implementation=CuratorLeaderElectionManager
Expand All @@ -140,7 +140,7 @@ commands:
nifi.content.claim.max.appendable.size=1 MB
nifi.content.repository.always.sync=false
nifi.content.repository.archive.enabled=true
nifi.content.repository.archive.max.retention.period=
nifi.content.repository.archive.max.retention.period=3 days
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.directory.default=/stackable/data/content
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
Expand Down
4 changes: 4 additions & 0 deletions tests/templates/kuttl/upgrade/02-install-nifi.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,10 @@ spec:
gracefulShutdownTimeout: 1m
logging:
enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}
configOverrides:
"nifi.properties":
# Quicker startup, and we only have a single node
"nifi.cluster.flow.election.max.wait.time": "10 secs"
roleGroups:
default:
config: {}
Expand Down
Loading