NFS kernel cutover: NLM registration fix + worker scale-up campaign changes - #113
Closed
anchapin wants to merge 13 commits into
Closed
NFS kernel cutover: NLM registration fix + worker scale-up campaign changes#113anchapin wants to merge 13 commits into
anchapin wants to merge 13 commits into
Conversation
- Increase minReplicas from 2 to 10 for faster initial capacity - Increase maxReplicas from 20 to 1000 for larger scale-up potential - Lower targetCPUUtilizationPercentage from 50% to 40% for earlier scaling - Reduce stabilizationWindowSeconds from 3600 to 600 for faster reaction - Reduce scaleUpStabilizationWindowSeconds from 300 to 120 for faster scale-up - Increase scalePolicyValue from 25 to 100 pods/min for larger bursts - Add scaleUpPercentPolicyValue: 200 for exponential scale-up on large spikes - Set scaleDownPolicyValue to 10 to throttle scale-down independently
…r-03 This config had drifted from prod: wrong region (us-east-2 vs us-west-2), cluster name, EKS version, single-AZ worker nodegroup vs the three actual per-AZ spot nodegroups, and outdated ASG max sizes. Reconciled during remediation of a worker scale-up incident (pods stalled at 2378/10000) caused by ASG MaxSize=50/AZ and VPC subnet IP exhaustion. Updates: - region/cluster name/version to match openstudio-server-03 - split single worker-node-group-spot into the three real per-AZ groups (2a/2c/2d) with maxSize raised on 2a/2c (60/120) where spot capacity and subnet IPs are available; 2d left at 50 since its subnet has 0 free IPs and spot requests there fail with UnfulfillableCapacity - vpc-cni addon now enables ENABLE_PREFIX_DELEGATION to reduce per-node EC2 IP-assignment API calls (this was applied live via 'kubectl set env ds aws-node' during the incident; now reflected here so a cluster rebuild matches)
priority_low.yaml checked for an existing "high-priority" PriorityClass before creating "low-priority", instead of checking for "low-priority" itself. On any upgrade where high-priority already existed, Helm would skip creating low-priority entirely. This caused worker pods (which require priorityClassName: low-priority) to fail creation with "no PriorityClass with name low-priority was found" once the class was missing from the cluster, capping the worker deployment well below the HPA's desired replica count.
Snapshot of uncommitted work on faster-scaleup prior to replacing the userspace ganesha nfs-server-provisioner with a kernel nfsd deployment (see HANDOFF-OPTION-C-NFS.md).
…OPTION-C-NFS.md) - Gate vendored nfs-server-provisioner subchart behind nfsServerProvisioner.enabled (default true; false for openstack) - New openstack-only kernel-NFS templates (templates/nfs/nfs-kernel-*): Deployment (256-thread rpc.nfsd, v3-only, privileged, Recreate strategy, TCP-2049 liveness + nfsd-th readiness probes), ClusterIP Service with the legacy 6-port contract, /etc/exports ConfigMap, backing-claim reuse of nfs-pvc-data, static Retain PV for fresh installs - Pin NLM to 32803 via initContainer sysctl so v3 locking works through the VIP - Update pre-delete hook drain exclusions to cover both NFS deployments - docker/nfs-kernel-server/: self-hostable Ubuntu 22.04 image source - docs/nfs-kernel-server-migration.md: ADR + operational runbook rules - Chart version 0.7.0 -> 0.8.0
…ript
Live-cutover findings on openstudio-server-azimuth-openstack:
- Helm4 SSA prunes dropped subchart resources, so the legacy
openstudio-server-nfs-server-provisioner Service was deleted by the first
upgrade; the kernel Service now ADOPTS its exact name + pinned ClusterIP
172.28.43.124 (nfsKernelServer.service.{name,clusterIP}) so the immutable
client PV keeps mounting unchanged -- no claim/PV surgery needed.
- gists/nfs-server stock entrypoint is hardcoded v4-only/8-thread and rewrites
/etc/exports: replaced via command override with our supervisor script
(v3-only rpc.nfsd -N 4 -N 4.1 -N 4.2, 256 threads, mountd 20048).
- Verified live: rpcinfo shows nfs v3@2049/nlockmgr@32803/mountd@20048,
th=256, prior data readable/writable through the VIP, liveness probe
restarts pod after rpc.nfsd 0, 500->1000->3000->8580-worker staged ramp all
mounts healthy (only transient single-shot mount.nfs races on freshly
joined nodes).
…oint override, ramp behavior)
…ork mode Second same-day incident: at ~900 concurrent mounts through the ClusterIP, established client flows blackholed mid-write (clients killable-D in rpc_wait) while nfsd, Cinder volume, and fresh connections all tested healthy. Same failure signature previously blamed on ganesha -- the NAT dataplane is the common factor. - hostNetwork dataplane implemented (nfsKernelServer.hostNetwork / hostNetworkNodeName, hostname+group pinned affinity) but blocked by the web nodes' OpenStack security groups (direct :2049 filtered); documented infra ask. - worker.replicas 9000 -> 300 pending the SG change; batch6647 rerun needs <=1000 anyway.
OpenStack CLI verification showed the shared worker security group already allows all intra-cluster ingress (self-referencing rule) -- no SG change was ever needed. The actual failure of the first hostNetwork attempt: the Ubuntu node image ships rpcbind/statd (nfs-common defaults) which squat on :111, so our mountd registered into an invisible portmap table and remote mounts failed with 'requested NFS version or transport protocol is not supported'. - start script now stops/disables node-level rpcbind/rpcbind.socket/ rpc-statd via nsenter into host systemd (pod gains hostPID in hostNetwork mode) - readiness probe now requires mountd registration (rpcinfo 100005) on top of the nfsd thread counter - verified from a worker node: direct mount to node IP + 221MB/s fsync'd write, zero NAT hops - client PV repointed to the pinned web node IP; all client pods recycled; queued jobs resumed from redis, orphaned-starts re-dispatched
Grounded in verified live state (2026-08-23 ~19:30 UTC): dataplane done, 600-worker ramp in flight, batch6647 draining. Documents the 175-node MachineDeployment ceiling, quota/IP audit needs, NFS+Cinder headroom questions, pipeline singletons, Resque silent-loss process rules, and the staged-ramp execution plan with gates.
…lback 2026-08-23 fleet-wide wedge: init container modprobed lockd before the pod's rpcbind existed, so nlockmgr(100021) was never registered and every client flock() hung forever while data ops looked healthy. start-nfsd.sh now reloads lockd after rpcbind is up, falls back to injecting PMAP entries via pmap-set-nlm.py, and fails loudly otherwise. Liveness probe now checks port 100021. Validated: helm template with provider=openstack + nfsKernelServer.enabled; rendered pmap-set-nlm.py passes py_compile.
… chart to 0.8.6 Move hardcoded CronJob resources to .Values.stuckNodeRemediation.resources (500m/512Mi requests, 1 CPU/2Gi limits) sized for mass-scale-up event-churn OOMs observed during the 9000-worker ramp campaign. Add /.venv/ to .gitignore.
anchapin
changed the title
This was referenced Aug 24, 2026
…aware ramp - Switch NFS export sync->async: at fleet scale ext4 journal on backing Cinder volume serialized all nfsd threads into D-state (fleet freeze at ~3k writers); async coalesces batched commits (41ms fsync @1k observed) - Recalibrate passenger_memory_per_process 250->512 and add PASSENGER_MAX_REQUESTS=1000 recycling: hot processes measured 1.4GB RSS during 2026-08-24 upload storm, invalidating old MAX_POOL math - Raise default web memory 2Gi->10Gi so corrected formula keeps usable pool size for bare installs; resize openstack web node to 12cpu/192Gi - Add scripts/storage-aware-ramp.sh: staged health-gated worker ramp with fsync/nfsd-D-state/queue-drain gates and auto-rollback
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Branch carries the faster-scale-up campaign work plus today's fixes:
fix(nfs): register nlockmgr with pod rpcbind; add pmap-set-nlm.py fallback
lockdbefore the pod's rpcbind existed, sonlockmgr(100021)was never registered and every clientflock()hung forever while data ops looked healthy (sims ran fine until the completion path took its first file lock ~1h in).start-nfsd.shnow reloads lockd after rpcbind is up, falls back to injecting PMAP entries directly viapmap-set-nlm.py(minimal PMAPv2 SET for 100021 v1/v3/v4 on tcp+udp), and fails loudly if nlockmgr still isn't registered.chore: externalize stuck-node-remediation resources into values
.Values.stuckNodeRemediation.resources(500m/512Mi requests, 1 CPU/2Gi limits), sized for cluster-widekubectl get events -o jsonchurn at thousands of replicas (OOMKilled at 256Mi in the 2026-08-20 incident, again at 1Gi during 2026-08-22 ramp prep)./.venv/gitignored.Earlier commits on this branch: userspace Ganesha → kernel nfsd migration, hostNetwork NFS dataplane, HPA scale-up tuning, worker cap + handoff/campaign docs.
Validation
helm templatewith--set provider.name=openstack --set nfsKernelServer.enabled=true: clean renderpmap-set-nlm.pyextracted, passespython3 -m py_compile;PORTcorrectly substituted (32803)Follow-ups
redis_layout_check(upstream fix needed)openstorage-storage-prunerCronJob🤖 Generated with opencode