Vm starting issue with InsufficientServerCapacityException

[TheKunalSen]

My setup was working fine then a sudden ups power failure made every machine off

Then when I turned on evrything again my vms were still at running state then i stopped those vms , rebooted the system vms and also deleted the vr now when I tried to start the vms again I am getting this error of unable to start vm, cannot find vm deployment

Note that all the system vms and vrs are working fine

All state up and all

In in logs I'm getting these

,"cmdInfo":"rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAzXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAA3BzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAANzcQB-AAgAAAAAAAAACnBwcHEAfgAKcHBw","cmdVersion":0,"completeMsid":null,"created":"Mon Jun 22 07:14:32 UTC 2026","id":25934,"initMsid":163763490546348,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"a57c6ef2-84b8-408c-932c-b68187008032"}, job origin: 25933 com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":205,"instanceName":"i-2-205-VM","state":"Starting","type":"User","uuid":"e407da92-1405-4741-b696-9658a409e0d6"}Scope=interface com.cloud.dc.DataCenter; id=3
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1454)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5951)
        at jdk.internal.reflect.GeneratedMethodAccessor306.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:6075)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
2026-06-22 07:14:34,460 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-39:[ctx-3537b865, job-25933/job-25934]) (logid:74b9be61) Complete async job-25934, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyADdjb20uY2xvdWQuZXhjZXB0aW9uLkluc3VmZmljaWVudFNlcnZlckNhcGFjaXR5RXhjZXB0aW9uVk1PcAAAABACAAFaABVhZmZpbml0eUdyb3Vwc0FwcGxpZWR4cgAxY29tLmNsb3VkLmV4Y2VwdGlvbi5JbnN1ZmZpY2llbnRDYXBhY2l0eUV4Y2VwdGlvblZNT3AAAAAVAgACTAACaWR0ABBMamF2YS9sYW5nL0xvbmc7TAAFc2NvcGV0ABFMamF2YS9sYW5nL0NsYXNzO3hyACJjb20uY2xvdWQuZXhjZXB0aW9uLkNsb3VkRXhjZXB0aW9ueeiOG_HkRMcCAAJMAAtjc0Vycm9yQ29kZXQAE0xqYXZhL2xhbmcvSW50ZWdlcjtMAAZpZExpc3R0ABVMamF2YS91dGlsL0FycmF5TGlzdDt4cgATamF2YS5sYW5nLkV4Y2VwdGlvbtD9Hz4aOxzEAgA

cityException: Unable to create a deployment for VM instance {"id":25,"instanceName":"i-2-25-VM","state":"Starting","type":"User","uuid":"d8528608-1868-44c3-b235-4deee0aa3c5c"}Scope=interface com.cloud.dc.DataCenter; id=3
2026-06-22 08:11:17,785 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-1:[ctx-6d9e4b71, job-25991/job-25992]) (logid:2ef0537b) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 25, job origin: 25991
2026-06-22 08:11:17,785 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-1:[ctx-6d9e4b71, job-25991/job-25992]) (logid:2ef0537b) Unable to complete AsyncJob {"accountId":2,"cmd":"com.cloud.vm.VmWorkStart","cmdInfo":"rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAGXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAA3BzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAANzcQB-AAgAAAAAAAAABXBwcHEAfgAKcHBw","cmdVersion":0,"completeMsid":null,"created":"Mon Jun 22 08:10:34 UTC 2026","id":25992,"initMsid":163763490546348,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"47f22eb4-eac8-4def-bf55-1f2700027cbc"}, job origin: 25991 com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":25,"instanceName":"i-2-25-VM","state":"Starting","type":"User","uuid":"d8528608-1868-44c3-b235-4deee0aa3c5c"}Scope=interface com.cloud.dc.DataCenter; id=3
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1454)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5951)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:6075)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2026-06-22 08:11:17,793 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:[ctx-6d9e4b71, job-25991/job-25992]) (logid:2ef0537b) Complete async job-25992, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyADdjb20uY2xvdWQuZXhjZXB0aW9uLkluc3VmZmljaWVudFNlcnZlckNhcGFjaXR5RXhjZXB0aW9uVk1PcAAAABACAAFaABVhZmZpbml0eUdyb3Vwc0FwcGxpZWR4cgAxY29tLmNsb3VkLmV4Y2VwdGlvbi5JbnN1ZmZpY2llbnRDYXBhY2l0eUV4Y2VwdGlvblZNT3AAAAAVAgACTAACaWR0ABBMamF2YS9sYW5nL0xvbmc7TAAFc2NvcGV0ABFMamF2YS9sYW5nL0NsYXNzO3hyACJjb20uY2xvdWQuZXhjZXB0aW9uLkNsb3VkRXhjZXB0aW9ueeiOG_HkRMcCAAJMAAtjc0Vycm9yQ29kZXQAE0xqYXZhL2xhbmcvSW50ZWdlcjtMAAZpZExpc3R0ABVMamF2YS91dGlsL0FycmF5TGlzdDt4cgATamF2YS5sYW5nLkV4Y2VwdGlvbtD9Hz4aOxzEAgAAeHIAE2phdmEubGFuZy5UaHJvd2FibGXVxjUnOXe4ywMABEwABWNhdXNldAAVTGphdmEvbGFuZy9UaHJvd2FibGU7TAANZGV0YWlsTWVzc2FnZXQAEkxqYXZhL2xhbmcvU3RyaW5nO1sACnN0YWNrVHJhY2V0AB5bTGphdmEvbGFuZy9TdGFja1RyYWNlRWxlbWVudDtMABRzdXBwcmVzc2VkRXhjZXB0aW9uc3QAEExqYXZhL3V0aWwvTGlzdDt4cHEAfgANdAChVW5hYmxlIHRvIGNyZWF0ZSBhIGRlcGxveW1lbnQgZm9yIFZNIGluc3RhbmNlIHsiaWQiOjI1LCJpbnN0YW5jZU5hbWUiOiJpLTItMjUtVk0iLCJzdGF0ZSI6IlN0YXJ0aW5nIiwidHlwZSI6IlVzZXIiLCJ1dWlkIjoiZDg1Mjg2MDgtMTg2OC00NGMzLWIyMzUtNGRlZWUwYWEzYzVjIn11cgAeW0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABVzcgAbamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAAhCAAZmb3JtYXRJAApsaW5lTnVtYmVyTAAPY2xhc3NMb2FkZXJOYW1lcQB-AApMAA5kZWNsYXJpbmdDbGFzc3EAfgAKTAAIZmlsZU5hbWVxAH4ACkwACm1ldGhvZE5hbWVxAH4ACkwACm1vZHVsZU5hbWVxAH4ACkwADW1vZHVsZVZlcnNpb25xAH4ACnhwAQAABa50AANhcHB0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAEG9yY2hlc3RyYXRlU3RhcnRwcHNxAH4AEQEAABc_cQB-ABNxAH4AFHEAfgAVcQB-ABZwcHNxAH4AEQL____-cHQALWpkay5pbnRlcm5hbC5yZWZsZWN0Lk5hdGl2ZU1ldGhvZ

i am getting these error in the vr

root@r-752-VM:~# tail -50 /var/log/cloud.log
    handle.close()
OSError: [Errno 75] Value too large for defined data type
Mon Jun 22 08:51:07 AM UTC 2026 : VR config: executing failed: /opt/cloud/bin/update_config.py ip_associations.json.f419bae0-d22a-4f21-8077-ce859ba695db
Mon Jun 22 08:51:07 AM UTC 2026 : VR config: Flushing conntrack table
Mon Jun 22 08:51:07 AM UTC 2026 : VR config: Flushing conntrack table completed
Mon Jun 22 08:51:16 AM UTC 2026 : VR config: configuration format version 1.0
Mon Jun 22 08:51:16 AM UTC 2026 : VR config: creating file: /var/cache/cloud/firewall_rules.json.c3028bfd-8405-479b-9257-156ac425e46e
Mon Jun 22 08:51:16 AM UTC 2026 : VR config: create file success
Mon Jun 22 08:51:16 AM UTC 2026 : VR config: executing: /opt/cloud/bin/update_config.py firewall_rules.json.c3028bfd-8405-479b-9257-156ac425e46e
2026-06-22 08:51:16,398 INFO     update_config.py :: Processing incoming file => firewall_rules.json.c3028bfd-8405-479b-9257-156ac425e46e
2026-06-22 08:51:16,399 INFO     Processing JSON file firewall_rules.json.c3028bfd-8405-479b-9257-156ac425e46e
2026-06-22 08:51:16,399 INFO     Continuing with the processing of file '/var/cache/cloud/firewall_rules.json.c3028bfd-8405-479b-9257-156ac425e46e'
2026-06-22 08:51:16,399 INFO     Command of type firewallrules received
2026-06-22 08:51:16,400 INFO     Executing: ip addr show dev eth0
2026-06-22 08:51:16,403 INFO     Executing: ip addr show dev eth1
2026-06-22 08:51:16,406 INFO     Executing: ip addr show dev eth2
2026-06-22 08:51:16,408 INFO     Executing: ip addr show dev eth0
2026-06-22 08:51:16,411 INFO     Address found in DataBag ==> {'add': True, 'broadcast': '10.1.255.255', 'cidr': '10.1.0.1/16', 'device': 'eth0', 'gateway': '', 'netmask': '255.255.0.0', 'network': '10.1.0.0/16', 'nic_dev_id': '0', 'nw_type': 'guest', 'one_to_one_nat': False, 'public_ip': '10.1.0.1', 'size': '16', 'source_nat': False}
2026-06-22 08:51:16,411 INFO     Address 10.1.0.1/16 on device eth0 already configured
2026-06-22 08:51:16,411 INFO     Adding route table: 100 Table_eth0 to /etc/iproute2/rt_tables if not present
2026-06-22 08:51:16,411 INFO     Executing: ip rule show
2026-06-22 08:51:16,414 INFO     Executing: ip rule show
2026-06-22 08:51:16,417 INFO     Executing: ip link show eth0 | grep ' state '
2026-06-22 08:51:16,421 INFO     Executing2: arping -c 1 -I eth0 -A -U 10.1.0.1
OSError: [Errno 75] Value too large for defined data type

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/cloud/bin/update_config.py", line 147, in <module>
    process_file()
  File "/opt/cloud/bin/update_config.py", line 57, in process_file
    finish_config()
  File "/opt/cloud/bin/update_config.py", line 42, in finish_config
    returncode = configure.main(sys.argv)
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/cloud/bin/configure.py", line 1653, in main
    config.address().process()
  File "/opt/cloud/bin/cs/CsAddress.py", line 132, in process
    ip.post_configure(address)
  File "/opt/cloud/bin/cs/CsAddress.py", line 369, in post_configure
    CsRpsrfs(self.dev).enable()
  File "/opt/cloud/bin/cs/CsAddress.py", line 1018, in enable
    CsHelper.updatefile(filename, val, "w+")
  File "/opt/cloud/bin/cs/CsHelper.py", line 108, in updatefile
    handle.close()
OSError: [Errno 75] Value too large for defined data type
Mon Jun 22 08:51:16 AM UTC 2026 : VR config: executing failed: /opt/cloud/bin/update_config.py firewall_rules.json.c3028bfd-8405-479b-9257-156ac425e46e
Mon Jun 22 08:51:16 AM UTC 2026 : VR config: Flushing conntrack table
Mon Jun 22 08:51:16 AM UTC 2026 : VR config: Flushing conntrack table completed

Please help

@vishesh92
@weizhouapache
@DaanHoogland

[DaanHoogland]

just a hunch @TheKunalSen:

  File "/opt/cloud/bin/cs/CsAddress.py", line 1018, in enable
    CsHelper.updatefile(filename, val, "w+")

is the file system readonly for the VR or is the backing storage device full?

I don’t think the error report

OSError: [Errno 75] Value too large for defined data type

is actually significant.

[TheKunalSen]

@DaanHoogland
Nah the storage is not full at all

I have roughly 50tb of storage in nfs total
And also the vr got created automatically and I also restarted the network with cleanup

And evrything was running before the power cut

Can it be a nfs server file writing permission issue cause I am using a synology nas as nfs

even if i create a new network, the issue is the same i can not create new vm or start old vms

[weizhouapache]

@TheKunalSen
do you allocate VR with more CPUs ?

[arpanbht]

@weizhouapache @DaanHoogland
Hey guys, I have tried this method and it was working fine. But we were getting "virtual router os error" because the cpu architecture of our hosts are different, in the intel servers the guest.cpu.mode was set to custom and after removing this line we restarted the virtual router and everything worked then but now we cannot migrate newly created vm from the amd server to intel server but the vice versa is working. If we create any vm on intel server and then we migrate it on amd server and again migrate it back to the intel server then it is working.

what could be the solution so that we don't have to face these issues the migration would be seemless.

FYI : after researching a little we tried with these settings in agent.properties but still the same problem occuring

guest.cpu.mode=custom
guest.cpu.model=EPYC
guest.cpu.features=-svm

On amd server we put these lines in agent.properties