{"action":"created","issue":{"url":"https://api.github.com/repos/apache/cloudstack/issues/1180","labels_url":"https://api.github.com/repos/apache/cloudstack/issues/1180/labels{/name}","comments_url":"https://api.github.com/repos/apache/cloudstack/issues/1180/comments","events_url":"https://api.github.com/repos/apache/cloudstack/issues/1180/events","html_url":"https://github.com/apache/cloudstack/pull/1180","id":120711948,"number":1180,"title":"CLOUDSTACK-9112: Deploy VM failing frequently due to capacity calculation not synchron…","user":{"login":"harikrishna-patnala","id":3348673,"avatar_url":"https://avatars.githubusercontent.com/u/3348673?v=3","gravatar_id":"","url":"https://api.github.com/users/harikrishna-patnala","html_url":"https://github.com/harikrishna-patnala","followers_url":"https://api.github.com/users/harikrishna-patnala/followers","following_url":"https://api.github.com/users/harikrishna-patnala/following{/other_user}","gists_url":"https://api.github.com/users/harikrishna-patnala/gists{/gist_id}","starred_url":"https://api.github.com/users/harikrishna-patnala/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/harikrishna-patnala/subscriptions","organizations_url":"https://api.github.com/users/harikrishna-patnala/orgs","repos_url":"https://api.github.com/users/harikrishna-patnala/repos","events_url":"https://api.github.com/users/harikrishna-patnala/events{/privacy}","received_events_url":"https://api.github.com/users/harikrishna-patnala/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-12-07T06:55:22Z","updated_at":"2015-12-08T11:08:50Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/cloudstack/pulls/1180","html_url":"https://github.com/apache/cloudstack/pull/1180","diff_url":"https://github.com/apache/cloudstack/pull/1180.diff","patch_url":"https://github.com/apache/cloudstack/pull/1180.patch"},"body":"CLOUDSTACK-9112: deployVM thread is holding the global lock on network longer and cause delays and some improvements in the planner\r\n\r\nThere are some VM deployment failures happening when multiple VMs are deployed at a time, failures mainly due to NetworkModel code that iterates over all the vlans in the pod. This causes each deployVM thread to hold the global lock on Network longer and cause delays. This delay in turn causes more threads to choose same host and fail since capacity is not available on that host.\r\n\r\nFollowing are some changes required to be done to reduce delays during VM deployments which in turn causes some vm deployment failures when multiple VMs are launched at a time.\r\n\r\n  -  In Planner, remove the clusters that do not contain a host with matching service offering tag. This will save some iterations over clusters that dont have matching tagged host\r\n  -  In NetworkModel, do not query the vlans for the pod within the loop. Also optimized the logic to query the ip/ipv6\r\n  -  In DeploymentPlanningManagerImpl, do not process the affinity group if the plan has hostId provided.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/cloudstack/issues/comments/162851856","html_url":"https://github.com/apache/cloudstack/pull/1180#issuecomment-162851856","issue_url":"https://api.github.com/repos/apache/cloudstack/issues/1180","id":162851856,"user":{"login":"remibergsma","id":1630096,"avatar_url":"https://avatars.githubusercontent.com/u/1630096?v=3","gravatar_id":"","url":"https://api.github.com/users/remibergsma","html_url":"https://github.com/remibergsma","followers_url":"https://api.github.com/users/remibergsma/followers","following_url":"https://api.github.com/users/remibergsma/following{/other_user}","gists_url":"https://api.github.com/users/remibergsma/gists{/gist_id}","starred_url":"https://api.github.com/users/remibergsma/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/remibergsma/subscriptions","organizations_url":"https://api.github.com/users/remibergsma/orgs","repos_url":"https://api.github.com/users/remibergsma/repos","events_url":"https://api.github.com/users/remibergsma/events{/privacy}","received_events_url":"https://api.github.com/users/remibergsma/received_events","type":"User","site_admin":false},"created_at":"2015-12-08T11:08:50Z","updated_at":"2015-12-08T11:08:50Z","body":"LGTM based on these tests:\r\n\r\n```\r\nnosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=true \\\r\ncomponent/test_vpc_redundant.py \\\r\ncomponent/test_routers_iptables_default_policy.py \\\r\ncomponent/test_routers_network_ops.py \\\r\ncomponent/test_vpc_router_nics.py \\\r\nsmoke/test_loadbalance.py \\\r\nsmoke/test_internal_lb.py \\\r\nsmoke/test_ssvm.py \\\r\nsmoke/test_network.py\r\n\r\n```\r\n\r\nResult:\r\n\r\n```\r\nTest iptables default INPUT/FORWARD policies on VPC router ... === TestName: test_01_single_VPC_iptables_policies | Status : SUCCESS ===\r\nok\r\nTest redundant router internals ... === TestName: test_01_isolate_network_FW_PF_default_routes_egress_true | Status : SUCCESS ===\r\nok\r\nTest redundant router internals ... === TestName: test_02_isolate_network_FW_PF_default_routes_egress_false | Status : SUCCESS ===\r\nok\r\nTest redundant router internals ... === TestName: test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true | Status : SUCCESS ===\r\nok\r\nTest redundant router internals ... === TestName: test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false | Status : SUCCESS ===\r\nok\r\nCreate a VPC with two networks with one VM in each network and test nics after destroy ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===\r\nok\r\nCreate a VPC with two networks with one VM in each network and test default routes ... === TestName: test_02_VPC_default_routes | Status : SUCCESS ===\r\nok\r\nCheck the password file in the Router VM ... === TestName: test_isolate_network_password_server | Status : SUCCESS ===\r\nok\r\nCheck that the /etc/dhcphosts.txt doesn't contain duplicate IPs ... === TestName: test_router_dhcphosts | Status : SUCCESS ===\r\nok\r\nTest to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===\r\nok\r\nTest to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===\r\nok\r\nTest for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===\r\nok\r\nTest to verify access to loadbalancer haproxy admin stats page ... === TestName: test02_internallb_haproxy_stats_on_all_interfaces | Status : SUCCESS ===\r\nok\r\nTest create, assign, remove of an Internal LB with roundrobin http traffic to 3 vm's ... === TestName: test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 | Status : SUCCESS ===\r\nok\r\nTest SSVM Internals ... === TestName: test_03_ssvm_internals | Status : SUCCESS ===\r\nok\r\nTest CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===\r\nok\r\nTest stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===\r\nok\r\nTest stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===\r\nok\r\nTest reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===\r\nok\r\nTest reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===\r\nok\r\nTest destroy SSVM ... === TestName: test_09_destroy_ssvm | Status : SUCCESS ===\r\nok\r\nTest destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===\r\nok\r\nTest Remote Access VPN in VPC ... === TestName: test_vpc_remote_access_vpn | Status : SUCCESS ===\r\nok\r\nTest VPN in VPC ... === TestName: test_vpc_site2site_vpn | Status : SUCCESS ===\r\nok\r\nTest for port forwarding on source NAT ... === TestName: test_01_port_fwd_on_src_nat | Status : SUCCESS ===\r\nok\r\nTest for port forwarding on non source NAT ... === TestName: test_02_port_fwd_on_non_src_nat | Status : SUCCESS ===\r\nok\r\nTest for reboot router ... === TestName: test_reboot_router | Status : SUCCESS ===\r\nok\r\nTest for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_1_static_nat_rule | Status : SUCCESS ===\r\nok\r\nTest for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_2_nat_rule | Status : SUCCESS ===\r\nok\r\nTest for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_3_Load_Balancer_Rule | Status : SUCCESS ===\r\nok\r\n\r\n----------------------------------------------------------------------\r\nRan 33 tests in 17222.324s\r\n\r\nOK\r\n```\r\n\r\n\r\nAnd:\r\n\r\n```\r\nnosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=false \\\r\nsmoke/test_routers.py \\\r\nsmoke/test_network_acl.py \\\r\nsmoke/test_privategw_acl.py \\\r\nsmoke/test_reset_vm_on_reboot.py \\\r\nsmoke/test_vm_life_cycle.py \\\r\nsmoke/test_vpc_vpn.py \\\r\nsmoke/test_service_offerings.py \\\r\ncomponent/test_vpc_offerings.py \\\r\ncomponent/test_vpc_routers.py\r\n```\r\n\r\nResult:\r\n\r\n```\r\nTest router internal advanced zone ... === TestName: test_02_router_internal_adv | Status : SUCCESS ===\r\nok\r\nTest restart network ... === TestName: test_03_restart_network_cleanup | Status : SUCCESS ===\r\nok\r\nTest router basic setup ... === TestName: test_05_router_basic | Status : SUCCESS ===\r\nok\r\nTest router advanced setup ... === TestName: test_06_router_advanced | Status : SUCCESS ===\r\nok\r\nTest stop router ... === TestName: test_07_stop_router | Status : SUCCESS ===\r\nok\r\nTest start router ... === TestName: test_08_start_router | Status : SUCCESS ===\r\nok\r\nTest reboot router ... === TestName: test_09_reboot_router | Status : SUCCESS ===\r\nok\r\ntest_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_privategw_acl | Status : SUCCESS ===\r\nok\r\nTest reset virtual machine on reboot ... === TestName: test_01_reset_vm_on_reboot | Status : SUCCESS ===\r\nok\r\nTest advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===\r\nok\r\nTest Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===\r\nok\r\nTest Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===\r\nok\r\nTest Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===\r\nok\r\nTest Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===\r\nok\r\nTest Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===\r\nok\r\nTest destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===\r\nok\r\nTest recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===\r\nok\r\nTest migrate VM ... === TestName: test_08_migrate_vm | Status : SUCCESS ===\r\nok\r\nTest destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===\r\nok\r\nTest to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===\r\nok\r\nTest to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===\r\nok\r\nTest to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===\r\nok\r\nTest for delete account ... === TestName: test_delete_account | Status : SUCCESS ===\r\nok\r\nTest for Associate/Disassociate public IP address for admin account ... === TestName: test_public_ip_admin_account | Status : SUCCESS ===\r\nok\r\nTest for Associate/Disassociate public IP address for user account ... === TestName: test_public_ip_user_account | Status : SUCCESS ===\r\nok\r\nTest for release public IP address ... === TestName: test_releaseIP | Status : SUCCESS ===\r\nok\r\nTest create VPC offering ... === TestName: test_01_create_vpc_offering | Status : SUCCESS ===\r\nok\r\nTest VPC offering without load balancing service ... === TestName: test_03_vpc_off_without_lb | Status : SUCCESS ===\r\nok\r\nTest VPC offering without static NAT service ... === TestName: test_04_vpc_off_without_static_nat | Status : SUCCESS ===\r\nok\r\nTest VPC offering without port forwarding service ... === TestName: test_05_vpc_off_without_pf | Status : SUCCESS ===\r\nok\r\nTest VPC offering with invalid services ... === TestName: test_06_vpc_off_invalid_services | Status : SUCCESS ===\r\nok\r\nTest update VPC offering ... === TestName: test_07_update_vpc_off | Status : SUCCESS ===\r\nok\r\nTest list VPC offering ... === TestName: test_08_list_vpc_off | Status : SUCCESS ===\r\ntest_09_create_redundant_vpc_offering (integration.component.test_vpc_offerings.TestVPCOffering) ... === TestName: test_09_create_redundant_vpc_offering | Status : SUCCESS ===\r\nok\r\nTest start/stop of router after addition of one guest network ... === TestName: test_01_start_stop_router_after_addition_of_one_guest_network | Status : SUCCESS ===\r\nok\r\nTest reboot of router after addition of one guest network ... === TestName: test_02_reboot_router_after_addition_of_one_guest_network | Status : SUCCESS ===\r\nok\r\nTest to change service offering of router after addition of one guest network ... === TestName: test_04_chg_srv_off_router_after_addition_of_one_guest_network | Status : SUCCESS ===\r\nok\r\nTest destroy of router after addition of one guest network ... === TestName: test_05_destroy_router_after_addition_of_one_guest_network | Status : SUCCESS ===\r\nok\r\nTest to stop and start router after creation of VPC ... === TestName: test_01_stop_start_router_after_creating_vpc | Status : SUCCESS ===\r\nok\r\nTest to reboot the router after creating a VPC ... === TestName: test_02_reboot_router_after_creating_vpc | Status : SUCCESS ===\r\nok\r\nTests to change service offering of the Router after ... === TestName: test_04_change_service_offerring_vpc | Status : SUCCESS ===\r\nok\r\nTest to destroy the router after creating a VPC ... === TestName: test_05_destroy_router_after_creating_vpc | Status : SUCCESS ===\r\nok\r\n\r\n----------------------------------------------------------------------\r\nRan 42 tests in 8486.536s\r\n\r\nOK\r\n```\r\n\r\nHaven't tested the actual functionality, this just shows you didn't break anything else."}}