AMP Release 4.0.0 * Introduction * Alert of Future Changes * New Features * Bug Fixes * Known Issues * Backwards Compatibility * Upgrade Instructions ### Introduction Thanks you to our community who have contributed a lot of improvements and feedback! Thanks also go to Brooklyn's commercial users who have funded this development and made some major contributions. For more information, please visit http://www.cloudsoft.io/amp, http://docs.cloudsoft.io and http://brooklyn.apache.org This is a major upgrade from AMP version 3.3.x. ### Alert of Future Changes * Longer term, the persistence naming will be changed to avoid using package and class names in persisted state. This will improve backwards compatibility support moving forwards. A transformation tool will be provided for this update. * Brooklyn networking is being revisited, to incorporate work pioneered in the Clocker and Calico projects. This will allow networking requirements to be more easily described within an application, and the appropriate networking configured (be it VPCs, subnets, security groups or NAT'ing). There is a proposal at https://docs.google.com/document/d/1IrWLWunWSl_ScwY3MRICped8eJMjQEH1FbWZJcoK0Iw for the handling of multiple networks. This lays out a change in the semantics of sensors, to make it clearer which network an ip/port applies to. * The configuration options for both Azure and WinRM (especially for password and use of NTLM) are under review, and may be changed in a future release. Please treat these configuration options as "beta". * Code that was already deprecated in AMP 2.x and AMP 3.x will be deleted (incrementally). Additional code that has been deprecated will be deleted in a future release, in line with out deprecation policy. ### New Features * A new AMP UI has been created to enable users to better harness the power of AMP orchestration. The new UI improves usability with a modern responsive design and better access to information and tools within AMP. The UI better supports the different types of user: business users who are focused on deploying apps; application developers who are focused on composing apps or migrating apps to the cloud; and operations who are responsible for production deployment and management. The AMP classic UI is still available and has been moved to /amp-classic-ui, with links from the new UI. * A new website, http://docs.cloudsoft.io is available with guides and instructions detailing AMP functionality. * AMP now ships as an RPM and .deb, which installs AMP as a service. This is now the recommended way to install and run AMP. CentOS 7.x is recommended. * The modules within AMP are restructured to run as true OSGi bundles. AMP now runs inside a Karaf OSGi container. This significantly changes how server's command line options, as documented on docs.cloudsoft.io. * The Cloudsoft QA Framework, a tool for automated testing of YAML based blueprints is now an official part of AMP. This includes a scheduling and orchestration tool, test storage and a GUI to display and investigate results. * The Cloudsoft Container Service, a docker orchestration engine, is now an official part of AMP. This enables creation and in-life management of docker containers and swarms across multiple clouds and environments. * There is now an effector for opening inbound ports in a security group (if security groups are not supported by the given cloud api/provider, then it will return an error). Previously, such ports could only be easily opened by AMP at provisioning time. (See https://github.com/apache/brooklyn-server/pull/276) To enable this in your software-process entities (for newly provisioned entities), add the configuration option: effector.add.openInboundPorts: true * New SharedLocationSecurityGroupCustomizer This is a location customizer that creates a single shared security group for each entity it is applied to as well as an individual entity. The shared entity allows full TCP and UDP traffic within members of the group. (See https://github.com/apache/brooklyn-server/pull/292) It can be instantiated in YAML as given in the following example: services: - type: org.apache.brooklyn.entity.software.base.EmptySoftwareProcess brooklyn.config: provisioning.properties: customizers: - $brooklyn:object: type: org.apache.brooklyn.location.jclouds.networking.SharedLocationSecurityGroupCustomizer object.fields: {locationName: "test", tcpPortRanges: ["900-910", "915", "22"], cidr: "82.40.153.101/24"} * New SshCommandMembershipTrackingPolicy - a new member tracking policy with configurable SSH commands. The given ssh commands will be executed when a member joins or leaves the group. For example, this can be used to configure in YAML a load balancer as members of the target cluster are added/removed. (See https://github.com/apache/brooklyn-server/pull/287) * Improved $brooklyn:object() DSL. Extended $brooklyn:object() syntax to support calling of parameterised constructors and static factory methods. Added variants of EntityPredicates.config*() and attribute*() that accept strings. Added variants of EntityFunctions.config() and attribute() that accept strings. Added variants of EntityFunctions.config() and attribute() that accept format specifier. (See pull https://github.com/apache/brooklyn-server/pull/291) * BashCommands.dontRequireTtyForSudo() is now idempotent to prevent race condition with multiple entities that share the same configuration. (See pull https://github.com/apache/brooklyn-server/pull/283) * Implemented behaviour required to merge `brooklyn.parameters` sections of specifications in YAML files, allowing addition of new parameters to entity sub-types without having to specify the complete list again. (See pull https://github.com/apache/brooklyn-server/pull/229) * Added shell.env config for SSH sensor and effector. (See pull https://github.com/apache/brooklyn-server/pull/258) * Created ConditionalEntity to allow configurable addition of child entities. Allows entity creation to be controlled using a configuration key value. (See pull https://github.com/apache/brooklyn-server/pull/259) For example: - type: org.apache.brooklyn.entity.stock.ConditionalEntity brooklyn.config: conditional.entity.create: $brooklyn:scopeRoot().config("enable.loadBalancer") conditional.entity.spec: $brooklyn:entitySpec: type: load-balancer brooklyn.config: proxy.port: 8080 loadbalancer.serverpool: $brooklyn:entity("servers") * Complex objects are now serialized to JSON, instead of calling toString(), when building the shell environment variables for SoftwareProcess. (See pull https://github.com/apache/brooklyn-server/pull/216) For example: services: - type: org.apache.brooklyn.entity.software.base.VanillaSoftwareProcess brooklyn.config: mappings: key1: val1 key2: val2 env: MAPPINGS: $brooklyn:config("mappings") It will result in an environment variable of MAPPINGS={"key1":"val1","key2":"val2"} * Nginx config file generator now supports HTTP 1.1, and connection upgrade has been added. * The Cloudsoft Container Service now deploys Docker Swarm as multi-master, with replication and failover, using HA Proxy as a load balancer. It also includes autonomic policies to replace failed Swarm nodes and Swarm managers on failure. * The Cloudsoft Container Service now includes a DockerContainer entity type. This makes it easy to deploy a docker container (e.g. to Docker Swarm) using an images from docker-hub. For example: services: - type: io.cloudsoft.amp.containerservice.dockercontainer.DockerContainer brooklyn.config: docker.container.imageName: httpd docker.container.disable_ssh: true docker.container.ports: ["80"] ### Bug Fixes * BROOKLYN-323 After logging out from AMP UI there was an issue with logging in again with the same user in same browser tab. It occurred since AMP build 3.2.1. The issue is fixed in AMP 4.0.0. * BROOKLYN-320 Tomcat8Server entity fails to start if NSS package 3.15 was installed (e.g. on some CentOS 6.5 images). This stopped working due to upgrades in the default Java installed by the Tomcat entity. NSS is now upgraded automatically on yum-based systems. * BROOKLYN-300,BROOKLYN-287 MySQL provisioning fails on CentOS, if perl was not pre-installed * BROOKLYN-322 On rebind, entity temporarily (incorrectly) reports serviceUp=false Previously when rebinding to many entities, some software processes were temporarily showing serviceUp=false before going back to showing the correct serviceUp=true. This was causing some feeds to throw an exception. * BROOKLYN-319 RabbitMQ intermittent restart failure * Fixed NPE while generating Nginx config when no SSL is configured. (See https://github.com/apache/brooklyn-library/pull/59) * Fix for CreateUserPolicy for CentOS 7. Previously it was failing because of the wrong usage of `service ssh reload` instead of `service sshd reload` which is the proper way for CentOS 7. (See https://github.com/jclouds/jclouds/pull/991) * Fix NodeJsWebAppSshDriver to obtain properly MAIN_URL when PortForwardManager is used (https://github.com/apache/brooklyn-library/pull/48) ### Known Issues * AMP-1233 Windows blueprints fail to execute WinRM commands, due to binding issues in karaf. * AMP-1202 The RPM install does not work on CentOS 6.x. Instead, CentOS 7.x is recommended. * Tosca support is disabled in this release. Support for Tosca will be re-added in a future release soon. * AMP-1206 Modifying etc/org.apache.brooklyn.osgilauncher.cfg causes a reload of AMP OSGi bundles, which causes the management context to be stopped and restarted. It does not restart correctly, leaving AMP in a state where it is not running, but the karaf process is still running. It is strongly recommended to first stop AMP before modifying the configuration file, and then restart AMP. * AMP-1215,AMP-1226,AMP-1230 VM provisioning can fail, e.g. with the quick launch blueprints, when using the most basic location configuration. For example, the blueprint may require a particular OS distro, but the location picks an incompatible image, or the "login user" for initially ssh'ing to the VM may be incorrectly inferred for the chosen VM image. Also, the default hardware size of 512M is insufficient for most blueprint (which can be fixed by setting the config option `minRam`). * AMP-1220 The default logging configuration writes the jclouds wire log to log/amp.debug.log. If a lot of calls are made to the cloud (particularly if using S3 as the persistence store), this can quickly fill the debug log. In a future release, the logging will be split into separate files. * AMP-1221 The "VM customization" module is missing from AMP 4.0. The main implication is that the VcloudDirectorImageChooser is not included. * AMP-1227 Use of the AMP jumphost for ssh command execution has problems, due to issues with the correct classloading in Karaf. * AMP-1235 When using an empty brooklyn.properties, if the "reload brooklyn.properties" is used (either in the classic UI or via the REST api), it reloads the HTTP login credentials. If no credentials are defined in brooklyn.properties, then this means the original default credentials will no longer work. * AMP-877 the previous `./bin/amp copy-state` command is not available in Karaf. If required, an older version of AMP 3.x can be used to execute this command. * AMP-1043 the previous `./bin/amp cloud-compute` command is not available in Karaf. If required, an older version of AMP 3.x can be used to execute this command. * AMP-883 the previous `./bin/amp generate-password` command is not available in Karaf. If required, an older version of AMP 3.x can be used to execute this command. * The new UI has some features missing that were present in the classic UI. The workaround is to access these via the classic UI, or the `br` CLI, or the REST api: * There is no "reload properties" button, to reload the brooklyn.properties * No "expunge" and "unmanage" button, to complete remove an entity. Calling the "stop" effector on a top-level app with attempt to gracefully stop and unmanage the entity, but if stop fails then unmanage will not be done. * AMP will occasionally report NullPointerException in org.apache.cxf.transport.http.async. These are not harmful and can be ignored. * BROOKLYN-317 VM provisioning can fail in AWS-EC2 (because api did not return public IP). The workaround is to configure machineCreateAttempts on the location, setting it to a number greater than 1. It will thus retry if the problem is encountered. * Windows blueprints on vCloud Director sometimes fail. This is because the VM customizations performed by vCloud Director can include a reboot. This sometimes does not happen until several minutes after the VM is available, which can happen mid-way through installation of the software components. A workaround is available for this behaviour. Please check waitWindowsToStart parameter http://docs.cloudsoft.io/blueprints/base-blueprints/winrm/index.html#webserviceexception-could-not-send-message * It is strongly recommended to use WinRM v4 or greater. Older versions of WinRM (e.g. v3 on Windows 2008rc2) may fail. See COMPOSEC-1200. * For Windows VMs on Azure and GCE, the "Administrator" user is disabled. * https://issues.apache.org/jira/browse/BROOKLYN-243: On restarting an entity's process via the restart effector, there is a rare race-condition where the entity is not reported at "service.isUp" true, even though the process is running. * When using BYON to deploy multiple components to a single VM, the recommended way is to use the `SameServerEntity`. If instead the same machine were to be listed multiple times, then concurrent conflicting commands could be executed on that machine. For example, if multiple `iptables` commands are executed to add rules, then these interfere so that some rules may not be applied. * The following problems have been encountered in vCloud Director: * https://github.com/cloudsoft/jclouds-vcloud-director/issues/12 Failure terminating VM in vCloud Air: InvalidCacheLoadException when looking up the VM * https://github.com/cloudsoft/jclouds-vcloud-director/issues/23 VM delete failed; seemingly it did not wait for the stop to complete. * Windows provisioning to vCloud Air failed. The guest customization defined in the vApp Template appears not to be applied. The VM has no password, requires immediate password reset, and has not had the init script executed to configure WinRM. * Timeout waiting for VM to be provisioned, e.g. failing with a message like: "not composed within 1200000 ms" The following configuration can increase this timeout: brooklyn.location.named.MyCanopyLocation.jclouds.compute.timeout.node-running=3600000 * The following are known issues with jclouds-azure: * Private VM images are not supported. This is non-trivial to support, because private images and public images are modelled very differently in Azure. * The classic REST API is used, rather than the new Azure Resource Manager REST API. This classic API will likely be deprecated; Microsoft encourage the use of the new API. * Warning when deleting a VM. The error reported was: "A disk with name ... is currently in use by virtual machine ..." Those messages are to be considered warnings as they are produced by a retry logic that attempts to delete the disk while the vm is still being deleted. * Azure does not support port ranges for public endpoints; moreover Azure limits the number of endpoints per cloud service and per VM (150 per cloud service and 50 per VM). This causes provisioning to fail if a blueprint requires a large number of firewall rules (aka public endpoints). See https://social.msdn.microsoft.com/forums/azure/en-US/4c093e33-eb86-4639-b325-c295fa8bfc5d/vm-endpoint-and-port-range and https://feedback.azure.com/forums/216843-virtual-machines/suggestions/7120093-increase-150-endpoints-limit-for-a-cloud-service for more details * In the persisted state, it is possible to have "dangling references" - e.g. the persisted state for an entity references another entity that does not exist (because that other entity is in the process of being created or deleted, for example). On startup, if the number of dangling references is too large then startup will fail. One can configure the tolerable number of dangling references. In the example below, each pair indicates for a given number of items how many dangling references are acceptable. This will be interpolated and extrapolated for other numbers of items. See the docs in https://github.com/apache/incubator-brooklyn/blob/0.8.0-incubating/utils/common/src/main/java/org/apache/brooklyn/util/collections/QuorumCheck.java#L87-#L112 For example, one could set in brooklyn.properties: rebind.failureMode.danglingRefs.minRequiredHealthy=[[0,0],[2,0],[3,1],[10,1],[100,10]] ### Backwards Compatibility Now that AMP runs inside a Karaf container, it entirely changes the mechanism for launching AMP. This will impact any custom scripting around the launching of AMP, and the supplying of command line arguments. Use of the lib/dropins and lib/patch folders will no longer work (because Karaf does not support that kind of classlaoding). Instead, code must be built and installed as OSGi bundles. There is a proposal at https://docs.google.com/document/d/1IrWLWunWSl_ScwY3MRICped8eJMjQEH1FbWZJcoK0Iw for the handling of multiple networks. This lays out a change in the semantics of sensors, to make it clearer which network an ip/port applies to. Some entities have been changed to follow this convention, and others will be changed in future releases. ### Upgrade Instructions Use of RPM and DEB is now recommended, rather than the tar.gz. CentOS 7.x is recommended over CentOS 6.x (indeed the RPM will not work on CentOS 6.x). This entirely replaces the previous install. If binding to existing persisted state, an additional command is required to update the existing catalog with the AMP 4.0 versions. Assuming AMP has been installed to /opt/amp (as is done by the RPM and DEB): br add-catalog /opt/amp/catalog/catalog.bom All existing custom jars previously added to lib/plugins (e.g. for Java-based entities) need to be converted to OSGi bundles, and installed in Karaf. The use of the "brooklyn.libraries" section in catalog.bom files will continue to work. User credentials should now be recorded in ${karaf.home}/etc/brooklyn.cfg. AMP will still read them from both ${karaf.home}/etc/brooklyn.cfg and ~/.brooklyn/brooklyn.properties