AMP Release 4.1.0 * Introduction * Alert of Future Changes * New Features * Bug Fixes * Known Issues * Backwards Compatibility * Upgrade Instructions ### Introduction Thank you to our community who have contributed a lot of improvements and feedback! Thanks also go to Brooklyn's commercial users who have funded this development and made some major contributions. For more information, please visit http://www.cloudsoft.io/amp-4, http://docs.cloudsoft.io and http://brooklyn.apache.org This is a major upgrade from AMP version 4.0.1. ### Alert of Future Changes * The persistence naming will be changed to avoid using package and class names in persisted state. This will improve backwards compatibility support moving forwards. A transformation tool will be provided for this update. * Brooklyn networking is being revisited, to incorporate work pioneered in the Clocker and Calico projects. This will allow networking requirements to be more easily described within an application, and the appropriate networking configured (be it VPCs, subnets, security groups or NAT'ing). There is a proposal at https://docs.google.com/document/d/1IrWLWunWSl_ScwY3MRICped8eJMjQEH1FbWZJcoK0Iw for the handling of multiple networks. This lays out a change in the semantics of sensors, to make it clearer which network an ip/port applies to. * The configuration options for both Azure and WinRM (especially for password and use of NTLM) are under review, and may be changed in a future release. Please treat these configuration options as "beta". * Code that was already deprecated in AMP 2.x and AMP 3.x will be deleted (incrementally). Additional code that has been deprecated will be deleted in a future release, in line with out deprecation policy. ### New Features * Clocker 2.0.0 (http://www.clocker.io) has been released, with open sourced blueprints that were first developed in the closed-source Cloudsoft Container Service. AMP 4.1.0 includes the Clocker blueprints deploying Docker Engine and Docker Swarm. * Configuration inheritance has been greatly improved. "Inheritance" in this context is when config values are defined on the super-type, or on the parent entity in the runtime management hierarchy. This controls how sub-types and/or an entity's children will use the config that is declared on its parent. See http://docs.cloudsoft.io/blueprints/syntax-entity-configuration.html * Adds ProxyEffector (as an entity initializer, for adding the effector to an entity). This forwards effector invocations from one entity to another. See https://github.com/apache/brooklyn-server/pull/370 In the toy YAML example below, the second entity ("source") will have an effector named "runPsProxy". Invoking this will cause the "runPs" effector to invoked on the first entity ("target")": location: ... services: - id: target type: org.apache.brooklyn.entity.software.base.EmptySoftwareProcess brooklyn.initializers: - type: org.apache.brooklyn.core.effector.ssh.SshCommandEffector brooklyn.config: name: "runPs" command: "ps" - id: source type: org.apache.brooklyn.entity.software.base.EmptySoftwareProcess brooklyn.initializers: - type: org.apache.brooklyn.core.effector.ProxyEffector brooklyn.config: name: "runPsProxy" targetEntity: $brooklyn:entity("target") targetEffector: runPs * The InvokeEffectorOnSensorChange policy has been added. This will trigger a given effector call when a sensor changes. See https://github.com/apache/brooklyn-server/pull/308 In the toy YAML example below, the effector "createLog" will be invoked each time the sensor "ps" changes. The sensor will be set every 10 seconds by the given SshCommandSensor: location: ... services: - type: org.apache.brooklyn.entity.software.base.EmptySoftwareProcess brooklyn.initializers: - type: org.apache.brooklyn.core.sensor.ssh.SshCommandSensor brooklyn.config: name: "ps" command: "ps" period: 10s - type: org.apache.brooklyn.core.effector.ssh.SshCommandEffector brooklyn.config: name: "createLog" command: "ps >> /tmp/test.log" brooklyn.policies: - type: org.apache.brooklyn.policy.InvokeEffectorOnSensorChange brooklyn.config: sensor: $brooklyn:sensor("ps") effector: "create log" * JcloudsLocation: Ability to supply external implementation for the socket-reachability check See https://github.com/apache/brooklyn-server/pull/332 * TestWinrmCommand has been added to the test-framework. This allows WinRM commands to be executed as part of automated tests, and assertions to be made about the results. See https://github.com/apache/brooklyn-server/pull/343 * REST API for retrieving config now includes whether the key is "pinned" and any constraints. "Pinned" means that the config key will always be displayed when presenting as editable in the catalog. In YAML, one can include in the brooklyn.parameters declaration: pinned: true In Java, one can annotate a config key with: @CatalogConfig(pinned = true) See https://github.com/apache/brooklyn-server/pull/358 * OnPublicNetworkEnricher: additional configuration added to specify which sensor contains the entity's public IP. See https://github.com/apache/brooklyn-server/pull/379 * Add "disable" flag to SharedLocationSecurityGroupCustomizer See https://github.com/apache/brooklyn-server/pull/361 * BindDNS entity improvements: support configurable source sensor for the IP address (so we can do private or public); improvements for OS portability. See https://github.com/apache/brooklyn-library/pull/62 * The latest AMP Jumphost now supports a "reachability check", to determine whether an IP:port is reachable from the jumphost. ### Bug Fixes * BROOKLYN-325 On rebind/failover when entity is starting/stopping, report as "on-fire" with explanation See https://github.com/apache/brooklyn-server/pull/356 and https://github.com/apache/brooklyn-server/pull/364 * BROOKLYN-349: fix YAML DSL resolution, when used in location config * BROOKLYN-351: fix jclouds provider/api caching, so new providers adding in OSGi will be picked up. * BROOKLYN-354: Rebind failed (when an NingxController entity's hostname sensor was null) See https://github.com/apache/brooklyn-server/pull/360 * BROOKLYN-350: PostgresSQL to handle yum repos enforcing https See https://github.com/apache/brooklyn-library/pull/64 * BROOKLYN-357: Remove lock on DSL object evaluation See https://github.com/apache/brooklyn-server/pull/383 * BROOKLYN-358: add jclouds:swift to Karaf distro See https://github.com/apache/brooklyn-server/pull/374 * BROOKLYN-360: jclouds extensions config coerced from string. This is important for use in brooklyn.properties. See https://github.com/apache/brooklyn-server/pull/377 * AMP-1220 The default logging configuration now writes the jclouds wire log to a separate log file: log/jclouds.debug.log. * AMP-1365: Nginx advertises main.uri.mapped.public and main.uri.mapped.subnet. Previously it only advertised main.uri, which was the subnet address. In clouds like GCE and Azure, this advertised URI was therefore not reachable by the end user. See https://github.com/apache/brooklyn-library/pull/66 * Fix Docker Swarm manager's load balancer configuration. https://github.com/brooklyncentral/clocker/pull/357 * For jclouds VM provisioning, use same customizer instance for each step. Previously a new instance was created for each call, which caused problems for stateful customizers like SharedLocationSecurityGroupCustomizer. See https://github.com/apache/brooklyn-server/pull/371 * Fix config key ordering in the UI, based on the "priority" specified on those keys. See https://github.com/apache/brooklyn-server/pull/358 * For jclouds VM provisioning, fix choice of IP when has multiple public and/or private IPs. See https://github.com/apache/brooklyn-server/pull/350 See https://github.com/apache/brooklyn-server/pull/366 * Fix loading of UsageListener and jclouds imageChooser, so works in Karaf See https://github.com/apache/brooklyn-server/pull/342 * Fix use of SharedLocationSecurityGroupCustomizer.udpPortRanges in YAML. Previously, specifying UDP ports in YAML failed. See https://github.com/apache/brooklyn-server/pull/348 * When provisioning VMs using jclouds, ensure relevant config values are subsequently used by the machine. See https://github.com/apache/brooklyn-server/pull/350 * Fix `installDevUrandom` in jclouds VM provisioning, where root user is subsequently used for ssh'ing. See https://github.com/apache/brooklyn-server/pull/350 * DynamicGroup filters entities in its app, not the management context See https://github.com/apache/brooklyn-server/pull/369 * Fix JcloudsLocationSecurityGroupCustomizer for azure (so doesn't fail, trying to configure ICMP) See https://github.com/apache/brooklyn-server/pull/353 * Fix RPM install on CentOS 6. Previously, when the AMP service was stopped, the Karaf container was left running. * Fix use of metering service in Karaf-based AMP, so can push metering information. * Fix some (minor) memory leaks in YAML blueprint parsing See https://github.com/apache/brooklyn-server/pull/331 * Fix deadlock in DynamicGroup See https://github.com/apache/brooklyn-server/pull/306 * On catalog-reset, ensure locations added via brooklyn.properties are still available See https://github.com/apache/brooklyn-server/pull/307 * AMP-1221: Adds the "VM customization" module to Karaf (e.g. so that VcloudDirectorImageChooser is available). * AMP-1227: Fixed the AMP jumphost in Karaf. ### Known Issues * BROOKLYN-364: Provisioning in Softlayer can cause an OutOfMemoryError, especially when deploying multiple apps concurrently or large apps. See http://stackoverflow.com/questions/40050395/softlayer-api-call-to-virtualguests-returns-huge-duplicated-response * The AMP UI has primarily been tested in Chrome and Firefox. These are the recommended browsers. In Safari, it has been observed that the quick launch "Add Config" does not work - there may be other unknown issues. For users of Internet Explorer, the recommendation is to use Chrome or Firefox instead; or failing that to upgrade to Microsoft Edge - but that is not officially supported. * BROOKLYN-356: use of the Transformer enricher can sometimes fail to populate the target sensor, particularly on a heavily loaded AMP server. This is when "targetValue" is used with the YAML DSL, and the value is not evaluated promptly. * AMP-1254 Alien4Cloud app deployment fails when an app is composed in the A4C UI. * AMP-1249 After installing the Tosca feature in Karaf, an additional `bin/stop` and `bin/start` may be required. * AMP-1248 On some environments, the `bin/client` tool for connecting to Karaf may see: IllegalStateException: Unable to negotiate key exchange for kex algorithms The client can be accessed alternatively by using `ssh -p 8101 karaf@localhost` (with default password being karaf). * AMP-1250 The latest OpenStack (Mitaka) requires use of the location type `openstack-mitaka-nova`, rather than the vanilla `openstack-nova`. * AMP-1226 The username to use when first logging into a new VM is sometimes inferred incorrectly. The solution is to explicitly specify the username in the location's configuration, e.g. using `loginUser: ubuntu`. For further information, see http://docs.cloudsoft.io/locations/reference/#os-initial-login-and-setup and http://docs.cloudsoft.io/operations/troubleshooting/deployment.html * AMP-1213 When upgrading AMP, using existing persisted state, it will re-use the existing catalog from that persisted state and will not add new versions of the catalog items that ship with the upgraded AMP. As described in the "Upgrade Instructions", the workaround is to manually trigger the addition of those catalog items by running: br add-catalog /opt/amp/catalog/catalog.bom * AMP-1215,AMP-1226 VM provisioning can fail, e.g. with the quick launch blueprints, when using the most basic location configuration. For example, the blueprint may require a particular OS distro, but the location picks an incompatible image, or the "login user" for initially ssh'ing to the VM may be incorrectly inferred for the chosen VM image. * AMP-1235 When using an empty brooklyn.properties, if the "reload brooklyn.properties" is used (either in the classic UI or via the REST api), it reloads the HTTP login credentials. If no credentials are defined in brooklyn.properties, then this means the original default credentials will no longer work. * AMP-877 the previous `./bin/amp copy-state` command is not available in Karaf. If required, an older version of AMP 3.x can be used to execute this command. * AMP-1043 the previous `./bin/amp cloud-compute` command is not available in Karaf. If required, an older version of AMP 3.x can be used to execute this command. * AMP-883 the previous `./bin/amp generate-password` command is not available in Karaf. If required, an older version of AMP 3.x can be used to execute this command. * The new UI has some features missing that were present in the classic UI. The workaround is to access these via the classic UI, or the `br` CLI, or the REST api: * There is no "reload properties" button, to reload the brooklyn.properties * No "expunge" and "unmanage" button, to complete remove an entity. Calling the "stop" effector on a top-level app with attempt to gracefully stop and unmanage the entity, but if stop fails then unmanage will not be done. * The AMP log will occasionally report NullPointerException in org.apache.cxf.transport.http.async. These are not harmful and can be ignored. * BROOKLYN-317 VM provisioning can fail in AWS-EC2 (because api did not return public IP). The workaround is to configure machineCreateAttempts on the location, setting it to a number greater than 1. It will thus retry if the problem is encountered. * Windows blueprints on vCloud Director sometimes fail. This is because the VM customizations performed by vCloud Director can include a reboot. This sometimes does not happen until several minutes after the VM is available, which can happen mid-way through installation of the software components. A workaround is available for this behaviour. Please check waitWindowsToStart parameter http://docs.cloudsoft.io/blueprints/base-blueprints/winrm/index.html#webserviceexception-could-not-send-message * It is strongly recommended to use WinRM v4 or greater. Older versions of WinRM (e.g. v3 on Windows 2008rc2) may fail. See COMPOSEC-1200. * For Windows VMs on Azure and GCE, the "Administrator" user is disabled. * https://issues.apache.org/jira/browse/BROOKLYN-243: On restarting an entity's process via the restart effector, there is a rare race-condition where the entity is not reported at "service.isUp" true, even though the process is running. * When using BYON to deploy multiple components to a single VM, the recommended way is to use the `SameServerEntity`. If instead the same machine were to be listed multiple times, then concurrent conflicting commands could be executed on that machine. For example, if multiple `iptables` commands are executed to add rules, then these interfere so that some rules may not be applied. * The following problems have been encountered in vCloud Director: * https://github.com/cloudsoft/jclouds-vcloud-director/issues/12 Failure terminating VM in vCloud Air: InvalidCacheLoadException when looking up the VM * https://github.com/cloudsoft/jclouds-vcloud-director/issues/23 VM delete failed; seemingly it did not wait for the stop to complete. * Windows provisioning to vCloud Air failed. The guest customization defined in the vApp Template appears not to be applied. The VM has no password, requires immediate password reset, and has not had the init script executed to configure WinRM. * Timeout waiting for VM to be provisioned, e.g. failing with a message like: "not composed within 1200000 ms" The following configuration can increase this timeout: brooklyn.location.named.MyCanopyLocation.jclouds.compute.timeout.node-running=3600000 * The following are known issues with jclouds-azure: * Private VM images are not supported. This is non-trivial to support, because private images and public images are modelled very differently in Azure. * The classic REST API is used, rather than the new Azure Resource Manager REST API. This classic API will likely be deprecated; Microsoft encourage the use of the new API. * Warning when deleting a VM. The error reported was: "A disk with name ... is currently in use by virtual machine ..." Those messages are to be considered warnings as they are produced by a retry logic that attempts to delete the disk while the vm is still being deleted. * Azure does not support port ranges for public endpoints; moreover Azure limits the number of endpoints per cloud service and per VM (150 per cloud service and 50 per VM). This causes provisioning to fail if a blueprint requires a large number of firewall rules (aka public endpoints). See https://social.msdn.microsoft.com/forums/azure/en-US/4c093e33-eb86-4639-b325-c295fa8bfc5d/vm-endpoint-and-port-range and https://feedback.azure.com/forums/216843-virtual-machines/suggestions/7120093-increase-150-endpoints-limit-for-a-cloud-service for more details * In the persisted state, it is possible to have "dangling references" - e.g. the persisted state for an entity references another entity that does not exist (because that other entity is in the process of being created or deleted, for example). On startup, if the number of dangling references is too large then startup will fail. One can configure the tolerable number of dangling references. In the example below, each pair indicates for a given number of items how many dangling references are acceptable. This will be interpolated and extrapolated for other numbers of items. See the docs in https://github.com/apache/incubator-brooklyn/blob/0.8.0-incubating/utils/common/src/main/java/org/apache/brooklyn/util/collections/QuorumCheck.java#L87-#L112 For example, one could set in brooklyn.properties: rebind.failureMode.danglingRefs.minRequiredHealthy=[[0,0],[2,0],[3,1],[10,1],[100,10]] ### Backwards Compatibility For backwards compatibility with 3.x, see also the release notes for AMP 4.0.0. The rules for configuration inheritance have changed. The defaults are believed to now be more intuitive (and less of a surprise to users!). For more details, see: http://docs.cloudsoft.io/blueprints/syntax-entity-configuration.html DynamicGroup now filters its entities for just those in the same app, rather than everything in the management context. The jackson version has been upgraded to 2.7.5 (from 2.4.5). See https://github.com/apache/brooklyn-server/pull/339 There is a proposal at https://docs.google.com/document/d/1IrWLWunWSl_ScwY3MRICped8eJMjQEH1FbWZJcoK0Iw for the handling of multiple networks. This lays out a change in the semantics of sensors, to make it clearer which network an ip/port applies to. Some entities have been changed to follow this convention, and others will be changed in future releases. ### Upgrade Instructions If upgrading from 3.x, see also the release notes for AMP 4.0.0. Use of RPM and DEB is now recommended, rather than the tar.gz. CentOS 7.x is recommended, though CentOS 6 and Ubuntu 16.04 are also tested. This entirely replaces the previous install. If binding to existing persisted state, an additional command is required to update the existing catalog with the AMP 4.1 versions. Assuming AMP has been installed to /opt/amp (as is done by the RPM and DEB): br add-catalog /opt/amp/catalog/catalog.bom