AMP Release 4.2.0 * Introduction * Alert of Future Changes * New Features * Bug Fixes * Known Issues * Backwards Compatibility * Upgrade Instructions ### Introduction Thank you to our community who have contributed a lot of improvements and feedback! Thanks also go to Brooklyn's commercial users who have funded this development and made some major contributions. For more information, please visit http://www.cloudsoft.io/amp-4, http://docs.cloudsoft.io and http://brooklyn.apache.org This is a major upgrade from AMP version 4.1.0. ### Alert of Future Changes * The persistence naming will be changed to avoid using package and class names in persisted state. This will improve backwards compatibility support moving forwards. A transformation tool will be provided for this update. * Brooklyn networking is being revisited, to incorporate work pioneered in the Clocker and Calico projects. This will allow networking requirements to be more easily described within an application, and the appropriate networking configured (be it VPCs, subnets, security groups or NAT'ing). There is a proposal at https://docs.google.com/document/d/1IrWLWunWSl_ScwY3MRICped8eJMjQEH1FbWZJcoK0Iw for the handling of multiple networks. This lays out a change in the semantics of sensors, to make it clearer which network an ip/port applies to. * The configuration options for both Azure and WinRM (especially for password and use of NTLM) are under review, and may be changed in a future release. Please treat these configuration options as "beta". * Code that was already deprecated in AMP 2.x and AMP 3.x will be deleted (incrementally). Additional code that has been deprecated will be deleted in a future release, in line with out deprecation policy. ### New Features * Added support for the launching of Kubernetes clusters, and deploying to kubernetes clusters as a location target. The Kubernetes cluster template is a quick launch item in the AMP catalog, to deploy to a running kubernetes cluster, use a location in the following form: brooklyn.catalog: id: k8s-cluster name: "Kubernetes Cluster" itemType: location item: type: kubernetes brooklyn.config: endpoint: https://:/ identity: "guest" credential: "guest" * The ability to enable or disable networking config has been added to the riak entity through the riak.networking.internal key. * BROOKLYN-381: Added support for nested DSL - allows DSL to look up values on other entities. Particularly useful for members of clusters. For example: $brooklyn:entity(config("targetId")).attributeWhenReady("main.uri") * Improved error reporting and timeout management in the QA Framework. New tests can override the global default of 1 hour and a full log of errors is returned. Failed entities can also automatically be left running for further inspection. * Improved support for QA entities, including extended HTTP testing support with HTTP POST and GET, and self signed certificates. * BROOKLYN-380: Add DynamicFabric.includeInitialChildren config to say whether the existing children of a DynamicRegionsFabric should be treated as "members", or whether new members should be created for each initial location. * Create and attach additional volumes in AWS, OpenStack and vCloudDirector. This can be done at provisioning time by using a cusotmizer. It can also be done after the application is deployed, through an effector. To attach one or multiple volumes during provisioning, you should use: io.brooklyn.blockstore.brooklyn-blockstore:brooklyn.location.blockstore.NewVolumeCustomizer A yaml example is shown below: brooklyn.config: provisioning.properties: customizers: - $brooklyn:object: type: io.brooklyn.blockstore.brooklyn-blockstore:brooklyn.location.blockstore.NewVolumeCustomizer object.fields: volumes: - blockDevice: sizeInGb: 3 deviceSuffix: 'h' deleteOnTermination: true tags: brooklyn: br-example-val-test-1 filesystem: mountPoint: /mount/brooklyn/h filesystemType: ext3 - blockDevice: sizeInGb: 3 deviceSuffix: 'g' deleteOnTermination: true tags: brooklyn: br-example-val-test-1 filesystem: mountPoint: /mount/brooklyn/g filesystemType: ext3 To attach an effector for creating extra hdd, you should add this initializer to the yaml: brooklyn.initializers: - type: io.brooklyn.blockstore.brooklyn-blockstore:brooklyn.location.blockstore.effectors.ExtraHddBodyEffector The expected effector argument value is json map applicable to NewVolumeCustomizer's fields. For example: { "blockDevice": { "sizeInGb": 3, "deviceSuffix": "h", "deleteOnTermination": true, "tags": { "brooklyn": "br-example-test-1" } }, "filesystem": { "mountPoint": "/mount/brooklyn/h", "filesystemType": "ext3" } } Notice that the volumes will not be automatically deleted on termination. To delete volumes created from old machines you should go to the cloud provider web console. See http://docs.cloudsoft.io/locations/reference/index.html ### Bug Fixes * Zookeeper and ZookeeperEnsemble have updated zookeeper.endpoints which fix the AMP is-up check. See https://issues.apache.org/jira/browse/ZOOKEEPER-390. * AMP-1437: With high latency connections the App Inspector UI module backed up connections and caused browser errors. This has been fixed through migration to RXJS. * BROOKLYN-230: br app only outputs location details when the app has a location. This stops the command line tool erroring for apps without a location. * The QA Framework only tests root items within a catalog now rather than all items specified in the testing blueprint. * BROOKLYN-399: only looks in specific regions of AWS for AMIs significantly speeding it up. * Set proper datastore.url in ElasticSearch. There is no longer a need to specify "subnet.publiclyForwardedPorts" for the datastore.url. See https://github.com/apache/brooklyn-library/pull/67 * BROOKLYN-382: get-catalog performance improvements. Previously it would take several seconds to retrieve a catalog listing of 100+ catalog items; now the first call will take that long, and subsequent calls will be much faster. See https://github.com/apache/brooklyn-server/pull/423 * Fix reporting of HttpResponse header, particularly when using HttpExecutor. See https://github.com/apache/brooklyn-server/pull/441 * BROOKLYN-375: avoid high CPU usage due to inefficient garbage collection of soft references. This adds a JVM configuration option: -XX:SoftRefLRUPolicyMSPerMB=1 See https://github.com/apache/brooklyn-server/pull/413 * BROOKLYN-383: fix json serialisation of tasks. See https://github.com/apache/brooklyn-server/pull/424 * BROOKLYN-379: Fixes rebind error when an entity type references a pre-installed OSGi bundle. * BROOKLYN-364: more efficient Softlayer API call when retrieving instance list. This avoids an OutOfMemoryError when there are many VMs running in the Softlayer account. * BROOKLYN-352: fixed ConcurrentModificationException. This occurred rairly, when using $brooklyn:entity on a fairly complex blueprint on a cpu starved machine. * Add config maxConcurrentChildCommands to DynamicCluster. This option configures the maximum number of simultaneous Startable effector invocations that will be made on members of the group. * Disable jclouds wire logging (so http response bodies not logged). Motivated in part by BROOKLYN-364. See https://github.com/apache/brooklyn-server/pull/395 * Ensure runtime exceptions in task execution always logged. See https://github.com/apache/brooklyn-server/pull/391 * Fix log message of memory used by the JVM See https://github.com/apache/brooklyn-server/pull/384 * BROOKLYN-325 On rebind/failover when entity is starting/stopping, report as "on-fire" with explanation. Currently the entity will be set "on-fire" and will alert if provisioning/termination aborted on rebind. * Fix error caused when attempting to use a Softlayer VM which has been set up with multiple credentials. See https://github.com/cloudsoft/jclouds/commit/0248931cc191dc75a3c9d370cf6a13fdb3b75912 * Fix http over AMP Jumphost - the HTTP Response headers were not included in the response message. * Fix AMP persistence when HttpProxiedClient was used (i.e. for executing http over the jumphost). Previously, the client was not correctly persisted, which could cause problems on rebind. * Fix RPM install, so that `systemctl enable amp.service` (and disable) work. * BROOKLYN-405: Passwords in environment variables logged by brooklyn.SSH debug * BROOKLYN-396: Exception resizing DynamicCluster after rebind when members use localhost * BROOKLYN-394: "Request limit exceeded" on Amazon * BROOKLYN-386: NPE on rebind calling CreateUserPolicy.addUser * BROOKLYN-384: ZookeeperEnsemble configuration error * BROOKLYN-373: Using br against a single deployed BasicApplication can panic * Updated MariaDB version from 5.5.40 to 5.5.53. This was due to an error within MariaDB causing startup failures on CentOS 7. ### Known Issues * BROOKLYN-402: jclouds-google-compute fails to provision windows VM: json parse in ResetWindowsPassword This appears to be a change in GCE, causing a failure to parse the json response. It thus also affects previous releases (e.g. 3.5.0) where provisioning was previously successful. * The creation and attaching of volumes does not work in OpenStack Mitaka, but does work in OpenStack Liberty. * BROOKLYN-406: Coercion using fromMap(map) - exception thrown away, and value not coerced. For example, if using NewVolumeCustomizer but the yaml input has missing (or misaligned) fields, it may not convert the VolumeOptions, causing a ClassCastException instead. * The AMP UI has primarily been tested in Chrome and Firefox. These are the recommended browsers. In Safari, it has been observed that the quick launch "Add Config" does not work - there may be other unknown issues. For users of Internet Explorer, the recommendation is to use Chrome or Firefox instead; or failing that to upgrade to Microsoft Edge - but that is not officially supported. * Stock MariaDB fails on CentOS 7.2.x. As a workaround please use a later version of MariaDB by specifying the following brooklyn config for the entity: brooklyn.config: install.version: 5.5.53 * BROOKLYN-356: use of the Transformer enricher can sometimes fail to populate the target sensor, particularly on a heavily loaded AMP server. This is when "targetValue" is used with the YAML DSL, and the value is not evaluated promptly. * AMP-1254 Alien4Cloud app deployment fails when an app is composed in the A4C UI. * AMP-1249 After installing the Tosca feature in Karaf, an additional `bin/stop` and `bin/start` may be required. * AMP-1248 On some environments, the `bin/client` tool for connecting to Karaf may see: IllegalStateException: Unable to negotiate key exchange for kex algorithms The client can be accessed alternatively by using `ssh -p 8101 karaf@localhost` (with default password being karaf). * AMP-1250 The latest OpenStack (Mitaka) requires use of the location type `openstack-mitaka-nova`, rather than the vanilla `openstack-nova`. * AMP-1226 The username to use when first logging into a new VM is sometimes inferred incorrectly. The solution is to explicitly specify the username in the location's configuration, e.g. using `loginUser: ubuntu`. For further information, see http://docs.cloudsoft.io/locations/reference/#os-initial-login-and-setup and http://docs.cloudsoft.io/operations/troubleshooting/deployment.html * AMP-1213 When upgrading AMP, using existing persisted state, it will re-use the existing catalog from that persisted state and will not add new versions of the catalog items that ship with the upgraded AMP. As described in the "Upgrade Instructions", the workaround is to manually trigger the addition of those catalog items by running: br add-catalog /opt/amp/catalog/catalog.bom * AMP-1215,AMP-1226 VM provisioning can fail, e.g. with the quick launch blueprints, when using the most basic location configuration. For example, the blueprint may require a particular OS distro, but the location picks an incompatible image, or the "login user" for initially ssh'ing to the VM may be incorrectly inferred for the chosen VM image. * AMP-1235 When using an empty brooklyn.properties, if the "reload brooklyn.properties" is used (either in the classic UI or via the REST api), it reloads the HTTP login credentials. If no credentials are defined in brooklyn.properties, then this means the original default credentials will no longer work. * AMP-877 the previous `./bin/amp copy-state` command is not available in Karaf. If required, an older version of AMP 3.x can be used to execute this command. * AMP-1043 the previous `./bin/amp cloud-compute` command is not available in Karaf. If required, an older version of AMP 3.x can be used to execute this command. * AMP-883 the previous `./bin/amp generate-password` command is not available in Karaf. If required, an older version of AMP 3.x can be used to execute this command. * The new UI has some features missing that were present in the classic UI. The workaround is to access these via the classic UI, or the `br` CLI, or the REST api: * There is no "reload properties" button, to reload the brooklyn.properties * No "expunge" and "unmanage" button, to complete remove an entity. Calling the "stop" effector on a top-level app with attempt to gracefully stop and unmanage the entity, but if stop fails then unmanage will not be done. * The AMP log will occasionally report NullPointerException in org.apache.cxf.transport.http.async. These are not harmful and can be ignored. * BROOKLYN-317 VM provisioning can fail in AWS-EC2 (because api did not return public IP). The workaround is to configure machineCreateAttempts on the location, setting it to a number greater than 1. It will thus retry if the problem is encountered. * Windows blueprints on vCloud Director sometimes fail. This is because the VM customizations performed by vCloud Director can include a reboot. This sometimes does not happen until several minutes after the VM is available, which can happen mid-way through installation of the software components. A workaround is available for this behaviour. Please check waitWindowsToStart parameter http://docs.cloudsoft.io/blueprints/base-blueprints/winrm/index.html#webserviceexception-could-not-send-message * It is strongly recommended to use WinRM v4 or greater. Older versions of WinRM (e.g. v3 on Windows 2008rc2) may fail. See COMPOSEC-1200. * For Windows VMs on Azure and GCE, the "Administrator" user is disabled. * https://issues.apache.org/jira/browse/BROOKLYN-243: On restarting an entity's process via the restart effector, there is a rare race-condition where the entity is not reported at "service.isUp" true, even though the process is running. * When using BYON to deploy multiple components to a single VM, the recommended way is to use the `SameServerEntity`. If instead the same machine were to be listed multiple times, then concurrent conflicting commands could be executed on that machine. For example, if multiple `iptables` commands are executed to add rules, then these interfere so that some rules may not be applied. * The following problems have been encountered in vCloud Director: * https://github.com/cloudsoft/jclouds-vcloud-director/issues/12 Failure terminating VM in vCloud Air: InvalidCacheLoadException when looking up the VM * https://github.com/cloudsoft/jclouds-vcloud-director/issues/23 VM delete failed; seemingly it did not wait for the stop to complete. * Windows provisioning to vCloud Air failed. The guest customization defined in the vApp Template appears not to be applied. The VM has no password, requires immediate password reset, and has not had the init script executed to configure WinRM. * Timeout waiting for VM to be provisioned, e.g. failing with a message like: "not composed within 1200000 ms" The following configuration can increase this timeout: brooklyn.location.named.MyCanopyLocation.jclouds.compute.timeout.node-running=3600000 * The following are known issues with jclouds-azure: * JcloudsSecurityGroupCustomizer does not work with azure, entities such as Riak or clocker will not work with this and a fixed, pre-configured network must be specified instead. * Private VM images are not supported. This is non-trivial to support, because private images and public images are modelled very differently in Azure. * The classic REST API is used, rather than the new Azure Resource Manager REST API. This classic API will likely be deprecated; Microsoft encourage the use of the new API. * Warning when deleting a VM. The error reported was: "A disk with name ... is currently in use by virtual machine ..." Those messages are to be considered warnings as they are produced by a retry logic that attempts to delete the disk while the vm is still being deleted. * Azure does not support port ranges for public endpoints; moreover Azure limits the number of endpoints per cloud service and per VM (150 per cloud service and 50 per VM). This causes provisioning to fail if a blueprint requires a large number of firewall rules (aka public endpoints). See https://social.msdn.microsoft.com/forums/azure/en-US/4c093e33-eb86-4639-b325-c295fa8bfc5d/vm-endpoint-and-port-range and https://feedback.azure.com/forums/216843-virtual-machines/suggestions/7120093-increase-150-endpoints-limit-for-a-cloud-service for more details * In the persisted state, it is possible to have "dangling references" - e.g. the persisted state for an entity references another entity that does not exist (because that other entity is in the process of being created or deleted, for example). On startup, if the number of dangling references is too large then startup will fail. One can configure the tolerable number of dangling references. In the example below, each pair indicates for a given number of items how many dangling references are acceptable. This will be interpolated and extrapolated for other numbers of items. See the docs in https://github.com/apache/incubator-brooklyn/blob/0.8.0-incubating/utils/common/src/main/java/org/apache/brooklyn/util/collections/QuorumCheck.java#L87-#L112 For example, one could set in brooklyn.properties: rebind.failureMode.danglingRefs.minRequiredHealthy=[[0,0],[2,0],[3,1],[10,1],[100,10]] ### Backwards Compatibility For backwards compatibility with 3.x, see also the release notes for AMP 4.0.0. For backwards compatibility with 4.0.x, see also the release notes for AMP 4.1.0. There is a proposal at https://docs.google.com/document/d/1IrWLWunWSl_ScwY3MRICped8eJMjQEH1FbWZJcoK0Iw for the handling of multiple networks. This lays out a change in the semantics of sensors, to make it clearer which network an ip/port applies to. Some entities have been changed to follow this convention, and others will be changed in future releases. ### Upgrade Instructions If upgrading from 3.x, see also the release notes for AMP 4.0.0. Use of RPM and DEB is now recommended, rather than the tar.gz. CentOS 7.x is recommended, though CentOS 6 and Ubuntu 16.04 are also tested. This entirely replaces the previous install. If binding to existing persisted state, an additional command is required to update the existing catalog with the AMP 4.1 versions. Assuming AMP has been installed to /opt/amp (as is done by the RPM and DEB): br add-catalog /opt/amp/catalog/catalog.bom