# AMP Release 4.4.0 * Introduction * Alert of Future Changes * New Features * Bug Fixes * Known Issues * Backwards Compatibility * Upgrade Instructions ### Introduction Thank you to our community who have contributed a lot of improvements and feedback! Thanks also go to Brooklyn's commercial users who have funded this development and made some major contributions. For more information, please visit http://www.cloudsoft.io/amp-4, https://docs.cloudsoft.io and http://brooklyn.apache.org This is a major upgrade from AMP version 4.3.0. ### Alert of Future Changes * The classic UI will be deleted in a future AMP release. Use of the new AMP web-console, added in AMP 4.0, is strongly recommended. * The use of `~/.brooklyn/brooklyn.properties` is discouraged - use of `/etc/amp/brooklyn.cfg` is instead recommended. In a future major release, support for the former will likely be removed. * The persistence naming will be changed to avoid using package and class names in persisted state. This will improve backwards compatibility support moving forwards. A transformation tool will be provided for this update. * Brooklyn networking is being revisited, to incorporate work pioneered in the Clocker and Calico projects. * This allows networking requirements to be more easily described within an application, and the appropriate networking configured (be it VPCs, subnets, security groups or NAT'ing). Currently, this networking feature works only on some clouds, i.e. AWS and OpenStack. Future releases will expand this support to other clouds. * There is a proposal at https://docs.google.com/document/d/1IrWLWunWSl_ScwY3MRICped8eJMjQEH1FbWZJcoK0Iw for the handling of multiple networks. This lays out a change in the semantics of sensors, to make it clearer which network an ip/port applies to. * The configuration options for both Azure and WinRM (especially for password and use of NTLM) are under review, and may be changed in a future release. Please treat these configuration options as "beta". * Code that was already deprecated in AMP 2.x and AMP 3.x will be deleted (incrementally). Additional code that has been deprecated will be deleted in a future release, in line with out deprecation policy. * Code using `ManagementContext.getConfig()` to modify the management context's config should migrate to use the new scratchpad area, as the `getConfig()` method will become read-only in a future release. Note that the scratchpad is not reset on reloading `brooklyn.properties`. ### New Features * Blueprints for deploying AMP instances have been revamped. * The catalog item for deploying an AMP instance has been rewritten as a pure-yaml blueprint (named "amp-node"). Use of the old org.apache.brooklyn.entity.brooklynnode.* is strongly discouraged (e.g. it does not correctly handle karaf deployments). * A new catalog item has been added for deploying a set of independent AMP nodes (named "amp-cluster"). * AMP Networking: This feature allows application designers to model their security requirements directly in blueprints, specifying security "networks" to which entities belong, and the permitted access from network to network. AMP takes care of creating an implementation enforcing these restrictions using the mechanisms of the underlying cloud. See [the documentation](https://docs.cloudsoft.io/blueprints/yaml-syntax/networking/index.html). Currently, this networking feature works only on some clouds, i.e. AWS and OpenStack. Future releases will expand this support to other clouds. * Added `org.apache.brooklyn.feed.windows.CmdFeed` analogically to `org.apache.brooklyn.feed.ssh.SshFeed`. Provides a feed of attribute values, by polling windows commands over WinRM. * Added `org.apache.brooklyn.core.sensor.windows.WinRmCommandSensor` initializer. It can create a sensor by polling a Windows command and obtaining its stdout as a sensor string value. Can be used in YAML, see https://docs.cloudsoft.io/blueprints/syntax-yaml-reference.html and https://docs.cloudsoft.io/blueprints/java/feeds.html#windows-performance-counter-feed * Network security groups can be created automatically during deployment and adding multiple provisioned VMs to that group when they are configured with `org.apache.brooklyn.location.jclouds.networking.SharedLocationSecurityGroupCustomizer`. The customizer works only for Azure ARM and AWS. see https://docs.cloudsoft.io/locations/location-customizers.html * BROOKLYN-433: Improved support for config key constraints in YAML blueprints. It now supports regular expressions as well as the use of `$brooklyn:object` to create a Predicate. See https://docs.cloudsoft.io/blueprints/syntax-entity-configuration.html * BROOKLYN-421: Add DSL for `$brooklyn:urlEncode(...)`. See https://docs.cloudsoft.io/blueprints/syntax-yaml-reference.html#dsl-commands * Config key of type `List` and `Map` that are using DSL functions for their keys/values failed to resolve and coerce the value in certain cases. This fixes it. * Added the `JcloudsLocation` configuration option `connectivityResolver`, whose value should be an instance of `org.apache.brooklyn.location.jclouds.ConnectivityResolver`. This is used to determine the address and credentials with which a VM should be contacted. The default it to behave as per previous AMP releases. However, it can be used to control whether to prefer a VM's private address etc on a per-entity basis. * QA Framework: Add `additionalBrooklynConfig` to `TestScheduler`. When the effector is called a user can now specify optional additionalBrooklynConfig as a map. These will be added as `brooklyn.config` when running the test. * Create and attach additional volumes in Azure ARM. This can be done at provisioning time by using a customizer. It can also be done after the application is deployed, through an effector. See https://docs.cloudsoft.io/locations/misc/index.html#attaching-additional-volumes Note that the volumes will not be automatically deleted on termination. To delete volumes created from old machines you should go to the cloud provider web console. * Web-console improvements: * The web-console includes a logout feature through basic authentication, similar to how the classic UI supported logout. * Background tasks are now displayed within the activity tab in the App inspector module. A new filter has also been created to display background tasks only. * The web-console has a new module called `Blueprint importer` that allows a user to add catalog items that are in the .bom yaml format. * The web-console has a new module called `Groovy console` that allows a user to execute arbitrary groovy code at runtime. This is useful for debugging purposes. * The web-console has a new module called `REST API Docs` that allows a user to see and test REST endpoints provided by AMP. This features if provided through Swagger UI. * The web-console has better error handling when deploying blueprints from the `Blueprint Composer`. For example, there is a Java stacktrace visualiser. * The web-console can now display external UI modules. This can be dynamically added via an enricher or Karaf configuration. * The `Catalog` module now supports the visualisation of versioned catalog items. * The "about" page also provides information about the HA status of AMP. * For modifying a management context's config, there is a new scratchpad area, see org.apache.brooklyn.api.mgmt.ManagementContext.getScratchpad(). * Added `DurationPredicates` which contains `Predicates` for common `Duration` comparison operations. * The `Repeater` utility class is configurable via `.threaded()`, to execute the repeated job in a different thread. This allows a timeout to be enforced, even if the job does not complete in a timely manner. * Upgraded Karaf to version 4.0.8 (previously version 4.0.4). ### Bug Fixes * Patch jclouds azurecompute-arm provider, to incorporate recent fixes made in jclouds. These include: * Fix generateStorageAccountName: the storageAccountName was used instead of the sanitizedStorageAccountName, to avoid the use of illegal characters. * JCLOUDS-1231: Implement the SecurityGroupExtension and the inbound-ports feature. * Cleanup legacy code and introduce the resource group cache: See https://github.com/cloudsoft/jclouds-labs/commit/6fcea4dcef26976779c3e2eb62829bf7c37e3baf * JCLOUDS-1229: Enable the rate limit module by default, to retry if rate-limited. (Previously, this was worked around in AMP 4.3 to explicitly enable it). * Inspect all known OSFamilies in VMImageToImage. * Performance improvements misc: improve cleanup performances by using resourceGroupApi.resources() API. * Cleanup orphaned availability sets. * Adds new available Azure locations, according to https://azure.microsoft.com/en-ca/regions/ * Fix `NullPointerException` in ARM `VirtualMachineToNodeMetadata`. * `AzureComputeSecurityGroupExtension`: Checks existence of node before getting SecurityGroups and Images. * `AzureComputeSecurityGroupExtension`: Better rules creation and explicitly test for the ALL protocol. * BROOKLYN-434: Workaround for sshd configuration reload on CentOS 7 and RHEL 7. See https://bugzilla.redhat.com/show_bug.cgi?id=1381997 * Update CouchDB Node to support CouchDB 2.0+. See https://github.com/apache/brooklyn-library/pull/92 * For AWS windows provisioning, set the username correctly. Previously, the machine location's username was empty. See https://github.com/apache/brooklyn-server/pull/580 * BROOKLYN-323: Use http WWW-Authorization header properly in karaf. * BROOKLYN-452: Brooklyn may fail to deploy with certain CentOS 7 images A change in CentOS 7 prevented normal deployments if Brooklyn was configured not to use jClouds for the initial ssh-based setup. This was due to a change in the sudo configuration on RHEL CentOS 7 images. * Better error reporting for the REST API. Populate error field of `ApiError`, and better logging on REST errors. See https://github.com/apache/brooklyn-server/pull/571 * Apply class renames mapping to white-listed bundles. The rename was used for loading from app classpath, but not for the white-list fallbacks. See https://github.com/apache/brooklyn-server/pull/569 * Fix application start problems. A bug was introduced where if there are errors in a custom `Application.doStart(...)`, the start effector fails but the application does not. This corrects that using a special exception to allow children problems to pass through. See https://github.com/apache/brooklyn-server/pull/568 * Background tasks published by sensors now have a name with relevant information instead of being blank. See https://github.com/apache/brooklyn-server/pull/561 * Fix json serialization (e.g. for REST api responses) of `Multimap` for non-string keys. See https://github.com/apache/brooklyn-server/pull/553 * Fix an issue in `SecurityGroupEditor` to allow for different jclouds prefixes on different clouds. See https://github.com/apache/brooklyn-server/pull/549 * BROOKLYN-425: Rebinding to `MachineEntity` failed to reconnect sensor feeds. This fixes the issue. * Fix unexpected `AddChildrenEffector` behaviour. https://github.com/apache/brooklyn-server/pull/521 * BROOKLYN-267: brooklyn.parameter default value now picked up via inherited config. * Brooklyn Client: Add flag for connecting to servers using self-sign SSL. * Container service: Set generated deployment id to lower case. * Container service: Add new `KubernetesResource` entity for loading YAML directly * Clocker: Fix HAProxy configureEndpoints effector. The HAProxy entity configures its targets by executing an effector called configureEndpoints. Previously it did not check that HAProxy was actually installed before attempting to restart the process. This could lead to an infinite wait for a non-existent process to die. * Clocker: Remove dependency on external resources. This let the Kubernetes catalog entries load when there is no external (public Internet) connectivity available to AMP. * Update `hello-world-sql` example webapp to support Redis datastore. * Generalised `Entities.waitFor()`. Provides general-purpose blocking wait on any entity predicate. * Fix jBoss logo in the UI quick launch. The correct resource name is now jboss-logo.png * The way DSL (i.e. `$brooklyn:`) in entity configuration is evaluated has been improved, and made more consistent. The biggest change is how this works for configuration inheritance (both in sub-types and runtime management hierarchy). There were previously inconsistencies in whether the default value in `brooklyn.parameters` would be picked up. The DSL is now always evaluated in the context of the entity that declares the config value (rather that the entity that uses the key, in the case of runtime management inheritance). See http://docs.cloudsoft.io/blueprints/syntax-entity-configuration.html ### Known issues * BROOKLYN-431: br cli does not show error message when authentication fails * BROOKLYN-450: Brooklyn deployments fail on Azure ARM when VM hostname is changed Microsoft Azure Linux agent detects if machine's hostname is changed, initiates an update against the Azure cloud and restarts the machine's network scripts. It can lead to network disruptions and failing deployments, because of an inability for installing software packages from remote repositories and/or downloading of required assets from remote locations. You can observe messages like `Could not resolve host` in the error log. The proposed workaround is to configure the Azure Linux agent not to check for hostname changes and prevent the network restart. It can be achieved via changing the agent's configuration available in `/etc/waagent.conf` and setting the required parameter `Provisioning.MonitorHostName` from `y` to `n`. Azure Linux agent needs to be restarted after the change by executing its service script with `restart` option. Such restart for RedHat-based distributions (RHEL or CentOS) is possible with the following commands RedHat / CentOS 6 - `service waagent restart` RedHAt / CentOS 7 - `systemctl restart waagent` AMP can automate it either with [setup.script](https://docs.cloudsoft.io/locations/reference/#os-setup) or by using `pre.install.command` like ``` brooklyn.config: pre.install.command: | if [ -f /etc/waagent.conf ]; then sudo sed -i.bak 's/Provisioning\.MonitorHostName\=y/Provisioning\.MonitorHostName\=n/g' /etc/waagent.conf sudo service waagent restart fi ``` * BROOKLYN-454: DSL config evaluation fails for inherited entity * The creation and attaching of volumes does not work in OpenStack Mitaka. See https://docs.cloudsoft.io/locations/misc/index.html#attaching-additional-volumes * The AMP UI has primarily been tested in Chrome and Firefox. These are the recommended browsers. In Safari, it has been observed that the quick launch "Add Config" does not work - there may be other unknown issues. For users of Internet Explorer, the recommendation is to use Chrome or Firefox instead; or failing that to upgrade to Microsoft Edge - but that is not officially supported. * On some environments, the `bin/client` tool for connecting to Karaf may see: IllegalStateException: Unable to negotiate key exchange for kex algorithms The client can be accessed alternatively by using `ssh -p 8101 karaf@localhost` (with default password being karaf). * The username to use when first logging into a new VM is sometimes inferred incorrectly. The solution is to explicitly specify the username in the location's configuration, e.g. using `loginUser: ubuntu`. For further information, see https://docs.cloudsoft.io/locations/reference/#os-initial-login-and-setup and https://docs.cloudsoft.io/operations/troubleshooting/deployment.html * When upgrading AMP, using existing persisted state, it will re-use the existing catalog from that persisted state and will not add new versions of the catalog items that ship with the upgraded AMP. As described in the "Upgrade Instructions", the workaround is to manually trigger the addition of those catalog items by running: br catalog add /opt/amp/catalog/catalog.bom * VM provisioning can fail, e.g. with the quick launch blueprints, when using the most basic location configuration. For example, the blueprint may require a particular OS distro, but the location picks an incompatible image, or the "login user" for initially ssh'ing to the VM may be incorrectly inferred for the chosen VM image. * the previous `./bin/amp copy-state` command is not available in Karaf. If required, an older version of AMP 3.x can be used to execute this command. * the previous `./bin/amp generate-password` command is not available in Karaf. If required, an older version of AMP 3.x can be used to execute this command. * There is no "reload properties" button in the new UI to reload the brooklyn.properties * The AMP log will occasionally report `NullPointerException` in `org.apache.cxf.transport.http.async`. These are not harmful and can be ignored. * Windows blueprints on vCloud Director sometimes fail. This is because the VM customizations performed by vCloud Director can include a reboot. This sometimes does not happen until several minutes after the VM is available, which can happen mid-way through installation of the software components. A workaround is available for this behaviour. Please check waitWindowsToStart parameter https://docs.cloudsoft.io/blueprints/base-blueprints/winrm/index.html#webserviceexception-could-not-send-message * It is strongly recommended to use WinRM v4 or greater. Older versions of WinRM (e.g. v3 on Windows 2008rc2) may fail. * For Windows VMs on Azure and GCE, the "Administrator" user is disabled. * BROOKLYN-243: On restarting an entity's process via the restart effector, there is a rare race-condition where the entity is not reported at "service.isUp" true, even though the process is running. * When using BYON to deploy multiple components to a single VM, the recommended way is to use the `SameServerEntity`. If instead the same machine were to be listed multiple times, then concurrent conflicting commands could be executed on that machine. For example, if multiple `iptables` commands are executed to add rules, then these interfere so that some rules may not be applied. * The following problems have been encountered in vCloud Director: * https://github.com/cloudsoft/jclouds-vcloud-director/issues/42 When rate-limited (getting a OPERATION_LIMITS_EXCEEDED response), the request is automatically retried with an exponential back-off. However, the command may have partially completed. This means an artifact with the given may have been partially created, causing a "duplicate name" error when retrying. * https://github.com/cloudsoft/jclouds-vcloud-director/issues/12 Failure terminating VM in vCloud Air: InvalidCacheLoadException when looking up the VM * https://github.com/cloudsoft/jclouds-vcloud-director/issues/23 VM delete failed; seemingly it did not wait for the stop to complete. * Windows provisioning to vCloud Air failed. The guest customization defined in the vApp Template appears not to be applied. The VM has no password, requires immediate password reset, and has not had the init script executed to configure WinRM. * Timeout waiting for VM to be provisioned, e.g. failing with a message like: "not composed within 1200000 ms" The following configuration can increase this timeout: brooklyn.location.named.MyVcdLocation.jclouds.compute.timeout.node-running=3600000 * The following are known issues with Azure Compute Classic: * JcloudsSecurityGroupCustomizer does not work with Azure, entities such as Riak or clocker will not work with this and a fixed, pre-configured network must be specified instead. * Private VM images are not supported. This is non-trivial to support, because private images and public images are modelled very differently in Azure. * Warning when deleting a VM. The error reported was: A disk with name ... is currently in use by virtual machine ... Those messages are to be considered warnings as they are produced by a retry logic that attempts to delete the disk while the VM is still being deleted. * Azure Compute Classic does not support port ranges for public endpoints; moreover it limits the number of endpoints per cloud service and per VM (150 per cloud service and 50 per VM). This causes provisioning to fail if a blueprint requires a large number of firewall rules (aka public endpoints). See https://social.msdn.microsoft.com/forums/azure/en-US/4c093e33-eb86-4639-b325-c295fa8bfc5d/vm-endpoint-and-port-range and https://feedback.azure.com/forums/216843-virtual-machines/suggestions/7120093-increase-150-endpoints-limit-for-a-cloud-service for more details. * The following are known issues with Azure ARM: * The following additional keys need to be specified in the location in order for it to work (substituting your choice of publisher and region): ``` jclouds.azurecompute.arm.publishers: OpenLogic region: southeastasia loginUser: amp templateOptions: overrideAuthenticateSudo: true ``` The publishers is any item from the list available here: https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-linux-cli-ps-findimage The region is any region from the list available here: https://azure.microsoft.com/en-us/regions/ The `loginUser` can be anything, as long as it's specified. * VM provisioning and deletion via jclouds can be very slow on ARM, particularly if there are many images to choose from and/or many storage accounts (i.e. many volumes). Setting a single publisher speeds this up considerably. * An issue has been identified with the upgrade to jclouds 2.0 which prevents the deployment of Windows blueprints to Azure Classic and Azure ARM. This issue only affects Windows blueprints * ARM rate limiting can introduce significant delays, particularly while performing concurrent actions. Rate limiting is applied at the account level, so different users of the same Azure account can cause limits to be applied to each other. For more information see the Azure ARM documentation here: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-request-limits One area which can improve multiple user access of Azure ARM is to increase the number of times AMP retries a request before failing. This can be set in the location or provisioning properties using the following key: jclouds.max-retries: 100 Also see https://issues.apache.org/jira/browse/JCLOUDS-1230 for a discussion of reducing the number of API calls. * In the persisted state, it is possible to have "dangling references" - e.g. the persisted state for an entity references another entity that does not exist (because that other entity is in the process of being created or deleted, for example). On startup, if the number of dangling references is too large then startup will fail. One can configure the tolerable number of dangling references. In the example below, each pair indicates for a given number of items how many dangling references are acceptable. This will be interpolated and extrapolated for other numbers of items. See the docs in https://github.com/apache/incubator-brooklyn/blob/0.8.0-incubating/utils/common/src/main/java/org/apache/brooklyn/util/collections/QuorumCheck.java#L87-#L112 For example, one could set in `/etc/amp/brooklyn.cfg`: rebind.failureMode.danglingRefs.minRequiredHealthy=[[0,0],[2,0],[3,1],[10,1],[100,10]] ### Backwards Compatibility * AMP 4.4 includes significant improvements to DSL evaluation and config inheritance. This may require some upgrades to larger YAML blueprints which rely upon this. The most impactful change is to the evaluation of `$brooklyn` DSL and config inheritance. For further information on these improvements, please refer to: https://docs.cloudsoft.io/blueprints/syntax-entity-configuration.html * The `EntityFactory` (deprecated in AMP 3) has been deleted. Instead, use `EntitySpec`. * Support for the "AMP of AMPs" pattern has been withdrawn. The main purpose of those blueprints was to support a very specific multi-tenancy use-case, where different tenants would be routed to different AMP instances. However, this feature was insufficient to support the requirements that come from the wider AMP community. This feature is being replaced with new "AMP Cluster" functionality, which will better support the requirements demanded by customers (such as scalability). * As described under "Alert of Future Changes", code using `ManagementContext.getConfig()` to modify the management context's config should migrate to use the new scratchpad area, as the `getConfig()` method will become read-only. * The link to the classic UI has been removed from AMP. It is still available at ${BASE_URI}/amp-classic-ui/ Use of the classic UI **is strongly discouraged**. It will be deleted in a future release. For backwards compatibility with 3.x, see also the release notes for AMP 4.0.0. For backwards compatibility with 4.0.x or 4.1.x, see also the release notes for AMP 4.1.0, 4.2.0 and 4.3.0. ### Upgrade Instructions See detailed upgrade instructions at https://docs.cloudsoft.io/operations/upgrade.html Note that from AMP 4.3 onward the RPM and DEB packages use the standard file system layout for persisted state and log files: - By default, persisted state is located at `/var/lib/amp`. - The `persistenceDir` and `persistenceLocation` are configured in the file `/etc/amp/org.apache.brooklyn.osgilauncher.cfg`. - The persistence details will be logged in `/var/log/amp/amp.info.log` at startup time. Also, for the tar.gz install the log is now in `./data/log/*` rather than `./log/*`. Note also that from AMP 4.3 onward only Java SE 8 is supported, therefore existing Java SE 7 installations should be upgraded to version 8. Use of RPM and DEB is now recommended, rather than the tar.gz. CentOS 7.x is recommended, though CentOS 6 and Ubuntu 16.04 are also tested. If binding to existing persisted state, an additional command is required to update the existing catalog with the latest AMP versions. Assuming AMP has been installed to /opt/amp (as is done by the RPM and DEB): br add-catalog /opt/amp/catalog/catalog.bom