AMP Release 4.3.0 * Introduction * Alert of Future Changes * New Features * Bug Fixes * Known Issues * Backwards Compatibility * Upgrade Instructions ### Introduction Thank you to our community who have contributed a lot of improvements and feedback! Thanks also go to Brooklyn's commercial users who have funded this development and made some major contributions. For more information, please visit http://www.cloudsoft.io/amp-4, http://docs.cloudsoft.io and http://brooklyn.apache.org This is a major upgrade from AMP version 4.2.0. ### Alert of Future Changes * The use of `~/.brooklyn/brooklyn.properties` is discouraged - use of `/etc/amp/brooklyn.cfg` is instead recommended. In a future major release, support for the former will likely be removed. * The persistence naming will be changed to avoid using package and class names in persisted state. This will improve backwards compatibility support moving forwards. A transformation tool will be provided for this update. * Brooklyn networking is being revisited, to incorporate work pioneered in the Clocker and Calico projects. This will allow networking requirements to be more easily described within an application, and the appropriate networking configured (be it VPCs, subnets, security groups or NAT'ing). There is a proposal at https://docs.google.com/document/d/1IrWLWunWSl_ScwY3MRICped8eJMjQEH1FbWZJcoK0Iw for the handling of multiple networks. This lays out a change in the semantics of sensors, to make it clearer which network an ip/port applies to. * The configuration options for both Azure and WinRM (especially for password and use of NTLM) are under review, and may be changed in a future release. Please treat these configuration options as "beta". * Code that was already deprecated in AMP 2.x and AMP 3.x will be deleted (incrementally). Additional code that has been deprecated will be deleted in a future release, in line with out deprecation policy. ### New Features * AMP now includes jclouds 2.0.0. jclouds 2.0.0 represents a major milestone for the project, providing support for new compute providers and new blob store providers. See https://jclouds.apache.org/releasenotes/2.0.0/ * AMP now supports Azure Resource Manager (ARM) locations. See http://docs.cloudsoft.io/locations/reference/index.html#azure-compute-arm See "Known issues" for caveats. * AMP now supports OpenStack Swift version 2. See http://docs.cloudsoft.io/operations/persistence/index.html#object-store-persistence * AMP Container Service now supports deploying apps to Red Hat OpenShift. See http://docs.cloudsoft.io/ccs/locations/index.html#openshift-location * Default security: on a fresh installation, when no security configuration is supplied, AMP's web-console and REST api uses a default username and password (admin:password). See http://docs.cloudsoft.io/tutorials/tutorial-get-amp-running.html#setting-web-credentials * Web-console improvements: * The web-console has a new "about" page, which gives extra version information. * When connecting to a standby AMP instance, additional information is shown in the HA status window. * Improvements to RPM and DEB packages. * The amp service **is not started automatically** upon installation. * Install each version of AMP into its own folder `/opt/amp-`. This means that all data and binaries are kept, so that rollback is possible. * Symlink `/opt/amp` to `/opt/amp-` currently installing/upgrading * Configuration files are now installed under `/etc/amp`. Upon update, conflicts will be handled as follow: * for RPM, if a file has been edited and the new version is different that the base old version, the user edited version will be kept as-is and the new version will be written at `.rpmnew`. Otherwise, the new version is applied. This will be reported in the console when doing the `yum update ` * for DEB, the `dpkg` tool will ask the user what to do * Persistence data are written on `/var/lib/amp` * Log files are stored in `/opt/amp-/data/log`. A symlink from `/var/log/amp` to this directory is also created. * Home directory of the amp user uses the default path, given by the OS, rather than /opt/amp See http://docs.cloudsoft.io/operations/paths.html for details. * AMP vagrant now uses CentOS 7, rather than Ubuntu. * AMP Container Service: deploying to Kubernetes now supports cert-data and cert-files (url, classpath or file). See http://docs.cloudsoft.io/ccs/locations/index.html#kubernetes-location * AMP Container Service: Adds docker container environment config key. Adds a new configuration key `docker.container.environment` that is set on entities, and used by `JcloudsDockerLocation` and `KubernetesLocation` to set environment variables to be passed to a Docker container at startup. See http://docs.cloudsoft.io/ccs/locations/index.html#docker-container-location * AMP Container Service: For the `cloudsoft/*` images in Docker Hub, AMP now generates a random container password and sets that automatically (or alternatively a password can be supplied explicitly). * A new Karaf configuration file is supported for supplying class renames: `${KARAF_ETC_DIR}/org.apache.brooklyn.persistence.class.rename.properties`. * A new LocationEntity for creating location-specific entities. A new type that allows configuring different entities depending on the location being deployed to. Uses configuration like this: - type: org.apache.brooklyn.entity.stock.LocationEntity brooklyn.config: location.entity.spec.default: $brooklyn:entitySpec: type: traefik-load-balancer location.entity.spec.aws-ec2: $brooklyn:entitySpec: type: elastic-load-balancer location.entity.spec.gce: $brooklyn:entitySpec: type: google-load-balancer The map entries can also be class names like JcloudsLocation or ISO 3166 country codes like `UK`. * Added functionality to overwrite the private IP/hostname that AMP will use to talk to the VM with the public IP/hostname This is enabled by setting the config key `useMachinePublicAddressAsPrivateAddress: true`. * BROOKLYN-317: Change default `machineCreateAttempts` to 2 for jclouds locations. * Location of the default catalog.bom is now configurable in `etc/org.apache.brooklyn.osgilauncher.cfg`. ### Bug Fixes * BROOKLYN-418: Fix MariaDB install. * Advanced-networking had duplicate (conflicting) package names coming from different bundles; this caused vCloud Director NAT to fail in karaf. * RPM install: update permission check, because centos6 installs its files to "wheel" group. * UI fixes: * AMP-1569: Password sensor value not being blurred in the UI. * AMP-1253: Children entities' statuses are now updated dynamically in the app dashboard. * AMP-1378: Fix effector modal not showing available locations. * AMP-1429: UI: Error when entering a Boolean location configuration with default `false` in: - Blueprint composer - Quick launch - Location manager * Fix the "Edit blueprint" button on the catalog module. * AMP-1323: Add openstack mitaka icon for the location UI module. * AMP-1252: UI: fix unresolvable types presentation: - sets an icon when a entity, policy or enricher type cannot be found - adds an entry to the entity describing the problem - displays these problems on in the spec viewer * AMP-1246: Blueprint composer - Move edit/delete buttons to the node. * Disabled "Deploy" button during the deploying of a blueprint. * The "blocking task" block contains links to the block task and the associated entity. * Always display status, even if it is unknown. * Sensor filtering: fix the regex to match if a key is contained rather than strictly equals. * AMP-1521: UI menu items allow to Expunge and Unmanage Apps. * Container Service: Stop Firewalld. Some CentOS 7 images have firewalld instead of iptables (Rightscale for example), this change allows k8s to work on such images by stopping firewalld. * Container Service: when deploying a Kubernetes infrastructure, the load balancer URI now uses `host.name` for AWS. * Container Service: the etcd components now specify CentOS7 version. * AbstractMemberTrackingPolicy incorrectly called ENTITY_ADDED on group members when rebinding. * BROOKLYN-424: Use stripped version of `ScriptBytecodeAdapter.isCase`. This avoids calling (inefficient) dynamic groovy code. * BROOKLYN-422: Using AWS availability zone as location region causes IllegalStateException during deployment. The problem was introduced in AMP 4.2.0, when fixing BROOKLYN-399. * AMP-1442: Tomcat entity was failing to poll over JMX for sensor values due to missing jmx dependency in karaf. * BROOKLYN-410: rebind locations from bundles. * BROOKLYN-406: report failure if coercion's fromMap throws exception. * BROOKLYN-405: now ssh doesn't log password environment variables. * BROOKLYN-379: Fix rebind when bundle prefixes are used to load classes. * BROOKLYN-426: NoClassDefFoundError from whitelisted bundle causes catalog listing to fail. * AMP-1472: Problem provisioning Windows on GCE. This stopped working at the time of the 4.2.0 release, and is fixed by the upgrade to jclouds 2. ### Known Issues * BROOKLYN-431: br cli does not show error message when authentication fails * The creation and attaching of volumes does not work in OpenStack Mitaka. See http://docs.cloudsoft.io/locations/misc/index.html#attaching-additional-volumes * The AMP UI has primarily been tested in Chrome and Firefox. These are the recommended browsers. In Safari, it has been observed that the quick launch "Add Config" does not work - there may be other unknown issues. For users of Internet Explorer, the recommendation is to use Chrome or Firefox instead; or failing that to upgrade to Microsoft Edge - but that is not officially supported. * AMP-1248: On some environments, the `bin/client` tool for connecting to Karaf may see: IllegalStateException: Unable to negotiate key exchange for kex algorithms The client can be accessed alternatively by using `ssh -p 8101 karaf@localhost` (with default password being karaf). * AMP-1226: The username to use when first logging into a new VM is sometimes inferred incorrectly. The solution is to explicitly specify the username in the location's configuration, e.g. using `loginUser: ubuntu`. For further information, see http://docs.cloudsoft.io/locations/reference/#os-initial-login-and-setup and http://docs.cloudsoft.io/operations/troubleshooting/deployment.html * AMP-1213: When upgrading AMP, using existing persisted state, it will re-use the existing catalog from that persisted state and will not add new versions of the catalog items that ship with the upgraded AMP. As described in the "Upgrade Instructions", the workaround is to manually trigger the addition of those catalog items by running: br catalog add /opt/amp/catalog/catalog.bom * AMP-1215, AMP-1226: VM provisioning can fail, e.g. with the quick launch blueprints, when using the most basic location configuration. For example, the blueprint may require a particular OS distro, but the location picks an incompatible image, or the "login user" for initially ssh'ing to the VM may be incorrectly inferred for the chosen VM image. * AMP-877: the previous `./bin/amp copy-state` command is not available in Karaf. If required, an older version of AMP 3.x can be used to execute this command. * AMP-883: the previous `./bin/amp generate-password` command is not available in Karaf. If required, an older version of AMP 3.x can be used to execute this command. * The new UI has some features missing that were present in the classic UI. The workaround is to access these via the classic UI, or the `br` CLI, or the REST api: * There is no "reload properties" button, to reload the brooklyn.properties * The AMP log will occasionally report `NullPointerException` in `org.apache.cxf.transport.http.async`. These are not harmful and can be ignored. * Windows blueprints on vCloud Director sometimes fail. This is because the VM customizations performed by vCloud Director can include a reboot. This sometimes does not happen until several minutes after the VM is available, which can happen mid-way through installation of the software components. A workaround is available for this behaviour. Please check waitWindowsToStart parameter http://docs.cloudsoft.io/blueprints/base-blueprints/winrm/index.html#webserviceexception-could-not-send-message * It is strongly recommended to use WinRM v4 or greater. Older versions of WinRM (e.g. v3 on Windows 2008rc2) may fail. * For Windows VMs on Azure and GCE, the "Administrator" user is disabled. * BROOKLYN-243: On restarting an entity's process via the restart effector, there is a rare race-condition where the entity is not reported at "service.isUp" true, even though the process is running. * When using BYON to deploy multiple components to a single VM, the recommended way is to use the `SameServerEntity`. If instead the same machine were to be listed multiple times, then concurrent conflicting commands could be executed on that machine. For example, if multiple `iptables` commands are executed to add rules, then these interfere so that some rules may not be applied. * The following problems have been encountered in vCloud Director: * https://github.com/cloudsoft/jclouds-vcloud-director/issues/42 When rate-limited (getting a OPERATION_LIMITS_EXCEEDED response), the request is automatically retried with an exponential back-off. However, the command may have partially completed. This means an artifact with the given may have been partially created, causing a "duplicate name" error when retrying. * https://github.com/cloudsoft/jclouds-vcloud-director/issues/12 Failure terminating VM in vCloud Air: InvalidCacheLoadException when looking up the VM * https://github.com/cloudsoft/jclouds-vcloud-director/issues/23 VM delete failed; seemingly it did not wait for the stop to complete. * Windows provisioning to vCloud Air failed. The guest customization defined in the vApp Template appears not to be applied. The VM has no password, requires immediate password reset, and has not had the init script executed to configure WinRM. * Timeout waiting for VM to be provisioned, e.g. failing with a message like: "not composed within 1200000 ms" The following configuration can increase this timeout: brooklyn.location.named.MyVcdLocation.jclouds.compute.timeout.node-running=3600000 * The following are known issues with Azure Compute Classic: * JcloudsSecurityGroupCustomizer does not work with Azure, entities such as Riak or clocker will not work with this and a fixed, pre-configured network must be specified instead. * Private VM images are not supported. This is non-trivial to support, because private images and public images are modelled very differently in Azure. * Warning when deleting a VM. The error reported was: "A disk with name ... is currently in use by virtual machine ..." Those messages are to be considered warnings as they are produced by a retry logic that attempts to delete the disk while the VM is still being deleted. * Azure Compute Classic does not support port ranges for public endpoints; moreover it limits the number of endpoints per cloud service and per VM (150 per cloud service and 50 per VM). This causes provisioning to fail if a blueprint requires a large number of firewall rules (aka public endpoints). See https://social.msdn.microsoft.com/forums/azure/en-US/4c093e33-eb86-4639-b325-c295fa8bfc5d/vm-endpoint-and-port-range and https://feedback.azure.com/forums/216843-virtual-machines/suggestions/7120093-increase-150-endpoints-limit-for-a-cloud-service for more details * The following are known issues with Azure ARM: * The following additional keys need to be specified in the location in order for it to work (substituting your choice of publisher and region): ``` jclouds.azurecompute.arm.publishers: OpenLogic region: southeastasia loginUser: amp templateOptions: overrideAuthenticateSudo: true ``` The publishers is any item from the list available here: https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-linux-cli-ps-findimage The region is any region from the list available here: https://azure.microsoft.com/en-us/regions/ The `loginUser` can be anything, as long as it's specified. * VM provisioning and deletion via jclouds can be very slow on ARM, particularly if there are many images to choose from and/or many storage accounts (i.e. many volumes). Setting a single publisher speeds this up considerably. * ARM rate-limiting can introduce significant delays, particularly while performing concurrent actions. Also see https://issues.apache.org/jira/browse/JCLOUDS-1230 for a discussion of reducing the number of API calls. * In the persisted state, it is possible to have "dangling references" - e.g. the persisted state for an entity references another entity that does not exist (because that other entity is in the process of being created or deleted, for example). On startup, if the number of dangling references is too large then startup will fail. One can configure the tolerable number of dangling references. In the example below, each pair indicates for a given number of items how many dangling references are acceptable. This will be interpolated and extrapolated for other numbers of items. See the docs in https://github.com/apache/incubator-brooklyn/blob/0.8.0-incubating/utils/common/src/main/java/org/apache/brooklyn/util/collections/QuorumCheck.java#L87-#L112 For example, one could set in `/etc/amp/brooklyn.cfg`: rebind.failureMode.danglingRefs.minRequiredHealthy=[[0,0],[2,0],[3,1],[10,1],[100,10]] ### Backwards Compatibility This release only supports Java SE 8. For backwards compatibility with 3.x, see also the release notes for AMP 4.0.0. For backwards compatibility with 4.0.x or 4.1.x, see also the release notes for AMP 4.1.0 and 4.2.0. The RPM and DEB packages have been significantly improved. See "New Features" section. Of particular importance for backwards compatibility is the new location of the files (for configuration, persisted state and logging), and that the AMP service is no longer started automatically as part of install. The default username and password (for when no security configuration is supplied) have been changed to admin:password. Users are strongly encouraged to configure security. This changes will have no affect on users who have configured security. There is a proposal at https://docs.google.com/document/d/1IrWLWunWSl_ScwY3MRICped8eJMjQEH1FbWZJcoK0Iw for the handling of multiple networks. This lays out a change in the semantics of sensors, to make it clearer which network an ip/port applies to. Some entities have been changed to follow this convention, and others will be changed in future releases. Within advanced-networking, some packages have been renamed to avoid package conflicts in different OSGi bundles. The class renames are as follows. brooklyn.networking.subnet.PortForwarder : brooklyn.networking.common.subnet.PortForwarder brooklyn.networking.subnet.PortForwarderAsync : brooklyn.networking.common.subnet.PortForwarderAsync brooklyn.networking.subnet.PortForwarderAsyncImpl : brooklyn.networking.common.subnet.PortForwarderAsyncImpl brooklyn.networking.subnet.PortForwarderAsyncImpl$DeferredExecutor : brooklyn.networking.common.subnet.PortForwarderAsyncImpl$DeferredExecutor brooklyn.networking.subnet.PortForwarderClient : brooklyn.networking.common.subnet.PortForwarderClient brooklyn.networking.vclouddirector.CustomSSLSocketFactory : brooklyn.networking.vclouddirector.nat.CustomSSLSocketFactory brooklyn.networking.vclouddirector.FakeSSLSocketFactory : brooklyn.networking.vclouddirector.nat.FakeSSLSocketFactory brooklyn.networking.vclouddirector.NatPredicates : brooklyn.networking.vclouddirector.nat.NatPredicates brooklyn.networking.vclouddirector.NatService : brooklyn.networking.vclouddirector.nat.NatService brooklyn.networking.vclouddirector.PortForwardingConfig : brooklyn.networking.vclouddirector.nat.PortForwardingConfig `openstack-nova` now supports Mitaka. Use of `openstack-mitaka-nova` is deprecated (it will be automatically translated to `openstack-nova`). `jclouds:swift` no longer exists. Instead use `jclouds:openstack-swift`. For Softlayer, this requires an additional config options (see http://docs.cloudsoft.io/operations/persistence/index.html#object-store-persistence) Several AMP dependencies have been upgraded: * The fabric8.io library has been upgraded to 1.4.27 (from 1.4.20), used for deploying to Kubernetes and OpenShift. * Updates Google Guava to 18.0 (from 16.0.1) * Updates FreeMarker to 2.3.25 (from 2.3.22), which adds support for iterating over map entries. * Updates Angular to 1.6.1. BROOKLYN-317: When provisioning VM(s) in a jclouds location, the default `machineCreateAttempts` has been changed from 1 to 2. This means that if the first attempt fails (e.g. the VM is dead-on-arrival, or the location is misconfigured causing failures), a second attempt will be made to create a new VM. Use of the location configuration `machineCreateAttempts: 1` will restore the original behaviour. Within the Container Service for deploying apps into Kubernetes: * packages have been renamed from `io.cloudsoft.amp.container.kubernetes.*` to `io.cloudsoft.amp.containerservice.kubernetes.*`. * Config key names have been changed to remove the "kubernetes" prefix. See http://docs.cloudsoft.io/ccs/locations/index.html#kubernetes-location ### Upgrade Instructions See detailed upgrade instructions at http://docs.cloudsoft.io/operations/upgrade.html Note that from AMP 4.3 onward the RPM and DEB packages use the standard file system layout for persisted state and log files: - By default, persisted state is located at `/var/lib/amp`. - The `persistenceDir` and `persistenceLocation` are configured in the file `/etc/amp/org.apache.brooklyn.osgilauncher.cfg`. - The persistence details will be logged in `/var/log/amp/amp.info.log` at startup time. Also, for the tar.gz install the log is now in `./data/log/*` rather than `./log/*`. Note also that from AMP 4.3 onward only Java SE 8 is supported, therefore existing Java SE 7 installations should be upgraded to version 8. If upgrading from 3.x, see also the release notes for AMP 4.0.0. Use of RPM and DEB is now recommended, rather than the tar.gz. CentOS 7.x is recommended, though CentOS 6 and Ubuntu 16.04 are also tested. This entirely replaces the previous install. If binding to existing persisted state, an additional command is required to update the existing catalog with the latest AMP versions. Assuming AMP has been installed to /opt/amp (as is done by the RPM and DEB): br add-catalog /opt/amp/catalog/catalog.bom