# AMP Release 5.1.0

* [Introduction](#introduction)
* [New Features](#new-features)
* [Bug Fixes](#bug-fixes)
* [Known Issues](#known-issues)
* [Backwards Compatibility](#backwards-compatibility)
* [Deprecation Announcements](#deprecation-announcements)
* [Installation and Upgrade Instructions](#installation-and-upgrade-instructions)

## Introduction

This release of AMP includes:

* The new `aws-services-pack` allowing AMP to configure and manage a host of services in Amazon Web Services
  as part of application blueprints

* Major UI improvements including bundles in the catalog and responsiveness

* Over 30 other improvements such as Windows blockstore and reliability, maximum size on clusters, and OSGi and persistence fixes

More information on these changes is below.

Thank you to our customers and users for their feedback and support.
Thanks also go to the Apache Brooklyn community for the many code contributions and
evolution of ideas at the heart of this product.

For more information, visit https://cloudsoft.io/amp, https://docs.cloudsoft.io
and http://brooklyn.apache.org .

## New Features

### AMP Web Console

#### Improved UI responsiveness

Previously it could take up to 30s to populate the palette and catalog view when many items are present in the catalog.  
This release updates the AMP UI to use new REST endpoints which are much more efficient.
The code now relies on the containing bundle when working with types / catalog items throughout the UI
(in Composer, Quick Launch, Catalog, elsewhere), aligning the client-side model with the server-side approach
of making bundles the primary means of grouping catalog items.

This change should be transparent to usersm apart from dramatically reducing the population time of the palette and catalog view.


#### "Bundle centric" UI

This updates the catalog UI to be "bundle centric":

- The main view (which was previously composed of 4 tabs for applications, entities, enrichers and policies) is replaced by a simple list where all the bundles are listed. Those items are expandables to see the main types within it. The list can be toggle to see either bundles or types directly.
- A new view has been added to see bundle details. It includes a list of all types defined within the bundle with filtering/sorting available.
- The type detail view has been updated to include the bundle link.

#### UI Dashboard improvements

Search has been improved:

- if the search is a classic string, it search into the entire dataset (same as before)
- if the search start by `<column-id>:blabla`, it will search for items containing `blabla` within the `column-id`.
- the search box auto-completes for searchable columns, not hidden
- filters and search string are stored into `sessionStorage`

The table can be ordered by any column.

Choice of columns to show:

- new columns can be defined.
- hide/show columns (state saved into `sessionStorage`).
- resize feature for columns.

The kilt visualisation can now go full width as a modal pop-out.

AMP includes a `DashboardAggregator` policy, which aggregates dashboard sensors from children and merges the results
for an application. Its usage still requires that configuration or sensors are set explicitly on the child entities.

### REST API

There are new REST endpoints: `/v1/catalog/bundles` and `/v1/catalog/types`. These provide a bundle-centric
view and management. They are also more efficient than the old `/v1/catalog` endpoints.

All REST API received by the server are now logged using `org.apache.brooklyn.REST`. All calls that can make modifications
(POST, PUT, DELETE) are logged at debug, while read-only calls (GET, HEAD) are logged at trace.

REST `/applications/fetch` can request selected sensors
This allows clients to efficiently get selected sensors they are interested in.
It does this by adding a `@JsonAnyGetter` map to `EntitySummary`, as done for `TypeSummary` and elsewhere, returning data in `sensors` key.


### AMP Cluster

#### Catalog Bundles

AMP Cluster now supports adding bundles to the catalog, and deletion of bundles.

#### AMP Worker Placement Policies and Labels

AMP Workers can now have labels, and placement policies can constrain which Worker can deploy an application based on labels
on the worker and on the application being deployed. This can for instance restrict workers to specific tenants
(isolation), geographies and clouds (proximity), or workloads (dev v prod).

#### Multiple Overseers

AMP Cluster now supports "multiple overseers". This gives a hierarchy of AMPs for deploying and managing
the AMP Cluster. The AMP Cluster blueprint can be deployed to a "Global Overseer", which provisions multiple
AMP Overseers, which can then each deploy a set of AMP Workers.

This allows for greatly improved scale. For example, each AMP overseer could deploy 100s of AMP Workers,
with the Global Overseer deploying many such AMP Overseers. Thus 1000s of AMP Workers can be supported.


### AWS Services Pack

The AWS Services Pack provides entities to create and manage AWS services.

See documentation at https://docs.cloudsoft.io/blueprints/yaml-syntax/aws-services-pack/

### Windows Support

* Added Windows support for attaching additional disks. This is done using the existing
  `io.brooklyn.blockstore.brooklyn-blockstore:brooklyn.location.blockstore.NewVolumeCustomizer`
  See https://docs.cloudsoft.io/locations/misc/index.html#attaching-additional-volumes

* Support for preventing concurrent execution of WinRM install steps, by adding mutex.
  This approach is similar to how it is already handled for SSH.
  See https://github.com/apache/brooklyn-server/pull/935

### Catalog

* Allow deprecation of individual catalog items within a .bom file. Previously, deprecation worked only at the top `brooklyn.catalog` level.

* BROOKLYN-421: Catalog libraries now supports externalized config for basic-auth credentials in the url,
  for YAML-based .bom files.

### Miscellaneous

* Adds `SetLimitsCustomizer` to easily set ulimits for CentOS and RHEL.
  See https://github.com/apache/brooklyn-server/pull/894

* DynamicCluster: only resize to maxSize
  The maxSize configuration is a hard-limit for how large a cluster can grow. This includes when
  calling the `resize` effector explicitly, either manually or by a policy.

* jclouds polling/backoff: increase default delays
  The defaults now work better on AWS, polling less frequently and backing off for longer when rate-limited.

* CreatePasswordSensor now supports `password.character.groups`.
  Character groups are groups like "capital letters", "numbers", etc. This option allows a blueprint to require that the generated password contains at least one character from every group.
  It is common (e.g. Microsoft default password policy) to see a requirement like "must contain characters from at least 3 groups". Previously, this sensor could not guarantee that this would happen.
  Use it like this:

  ```yaml
        - type: org.apache.brooklyn.core.sensor.password.CreatePasswordSensor
          brooklyn.config:
            name: my.password
            password.length: 10
            password.character.groups:
             - "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
             - "abcdefghijklmnopqrstuvwxyz"
             - "1234567890"
             - "-_=+[{]};:|`~,<.>/?!@#$%^&*()"
  ```

* The list of quick launch blueprints has been updated, and now includes Alfresco content management.

* AMP Node: adds entity types for catalog additions and application deployments


## Bug Fixes

* `br` cli:

  * Handle usage only flag parameters are passed in, e.g. `br -v`

  * BROOKLYN-572: Handle directory/path arguments correctly on Windows, such as when adding a directory as a catalog bundle

* Blueprint Fixes

  * Fix NPE in CouchbaseCluster enricher which happens when the enricher is triggered with the sensor `couchbase.cluster.clusterEntities` that has a null value.

  * Update openssl prior to installing nodejs.  Changes to how nodejs/npm packaged on centos mean we need to do this.  Fix recommended [here](https://stackoverflow.com/questions/46473376/node-relocation-error-node-symbol-ssl-set-cert-cb-version-libssl-so-10-not-d)

  * Fix the launch command for the metering microservice.
    - Ensure the download url of the jar is required
    - Ensure that we create the bundle correctly

  * Fix ELK blueprint for cluster deployment of ElasticSearch: previously the URI was not advertised correctly.

  * Brooklyn Etcd Fixes
    * Fix rebind / classloading of catalog.
      Previously, if had two etcd bundles installed in Brooklyn, they both use the
      most recent one for `etcd/etcd.bom` because the classpath includes
      the bundle name. It should instead load it from the current bundle.

  * MySql Fixes
    * Improve support for MySQL binding to a pre-populated data directory with an existing password

* REST API Fixes

  * BROOKLYN-167: Fix reporting of master ID when web API used before persistence is ready.
    ServerResource#getHighAvailabilityPlaneStates obtains a new plane sync
    record when the master node ID in the first one it obtained is null.
    This generally only happens when the web-console tries to obtain the
    HA status before the server has started up.

  * Delete bundle: handle pre-installed karaf bundles.  Previously, if you tried to delete a bundle that was pre-installed in brooklyn karaf then the REST call would return a 500 response.

  * BROOKLYN-571: REST API for tasks could throw IllegalArgumentException when entity is unmanaged.

* AMP UI Fixes

  * AMP-1937: AMP UI name truncated after '#' character when renaming entity

  * AMP-1985: AMP UI navigation error with Management tab
    Previously it would redirect the user to the `Management` page, if the user was previously viewing an adjunct
    detail and switched entities

  * AMP-1907: Fix blueprint composer usability issues
    This fixes the following issues:
    - YAML validation in YAML editor was previously validating against two possible schemas (for catalog versus blueprint).
      This lead to an over-complication of error messages being shown to the user.
    - YAML auto-complete was not working properly in some cases.
      - A port range value (e.g. `8080+` or `8000-9000`) what incorrectly wrapped into double quotes.
      - When the composer didn't find a location, a warning counter was displayed but there was no message.

  * AMP-1958: Don't automatically tail 'env' and 'stdin' streams

  * Location bug fixes in Blueprint composer.
    - When typing freeform, the location was not properly assigned
    - When using drag&drop, the location was not properly assigned due to the format of the underlying object
    - When flipping YAML => graphical with a location that is a simple string but not present in the catalog, the composer was showing an error. It now shows a warning rather than an error with a specific `snackbar` upon deployment saying that the location is not in the local catalog. A user can now choose to click on deploy anyway and bypass the warning (which was not allowed before)

  * fix tasks passed to activity detail view so kilt is full.  Activity details view only showed 1 level, unlike root activities view which showed a nice kilt.

  * BROOKLYN-561: coerce empty string to URI as null

* AMP Cluster Fixes

  * Fix errors with `amp-cluster catalog add` when run on Windows

  * Fix deploy: do not save app if failing with 500 because no AMP Worker available

  * AMP-2005: amp-cluster catalog: replay catalog modification on newly added AMP Worker

* Persistence / rebind Fixes

  * BROOKLYN-566, AMP-1982: Rebind fails for historic state of SaltEntity (from 0.11.0)

  * BROOKLYN-569, AMP-1942: Aggregator enricher could be lost on rebind (due to ClassCastException)

  * BROOKLYN-568, AMP-1983: Rebind failed: NPE deserializing SshFeed

  * BROOKLYN-567, AMP-1984: StackOverflowError rebinding to DynamicCluster (after many added/deleted members)

  * BROOKLYN-549, AMP-2008: Rebind policy fails when uses DSL to reference owning entity

  * AMP-1960: force bundle upgrade fails to upgrade `brooklyn.libraries` reference

  * BROOKLYN-545: Rebind failure after using cluster with `enableAvailabilityZones`

* Other Fixes

  * For Jumphost when using a message handler customizer with WinRM, it was not being passed all metadata

  * BROOKLYN-574: race condition where LoopOverGroupMembersTestCase failed even though child entities succeeded

  * BROOKLYN-573, AMP-2040: Repeater logs exception stacktrace many times (at debug)

  * AMP-1963: ElectPrimaryPolicy was not listed in catalog

  * Fix VanillaSoftwareProcess stop and then start when ssh monitoring was disabled. Previously, the entity would not be marked as 'healthy' after start.

  * Fix mvn config in karaf, for installing .kar artifacts by placing them in the deploy repo.

  * BROOKLYN-576, AMP-2037: reduce the excessive debug logging

  * BROOKLYN-570: Brooklyn hangs with custom java entity synchronizing on self

  * Fix SSH environment variables of type `Duration`. Now rendered as for example `5m`,
    whereas previously it was rendered as the json `{"delay": {"nanos": 30000000}}`.

  * memberSpec configuration is no longer re-inherited.
    This is important if using a cluster of clusters, for example.

## Known Issues

See Apache Brooklyn JIRA for list of known issues relating to Brooklyn and thus AMP.

Some issues are called out below, with suggested workarounds:

* BROOKLYN-559: on restart/rebind, the backup directory for persisted state (created automatically)
  may be missing some bundle JAR files

  The AMP upgrade instructions recommend making a manual backup copy of persisted state,
  and not use the automatically created backup directory. If this is done the bug will not affect users.

  If it is desired to revert to automatically created backup copies of persisted state,
  any extra bundles that had been installed to AMP these should be copied from the regular
  persisted state `bundles/` folder into the backup copy being used.
  Alternatively references to any such bundles can be added to the initial catalog as `brooklyn.libraries`.

* BROOKLYN-555, BROOKLYN-556, BROOKLYN-557: for some software process entities, calling stop to
  terminate the VM and then calling start may break some entities.

  If terminating the VM causes problems, it is recommended to instead replace the entity.

* BROOKLYN-550: some blueprints can break if AMP is running as a user whose name
  matches a pre-existing privileged user on machines it is provisioning.

  It is recommended to run AMP as an `amp` user or other normal user,
  and not to run it as `ec2-user`, `centos`, `ubuntu` or `root`.
  Alternatively locations can specify an explicit `user: ...` config.


## Backwards Compatibility

* From 5.0.0 onwards, AMP will always first load the "initial catalog" (from the `etc/default.catalog.bom`
  file). If persisted state contains bundle (or legacy catalog items), these are then added.

* When rebinding to old persisted state that includes Java code in bundles compiled against old version(s)
  of AMP, it is important to forcibly replace these with new versions of the bundles compiled against
  the latest AMP. This is described in more detail in the upgrade instructions.

* `org.apache.brooklyn.util.core.mutex.WithMutexes.acquireMutex()` no longer throws a checked
  `InterruptedException`, but instead throws a `RuntimeInterruptedException`. Code that calls
  this (internal) method and that catches or expects the `InterruptedException` will need to be
  modified.

* The REST endpoints `/v1/catalog/bundles` and `/v1/catalog/types` are new to 5.1.0.
  Scripts that use this will not be compatible with older versions of AMP.

* For backwards compatibility with 3.x and 4.x, please see the intermediate release notes.

## Deprecation Announcements

Support for the Docker Swarm blueprint has been withdrawn. This blueprint was outdated, not following
the latest best practices for deploying Docker Swarm.

## Installation and Upgrade Instructions

To install see https://docs.cloudsoft.io/operations/production-installation.html

To upgrade existing installations see https://docs.cloudsoft.io/operations/upgrades/