PendingReleaseNotes

12.2.13
-------
* Ceph now packages python bindings for python3.6 instead of
  python3.4, because EPEL7 recently switched from python3.4 to
  python3.6 as the native python3. see the `announcement <https://lists.fedoraproject.org/archives/list/epel-announce@lists.fedoraproject.org/message/EGUMKAIMPK2UD5VSHXM53BH2MBDGDWMO/>_`
  for more details on the background of this change.

* We now have telemetry support via a ceph-mgr module. The telemetry module is
  absolutely on an opt-in basis, and is meant to collect generic cluster
  information and push it to a central endpoint. By default, we're pushing it
  to a project endpoint at https://telemetry.ceph.com/report, but this is
  customizable using by setting the 'url' config option with::

    ceph telemetry config-set url '<your url>'

  You will have to opt-in on sharing your information with::

    ceph telemetry on

  You can view exactly what information will be reported first with::

    ceph telemetry show

  Should you opt-in, your information will be licensed under the
  Community Data License Agreement - Sharing - Version 1.0, which you can
  read at https://cdla.io/sharing-1-0/

  The telemetry module reports information about CephFS file systems,
  including:

    - how many MDS daemons (in total and per file system)
    - which features are (or have been) enabled
    - how many data pools
    - approximate file system age (year + month of creation)
    - how much metadata is being cached per file system

  As well as:

    - whether IPv4 or IPv6 addresses are used for the monitors
    - whether RADOS cache tiering is enabled (and which mode)
    - whether pools are replicated or erasure coded, and
      which erasure code profile plugin and parameters are in use
    - how many RGW daemons, zones, and zonegroups are present; which RGW frontends are in use
    - aggregate stats about the CRUSH map, like which algorithms are used, how big buckets are, how many rules are defined, and what tunables are in use

12.2.12
-------
* In 12.2.9 and earlier releases, keyring caps were not checked for validity,
  so the caps string could be anything. As of 12.2.10, caps strings are
  validated and providing a keyring with an invalid caps string to, e.g.,
  "ceph auth add" will result in an error.

12.2.11
-------
* `cephfs-journal-tool` makes rank argument (--rank) mandatory. Rank is
  of format `filesystem:rank`, where `filesystem` is the cephfs filesystem
  and `rank` is the MDS rank on which the operation is to be executed. To
  operate on all ranks, use `all` or `*` as the rank specifier. Note that,
  operations that dump journal information to file will now dump to per-rank
  suffixed dump files. Importing journal information from dump files is
  disallowed if operation is targetted for all ranks.

* The MDS cache trimming is now throttled. Dropping the MDS cache
  via the `ceph tell mds.<foo> cache drop` command or large reductions in the
  cache size will no longer cause service unavailability.

* The CephFS MDS behavior with recalling caps has been significantly improved
  to not attempt recalling too many caps at once, leading to instability.
  MDS with a large cache (64GB+) should be more stable.

* MDS now provides a config option "mds_max_caps_per_client" (default: 1M) to
  limit the number of caps a client session may hold. Long running client
  sessions with a large number of caps have been a source of instability in the
  MDS when all of these caps need to be processed during certain session
  events. It is recommended to not unnecessarily increase this value.

* The MDS config mds_recall_state_timeout has been removed. Late client recall
  warnings are now generated based on the number of caps the MDS has recalled
  which have not been released. The new configs mds_recall_warning_threshold
  (default: 32K) and mds_recall_warning_decay_rate (default: 60s) sets the
  threshold for this warning.

>= 12.1.2
---------
* When running 'df' on a CephFS filesystem comprising exactly one data pool,
  the result now reflects the file storage space used and available in that
  data pool (fuse client only).
* Added new commands "pg force-recovery" and "pg-force-backfill". Use them
  to boost recovery or backfill priority of specified pgs, so they're
  recovered/backfilled before any other. Note that these commands don't
  interrupt ongoing recovery/backfill, but merely queue specified pgs
  before others so they're recovered/backfilled as soon as possible.
  New commands "pg cancel-force-recovery" and "pg cancel-force-backfill"
  restore default recovery/backfill priority of previously forced pgs.


12.2.1
------

* Clusters will need to upgrade to 12.2.1 before upgrading to any
  Mimic 13.y.z version (either a development release or an eventual
  stable Mimic release).

- *CephFS*:

  * Limiting MDS cache via a memory limit is now supported using the new
    mds_cache_memory_limit config option (1GB by default).  A cache reservation
    can also be specified using mds_cache_reservation as a percentage of the
    limit (5% by default). Limits by inode count are still supported using
    mds_cache_size. Setting mds_cache_size to 0 (the default) disables the
    inode limit.

* The maximum number of PGs per OSD before the monitor issues a
  warning has been reduced from 300 to 200 PGs.  200 is still twice
  the generally recommended target of 100 PGs per OSD.  This limit can
  be adjusted via the ``mon_max_pg_per_osd`` option on the
  monitors.  The older ``mon_pg_warn_max_per_osd`` option has been removed.

* Creating pools or adjusting pg_num will now fail if the change would
  make the number of PGs per OSD exceed the configured
  ``mon_max_pg_per_osd`` limit.  The option can be adjusted if it
  is really necessary to create a pool with more PGs.

12.2.3
------

- *RBD*:

  * The RBD C API's rbd_discard method now enforces a maximum length of
    2GB to match the C++ API's Image::discard method. This restriction
    prevents overflow of the result code.

- *CephFS*:

  * The CephFS client now catches failures to clear dentries during startup
    and refuses to start as consistency and untrimmable cache issues may
    develop. The new option client_die_on_failed_dentry_invalidate (default:
    true) may be turned off to allow the client to proceed (dangerous!).

12.2.5
------

- *CephFS*:

  * Upgrading an MDS cluster to 12.2.3+ will result in all active MDS
    exiting due to feature incompatibilities once an upgraded MDS comes online
    (even as standby). Operators may ignore the error messages and continue
    upgrading/restarting or follow this upgrade sequence:

    Reduce the number of ranks to 1 (`ceph fs set <fs_name> max_mds 1`),
    deactivate all other ranks (`ceph mds deactivate <fs_name>:<n>`), shutdown
    standbys leaving the one active MDS, upgrade the single active MDS, then
    upgrade/start standbys. Finally, restore the previous max_mds.

    See also: https://tracker.ceph.com/issues/23172

* *rados list-inconsistent-obj format changes:*

  * Various error strings have been improved.  For example, the "oi" or "oi_attr"
    in errors which stands for object info is now "info" (e.g. oi_attr_missing is
    now info_missing).

  * The object's "selected_object_info" is now in json format instead of string.

  * The attribute errors (attr_value_mismatch, attr_name_mismatch) only apply to user
    attributes.  Only user attributes are output and have the internal leading underscore
    stripped.

  * If there are hash information errors (hinfo_missing, hinfo_corrupted,
    hinfo_inconsistency) then "hashinfo" is added with the json format of the
    information.  If the information is corrupt then "hashinfo" is a string
    containing the value.

  * If there are snapset errors (snapset_missing, snapset_corrupted,
    snapset_inconsistency) then "snapset" is added with the json format of the
    information.  If the information is corrupt then "snapset" is a string containing
    the value.

  * If there are object information errors (info_missing, info_corrupted,
    obj_size_info_mismatch, object_info_inconsistency) then "object_info" is added
    with the json format of the information instead of a string.  If the information
    is corrupt then "object_info" is a string containing the value.

* *rados list-inconsistent-snapset format changes:*

  * Various error strings have been improved.  For example, the "ss_attr" in
    errors which stands for snapset info is now "snapset" (e.g. ss_attr_missing is
    now snapset_missing).  The error snapset_mismatch has been renamed to snapset_error
    to better reflect what it means.

  * The head snapset information is output in json format as "snapset."  This means that
    even when there are no head errors, the head object will be output when any shard
    has an error.  This head object is there to show the snapset that was used in
    determining errors.


12.2.9
------
* 12.2.9 contains the pg hard hard limit patches(https://tracker.ceph.com/issues/23979).
  A partial upgrade during recovery/backfill, can cause the osds on the previous version,
  to fail with assert(trim_to <= info.last_complete). The workaround for users is to
  upgrade and restart all OSDs to a version with the pg hard limit, or only upgrade
  when all PGs are active+clean. This patch will be reverted in 12.2.10, until
  a clean upgrade path is added to the pg log hard limit patches.

  See also: http://tracker.ceph.com/issues/36686

12.2.11
-------

* The default memory utilization for the mons has been increased
  somewhat.  Rocksdb now uses 512 MB of RAM by default, which should
  be sufficient for small to medium-sized clusters; large clusters
  should tune this up.  Also, the ``mon_osd_cache_size`` has been
  increase from 10 OSDMaps to 500, which will translate to an
  additional 500 MB to 1 GB of RAM for large clusters, and much less
  for small clusters.

* New CephFS file system attributes session_timeout and session_autoclose
  are configurable via `ceph fs set`. The MDS config options
  mds_session_timeout, mds_session_autoclose, and mds_max_file_size are now
  obsolete.

* This release fixes the pg log hard limit bug(https://tracker.ceph.com/issues/23979).
  A flag called pglog_hardlimit has been introduced. It is off by default.
  This flag enables the feature that limits the length of the pg log. Users should run
  'ceph osd set pglog_hardlimit' after completely upgrading to 12.2.11. Once all the OSDs
  have this flag set, the length of the pg log will be capped by a hard limit. We do not
  recommend unsetting this flag beyond this point.

* A health warning is now generated if the average osd heartbeat ping
  time exceeds a configurable threshold for any of the intervals
  computed.  The OSD computes 1 minute, 5 minute and 15 minute
  intervals with average, minimum and maximum values.  New configuration
  option ``mon_warn_on_slow_ping_ratio`` specifies a percentage of
  ``osd_heartbeat_grace`` to determine the threshold.  A value of zero
  disables the warning.  New configuration option
 ``mon_warn_on_slow_ping_time`` specified in milliseconds over-rides the
  computed value, causes a warning
  when OSD heartbeat pings take longer than the specified amount.
  New admin command ``ceph daemon mgr.# dump_osd_network [threshold]`` command will
  list all connections with a ping time longer than the specified threshold or
  value determined by the config options, for the average for any of the 3 intervals.
  New admin command ``ceph daemon osd.# dump_osd_network [threshold]`` will
  do the same but only including heartbeats initiated by the specified OSD.

* The configuration value ``osd_calc_pg_upmaps_max_stddev`` used for upmap
  balancing has been removed. Instead use the mgr balancer config
  ``upmap_max_deviation`` which now is an integer number of PGs of deviation
  from the target PGs per OSD.  This can be set with a command like
  ``ceph config set mgr mgr/balancer/upmap_max_deviation 2``.  The default
  ``upmap_max_deviation`` is 1.  There are situations where crush rules
  would not allow a pool to ever have completely balanced PGs.  For example, if
  crush requires 1 replica on each of 3 racks, but there are fewer OSDs in 1 of
  the racks.  In those cases, the configuration value can be increased.