Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-18771

optimize S3 blobstores for intelligent tiering

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: New
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.14.0
    • Fix Version/s: None
    • Component/s: Blobstore, S3
    • Labels:
    • Notability:
      4

      Description

      AWS S3 supports various object storage classes.

      By default the S3 Blobstore implementation uses multi-part HTTP uploads of S3 objects with the default Storage Class of "STANDARD" for all objects.

      AWS S3 supports a feature called intelligent tiering of objects which dynamically tiers objects. To fully make use of this, one can set the storage class explicitly for objects when they are uploaded to INTELLIGENT_TIERING. Currently repository manager does not offer any configurability or optimization for using intelligent tiering storage classes.

      From https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html#sc-dynamic-data-access :

      Storage Class That Automatically Optimizes Frequently and Infrequently Accessed Objects
      The INTELLIGENT_TIERING storage class is designed to optimize storage costs by automatically moving data to the most cost-effective storage access tier, without performance impact or operational overhead. INTELLIGENT_TIERING delivers automatic cost savings by moving data on a granular object level between two access tiers, a frequent access tier and a lower-cost infrequent access tier, when access patterns change. The INTELLIGENT_TIERING storage class is ideal if you want to optimize storage costs automatically for long-lived data when access patterns are unknown or unpredictable.

      The INTELLIGENT_TIERING storage class stores objects in two access tiers: one tier that is optimized for frequent access and another lower-cost tier that is optimized for infrequently accessed data. For a small monthly monitoring and automation fee per object, Amazon S3 monitors access patterns of the objects in the INTELLIGENT_TIERING storage class and moves objects that have not been accessed for 30 consecutive days to the infrequent access tier. There are no retrieval fees when using the INTELLIGENT_TIERING storage class. If an object in the infrequent access tier is accessed, it is automatically moved back to the frequent access tier. No additional tiering fees apply when objects are moved between access tiers within the INTELLIGENT_TIERING storage class.

      Note

      The INTELLIGENT_TIERING storage class is suitable for objects larger than 128 KB that you plan to store for at least 30 days. If the size of an object is less than 128 KB, it is not eligible for auto-tiering. Smaller objects can be stored, but they are always charged at the frequent access tier rates in the INTELLIGENT_TIERING storage class. If you delete an object before the 30-day minimum, you are charged for 30 days. For pricing information, see Amazon S3 Pricing.

      References

      https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html#sc-howtoset
      https://aws.amazon.com/about-aws/whats-new/2018/11/s3-intelligent-tiering/
      https://aws.amazon.com/blogs/aws/new-automatic-cost-optimization-for-amazon-s3-via-intelligent-tiering/
      https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html
      https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html#object-metadata
      https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html

      Expected

      Allow optimization of S3 blobstore objects for storing intelligent tiering type storage classes.

      What this actually means is yet to be defined - it could be as simple as allowing configurability of storage classes per blobstore, or something a little more dynamic.

      Using S3 Lifecycle Rules Instead

      S3 Lifecycle Rules may be a workaround to implementing something in product. See

      See these links for using a lifecycle policy to handle transitioning objects to various tiers of storage classes:

      https://aws.amazon.com/blogs/aws/new-automatic-cost-optimization-for-amazon-s3-via-intelligent-tiering/
      https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            plynch Peter Lynch
            Last Updated By:
            Rich Seddon Rich Seddon
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:

                tigCommentSecurity.panel-title