This started out as a longer blog post, but then a lot of it boiled down to “read the fine documentation, Rachel”. So here's the short version.

Launched in December 2016, S3's Inventory Service is an alternative to using the ListObjects / ListObjectsV2 APIs for enumerating the objects in a bucket. You put an inventory configuration to your bucket (broadly speaking: which bit of S3 to list, where to put the results, and how often to do it), then sit back and wait for S3 itself to do all the hard work, so you don't have to. Great!

The documentation states where the inventory output goes:

destination-prefix/source-bucket/config-ID/YYYY-MM-DDTHH-MMZ/manifest.json
destination-prefix/source-bucket/config-ID/YYYY-MM-DDTHH-MMZ/manifest.checksum

And for the sake of brevity, let's cut to the chase: if you end your prefix with a “/” (either accidentally, or because like me you think you're being smart whereas in fact you simply haven't read the docs — good going, Rach), then due to a bug in the S3 Inventory service, your inventory will not be usable.

Specifically, I ended up with objects in S3 with keys like this:

s3-inventories//media/rachel-test-inventory/data/6eabc318-5ee0-41d9-b32b-a12b40a6f271.csv.gz
s3-inventories//media/rachel-test-inventory/data/b7dff5ea-c83d-4879-bc2a-0d0ced298356.csv.gz

whereas the manifest I got contained this (line breaks added for clarity):

{
  "files": [
    {
      "key": "s3-inventories/media/rachel-test-inventory/
                data/6eabc318-5ee0-41d9-b32b-a12b40a6f271.csv.gz",
      "size": 16486333,
      "MD5checksum": "3c94f6eed1fc3c2d057c098f355afffc"
    },
    {
      "key": "s3-inventories/media/rachel-test-inventory/
                data/b7dff5ea-c83d-4879-bc2a-0d0ced298356.csv.gz",
      "size": 20147436,
      "MD5checksum": "f0b39e0d85f0f5fb11bc5be73ecc26cf"
    }
  ]
}

The problem being that those double-slashes in the keys have become single slashes. On a Linux-ish filesystem, this would make no difference; on S3, it makes all the difference. The keys given in the manifest simply do not exist.

tl;dr: There's a bug in the S3 Inventory service which means that manifests are broken if the destination prefix ends with “/”. Solution: don't end your destination prefixes with “/”.