Erc721, IPFS and metadata woes

Apologies for the long description, I am going to try and be as thorough as I can be for clarity sake.

So we have a ERC721 contract deployed to mainnet that was previously tested on ropsten. For the most part things went according to plan except there is a problem with the metadata that we worked around but I would like to understand the problem so we can be sure to do it correctly next time.

The rub of the issue is that token metadata was not being loaded correctly. A token could be minted and manually added in Metamask mobile or be found on opensea but no attributes or image was found. We based this off the BAYC model.

So originally the contract baseURI would be set to a IPFS URI such as (without quotes):
"ipfs://[metadata dir CID]/"

So the tokenURI when queried would be something like:
""ipfs://[metadata dir CID]/13"

The JSON returned at that IPFS URI would look like:
"image": "ipfs://[image dir CID]/7",
"attributes": [{
"trait_type": "a",
"value": "1"
}, {
"trait_type": "b",
"value": "2"
}, {
"trait_type": "c",
"value": "3"
}, {
"trait_type": "d",
"value": "4"

Note that token ID 13 is resolving to image #7 in the example (both the metadata and images were uploaded in their own directories so they have different CIDs). If you look at BAYC contract and an example of their metadata ( its almost identical - only difference is their image URI points to a CID without a "/" at the end.

Now, from a data chain perspective everything seems to click - until you get to trying to get any of the metadata to load. We were not able to get opensea or any wallet to actually load any of the metadata or images until we changed both the base URI and image value in the token metadata to include a specific https gateway. For example:

BaseURI: "[metadata dir CID]/"
TokenURI: "[metadata dir CID]/13"
image property inside TokenURI metadata JSON: "[image dir CID]/7.png"

So, here are the 64k questions:

  1. Why did the original approach of just providing IPFS URI not work? Is there some problem with providing a postfix for the image index after the CID? And if so, why wouldn't the metadata properties load if the issue is just with the image property?
  2. Why would the BAYC contract work but not mine in this case since the model is almost identical?
  3. Why would I need to provide a https URI with a gateway prefix and can this be avoided?

In case it comes up, the metadata and images are pinned via pinata.

** One last aside - I know there is a ERC721 metadata standard but it seems like every NFT collection I see has metadata that does not adhere to it - i.e. putting image property at top level, etc. Anyone know why and if that matters?

Thanks in advance!