How to Access Amazon S3 Bucket Objects From OIC Integrations

 We connect to many different boundary systems from our OIC inegrations.

One such is Amazon S3 buckets.

In this blog we will see how we can create connection in OIC for S3, list object from S3, get object etc.

Based on the operation we want to do on S3 objects , we need to have policy to the user on the resources.

Below is a snippet for example, this has to be done and kept ready from AWS side:

 

 

resource "aws_iam_policy" "s3-xyz-oracle-rw" {
  name        = "s3.xyz-oracle-.rw"
  path        = "/"
  description = ""
  policy      = data.aws_iam_policy_document.s3-xyz-oracle-rw-policy.json
}
resource "aws_iam_policy_attachment" "s3-xyz-oracle-rw-policy-attachment" {
  name       = "s3.xyz-oracle-.rw-policy-attachment"
  policy_arn = aws_iam_policy.s3-xyz-oracle-rw.arn
  groups     = []
  users      = ["oracle_user"]
  roles      = []
}
data "aws_iam_policy_document" "s3-xyz-oracle-rw-policy" {
  statement {
    effect = "Allow"
    actions = [
      "s3:DeleteObject",
      "s3:GetObject",
      "s3:GetObjectMetaData",
      "s3:ListMultipartUploadParts",
      "s3:GetBucketLocation",
      "s3:GetObjectAcl",
      "s3:PutObject",
      "s3:ListBucket",
      "s3:GetObjects",
      "s3:ListObjects",
    ]
    resources = [
      "arn:aws:s3:::xyz-finance-internal-only/inbound_to_workday/*",
      "arn:aws:s3:::xyz-finance-internal-only/oracle_to_hive/*",
    ]
  }
}

 

To create S3 connection in OIC , we need  REST Adapter.

Connection URL: https://BucketName.s3.amazonaws.com/
Example: https://xyz-finance-internal-only.s3.amazonaws.com/

Security Policy: AWS Signature Version 4
Access key : Access Key of the service user
Secret key : Secret Key of the service user
AWS Region :  Select your aws region (for eg.  us-east-1)
Service : Select Service as S3 

Now let us see how we can use this connection in our integration for different operations

 List Object:

 

 

 

 

Provide the Sample XML schema as:

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult
    xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Name>xyz-finance-internal-only</Name>
    <Prefix>inbound_to_workday/staging/giftcard_cash_accounting</Prefix>
    <KeyCount>3</KeyCount>
    <MaxKeys>1000</MaxKeys>
    <IsTruncated>false</IsTruncated>
    <Contents>
        <Key>inbound_to_workday/staging/giftcard_cash_accounting/part-00000-1512efb4-e77c-4b04-aef2-28e0a22f0670-c000.csv</Key>
        <LastModified>2023-03-15T00:08:37.000Z</LastModified>
        <ETag>&quot;0bae6d2e84bvfva16d2f2b6c3fdbb2cec72&quot;</ETag>
        <Size>839</Size>
        <StorageClass>STANDARD</StorageClass>
    </Contents>
    <Contents>
        <Key>inbound_to_workday/staging/giftcard_cash_accounting/part-00000-60dd6411-8974-48b2-b1cc-a5d0ced72d63-c000.csv</Key>
        <LastModified>2023-04-15T00:15:03.000Z</LastModified>
        <ETag>&quot;36b0835f06e83b0876vdv47cdd944c69c9b&quot;</ETag>
        <Size>844</Size>
        <StorageClass>STANDARD</StorageClass>
    </Contents>
    <Contents>
        <Key>inbound_to_workday/staging/giftcard_cash_accounting/part-00000-f2a470d1-6ce7-4515-9a5e-65e6a4910d8f-c000.csv</Key>
        <LastModified>2023-05-15T00:08:00.000Z</LastModified>
        <ETag>&quot;1f4ed21595b1a847c4f285fdd6aea299768&quot;</ETag>
        <Size>840</Size>
        <StorageClass>STANDARD</StorageClass>
    </Contents>
</ListBucketResult>

Now let us see how it works during runtime:

Below file is preset in Amazon S3:











List Request:
 

Response:
 


Now let us see how to download this file:

Loop over Contents to download the file. In this case iteration will be two (one with file and other with directory only)

the directory one can be discarded from downloading based on size <Size>0</Size>.























Download Request and Response:
 

 

Until now we saw how to list and download file from S3.
Now let us see how we can upload a file to s3.

Method: PUT
Request Media Type: application/octet-stream

Response Media Type: application/octet-stream

Now let us see how it works during runtime:

Request:
 


Response:

Now let us verify if this file was uploaded successfully in Amazon S3.

From the below screenshot we can see that the above file was successfully uploaded to S3: