AWS: Reformatting DynamoDB Data

Intro

One of the quirks of DynamoDB that we learned earlier is that all of the data in Dynamo DB is stored with attribute values.

📘 DynamoDB Attribute Values

for example

{
    "Items": [
        {
            "UserId": {
                "S": "user002"
            },
            "income": {
                "N": "100000"
            },
            "height": {
                "N": "150"
            },
            "age": {
                "N": "33"
            }
        }
    ]
}

With N and S indicating string and number. And this gives us a bit of a problem when we get this data returned to us, we need to convert it back into a usable format.

There are a couple of ways to do this. First of all, Amazon has released a native tool in the SDK to handle converting data both into, and out of the DynamoDB called Document Client.

📘 AWS Document Client

Another way to handle this would be to do it manually. Let's take a look at each method.

Manual Reformat

Let us change our Lambda function to the following

Lambda/cyGetData/index.js

// Import the AWS SDK
var AWS = require("aws-sdk")
var dynamodb = new AWS.DynamoDB({
  region: "us-east-2",
  apiVersion: "2012-08-10",
})

exports.handler = (event, context, callback) => {
  const type = event.type

  // app will request all user data or single user data
  if (type === "all") {
    var params = {
      TableName: "compare-yourself",
    }

    dynamodb.scan(params, function (err, data) {
      if (err) {
        console.log(err, err.stack) // an error occurred
      } else {
        console.log(data) // successful response

        // map returned data to new variable
        const items = data.Items.map(item => {
          return {
            UserId: String(item.UserId.S),
            age: Number(item.age.N),
            height: Number(item.height.N),
            income: Number(item.income.N),
          }
        })

        // callback that new variable
        callback(null, items) // highlight-line
      }
    })
  } else if (type === "single") {
    callback(null, "Requested Single User Data")
  } else {
    callback(null, "Invalid Type Request in URL Parameter")
  }
}

Where we have simply created a new object from a map of the originally returned object, and then put that new object in the callback. This successfully returns the following.

result

[
  {
    "UserId": "user002",
    "age": 33,
    "height": 150,
    "income": 100000
  },
  {
    "UserId": "user001",
    "age": 33,
    "height": 150,
    "income": 100000
  },
  {
    "UserId": "user003",
    "age": 55,
    "height": 150,
    "income": 5670000
  }
]

Document Client Reformat

With this method we use the Document Client to abstract away the DynamoDB attribute values, and we can use it while sending and receiving data from the database.

📘 AWS Document Client

Lambda/cyGetData/index.js

// Import the AWS SDK
var AWS = require("aws-sdk")
var docClient = new AWS.DynamoDB.DocumentClient({ region: "us-east-2" }) // highlight-line

exports.handler = (event, context, callback) => {
  const type = event.type

  // app will request all user data or single user data
  if (type === "all") {
    var params = {
      TableName: "compare-yourself",
    }

    docClient.scan(params, function (err, data) { // highlight-line
      if (err) {
        // an error occurred
        console.log(err, err.stack)
      } else {
        // successful response
        console.log(data)

        callback(null, data)
      }
    })
  } else if (type === "single") {
    callback(null, "Requested Single User Data")
  } else {
    callback(null, "Invalid Type Request in URL Parameter")
  }
}

which successfully returns to us

{
  "Items": [
    {
      "UserId": "user002",
      "income": 100000,
      "height": 150,
      "age": 33
    },
    {
      "UserId": "user001",
      "income": 100000,
      "height": 150,
      "age": 33
    },
    {
      "UserId": "user003",
      "income": 5670000,
      "height": 150,
      "age": 55
    }
  ],
  "Count": 3,
  "ScannedCount": 3
}

Conclusion

Using the document client would be my preferred method, but it's good to keep in mind that we can do this all manually if we have an edge use case that requires manual conversion.

Comments

Recent Work

Basalt

basalt.software

Free desktop AI Chat client, designed for developers and businesses. Unlocks advanced model settings only available in the API. Includes quality of life features like custom syntax highlighting.

Learn More

BidBear

bidbear.io

Bidbear is a report automation tool. It downloads Amazon Seller and Advertising reports, daily, to a private database. It then merges and formats the data into beautiful, on demand, exportable performance reports.

Learn More