AWS: Retrieve DynamoDB Data With Lambda

Intro

In the previous post DynamoDB Intro we covered how to add items to a DynamoDB table using Lambda functions. Now we will cover a couple of methods for retrieving items from the database. We are building a sample application that stores and requests user data attributes such as height, weight and income.

Fetch Data Methods

We can start by referencing the 📘 SDK v2 API Guide: DynamoDB Methods where we can see several methods for retrieving data from DynamoDB.

getItem

getItem(params = {}, callback) ⇒ AWS.Request

The getItem operation returns a set of attributes for the item with the given primary key.

scan

scan(params = {}, callback) ⇒ AWS.Request

The scan operation returns one or more items and item attributes by accessing every item in a table or a secondary index.

query

query(params = {}, callback) ⇒ AWS.Request

The Query operation finds items based on primary key values.

Example: Fetch All User Data

Let us review our blank Lambda function here. This Lambda function will be handling requests to our applications API, and those requests are going to include a type. The type will either be all or single to indicate whether we are requesting the data for all users or a single user. And we can see that we have a simple if statement here in our function to send a different request based on the type.

Lambda/cyGetData/index.js

// Import the AWS SDK
var AWS = require("aws-sdk")
var dynamodb = new AWS.DynamoDB({
  region: "us-east-2",
  apiVersion: "2012-08-10",
})

exports.handler = (event, context, callback) => {
  const type = event.type

  // app will request all user data or single user data
  if (type === "all") {
    callback(null, "Requested All User Data")
  } else if (type === "single") {
    callback(null, "Requested Single User Data")
  } else {
    callback(null, "Invalid Type Request in URL Parameter")
  }
}

For now we will be working on the request if (type === 'all'). Therefore let us start by using the scan method.

scan method

Let us first define our parameters. The name of the table we are querying is compare-yourself.

var params = {
  TableName: "compare-yourself",
}

and then we can invoke the scan method

dynamodb.scan(params, function (err, data) {
      if (err) {
        console.log(err, err.stack) // an error occurred
      } else {
        console.log(data) // successful response
        callback(null, data) // highlight-line
      }

Note that we included our callback in the highlight above.

So our complete function looks like this

Lambda/cyGetData/index.js

// Import the AWS SDK
var AWS = require("aws-sdk")
var dynamodb = new AWS.DynamoDB({
  region: "us-east-2",
  apiVersion: "2012-08-10",
})

exports.handler = (event, context, callback) => {
  const type = event.type

  // app will request all user data or single user data
  if (type === "all") {
    var params = {
      TableName: "compare-yourself",
    }

    dynamodb.scan(params, function (err, data) {
      if (err) {
        console.log(err, err.stack) // an error occurred
      } else {
        console.log(data) // successful response
        callback(null, data)
      }
    })
  } else if (type === "single") {
    callback(null, "Requested Single User Data")
  } else {
    callback(null, "Invalid Type Request in URL Parameter")
  }
}

and we can make a test GET request to the API /compare-yourself/all and then look in the CloudWatch console logs and check for our data dump.

AccessDeniedException: User: arn:aws:sts::XXXXXXXXXXXXXXX:assumed-role/cyGetData-role-XXXXXXXX/cyGetData is
not authorized to perform: dynamodb:Scan on resource: arn:aws:dynamodb:us-east-2:XXXXXXXXXXXX:table/compare-yourself

So we can see that the IAM role for our Lambda function is not authorized to access the database. So let us go in and fix that just like we did in our last post.

And now we can see that the console log has come through for us.

And a Postman test returns the following JSON formatted data

{
    "Items": [
        {
            "UserId": {
                "S": "user002"
            },
            "income": {
                "N": "100000"
            },
            "height": {
                "N": "150"
            },
            "age": {
                "N": "33"
            }
        },
        {
            "UserId": {
                "S": "user001"
            },
            "income": {
                "N": "100000"
            },
            "height": {
                "N": "150"
            },
            "age": {
                "N": "33"
            }
        },
        {
            "UserId": {
                "S": "user003"
            },
            "income": {
                "N": "5670000"
            },
            "height": {
                "N": "150"
            },
            "age": {
                "N": "55"
            }
        }
    ],
    "Count": 3,
    "ScannedCount": 3
}

So this is working perfectly. The next steps in this application would be to make these GET and POST requests from our application instead of manually using Postman. We will get to that later, for now let us cover some more security best practices.

Currently with IAM permissions we have given these Lambda functions blanket access to DynamoDB. However in a production environment we would want to fine-tune these permissions to be read/write only and only for specific tables. We will get into that in the next post.

Example: Fetch Single User Data

Before the next example let us quickly do some cleanup here. We learned in another post about the AWS SDK Document Client

📘 AWS Document Client

Which reformats our data for us. So the following example will be using what we learned in our post about reformatting our DynamoDB requests and responses

📘 Ncoughlin: Reformatting DynamoDB Data

And I also have reformatted our code to be cleaner by creating objects for each of our request types, so that our if statement at the bottom is less crowded.

Lambda/cyGetData/index.js

// Import the AWS SDK
var AWS = require("aws-sdk")
var docClient = new AWS.DynamoDB.DocumentClient({ region: "us-east-2" })


// Handler
exports.handler = (event, context, callback) => {
  const type = event.type

  // Request type helpers
  let all = {
    params: {
      TableName: "compare-yourself",
    },
    request:
      (err, data) => {
        if (err) {
          // an error occurred
          console.log(err, err.stack)
        }
        else {
          // successful response
          console.log(data);
          callback(null, data);
        }
      }

  };

  let single = {

  };


  // app will request all user data or single user data
  if (type === "all") {
    docClient.scan(all.params, all.request) // highlight-line
  }
  else if (type === "single") {
    callback(null, "Requested Single User Data")
  }
  else {
    callback(null, "Invalid Type Request in URL Parameter")
  }
}

A couple other things to note here, is that we will now need to include a key in our request so that DynamoDB knows what data we are requesting, and in this case that is the UserId. Also note that the method to retrieve a single item using the document client is get and not getItem. Otherwise we will get an error that docClient.getItem() is not a function, because it is not a valid method. And lastly we need to expand the permissions of the cyGetData role that we have attached to this Lambda function, otherwise we will not have permission to run this function on the database.

And if we take all that into account we end up with this.

Lambda/cyGetData/index.js

// Import the AWS SDK
var AWS = require("aws-sdk")
var docClient = new AWS.DynamoDB.DocumentClient({ region: "us-east-2" })


// Handler
exports.handler = (event, context, callback) => {
  // Request type (all/single)
  const type = event.type;
  // extract User ID
  const userid = event.userid;

  // Request type helpers
  let all = {
    params: {
      TableName: "compare-yourself",
    },
    request:
      (err, data) => {
        if (err) {
          // an error occurred
          console.log(err, err.stack);
        }
        else {
          // successful response
          console.log(data);
          callback(null, data);
        }
      }

  };

  let single = {
    params: {
      TableName: "compare-yourself",
      Key: {
        "UserId": `${userid}`
      },

    },
    request:
      (err, data) => {
        if (err) {
          // an error occurred
          console.log(err, err.stack);
        }
        else {
          // successful response
          console.log(data);
          callback(null, data);
        }
      }
  };


  // app will request all user data or single user data
  if (type === "all") {
    docClient.scan(all.params, all.request);
  }
  else if (type === "single") {
    docClient.get(single.params, single.request);
  }
  else {
    callback(null, "Invalid Type Request in URL Parameter");
  }
};

which if we give the following data in a request

{
  "type": "single",
  "userid": "user001"
}

gives us this result

{
  "Item": {
    "UserId": "user001",
    "income": 100000,
    "height": 150,
    "age": 33
  }
}

So this is working perfectly. Note that our data is already returned to us in the correct data type/format thanks to the Document Client.

Comments

Recent Work

Basalt

basalt.software

Free desktop AI Chat client, designed for developers and businesses. Unlocks advanced model settings only available in the API. Includes quality of life features like custom syntax highlighting.

Learn More

BidBear

bidbear.io

Bidbear is a report automation tool. It downloads Amazon Seller and Advertising reports, daily, to a private database. It then merges and formats the data into beautiful, on demand, exportable performance reports.

Learn More