Resources
📘 AWS:DynamoDB Developer Documentation
📘 AWS:Choosing the right partition key
Intro
DynamoDB is the AWS brand of NoSQL database. It’s very similar to MongoDB, in the sense that it is a managed database service with a NoSQL format. Every AWS Account has one DynamoDB database, and then within that database you have tables. Each table is what you would traditionally think of as a database. Therefore for every new “database” that you would like to make in your account, you will make a new table.
Data Organization
DynamoDB table data is organized with three items. Keys, Attributes and Indexes.
Partition Key
DynamoDB’s are broken out into 10GB partitions, and each partition is required to have a unique partition ID.
A partition key should be a top level attribute, such as a brand or customer ID. Then all of the data for that particular brand or customer would go into that partition.
You can ream much more about partition ID’s here:
📘 AWS:Choosing the right partition key
The Partition Key is the Primary Key.
Adding Items Manually
In a typical use case we would be adding items to the database programmatically. We will be covering that extensively below. Let us briefly cover how to add an item to the database (table) manually using the console.
You simply have to select the table you want, and then use the create item button.
We will always start by adding the partition key, which as we discussed, is mandatory for every item in the database. For this example we have decided that the partition key will be the UserID.
Our little dummy application is going to keep track of three items for our users. Age, Height & Income. All of those items are numbers. When we click the append button we are prompted to select the data type of the data we are going to add. Therefore we will select number, and input some dummy data for our first user.
We append all three pieces of data and then save.
And we have added our first piece of data to the table. Of course this is completely impractical for anything useful. We want our application to be programmatically adding and removing data from the database. To accomplish that we will use the AWS SDK. Because we are writing our functions in Lambda, and Lambda is running Node, we will be using the AWS SDK for Node.
Adding Items Programmatically
AWS SDK v2
There is a new version of the SDK (v3) so i’m splitting these instructions up.
If we wanted to use the AWS SDK from our local machine we would need to install it. However because we are going to be calling the SDK with Lambda functions, we don’t need to worry about that. The SDK is pre-installed in Lambda for all programming languages. We will however need to import it in every function that requires it.
var AWS = require("aws-sdk")AWS SKD v3
There is a new version of the SDK that is intended to be more modular. Instead of including all the services in one package, they are separated by service so you would only import the ones you need.
⚠️ However version 3 of the SDK is not compatible with Lambda functions yet, so if you are writing code in Lambda use v2.
import {
DynamoDBClient,
BatchExecuteStatementCommand,
} from "@aws-sdk/client-dynamodb"
// a client can be shared by difference commands.
const client = new DynamoDBClient({ region: "us-east-2" })
const params = {
/** input parameters */
}
const command = new BatchExecuteStatementCommand(params).putItem
.putItem is the DynamoDB property that we will use to add an item to the database.
And the documentation gives the following example
/* This example adds a new item to the Music table. */
var params = {
Item: {
AlbumTitle: {
S: "Somewhat Famous",
},
Artist: {
S: "No One You Know",
},
SongTitle: {
S: "Call Me Today",
},
},
ReturnConsumedCapacity: "TOTAL",
TableName: "Music",
}
dynamodb.putItem(params, function (err, data) {
if (err) console.log(err, err.stack)
// an error occurred
else console.log(data) // successful response
/*
data = {
ConsumedCapacity: {
CapacityUnits: 1,
TableName: "Music"
}
}
*/
})So there are just a couple things there to take note of. First we can see in the params that we are saying which database(table) we want to add the information to. Also we are describing the item we would like to add. Which is an object. In our actual function this item will be receiving information dynamically from our API request. These are the bare minimum that we can describe in params. What the data is, and where it is going.
The name of our table is compare-yourself.
Therefore our parameters will look like so
const params = {
Item: {
UserId: {
S: event.userid,
},
age: {
N: event.age,
},
income: {
N: event.income,
},
height: {
N: event.height,
},
},
ReturnConsumedCapacity: "TOTAL",
TableName: "compare-yourself",
}Note that for each variable piece of data we have created another object and then used S or N to define the data-type, which in DynamoDB they call attribute value. This is required by DynamoDB, because it never assumed to know the data-type for anything coming in, we must specify it ourselves. For a full list of available attribute values see here:
What that means is that everything we send to DynamoDB needs to be formatted as a string in the POST request. If we sent the following data in the POST request.
{
"userid": "user002",
"age": 33,
"height": 150,
"income": 100000
}We would get this error
{
"errorType": "MultipleValidationErrors",
"errorMessage": "There were 3 validation errors:\n* InvalidParameterType: Expected params.Item['age'].N to be a string\n* InvalidParameterType: Expected params.Item['income'].N to be a string\n* InvalidParameterType: Expected params.Item['height'].N to be a string",
"trace": [
"MultipleValidationErrors: There were 3 validation errors:",
"* InvalidParameterType: Expected params.Item['age'].N to be a string",
"* InvalidParameterType: Expected params.Item['income'].N to be a string",
"* InvalidParameterType: Expected params.Item['height'].N to be a string",
" at ParamValidator.validate (/var/runtime/node_modules/aws-sdk/lib/param_validator.js:40:28)",
" at Request.VALIDATE_PARAMETERS (/var/runtime/node_modules/aws-sdk/lib/event_listeners.js:132:42)",
" at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
" at callNextListener (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:96:12)",
" at /var/runtime/node_modules/aws-sdk/lib/event_listeners.js:86:9",
" at finish (/var/runtime/node_modules/aws-sdk/lib/config.js:386:7)",
" at /var/runtime/node_modules/aws-sdk/lib/config.js:404:9",
" at EnvironmentCredentials.get (/var/runtime/node_modules/aws-sdk/lib/credentials.js:127:7)",
" at getAsyncCredentials (/var/runtime/node_modules/aws-sdk/lib/config.js:398:24)",
" at Config.getCredentials (/var/runtime/node_modules/aws-sdk/lib/config.js:418:9)"
]
}Which also means that we need to update our mapping template in API gateway to accept a string for every item. Otherwise we will throw an error in API Gateway for having our request body not match the template!
Which means that we need to change our body mapping model from this
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "CompareData",
"type": "object",
"properties": {
"userid": {"type": "string"},
"age": {"type": "integer"},
"height": {"type": "integer"},
"income": {"type": "integer"}
},
"required": ["userid","age", "height", "income"]
}to this
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "CompareData",
"type": "object",
"properties": {
"userid": {"type": "string"},
"age": {"type": "string"},
"height": {"type": "string"},
"income": {"type": "string"}
},
"required": ["userid","age", "height", "income"]
}and our mapping template in the integration request to this:
#set($inputRoot = $input.path('$'))
{
"userid" : "$inputRoot.userid",
"age" : "$inputRoot.age",
"height" : "$inputRoot.height",
"income" : "$inputRoot.income"
}And we finally are able to get the request through, except for one more problem!
AccessDeniedException
If you go back to the post about AWS IAM we discussed roles and permissions. The problem that we have here is that our role that we have assigned to our Lambda function does not have permission to access DynamoDB. Therefore we need to go into that role and add some new permissions.
You can start by assigning it the policy AmazonDynamoDBFullAccess which is overkill but will allow us to quickly test this function.
And that finally works and adds a new item to the database!
Final Lambda Function
Let’s do a final review of the Lambda function we have created here. This function is designed to receive an HTTP POST request from API Gateway, and then use that data to add a new item to a DynamoDB table.
// Import the AWS SDK
var AWS = require("aws-sdk")
var dynamodb = new AWS.DynamoDB({
region: "us-east-2",
apiVersion: "2012-08-10",
})
exports.handler = (event, context, callback) => {
const params = {
Item: {
UserId: {
S: event.userid,
},
age: {
N: event.age,
},
income: {
N: event.income,
},
height: {
N: event.height,
},
},
ReturnConsumedCapacity: "TOTAL",
TableName: "compare-yourself",
}
dynamodb.putItem(params, function (err, data) {
// an error occurred
if (err) {
console.log(err, err.stack)
callback(err)
}
// successful response
else {
console.log("Successfully added data")
console.log(data)
callback(null, data)
}
})
}Which successfully takes a POST request with this format:
{
"userid": "user002",
"age": "33",
"height": "150",
"income": "100000"
}
