AWS CloudTrail is a managed service to log, monitor, and retain events related to API calls across an AWS account. The AWS calls could be made as a result of some AWS Management Console action or some action initiated by another managed service console. Or, the API call could be from AWS SDK and command line tools. Tracking API calls is used for various purposes, including monitoring, auditing, and resources usage tracking. CloudTrail stores the data files generated by API events in an AWS S3 bucket. The S3 bucket could be created while creating the CloudTrail or separately, in advance. Optionally, an SNS (Simple Notification Service) Topic and Subscription can be associated with a CloudTrail to send notifications to a subscriber. The CloudTrail generated API calls or events can be logged to an AWS CloudWatch Log Group. Though CloudTrail logs and monitors API calls across an AWS account, we shall be discussing CloudTrail only in the context of AWS Relational Database Service (RDS). The RDS API operations include CreateDBInstance, CreateDBSnapshot, and DescribeDBInstances among others (Refer to http://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_Operations.html). The RDS API Call can be made from the RDS Console or using RDS API; we shall discuss both methods. The procedure to use CloudTrail to monitor API calls in RDS is as follows:

Create an S3 Bucket
Create a CloudTrail
Create an SNS Topic and Subscription
Associate the SNS Topic and Subscription with the CloudTrail
Create a CloudWatch Log Group
Configure the CloudWatch Log Group with the CloudTrail
Create a Database Instance on RDS from the RDS Console or with RDS API
Generate other RDS Events such as reboot a database instance, modify a database instance, and delete a database instance.
Receive AWS Notifications\
Search the CloudWatch Log Group for RDS-generated API Events

This article has the following sections:

Setting the Environment
Creating an S3 Bucket for Storing CloudTrail Log Files
Creating a CloudTrail
Creating an SNS Topic and Subscription
Configuring the SNS Topic with the CloudTrail
Logging CloudTrail Logs to CloudWatch
Creating an Oracle Database Instance on RDS Console
Searching the CloudWatch Logs for Events generated by RDS
Generating other RDS Console Events and searching CloudWatch
Creating a MySQL Database Instance using RDS API
Searching CloudWatch Logs for RDS API Events
Generating other RDS API Events and searching CloudWatch
Deleting a CloudTrail

Setting the Environment

The only prerequisite for using AWS CloudTrail and RDS is an AWS account, which may be created at https://aws.amazon.com/resources/create-account/. For making RDS API calls the AWS CLI must be installed. The Amazon Linux AMI based EC2 instance has AWS CLI pre-installed; we shall use it to make RDS API calls. Select the Amazon Linux AMI to create an EC2 instance as shown in Figure 1.

Figure 1. Selecting Amazon Linux AMI for EC2

Create and download a new key pair (awscli) for SSH login to the EC2 instance as shown in Figure 2.

Figure 2. Creating a New Key Pair

Obtain the Public IP Address of the EC2 instance from the AWS Console as shown in Figure 3.

Figure 3. Public IP Address of EC2 Instance

Copy the key pair awscli.pem to a local machine and set permissions to read-only (400).

chmod 400 awscli.pem
SSH Login to the EC2 instance.
ssh –i “awscli.pem” ec2-user@34.205.26.201

The Amazon Linux command prompt gets displayed, as shown in Figure 4.

Figure 4. SSH Login into EC2 Instance

Creating a CloudTrail

To create an AWS CloudTrail open URL https://aws.amazon.com/cloudtrail/ n a browser and click on Get started with AWS CloudTrail. Select the CloudTrail service from the Management Tools section or open URL https://console.aws.amazon.com/cloudtrail/home?region=us-east-1#/gettingStartedin a browser. Click on Get Started Now as shown in Figure 5.

Figure 5. Get Started Now

In the CloudTrail wizard specify a Trail name (RDS-CloudTrail) as shown in Figure 6 and select Apply trail to all regions>No. In Management events select All for Read/Write events.

Figure 6. Turn on CloudTrail

In Storage location we have the option to create a new S3 bucket or use an existing bucket. Because a policy must be applied to the S3 bucket to be used with CloudTrail, it is recommended to create a new S3 bucket, as the required policy is automatically applied to the bucket created. Select Create a new S3 bucket and specify an S3 bucket name (rds.s3) as shown in Figure 7. Optionally, a Log file prefix may be specified to make log files easier to browse. Select No for Encrypt log files. Select Yes for Enable log file validation. Log file validation is used to determine whether a log file was modified, deleted, or unmodified after CloudTrail delivered it. We shall configure SNS notification separately; select No for Send SNS notification for every log file delivery. Click on Create.

Figure 7. Specifying Storage location

A new CloudTrail gets created, as shown in Figure 8.

Figure 8. CloudTrail RDS-CloudTrail

A CloudTrail collects API calls from across the AWS account and not just from RDS or any other specific service. The automatically created S3 bucket is shown in Figure 9.

Figure 9. S3 Bucket rds.s3

Click on the rds.s3 bucket to access the log files stored. Click on the AWSLogs directory as shown in Figure 10.

Figure 10. AWSLogs

Click on the subdirectories to access the log files as shown in Figure 11.

Figure 11. Log Files stored in S3 Bucket

The CloudTrail created is editable and we shall add an SNS topic and also configure CloudWatch Logs in subsequent sections. Click on the RDS-CloudTrail link in the Trails table. The configuration for the trail gets displayed. Among the configuration sections listed are the Trail settings and Management events as shown in Figure 12.

Figure 12. RDS-CloudTrail Configuration

Scroll to display configuration for Data Events and Storage location as shown in Figure 13.

Figure 13. Data events and Storage location

Scroll further to display the option to configure CloudWatch Logs as shown in Figure 14. We shall configure CloudWatch logs in a subsequent section.

Figure 14. CloudWatch Logs Configure Option

Creating an SNS Topic and SNS Subscription

A Simple Notification Service (SNS) Topic is used as a communication channel to send messages and subscribe to notifications. To create an SNS Topic select the SNS service from the AWS Services>Messaging header. Click on Get Started in the Simple Notification Service page. In the SNS dashboard click on Create topic as shown in Figure 15.

Figure 15. Create topic

In the Create new topic dialog specify a Topic name (RDSCloudTrail) and click on Create topic as shown in Figure 16.

Figure 16. Create topic

A new topic gets created, as shown in Figure 17.

Figure 17. Topic RDSTrail created

The topic gets added to the Topics table, as shown in Figure 18. A topic ARN also gets listed; we shall need it to create a Subscription for the topic.

Figure 18. RDSCloudTrail Topic

To receive notifications from the topic an SNS Subscription needs to be created. To create a subscription select Subscriptions in the margin and click on Create subscription as shown in Figure 19.

Figure 19. Create subscription

In Create subscription dialog specify a Topic ARN, which should bes copied from the Topics table, as shown in Figure 20.

Figure 20. Create subscription

To receive Email notifications select Protocol as Email, as shown in Figure 21.

Figure 21. Selecting Protocol as Email

In Endpoint specify an Email address to get SNS notification at and click on Create subscription, as shown in Figure 22.

Figure 22. Creating a Subscription

A new subscription gets created. Initially the subscription is PendingConfirmation, as shown in Figure 23.

Figure 23. Subscription PendingConfirmation

In the confirmation message that is sent to the Email specified in the Endpoint, click on Confirm subscription, as shown in Figure 24.

Figure 24. Confirming Subscription

As the Subscription Confirmed message in Figure 25 indicates, the subscription gets confirmed. A subscription id is also listed.

Figure 25. Subscription confirmed

Click on the button to refresh the Subscriptions; the PendingConfirmation should be replaced with the Subscription ARN, as shown in Figure 26.

Figure 26. Confirmed Subscription

Configuring the SNS Topic with the CloudTrail

Next, we shall configure the SNS Topic with the CloudTrail. In the SNS Topics table, click on Actions and select Edit topic policy, as shown in Figure 27.

Figure 27. Selecting Actions>Edit topic policy

Click on the Advanced view tab. Copy and paste the SNS Topic policy for CloudTrail from http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-permissions-for-sns-notifications.html and substitute the resource Region, SNSTopicOwnerAccountId and SNSTopicName values as shown here: "Resource": "arn:aws:sns:Region:SNSTopicOwnerAccountId:SNSTopicName". Click on Update Policy as shown in Figure 28.

Figure 28. Applying SNS Topic policy for CloudTrail

In the CloudTrail configuration we previosuly selected No for Send SNS notification for every log file delivery, as shown in Figure 29.

Figure 29. Send SNS Notification Option

SelectYes for the Send notification option. A sub-option to create a new SNS topic or use an existing topic gets displayed. As we already created an SNS Topic, select No in Create a new SNS Topic. A new field SNS topic gets displayed, as show in Figure 30. Select the RDSCloudTrail topic and click on Save to save the configuration.

Figure 30. Configuring SNS Topic with CloudTrail

The topic policy gets edited, as indicated by the message in Figure 31.

Figure 31. Topic Policy edited

The Publish to SNS gets set to Yes and the SNS topic gets set to RDSCloudTrail, as shown in Figure 32.

Figure 32. CloudTrail configured with SNS Topic RDSCloudTrail

CloudTrail log messages start getting sent to the topic, as indicated by the Last SNS notification field in Figure 33.

Figure 33. Messages start getting sent to SNS

Logging CloudTrail Logs to CloudWatch

Next, we shall configure CloudWatch logs with the CloudTrail so that the CloudTrail log messages can be sent to and searched in CloudWatch. Specify an existing CloudWatch Logs log group or let CloudTrail create a new CloudWatch log group, as indicated by the message in Figure 34.

Figure 34. Configuring CloudWatch Logs

We shall create a new CloudWatch log group separately; for which, select the CloudWatch AWS service and select Logs in the margin. Click on Actions>Create log group as shown in Figure 35.

Figure 35. Actions>Create log group

In the Create log group dialog specify a Log Group Name (/aws/cloudtrail/rds) and click on Create log group as shown in Figure 36.

Figure 36. Create log group Dialog

A new CloudWatch log group gets created, as shown in Figure 37.

Figure 37. New CloudWatch Log Group added

In the CloudTrail configuration select the newly added CloudWatch log group for the CloudWatch logs configuration and click on Continue as shown in Figure 38.

Figure 38. Configuring CloudWatch Logs Group

For CloudTrail to send events to CloudWatch, a new role is created and the Role Summary is displayed in Figure 39. Click on Allow.

Figure 39. Configuring Role

The CloudWatch Logs get configured with the CloudTrail, as shown in Figure 40.

Figure 40. CloudWatch Logs configured with CloudTrail

Two streams get configured for the CloudTrail, as shown in Figure 41. Multiple log streams may be created if the volume of data from CloudTrail is large. The log streams names are in the format account_ID_CloudTrail_source_region.

Figure 41. CloudWatch Streams for CloudTrail

Select the stream with the suffix us-east-1. Initially the CloudTrail does not send any RDS API calls or events as we have not yet created any RDS DB instance. Search for an RDS event such as CreateDBInstance and no log data is listed, as shown in Figure 42.

Figure 42 RDS API Call CreateDBInstance does not list any logs

However, the CloudTrail has been configured with the CloudWatch Log group, as shown in Figure 43, and all API calls get sent to the CloudWatch log group.

Figure 43. CloudTrail configured with CloudWatch Log Group

Creating an Oracle Database Instance on RDS Console

Next, create an Oracle database instance on RDS. The procedure to create an Oracle DB instance on RDS is discussed in detail in an earlier tutorial and shall be skipped here. The created Oracle DB instance oradb is shown in Figure 44.

Figure 44. Oracle DB Instance on RDS

Searching the CloudWatch Logs for Events generated by RDS

The CloudWatch Log group configured with the CloudTrail may be searched for API calls sent to it. The log messages are listed in Figure 45.

Figure 45. Log Messages in CloudWatch

After an Oracle Database instance has been created on RDS, search for the CreateDBInstance API call again, and a CloudTrail log file gets listed, as shown in Figure 46. A CloudTrail log file consists of information about a single event (API call) in JSON format. Information about the AWS service of the API call, the API call event, the user making the API call, and the parameters of the call are included in the log file. Event data is listed in the Records array. The eventName element lists the actual API call name and an RDS event would have eventName CreateDBInstance, as shown in Figure 46. The event source for API calls from RDS have the eventSource as rds.amazon.com.

Figure 46. A single CloudTrail Log File for CreateDBInstance

CloudTrail logs may also be searched using other event parameters. With a search for “oradb”, the DB instance name lists two CloudTrail log files, as shown in Figure 47.

Figure 47. Searching CloudWatch Logs with DB Instance name “oradb”

When creating a DB instance, CreateDBInstance is not the only API call made. Several associated API calls get made, such as DescribeDBInstances, shown in Figure 48.

Figure 48. CloudTrail Log File for DescribeDBInstances

While the CloudTrail log files provide a lot of valuable information that may used for monitoring and debugging an RDS instance, several AWS CloudTrail partners such as AlertLogic, Boundary, CloudCheckr, LogEntries, Loggly, Cognizant, Cloudnexa, Datapipe and Cloudlytics are available to analyze the CloudTrail logs in more detail.

Generating other RDS Console Events and Searching CloudWatch

Creating a DB instance on RDS generates the CreateDBInstance API call and a few associated API calls, such as DescribeDBInstances. Other events on an RDS instance generate other API calls. If a DB instance is rebooted, as shown in Figure 49, the RebootDBInstance API call gets made.

Figure 49. Selecting Instance Actions>Reboot

To modify a DB instance select Modify as shown in Figure 50.

Figure 50. Selecting Instance Actions>Modify

In the Modify DB Instance, make some modifications, such as selecting the db.m3.large Instance Class and increasing the Allocated Storage to 100GB, as shown in Figure 51.

Figure 51. Modifying DB Instance

Click on Modify DB Instance as shown in Figure 52.

Figure 52. Modify DB Instance

The DB instance’s Status becomes “modifying”, as shown in Figure 53.

Figure 53. Status as modifying

Search for the API call ModifyDBInstance in the CloudWatch log group, and one or more log files get listed. The eventName is “ModifyDBInstance”, as shown in Figure 54.

Figure 54. CloudTrail Log File for ModifyDBInstance

A CloudTrail log file is generated not only for successful API calls but also for failed API calls. If an instance modification is made with some invalid configuration, an error message is included in the CloudTrail log file. Modifying a DB instance with oradb with some invalid configuration parameters generates the InvalidParameterCombinationException, as shown in Figure 55. An error message is also listed.

Figure 55. CloudTrail Log File with an Error Message

Multiple search terms may be specified to search for CloudTrail logs, as shown in Figure 56 using the two search terms “oradb” and “ModifyDBInstance”.

Figure 56. Specifying multiple search terms

Next, delete the oradb instance with Instance Actions>Delete as shown in Figure 57.

Figure 57. Instance Actions>Delete

In the confirmation dialog click on Delete and the Status message becomes “deleting”, as shown in Figure 58.

Figure 58. Status Message deleting

Next, search for the DeleteDBInstance API call, which should get sent to CloudTrail a while after starting the delete. If the first stream does not list the DeleteDBInstance API call or some other API call, search in another stream, as shown in Figure 59.

Figure 59. Selecting the other CloudWatch Log Group Stream to search

A search for the DeleteDBInstance API call lists a CloudTrail log file, as shown in Figure 60.

Figure 60. CloudTrail Log File for DeleteDBinstance Event

Creating a MySQL Database Instance Using RDS API

CloudTrail collects API calls not only from events generated from the RDS Console but also from command line RDS API calls. In this section we shall create a MySQL RDS DB instance using the command aws rds create-db-instance. An AWS region must be specified when using the aws rds create-db-instance command and the region may be configured using the aws configure command as listed:

[ec2-user@ip-172-30-0-51 ~]$ aws configure
AWS Access Key ID [****************53FA]:
AWS Secret Access Key [****************/EXF]:
Default region name [use-east-1]: us-east-1
Default output format [json]:

When invoking the CreateDBInstance call for MySQL database,certain naming requirements must be met. The master user name or password cannot be "mysql", which is a reserved word. MasterUserPassword must be at least 8 characters long. Run the following command to create a MySQL DB instance called mysql-db.

aws rds create-db-instance --db-instance-identifier mysql-db \
 --allocated-storage 20 --db-instance-class db.m1.small --engine mysql \
 --master-username dvohra --master-user-password mysqldv16

The output from the command is shown in Figure 61.

Figure 61. Output from aws rds create-db-instance

A MySQL RDS DB instance gets created, as shown n Figure 62.

Figure 62. DB Instance mysql-db Created

Even though the RDS DB instance is not created from the RDS Console the mysql-db instance gets listed on the RDS Console, as shown in Figure 63.
Figure 63. MySQL DB instance mysql-db

Searching CloudWatch Logs for RDS API Events

Search for the “CreateDBInstance” in CloudWatch Log group as shown in Figure 64 and include a second search term “mysql” so that the CreateDBInstance API calls from the other RDS DB instance for Oracle database do not get listed. CloudTrail log files for failed API calls are also listed.

Figure 64. Searching for CreateDBInstance for mysql

A CloudTrail log file for a CreateDBInstance API call which did not complete successfully because of username being reserved word “mysql” is shown in Figure 65.

Figure 65. CloudTrail Log File for CreateDBInstance API call to create a MySQL Database Instance on RDS

Generating other RDS API Events and searching CloudWatch

As for the events generated from RDS Console, direct RDS API calls for other events are also sent to CloudTrail. To modify the mysql-db instance, run the following command, which modifies the instance name and instance class and makes the instance multizone.

[ec2-user@ip-172-30-0-51 ~]$ aws rds modify-db-instance \
 --db-instance-identifier mysql-db \
 --allocated-storage 100 \
 --db-instance-class db.m1.large \
 --apply-immediately \
 --multi-az \
 --new-db-instance-identifier mysqldb

The RDS DB instance gets modified, as shown by command output in Figure 66.

Figure 66. Output from aws rds modify-db-instance

The DB instance Status becomes “modifying”, as shown in Figure 67.

Figure 67. RDS instance Status modifying

Search for ModifyDBInstance in the CloudWatch Log group, and CloudTrail log file/s MySQL database instance gets listed, as shown in Figure 68.

Figure 68. CloudTrail Log File for ModifyDBInstance

Even though the DB instance name is modified from mysql-db to mysqldb, a search for mysql-db in the CloudWatch also lists a log file, as shown in Figure 69, as the modification is made on mysql-db.

Figure 69. CloudTrail Log File for DescribeDBEngineVersion

Deleting a CloudTrail

To delete the RDS-CloudTrail icon, click on the Delete icon as shown in Figure 70.

Figure 70. Deleting a CloudTrail

Click on Delete in the confirmation dialog as show in Figure 71. As indicated in the Delete trail, dialog log files for RDS-CloudTrail won’t be received after the CloudTrail is deleted. And log files already sent won’t be deleted.

Figure 71. Delete trail Confirmation

In this tutorial we discussed proactive logging and monitoring of database instances on RDS using AWS CloudTrail for collecting API calls and CloudWatch log group for searching the CloudTrail log files.