Collecting S3 logs using the Lambda Function

S3 Buckets acts as scalable containers in which large volumes of data can be stored. Site24x7 uses the Lambda Function to look for new logs added in the S3 Buckets and sends it to Site24x7 for indexing. Learn more about log management with Site24x7.

You can also configure your logs to be collected from S3 buckets using SQS. To avoid the overhead of configuring SQS permissions, you can use Lambda Functions to collect your logs as describ
ed below. 

Define the Log Type

A Log Type is a clear definition of the format in which an application writes logs. Different applications (such as IIS, Cassandra, Apache, MySQL) may write logs in different formats. Defining them as Log Types groups logs from different applications to simplify access and assist in efficient searching. 

To create a log type, go to Admin > AppLogs > Log Types > Add Log Type.
You can choose from the list of AWS log types supported out-of-the-box:

You can also define a custom log type to group logs from various applications for easier access and more efficient searching.

1. If you choose Custom Log Type, enter a display name.
2. Provide sample log lines to discover the log pattern.
3. Save the log type.

Once you define a Log Type for your logs stored in your S3 bucket, list it under a Log Profile and start managing your logs by performing search queries. 

Create a Log Profile

A Log Profile enables you to associate log types to a particular log source.

To create a Log Profile, navigate to Admin > AppLogs > Log Profile > Add Log Profile, and follow the instructions below:

1. Profile Name: Enter a name for your Log Profile.
2. Choose the Log Type: Choose the Log Type of the S3 logs you would like to associate with this profile.
3. Log Source: Choose Amazon Lambda.
4. Timezone: Select a timezone for your logs.
5. Click Save.
6. Configure the Lambda function as described here.

Configure the Lambda Function

1. Sign in to Manage AWS Resources - AWS Management Console
   https://us-east-2.console.aws.amazon.com/lambda/home?region=us-east-2#/functions
2. Choose Lambda from the Services drop-down list, and choose Create function.
   
   
3. Select Author from scratch, define a name for the function, and and choose Python 3.12 (the latest available) as the Runtime. 
   
   
4. Permissions: You can either choose an existing IAM role,  or Create a new role from the AWS policy template. 
   Method 1: If you choose an existing IAM role, select the relevant role from the drop down menu shown.
   
   
   Method 2: If you choose to Create a new role from AWS policy template, enter the Role name and select Amazon S3 Object read-only permissions from the Policy templates drop-down. Click Create function. 
   
   
   Note

   In both cases—whether you choose an existing IAM role or create a new role from an AWS policy template—when you enable server-side encryption using AWS Key Management Service (KMS), you should provide the KMS decrypt permission for the KMS key (using its ARN).

   1. You can copy the KMS key ARN from the respective S3 bucket Properties tab by clicking on the click to copy icon.
      
      
   2. Make sure the policy templates are updated with the necessary details. To do this, go to the Identity and Access Management (IAM) menu, click Roles, and select the required role name. Ensure that sufficient permissions, such as the KMS key ARN (if required) and Lambda update function configuration, are included in the policy.  You can find sample policy templates below.
      
      
      
   Sample KMS policy template (This template is required only when KMS encryption is enabled for the S3 bucket.) 

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": "kms:Decrypt",
               "Resource": "arn:aws:kms:us-east-2:<AccountId>:key/<KMSKEY>"
           }
       ]
   }

   Sample Lambda update function configuration template 

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Action": [
                   "logs:CreateLogGroup",
                   "logs:CreateLogStream",
                   "logs:PutLogEvents"
               ],
               "Resource": "arn:aws:logs:::*",
               "Effect": "Allow"
           },
           {
               "Action": [
                   "s3:GetObject",
                   "s3:GetBucketNotification",
                   "s3:PutBucketNotification"
               ],
               "Resource": "arn:aws:s3:::<S3BucketName>",
               "Effect": "Allow"
           },
       ]
   }

5. Add triggers: Click Add trigger.
   
   
   Select the source as S3 bucket. Any log file added to the S3 bucket will be sent to Site24x7 by the Lambda function. 
   
   
6. Configure Triggers
   • Bucket:Enter the name of the S3 bucket from which logs will be collected.
   • Event type: Choose All object create events.
   • Acknowledge and Add.
   
7. In the window that opens, go to Code. Scroll to the editor and place the code provided in the link below:
   https://github.com/site24x7/applogs-aws-lambda/blob/master/s3/s3-sender.py
   
   
8. After entering the code, go to the Site24x7 web client. Navigate to Admin > Applogs > Log Profile, select the created Log Profile, and copy the displayed code for the logTypeConfig.
   
   
9. Back in the AWS Management Console
   • Go to Configuration.
   • Select Environment variables.
   • Click Edit.
   
10. Enter the key as logTypeConfig. Enter the value you copied from the Site24x7 Log Profile page. Click Save.
    
    
11. After saving the environment variables, go to Code. Click Deploy.
    
    
12. After a few minutes, you can go to the Site24x7 web console and select AppLogs from the left menu. On the AppLogs Search page, you can view the logs by searching with the log type as shown below:
    logtype="ELB Application Log"
    
    

Related log types

• CloudFront logs
• CloudTrail logs
• S3 logs