Patch Autoscaling Group Instances
Run the Automation
Patching of
Autoscaling Group Instances is different than patching normal/ standalone
instances. Companies uses SSM and Lambda functions to patch Autoscaling instances.
Steps s to patch the Autoscaling
instances.
1. Create SSM Parameters to store SSM
document inputs
2. Create IAM roles required by
different services.
3. 2 AWS lambda functions for changing
ASG configuration and ASG instances capacity.
4. Create SSM document for patching and
calling Lambda functions.
For patching
first SSM document creates new ec2 instance from the ami used by autoscaling
and apply the patching to temporarily created instance ( SSM patch baseline of
Linux distribution used for autoscaling, should be modified to cover all
vulnerabilities). SSM then create new AMI from the patched instance and send
ami id and other ASG details to 1st lambda function.
Lambda
function then creates new Launch configuration same as exiting LC used by ASG
and only replace the new AMI id. Creates two SSM parameter stores for storing
DesiredCapacity and MaxSize of Autoscaling Group. And attach the new launch
configuration to ASG.
Once the ASG
is updated with new launch configuration, increase the ASG max size and desired
capacity with existing desired size.
Ex: If
desired capacity is 2 and max size is 4. Increase desired capacity to 4 (2 +2)
and max size to 6 (4+2).
Once the size
is change ASG launch new instance using latest patched AMI of recently created
launch configuration. SSM will sleep for 5 minutes and call the second lambda
function which reduce the ASG instance size to its initial.
By default,
ASG while scaling in terminates the older instances and deletes the instances
which are not patched. And SSM updates
the updated AMI in parameter store for next event.
Note: Please make
sure that ssm agent is installed on launch configuration’s AMI. SSM agent is
used to install the patches.
Required SSM parameters:
- · SSM parameter for ASG name. Replace (TestASGPatchingASGNameParam) name in SSM document with your ssm parameter name.
- eg) TestASGPatchingASGNameParam = testasgssm-asg (Name of the AutoScaling group)
- · SSM parameter for launch
configuration AMI id. Replace (TestASGPatchingAMIParam) name in ssm document with your ssm
parameter name.
- eg) TestASGPatchingAMIParam = ami-022e28a46f85f8cb6
- · SSM parameter for subnet ID. This
subnet id is used to launch instance temp instance for patching AMI.
(subnet-id)
- eg) subnet-id = subnet-09f645fff8627bcd2 (Subnet ID of the EC2 Instance.
Required IAM Roles:
1. IAM role required by Lambda
functions. ASGSSMLambdaRoles with Trusted Entity: Lambda
o
Permission:
AWSLambdaExecute, AWSAutoScalingFull, AWSLambdaVPCAccessExecutionRole and
Custom permission for SSM parameter Read and update. Json policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"ssm:ListCommands",
"ssm:ListDocumentVersions",
"ssm:DescribeInstancePatches",
"ssm:ListInstanceAssociations",
"ssm:GetParameter",
"ssm:GetMaintenanceWindowExecutionTaskInvocation",
"ssm:DescribeAutomationExecutions",
"ssm:GetMaintenanceWindowTask",
"ssm:DescribeMaintenanceWindowExecutionTaskInvocations",
"ssm:DescribeAutomationStepExecutions",
"ssm:DescribeParameters",
"ssm:ListResourceDataSync",
"ssm:ListDocuments",
"ssm:GetConnectionStatus",
"ssm:GetMaintenanceWindowExecutionTask",
"ssm:GetMaintenanceWindowExecution",
"ssm:GetParameters",
"ssm:PutParameter",
"ssm:DescribeMaintenanceWindows",
"ssm:DescribeEffectivePatchesForPatchBaseline",
"ssm:DescribeDocumentPermission",
"ssm:ListCommandInvocations",
"ssm:GetAutomationExecution",
"ssm:DescribePatchGroups",
"ssm:GetDefaultPatchBaseline",
"ssm:DescribeDocument",
"ssm:DescribeMaintenanceWindowTasks",
"ssm:ListAssociationVersions",
"ssm:GetPatchBaselineForPatchGroup",
"ssm:PutConfigurePackageResult",
"ssm:DescribePatchGroupState",
"ssm:DescribeMaintenanceWindowExecutions",
"ssm:GetManifest",
"ssm:DescribeMaintenanceWindowExecutionTasks",
"ssm:DescribeInstancePatchStates",
"ssm:DescribeInstancePatchStatesForPatchGroup",
"ssm:GetDocument",
"ssm:GetInventorySchema",
"ssm:GetParametersByPath",
"ssm:GetMaintenanceWindow",
"ssm:DescribeInstanceAssociationsStatus",
"ssm:GetPatchBaseline",
"ssm:DescribeInstanceProperties",
"ssm:ListInventoryEntries",
"ssm:DescribeAssociation",
"ssm:GetDeployablePatchSnapshotForInstance",
"ssm:DescribeSessions",
"ssm:GetParameterHistory",
"ssm:DescribeMaintenanceWindowTargets",
"ssm:DescribePatchBaselines",
"ssm:DescribeEffectiveInstanceAssociations",
"ssm:GetInventory",
"ssm:DescribeActivations",
"ssm:GetCommandInvocation",
"ssm:DescribeInstanceInformation",
"ssm:ListTagsForResource",
"ssm:DescribeDocumentParameters",
"ssm:ListAssociations",
"ssm:DescribeAvailablePatches"
],
"Resource": "*"
}
]
}
|
Incase if you want more granular w.r.to ARNs,then REWRITE the same Json policy as
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"ssm:DescribeAssociation",
"ssm:GetDefaultPatchBaseline",
"ssm:DescribeDocument",
"ssm:GetParameterHistory",
"ssm:DescribeMaintenanceWindowTargets",
"ssm:DescribeMaintenanceWindowTasks",
"ssm:GetPatchBaselineForPatchGroup",
"ssm:GetParameters",
"ssm:GetParameter",
"ssm:DescribeMaintenanceWindowExecutions",
"ssm:GetMaintenanceWindowTask",
"ssm:PutParameter",
"ssm:DescribeInstanceInformation",
"ssm:ListTagsForResource",
"ssm:DescribeDocumentParameters",
"ssm:DescribeEffectivePatchesForPatchBaseline",
"ssm:GetDocument",
"ssm:GetMaintenanceWindow",
"ssm:GetParametersByPath",
"ssm:DescribeDocumentPermission",
"ssm:GetPatchBaseline",
"ssm:DescribeInstanceProperties"
],
"Resource": [
"arn:aws:ssm:us-east-1:A/C-No:parameter/*",
"arn:aws:ssm:us-east-1:A/C-No:patchbaseline/*",
"arn:aws:ssm:us-east-1:A/C-No:document/*",
"arn:aws:ssm:us-east-1:A/C-No:maintenancewindow/*",
"arn:aws:ssm:us-east-1:A/C-No:windowtask/*"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"ssm:GetAutomationExecution",
"ssm:DescribePatchGroups",
"ssm:ListDocumentVersions",
"ssm:ListCommands",
"ssm:ListAssociationVersions",
"ssm:DescribeInstancePatches",
"ssm:PutConfigurePackageResult",
"ssm:ListInstanceAssociations",
"ssm:DescribePatchGroupState",
"ssm:GetMaintenanceWindowExecutionTaskInvocation",
"ssm:DescribeAutomationExecutions",
"ssm:GetManifest",
"ssm:DescribeMaintenanceWindowExecutionTaskInvocations",
"ssm:DescribeMaintenanceWindowExecutionTasks",
"ssm:DescribeAutomationStepExecutions",
"ssm:DescribeInstancePatchStates",
"ssm:DescribeInstancePatchStatesForPatchGroup",
"ssm:DescribeParameters",
"ssm:ListResourceDataSync",
"ssm:GetInventorySchema",
"ssm:ListDocuments",
"ssm:DescribeInstanceAssociationsStatus",
"ssm:ListInventoryEntries",
"ssm:GetConnectionStatus",
"ssm:GetMaintenanceWindowExecutionTask",
"ssm:GetDeployablePatchSnapshotForInstance",
"ssm:DescribeSessions",
"ssm:GetMaintenanceWindowExecution",
"ssm:DescribePatchBaselines",
"ssm:DescribeEffectiveInstanceAssociations",
"ssm:GetInventory",
"ssm:DescribeActivations",
"ssm:GetCommandInvocation",
"ssm:DescribeMaintenanceWindows",
"ssm:ListAssociations",
"ssm:ListCommandInvocations",
"ssm:DescribeAvailablePatches"
],
"Resource": "*"
}
]
}
|
2. Instance role required by Temp EC2
Instance. TestASGPatchingEC2Role
o
Trusted Entity: EC2
3. Automation role for automation
service TestASGPatchingAutomationRole
o
Permission: AmazonSSMAutomationRole and Inline policy for passrole
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource":
"arn:aws:iam::*:role/*"
}
]
}
|
o
Trusted
Entity: EC2 service and SSM service
SSM document:
SSM document used for this automation:
Note: Please replace the existing ssm parameter names and role name
with name newly created ssm parameters and roles.
{
"description": "Systems Manager Automation Demo - Patch AMI and Update ASG",
"schemaVersion": "0.3",
"assumeRole": "{{AutomationAssumeRole}}",
"parameters": {
"AutomationAssumeRole": {
"type": "String",
"description": "(Required) The ARN of the role that allows Automation to perform the actions on your behalf.",
"default": "arn:aws:iam::{{global:ACCOUNT_ID}}:role/TestASGPatchingAutomationRole"
},
"sourceAMIid": {
"type": "String",
"default": "{{ssm:TestASGPatchingAMIParam}}",
"description": "(Required)Edit default value with name of your accounts ASG AMI parameter store. Valid format {{ssm:your param name}}"
},
"sourceAMIidParamName": {
"type": "String",
"default": "TestASGPatchingAMIParam",
"description": "(Required)Edit default value with name of your accounts ASG AMI parameter store. Valid format: param name"
},
"targetAMIname": {
"type": "String",
"description": "(optional)Name of new AMI",
"default": "patchedAMI-{{global:DATE_TIME}}"
},
"targetASG": {
"type": "String",
"default": "{{ssm:TestASGPatchingASGNameParam}}",
"description": "(Required)Edit default value with name of your accounts ASG name parameter store. Valid format: {{ssm:your param name}}"
},
"ec2InstanceRoleName": {
"type": "String",
"default": "TestASGPatchingEC2Role",
"description": "(Required)ec2 role name created as per document"
},
"SubnetID": {
"type": "String",
"description": "(Required) The SubnetID of VPC in which Automation will launch instance.Valid format: {{ssm:your param name}}",
"default": "{{ssm:subnet-id}}"
},
"ASGDesiredCountParamName": {
"type": "String",
"default": "ASGDesiredCapacity",
"description": "(optional)ssm parameter name to store ASG desired capacity. Lambda will create paramer with provided name"
},
"ASGMaxSizeParamName": {
"type": "String",
"default": "ASGMaxSize",
"description": "(optional)ssm parameter name to store ASG Max Size. Lambda will create paramer with provided name"
},
"lambdafunction1": {
"type": "String",
"default": "Automation-ASGPatchingTestLamda",
"description": "(Required)lambda function name which creates new LC and increase ASG capacity"
},
"lambdafunction2": {
"type": "String",
"default": "Automation-ASGPatchingTestLamb2",
"description": "(Required)lambda function name which reset the ASG size to its initial"
}
},
"mainSteps": [
{
"name": "startInstances",
"action": "aws:runInstances",
"timeoutSeconds": 1200,
"maxAttempts": 1,
"onFailure": "Abort",
"inputs": {
"ImageId": "{{ sourceAMIid }}",
"InstanceType": "t2.micro",
"SubnetId": "{{SubnetID}}",
"SecurityGroupIds": [
"sg-073186e7592b6a012"
],
"MinInstanceCount": 1,
"MaxInstanceCount": 1,
"IamInstanceProfileName": "{{ec2InstanceRoleName}}"
}
},
{
"name": "runPatchBaseline",
"action": "aws:runCommand",
"maxAttempts": 1,
"onFailure": "Continue",
"inputs": {
"DocumentName": "AWS-RunPatchBaseline",
"InstanceIds": [
"{{ startInstances.InstanceIds }}"
],
"Parameters": {
"Operation": "Install"
}
}
},
{
"name": "stopInstance",
"action": "aws:changeInstanceState",
"maxAttempts": 1,
"onFailure": "Continue",
"inputs": {
"InstanceIds": [
"{{ startInstances.InstanceIds }}"
],
"DesiredState": "stopped"
}
},
{
"name": "createImage",
"action": "aws:createImage",
"maxAttempts": 1,
"onFailure": "Continue",
"inputs": {
"InstanceId": "{{ startInstances.InstanceIds }}",
"ImageName": "{{ targetAMIname }}",
"NoReboot": true,
"ImageDescription": "AMI created by EC2 Automation"
}
},
{
"name": "terminateInstance",
"action": "aws:changeInstanceState",
"maxAttempts": 1,
"onFailure": "Continue",
"inputs": {
"InstanceIds": [
"{{ startInstances.InstanceIds }}"
],
"DesiredState": "terminated"
}
},
{
"name": "updateASGLaunchConfiguration",
"action": "aws:invokeLambdaFunction",
"timeoutSeconds": 1200,
"maxAttempts": 1,
"onFailure": "Abort",
"inputs": {
"FunctionName": "{{lambdafunction1}}",
"Payload": "{\"targetASG\":\"{{targetASG}}\", \"newAmiID\":\"{{createImage.ImageId}}\", \"ASGDesiredCapacity\":\"{{ASGDesiredCountParamName}}\",\"ASGMaxSize\":\"{{ASGMaxSizeParamName}}\"}"
}
},
{
"name": "sleep",
"action": "aws:sleep",
"inputs": {
"Duration": "PT5M"
}
},
{
"name": "updateASGtoInitialSize",
"action": "aws:invokeLambdaFunction",
"timeoutSeconds": 1200,
"maxAttempts": 1,
"onFailure": "Abort",
"inputs": {
"FunctionName": "{{lambdafunction2}}",
"Payload": "{\"targetASG\":\"{{targetASG}}\",\"ASGDesiredCapacity\":\"{{ASGDesiredCountParamName}}\",\"ASGMaxSize\":\"{{ASGMaxSizeParamName}}\"}"
}
},
{
"name": "updateASGami",
"action": "aws:executeAwsApi",
"inputs": {
"Service": "ssm",
"Api": "PutParameter",
"Name": "{{sourceAMIidParamName}}",
"Value": "{{createImage.ImageId}}",
"Type": "String",
"Overwrite": true
}
}
],
"outputs": [
"createImage.ImageId"
]
}
|
Lambda Functions:
Go to and create the function
Patching uses two lambda functions
o
First
lambda function creates launch configuration, attaching it to ASG and
increasing desired size and maxSize of ASG.
o
Second
lambda function for resizing the ASG to its initial size.
Please make sure that your lambda function name starts with Automation.
As AWS added permission to call the lambda functions starts with Automation in
its AmazonSSMAutomation Policy.
Note: Functions are written in python
language and compatible with python2.7 version. Attach the Lambda role created
in Role section to these functions at the time of launch. and increase the Timeout on Basic settings of Lambda from 3 seconds to 1 minute
Lambda function 1: Automation-ASGPatchingTestLamda
- Add a Tag,
- Select the Execution role as ASGSSMLambdaRoles,
- Set timeout as 1 minute in Basic Settings,
- Select the VPC, Subnets & Security groups,
- Paste this python script and create the Lambda function .
from __future__ import print_function
import json
import datetime
import time
import boto3
print('Loading function')
def lambda_handler(event, context):
print("Received event: " + json.dumps(event, indent=2))
# get autoscaling client
clientASG = boto3.client('autoscaling')
# get object for the ASG we're going to update, filter by name of target ASG
response = clientASG.describe_auto_scaling_groups(AutoScalingGroupNames=[event['targetASG']])
if not response['AutoScalingGroups']:
return 'No such ASG'
# get name of InstanceID in current ASG that we'll use to model new Launch Configuration after
sourceInstanceId = response.get('AutoScalingGroups')[0]['Instances'][0]['InstanceId']
# create LC using instance from target ASG as a template, only diff is the name of the new LC and new AMI
timeStamp = time.time()
timeStampString = datetime.datetime.fromtimestamp(timeStamp).strftime('%Y-%m-%d %H-%M-%S')
newLaunchConfigName = 'LC '+ event['newAmiID'] + ' ' + timeStampString
clientASG.create_launch_configuration(
InstanceId = sourceInstanceId,
LaunchConfigurationName=newLaunchConfigName,
ImageId= event['newAmiID'] )
# update ASG to use new LC
responseASG = clientASG.update_auto_scaling_group(AutoScalingGroupName = event['targetASG'],LaunchConfigurationName = newLaunchConfigName)
#get ASG objet to read ASG Values
#responseASG = clientASG.describe_auto_scaling_groups(AutoScalingGroupNames=[event['targetASG']])
#get desired instance capacity from ASG
desiredCapacity = response.get('AutoScalingGroups')[0]['DesiredCapacity']
#get max instance cound from ASG
maxSize = response.get('AutoScalingGroups')[0]['MaxSize']
#create ssm client
clientSSM = boto3.client('ssm')
#store the desiredCapacity value in parameter store for later use
clientSSM.put_parameter(
Name = event['ASGDesiredCapacity'],
Description = "Parameter store which stores the ASG desired capacity value. Please do not update param name",
Value = str(desiredCapacity),
Overwrite = True,
Type = 'String'
)
#store the MaxSize value in parameter store for later use
clientSSM.put_parameter(
Name = event['ASGMaxSize'],
Description = "Parameter store which stores the ASG Max count value. Please do not update param name",
Value = str(maxSize),
Type = 'String',
Overwrite = True
)
# update ASG to use new ASG size
response = clientASG.update_auto_scaling_group(AutoScalingGroupName = event['targetASG'],MaxSize = maxSize + desiredCapacity , DesiredCapacity = 2 * desiredCapacity )
return 'Updated ASG `%s` with new launch configuration `%s` which includes AMI `%s`.' % (event['targetASG'], newLaunchConfigName, event['newAmiID'])
|
Lambda function 2: Automation-ASGPatchingTestLamb2
- Add a Tag,
- Select the Execution role as ASGSSMLambdaRoles,
- Set timeout as 30 secounds in Basic Settings,
- Select the VPC, Subnets & Security groups,
- Paste this python script and create the Lambda function .
from __future__ import print_function
import json
import datetime
import time
import boto3
print('Loading function')
def lambda_handler(event, context):
print("Received event: " + json.dumps(event, indent=2))
# get autoscaling client
clientASG = boto3.client('autoscaling')
# get object for the ASG we're going to update, filter by name of target ASG
response = clientASG.describe_auto_scaling_groups(AutoScalingGroupNames=[event['targetASG']])
if not response['AutoScalingGroups']:
return 'No such ASG'
#get ASG objet to read ASG Values
#responseASG = clientASG.describe_auto_scaling_groups(AutoScalingGroupNames=[event['targetASG']])
#get desired instance capacity from ASG
desiredCapacity = response.get('AutoScalingGroups')[0]['DesiredCapacity']
#get max instance cound from ASG
maxSize = response.get('AutoScalingGroups')[0]['MaxSize']
#create ssm client
clientSSM = boto3.client('ssm')
#get the desiredCapacity value in parameter store
desiredCapacity = clientSSM.get_parameter(Name = event['ASGDesiredCapacity']).get('Parameter')['Value']
#get the MaxSize value in parameter store
maxSize = clientSSM.get_parameter(Name = event['ASGMaxSize']).get('Parameter')['Value']
# update ASG to use new ASG size
response = clientASG.update_auto_scaling_group(AutoScalingGroupName = event['targetASG'],MaxSize = int(maxSize), DesiredCapacity = int(desiredCapacity))
return 'Updated ASG `%s` with new DesiredCapacity of `%s` and new MaxSize of `%s`.' % (event['targetASG'], desiredCapacity, maxSize)
|
- Go to AWS Systems Manager >> Automation >> Execute Automation and search by Owner then owned by me,
- Select the SSM Document version and then click next,
- Check the input parameters and click the execute button.
- and then you should see,
Incase if you dont have Admin privilege on your AWS account. then you need the below permissions for the limited account.
1 | AWS Managed Policy | |||
2 | AWS Managed Policy | |||
3 | AWS Managed Policy | |||
4 | AWS Managed Policy | |||
5 | AWS Managed Policy | |||
6 | AWS Managed Policy | |||
7 | AWS Managed Policy | |||
8 | AWS Managed Policy | |||
9 | AWS Managed Policy | |||
10 | AWS Managed Policy | |||
11 | Custom Policy | |||
12 | Custom Policy | |||
13 | Custom Policy | Custom Policy under the ASGSSMLambdaRole |
LambdaListPolicies
ListPolicies
ASGSSMLambdaRoles --Custom Policy under the ASGSSMLambdaRole
No comments:
Post a Comment