Problem :
Logs are very important aspect of any microservice or any AWS service and in AWS if your rate of generating Cloudwatch logs is high ,then it can increase your AWS costs significantly. There should be some limit on the growth of the logs to keep costs under control.
Solution :
There
are multiple ways to manage the logging. Let's begin with some of the best
practices and then some framework level changes which can help in reducing the
overall logging.
Best Practices :
1. Most of the times we are only interested to find error scenarios from the logs so it is important to use log.error while logging error cases. Never use other logging levels for logging errors
2. Never use print statements instead use a proper logging framework to log the statements
2. Be diligent about the log statements and categorize them correctly at different log levels : DEBUG, INFO, ERROR
4. Don't log data unless there is a critical need for same. Even if required, the data should be printed only at DEBUG level
Framework level changes :
Apart from the best practices listed above, it is also required that we put some controls in place to keep a check on unwanted logging. Here are few services on which I have implemented logging framework .
- Lambda functions - Python/Java
- ECS services- in Java
It is important that we handle logging from both the services.
Idea is to keep the logging levels at minimum for all the development environments. So for most of the dev/test environments, logging will be kept at ERROR while UAT and Prod may run on INFO but can be increased to DEBUG temporarily for troubleshooting.
1. ECS services
Following steps should be taken to retrofit logging in all ECS components:
1. Add following properties in application.properties:
1 2 3 4 5 | logging.level.root=${LOG_LEVEL} LOG_LEVEL=INFO #endpoints.loggers.sensitive=false management.endpoints.web.exposure.include=health,info,loggers org.springframework.boot:type=Endpoint,name=Loggers |
2. Enable Spring Actuator framework by adding following dependencies in pom.xml:
pom.xml:
1 2 3 4 | <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> |
3. Make following changes in CF template of the project :
1. Add security group - Below is a sample config. Rename the variables as per project name and usage :
Preview:
2. HelloComponentSecurityGroup:
3. Type: AWS::EC2::SecurityGroup
4. Properties:
5. GroupName: !Sub "hello-component-${EnvironmentName}-sg"
6. GroupDescription: ECS security group
7. SecurityGroupIngress:
8. - IpProtocol: tcp
9. FromPort: 8080
10. ToPort: 8080
11. CidrIp: 10.0.0.0/8
12. SecurityGroupEgress:
13. - IpProtocol: tcp
14. FromPort: 0
15. ToPort: 1111
16. CidrIp: 0.0.0.0/0
17. VpcId:
18. Fn::ImportValue: "vpc-id"
19. Tags:
20. - Key: Company
21. Value: !Ref CompanyTag
22. - Key: StageName
23. Value: !Ref StageName
24. - Key: EnvironmentName
25. Value: !Ref EnvironmentName
26. - Key: Name
27. Value: !Sub "hello-component-${EnvironmentName}-sg"
28. Make sure to include the security group in ECSService block:
29. EcsService:
30. Type: 'AWS::ECS::Service'
31. Properties:
32. Cluster: !Sub ${ClusterName}-${EnvironmentName}
33. ServiceName: !Sub "${ServiceName}-service"
34. LaunchType: FARGATE
35. DesiredCount: !Ref ServiceDesiredCount
36. TaskDefinition: !Ref TaskDefinition
37. DeploymentConfiguration:
38. MaximumPercent: 200
39. MinimumHealthyPercent: 100
40. NetworkConfiguration:
41. AwsvpcConfiguration:
42. AssignPublicIp: DISABLED
43. ############## Include the statements below #############
44. SecurityGroups:
45. - !Ref HelloComponentSecurityGroup
46. Add another Environment variable in TaskDefinition → Environment :
47. Environment:
48. -
49. Name: LOG_LEVEL
Value: !Ref LogLevel ### (This LogLevel field should be defined in parameters and should be derived through config project)
2. Lambda Services
Let's try to define logging strategy for both Python and Java based Lambda functions:
2.1. Python based lambda
Python has inbuilt logging framework which can be used to log statements at different logging levels (much like in Java). Following steps can be used to enable the same:
1. import logging in main lambda function python file (file which has lambda_handler) and before beginning of the function write following statements:
import logging logger = logging.getLogger() logger.setLevel(os.environ['LOGGING_LEVEL'])
2. Use log.info, log.error or log.debug instead of context.log to log statements appropriately.
2.2. Java based Lambda
Unfortunately, for the lambda(s) built in Java, the standard context logging framework does not support different logging levels. So we will have to use aws-lambda-java-log4j framework to make logging inline with other compute resources.
1. Add following dependencies in pom.xml :
pom.xml:
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-log4j2</artifactId>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
</dependency>
2. Paste following file in src/main/resources/log4j.xml:
log4j.xml:
<?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd"> <log4j:configuration debug="true" xmlns:log4j='http://jakarta.apache.org/log4j/'> <appender name="file" class="org.apache.log4j.RollingFileAppender"> <param name="append" value="false" /> <param name="maxFileSize" value="10KB" /> <param name="maxBackupIndex" value="5" /> <!-- For Tomcat --> <param name="file" value="${catalina.home}/logs/myStruts1App.log" /> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n" /> </layout> </appender> <root> <level value="ERROR" /> <appender-ref ref="file" /> </root> </log4j:configuration>
We need to replace context.log statements to log.debug /log.info / log.error statements as applicable. But before we use log statements, we need to define the logger for the given class:
private static final Logger log = LogManager.getLogger(Hello.class);
....
log.error("Error: Exception processing input" + e.getMessage(), e);
