Logs collection from AWS Elasticbeanstalk to Splunk
What is AWS Elasticbeanstalk
AWS Elasticbeanstalk is an AWS service that comes as Platform as a Service (PaaS). It provides a standardised way to manage and deploy applications in the AWS cloud, so its primary purpose is to reduce management complexity without restricting control. In general, you just upload application and AWS Elasticbeanstalk takes care of the environment for the application, interconnection between required AWS resources, and provides monitoring and maintenance tools.
When Smartling adopted Service Oriented Architecture (SOA) we chose AWS Elasticbeanstalk as one of the options for Java based services — it provides teams with a reliable and scalable/redundant platform that is easy to use. Teams can manage their own infrastructure, do capacity planning and provisioning, and define deployments and monitoring. The whole point of SOA is ownership — when teams own not just code, but service infrastructure and does its own maintenance as well — this is an essential part of doing DevOps.
Logs collection in AWS Elasticbeanstalk
Once we decided to go with AWS Elasticbeanstalk the first problem that had to be solved was logs collection and aggregation so that those logs would be available for service team owners in near real time. AWS logs shipping is not great, so we decided to use Splunk.
In a nutshell, we needed to install the Splunk Universal Forwarder to instances in AWS Elasticbeanstalk and configure them to collect the required logs and forward them to Splunk Enterprise for consolidation.
This is where ebextensions is useful.
1. Configure your AWS Elasticbeanstalk environment with the following environment variables:
APPLICATION_NAME — name of an application as it will be shown in SplunkSPLUNK_FORWARDER_RPM_DOWNLOAD_URL — URL from where to download Splunk forwarderSPLUNK_SERVER_HOST — Splunk server host that will access logs from splunk forwarders
These can set these variable using AWS console or AWS Elasticbeanstalk configuration files.
Example:
APPLICATION_NAME = MyJavaApp-EBSPLUNK_FORWARDER_RPM_DOWNLOAD_URL = https://download.splunk.com/products/splunk/releases/6.2.2/universalforwarder/linux/splunkforwarder-6.2.2-255606-linux-2.6-x86_64.rpm SPLUNK_SERVER_HOST = 10.20.30.40
2. Add .ebextensions directory into the source bundle (usually zip file) of your application that is going to be deployed to AWS Elasticbeanstalk. Create 101splunk-fowarder.config in .ebextensions with the following contents:
Warning #1: This ebextension was tested with Amazon Linux. If other Linux is used, e.g. Ubuntu, you must alter it, accordingly.
Warning #2: Configuration expects application to write logs to /var/log/tomcat8. Ebextension will need to be changed if application logs are located somewhere else.
3. Deploy source bundle containing .ebextensions/101splunk-fowarder.config. During the deploy process, Splunk Fowarder will be installed, configured and started on every Elasticbeanstalk EC2 instance
Here are some Splunk query examples, note APPLICATION_NAME variable value (MyJavaApp-EB) is a part of search query:
host="MyJavaApp-EB/*"Splunk query to see only one log file at particular EC2 instance in AWS Elasticbeanstalk will be:
host="MyJavaApp-EB/i-1234567" source="/var/log/tomcat8/catalina.out"Logs collection from Docker in AWS Elasticbeanstalk
For applications running in ElasticBeanstalk under Docker there is a way to have log collection with Splunk as well.
1. Add Dockerrun.aws.json to source bundle:
This file instructs Docker to map a volume from an EC2 instance to Docker container.
2. Configure application in Docker to store logs into /var/log/myapplicationlogs directory and prepare logs directory for it. Here’s an example of a Dockerfile section for this:
...
RUN mkdir /var/log/myapplicationlogs
RUN chown jboss.jboss /var/log/myapplicationlogs
...
ADD logging.properties /opt/jboss/standalone/configuration
...3. Add /var/log/myapplicationlogs directory to Splunk Forwarder in .ebextensinons/101splunk-fowarder.config (see it above) and put this config into source bundle:
/opt/splunkforwarder/bin/splunk add monitor "/var/log/myapplicationlogs/*.log" -hostname "$splunk_logs_hostname" -sourcetype log4j4. Deploy an application using source bundle containing .ebextensinons/101splunk-fowarder.config.
