HOWTO: Host your own reporting platform on AWS¶

Metabase and Grafana are both quite capable reporting platforms. Both vendors offer you the ability to run the software for free on your own infrastructure, so it's quite an attractive option for companies who would like to have some business intelligence capability without breaking the bank. In this guide, I will show you how can use the Cloudformation template to deploy these reporting tools on your aws environment.

System Architecture¶

The stack is built around Fargate, running the Docker image, an Application Load Balancer, and a database, responsible for storing the platform configuration. What is not included on this archtecture, is the VPC itself.

graph TD
    subgraph "VPC"
        direction LR
        ALB[Application Load Balancer]
        ECS[ECS Cluster]
        Task[Fargate Task]
        RDS[(RDS PostgreSQL)]
        Secrets[Secrets Manager]
        CWLogs[CloudWatch Logs]

        ALB --> Task
        ECS --> Task
        Task --> RDS
        Secrets --> Task

        Secrets --> RDS
    end

    Task --> CWLogs

    Browser --> ALB
    Task --> Data[(External Data source)]

Before you deploy the stack, you will need to have a VPC up and running. The VPC need to allow the creation of a public IP address, or at the very least, allow outbound connectivity via a NAT gateway. If that's all good to go, you can deploy this Cloudformation Stack, which will create everything you need.

Parameters¶

You will be presented with a number of parameters. Most of them are self explanatory.

Parameter Name	Default	Description
`CreateDatabase`	True	Specify whether to create a database
`DBSize`	20	If `CreateDatabase` is `True` : Specify the size of the database in GB
`MultiAZ`	False	If `CreateDatabase` is `True` : Specify if Multi-AZ deployment for the database is required
`DBInstanceClass`	db.t3.small	If `CreateDatabase` is `True` : Database instance class
`DBName`	metabase	If `CreateDatabase` is `True` : Database name
`DBUsername`	metabase	If `CreateDatabase` is `True` : Database admin username
`DBPassword`	–	If `CreateDatabase` is `True` : Database admin password
`Image`	metabase/metabase	Specify the Docker image to deploy, either Metabase or Grafana
`VpcId`	–	The ID of the VPC where the container will be deployed
`SubnetIds`	–	The subnet IDs to use for the ALB and Fargate
`FargateSize`	1024.2048	The CPU/memory size to use for Fargate
`DesiredCount`	1	How many Fargate instances would you like to run?

Just testing?¶

If you just want to play around with the tools, I would suggest you run with CreateDatabase set to False, which will not spin up the RDS instance, which is a significant reduction in cost.

Warning

If you are running with CreateDatase set to False, the state, and any dashboards you create will be lost the second the Fargate container is terminated.

It takes around 15 minutes for the Cloudformation stack to fully deploy, so be patient while it boots up. Once it's up, you'll have access to the ALB. Just look at the "Outputs" of the stack to find the DNS name.

For Metabase, open the ALB, and follow the steps to configure the system. Continue with the configuration.
For Grafana, open the ALB and logon with admin / admin. Continue with the configuration.

Production scale¶

The database parameters allow you to enable things like MultiAZ (allowing the database to run across multiple availability zones), and increasing the instance size if you choose to. This will allow for full redundancy, and high performance of your instance.

The DesiredCount will also specify how many containers you would like to run. For production, I would suggest you go to at least 2. Also consider increasing the Fargate size, to allow the container more room to breath.

Warning

The database created in Cloudformation is the configuration database, not the database used to host your data sources. Make sure that your database backup is configured per your requirements, because if you loose this database, you loose all your dashboards and your reporting configuration.

What's next?¶

Configure DNS / Route 53¶

It is recommended that you update the DNS entry for the Load Balancer, so it is easier to reach.

SSL / TLS¶

By default the stack does not deploy a TLS listener, since doing TLS through CloudFormation can be very tricky, as you have to manually approve certificates in ACM, and create entries in Route53, which may not be hosted in the same AWS account. Getting the ALB updated to SSL and TLS is up to you.

Hosting it internally¶

In some cases, having the reporting solution publically available may not be an option. This is entirely possible, yet outside the scope of the current Cloudformation stack. Fargate will require internet access in some shape or form to retrieve the public images, so having access to a NAT gateway or internet gateway for outbound communication is a must-have, yet it is entirely possible to run the Fargate container on an internal network without any public IP addresses. Feel free to modify the stack to make it work for you.

Updating the security groups¶

By default, we will allow anyone on the internet to connect to the instance through the 0.0.0.0/0 CIDR ingress rule.

Is it really free?¶

Running the software on your own infrastructure is free, however you will incur the hosting fees. The basic template, with the small RDS instance will cost ~$80 US a month. If you go for MultiAZ, more instances, and a bigger database, the cost could go up to a few hundred dollars a month.

Do consider that the equivalent SaaS offering from these companies do vary from $5 p/m to $20p/m per user. There is no right or wrong answer if you should go with SaaS, or host it yourself. While hosting it yourself will have some cost appeal to it, you end up becoming responsible to support the platform. Consider the ongoing maintenance and support as part of your decision making process.

Cloudformation Stack