# Introduction

Mastodon requires a server (physical or virtual), and it’s also highly recommended to use an S3 bucket to store all local/remote media. It also requires a lot of time, resources and mental energy.

Any cloud provider should be fine. Try to find something local to your region. I’m in Sydney, so I’m using Linode, and Wasabi because they have Sydney based data centers, and offer good prices.

## Introduction to Scaling

Mastodon is built around multiple pieces of technology, and each of them can be tweaked to improve performance.

Key themes of this document is to highlight bottle necks and allow for scaling. All of these services can be run from a single box, but we HIGHLY recommend splitting them up as you start to scale (as soon as you can afford it).

Firstly, Please read the blogs written by Gargon and the official documentation on scaling. We will be using these for future reference.

\
[https://blog.joinmastodon.org/2017/04/scaling-mastodon/<br>](https://blog.joinmastodon.org/2017/04/scaling-mastodon/)<https://docs.joinmastodon.org/admin/scaling/>\
<https://gist.github.com/Gargron/aa9341a49dc91d5a721019d9e0c9fd11>

## Mastodon: Piece by Piece

A stable and high user Mastodon instance is made up of multiple pieces which are all working together.

* Patreon (or similar) to cover operating costs.
* DNS for internet facing resources.
* Email provider to send signups/password resets and other notifications.
* Cloud Storage (S3 compatable) to store and serve media (images/videos/avatars/etc).
* CDN to store and server static resources (js/css/svg/etc).
* VMs to host the Web servers (Frontend/API/Streaming/etc)
* VMs to host the Sidekiq Queue worker processes.
* VMs to host the Databases (Postgres) and Cache (Redis).
* Additional VMs might be required for monitoring/metrics and reporting.

Each of these pieces can be tuned, and optimised for cost/performance. I will make recommendations for all of these.

<figure><img src="/files/ZQiKx4wATvnBbIMSGelE" alt=""><figcaption><p>High Level diagram of Aus.Social (NOV 2022)</p></figcaption></figure>

<figure><img src="https://lh6.googleusercontent.com/_mWsfCArEKIr4YwIzI5t5X3yEwAa-XgUECQelS135_w6R6g60pAVbWIPuRE9Ob5Jdv2SNSkCIvnYfHDE421CNEyKMvRc5RWbPVgTAo-jc8u0jrE-l3GFW7TtyrB81JpsZHi-2LoU0CzMCm1a3mLLX9aCNDIonb_jtb-PhTUn0HjS9FwU3wp76ledOU9BPw" alt=""><figcaption><p>Generic diagram of a Mastodon Instance</p></figcaption></figure>


# Mental Health

## Technical Operational Stress

I've found the stress of having to maintain an instance to have a negative effect on my mental health, and requires a lot of time and resources to manage it.

Consider getting an ops team. Share the load.&#x20;

Also, if you are a single operator/admin of an instance, please write a guide and share your SSH key with Shamir's Secret Sharing or similar (this splits your document into pieces, and requires multiple pieces to rebuild it). In the unfortunate situation you find yourself dead or even on holiday, you might require the key to be rebuilt.

My plan is to write documentation on my instance and give this to a bunch of different instance admins in my region whom I have a strong community trust in... I'll give out 5 parts and require 3 to rebuild (in case somebody loses their piece).\
\
<https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing>

## Moderation Stress

Moderation sucks. It's depressing because the core reason it exists is to react and be an arbiter to the worse sides of the fediverse. People are not reporting good vibes and fun jokes.

You see the worse in people, and sometimes the reports are tramatic. Share the load and try not to dedicate yourself to both technical and moderation tasks.

It's also important to build a community of mods which share your instances and personal values. These people will speak for you. They will make decisions without you, and you need to trust them.


# Rules and Culture / Terms of Service

## Define your culture upfront

My instance is a progressive instance which requires people to have a shared values. My instance has a strong anti-brands/people only policy. You might feel that your instance should nuture a slightly different culture, but you need to think about what kind of instance you want as soon as possible.

* What's the minimal age to join?
* Do you allow NSFW? Do you allow Sex workers?
* Do you allow brands? or Companies?

## Examples

<https://mastodon.nz/rules.pdf><br>


# Cloud Servers (VMs)

VMs are generally pretty “affordable” these days.. But if you’re planning to scale to thousands of concurrent people on your instance, you will likely find yourself having multiple VMs.

Our goal with this guide is to optimise the VMs to run as high performance as possible on hopefully the smallest instances we can use.

## Example:&#x20;

Aus.Social currently has too many VMs, but for reference:

* PostgreSQL and Redis : 4 CPU / 8 GB VM
* ElasticSearch : 2 CPU / 4 GB VM
* Sidekiq : 6 CPU / 16 GB VM
* Nginx: 2 CPU / 4GB VM
* Puma/Streaming : Two x 4 CPU / 8 GB VM

This setup will allow me to increase the VMs as required but is currently running at around 50% utilisation on the Nginx / web server when hosting 10,000\~ active users (with average on and off usage).

<br>


# Cloud Media Storage (S3 Buckets)

There are plenty of cloud storage providers that support the AWS S3 bucket hosting.&#x20;

AWS S3 has 99.999% SLA, but also very very expensive. Don’t use it.

* [Amazon Simple Storage Service (S3)](https://aws.amazon.com/s3/)
* [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces)
* [Linode Object Storage](https://www.linode.com/products/object-storage/)
* [Vultr Object Storage](https://www.vultr.com/products/object-storage/)
* [Exoscale Simple Object Storage (SOS)](https://www.exoscale.com/object-storage/)
* [OVH Object Storage](https://www.ovhcloud.com/en-au/public-cloud/object-storage/)
* [Wasabi Cloud Storage](https://wasabi.com/s3-compatible-cloud-storage/)&#x20;
* [Backblaze Cloud Storage](https://www.backblaze.com/b2/cloud-storage.html)
* [Jortage Communal Cloud](https://jortage.com/) (huge fan of this service for US instances)
* Bunny.net Cloud Storage (coming 2023)

## Recommendations

* Just try to find one in your region… not a lot “tuning” that can be done on this front.
* I'd recommend putting a Media Subdomain (<https://media.aus.social>) or similar to front your traffic. [Read about that here](/basics/cloud-media-storage-s3-buckets/media-subdomain).

### AWS S3

Do not for any reason use AWS S3... they charge you for storage and upload/download.

<figure><img src="/files/2GJBR7clOQS2bhAk6HzE" alt=""><figcaption><p>I was using AWS S3, and once more than a few thousand people joined... it went from $30 to $1000s a month for something Wasabi charges $10</p></figcaption></figure>

### Wasabi

I'm using Wasabi, because they are very cheap. They have a history of outages in their primary regions, but in Sydney, I haven't experienced any issues with their service (as of writing).


# Media SubDomain

I'm using Wasabi as the backend, but I serve all content via a <https://mediacdn.aus.social/> URL.

This will allow me to quickly switch from Wasabi to BunnyS3 once they support it, by just updating the $s3\_backend path.

## Option 1: Forwarding only

This is my config for aus.social. It forwards all requests for media to the Wasabi bucket.\
\
Example: <https://mediacdn.aus.social/accounts/avatars/000/000/001/original/30ea1ea3b78c6690.jpg> -> <https://s3.ap-southeast-2.wasabisys.com/aussocial/accounts/avatars/000/000/001/original/30ea1ea3b78c6690.jpg>

```
server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name mediacdn.aus.social;
  root /var/www/html;

  keepalive_timeout 30;

  set $s3_backend 'https://s3.ap-southeast-2.wasabisys.com/aussocial';
  location / { return 302 $s3_backend$request_uri; }
  
  ssl_certificate /etc/letsencrypt/live/mediacdn.aus.social/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/letsencrypt/live/mediacdn.aus.social/privkey.pem; # managed by Certbot
}

```

## Option 2: Local Caching

This helps if the s3 backend such as Wasabi ever goes down, but puts extra load on the nginx server and required additional local storage.

Mastodon has official documentation to use nginx as a local cache by proxying object storage through nginx.&#x20;

<https://docs.joinmastodon.org/admin/optional/object-storage-proxy/><br>


# Cloud Email (SMTP)

There are plenty of cloud storage providers that support the email SMTP hosting.&#x20;

Unlike AWS S3, I highly recommend AWS SES.

## Known SaaS Providers

* Amazon Simple Email Service (SES)
* Mailgun
* SparkPost (Known to cancel Mastodon instances accounts for "spam")
* <https://www.zoho.com/zeptomail/>
* <https://www.smtp2go.com/> (recommended by pettingzoo)<br>


# ZeptoMail by Zoho


# AWS SES

SES is a basic SMTP based transactional email service. It's cheap, but also offers you very little in the way of logging or metrics.

AWS want you to pay move than a $1000 for insight into the SES platform which is generally included in the price for other providers.


# Static Content CDN

This is one of the most recommended steps and should be taken by all admins regardless of size.

## Why?

Using a CDN for static files will take pressure off the nginx/puma webserver when serving commonly available files like JS, CSS, SVG etc, and allows your web server to purely focus on the dynamic important stuff.

## Example

I'm running bunny.net CDN, and it's caching more than 95% of the static traffic. Which has dropped my CPU usage massively, and improved the \`end user\` experience for only a few bucks a month.

<figure><img src="/files/pG5mxRKQhOAO3Gd81l0G" alt=""><figcaption></figcaption></figure>


# Bunny.net CDN (Pull Zone)

## Stage 1: Setup Pull Zone&#x20;

### 1) Add new pull zone in Bunny.net

### 2) Create a unique name for the pullzone&#x20;

Example: `mastodoninstancestatic` or similar.

<figure><img src="/files/woPev23on9DIxfXLX5Hx" alt=""><figcaption></figcaption></figure>

### 3) Select `Origin Type URL`

### 4) Update **Origin URL**

Place the original FQDN of your instance in the Original URL.

<figure><img src="/files/X7LbAsFQnEI0DdgOcMQP" alt=""><figcaption></figcaption></figure>

### 5) Select your regions to replicate data

<figure><img src="/files/6ge5o9JO2RUSLKDxYHXA" alt=""><figcaption></figcaption></figure>

### 6) Add new Pull zone

## Stage 2: Configure Pull Zone.

### 1) Force SSL

<figure><img src="/files/e2rJ1XGRWiuG4uqjEJ0E" alt=""><figcaption></figcaption></figure>

### 2) Setup SafeHop (retries on failures)

<figure><img src="/files/20yQORyPsp9SjTwdi8PA" alt=""><figcaption></figcaption></figure>

### 3) Add additional Headers (CORS)

Add \`,js, gz\` to the Extension list.

Example: \`eot, ttf, woff, woff2, css, js, gz\`

<figure><img src="/files/TDAfgLIbM7oSaySaqDml" alt=""><figcaption></figcaption></figure>

## Stage 3: Configure Mastodon and NGINX

### 1) NGINX Headers

Add **`Access-Control-Allow-Origin`** header to nginx all locations.\
Example: `add_header Access-Control-Allow-Origin "https://aussocialstatic.b-cdn.net";`

<figure><img src="/files/hePqKFChjjWkDSK0qSEy" alt=""><figcaption><p>These paths are heavily used for static files... but the header might be required in multiple locations in the nginx conf</p></figcaption></figure>

### 2) Mastodon .env

Add `CDN_HOST` to .env.production&#x20;

Example: `CDN_HOST=https://aussocialstatic.b-cdn.net`

### 3) Reload nginx and restart mastodon.

## Troubleshooting

The OCR (Optical character recognition) might fail if you missed gz from the extension list.

<figure><img src="/files/UKEhnUIeMPdfFOn8XMoa" alt=""><figcaption></figcaption></figure>

## Notes

Do we need to add to svg to the CORs?


# Backups

PLEASE FOR THE LOVE OF ALL THAT IS HOLY. TAKE BACKUPS!

## References

<https://nutcroft.com/blog/how-to-backup-a-mastodon-instance/><br>


# Monitoring / Alerting

## Basic Monitoring

I'm running a monitoring service called statuscake.com to test my mastodon endpoints every 5 minutes and respond that the services are all online.

* <https://aus.social/about>
* <https://aus.social/api/v1/instance>
* <https://aus.social/api/v1/streaming/health>

These 3 endpoints should give a pretty basic amount of coverage for the internet facing Puma (FE/API) and NodeJS (Websocket Streaming) endpoints.&#x20;

I've connected statuscake up to my mobile, and get push notifications sent to my phone when one of these tests fail.

<figure><img src="/files/WZ0NxHNuoq61pgHnk9QG" alt=""><figcaption></figcaption></figure>

## Advanced Monitoring (metrics)

You can investigate using promethius and other dashboards to display real time and historical details from your instance.

<figure><img src="/files/XjEM1C09ES7VN8nTe0gj" alt=""><figcaption><p>tyr@pettingzoo.co example dashboard</p></figcaption></figure>

## APM (application performance monitoring)

There are more advanced monitoring tools that offer insight into the actual application low level code.&#x20;

I'd recommend investigating the different APM (application performance monitoring) tooling.

* NewRelic
* Sentry.io


# OS Tuning


# NGINX Tuning (Reverse Proxy)

Nginx tuning comes in two parts.. The “worker process” tuning, and the virtual server / Server Block tuning.

## NGINX official repo vs Distro repos

Alex Conner has recommended some NGINX tuning.

* Installing the mainline from the official nginx.org repo over the ubuntu stock repo.\
  official repo packages [http://nginx.org/en/linux\_packages.html#Ubun](http://nginx.org/en/linux_packages.html#Ubuntu)tu
* “The default systemd config from[ nginx.org](http://nginx.org) is pretty solid”


# NGINX Trouble Shooting

## Not enough worker processes

* Keep your eyes on the /var/log/nginx/errors.log\
  Problem: Not enough worker processes (Tells you the problem/solution)

Solution: Increase worker processes.

<figure><img src="https://lh5.googleusercontent.com/yxQhZHfmUkqmxOYtkhwAhRc3PKSw6XZd9t34Fqw5wk_6BkE9Yn9_ZvqqPhwzD9vobUHJpd6mKP9uwWbHJfSZCGnQ4Amu39AKkXVorEVxhXgIMkeu0GGBE5L_PIIQ0Gr3ClHj8-fHEpONXuHa2cvNjWmgKHt5me2N6WpI3K3QlWFp0gmjZ4v4mJoMJuyAkg" alt=""><figcaption></figcaption></figure>

## cache file has too long header

* `2022/11/30 08:18:38 [crit] 1392615#1392615: *4208360 cache file "/var/cache/nginx/8/aa/d974f05ff7f54ab0f4b21dbba7861aa8" has too long header, client: x.x.x.x, server: aus.social, request: "GET /api/v1/instance HTTP/2.0", host: "X.social"`

Solution: ??<br>

## Monitor the active connections

* Consider installing the status module to keep an eye on the active connections.

<figure><img src="https://lh5.googleusercontent.com/VjAazCLQ70bmYsz4bywwbv1rK2-nRx7j7KOVHN0l76524vVCj6A9NLUQq2xdZGdQmS0jilHF_6cFe5jJk5x_iF0P7SoSfVl-h4-7v0C5PVDoe9IemkfrAxEnhqXgoQMKfPsIjlgqqE_dxo1PoTAbG2NtPNrQuWtzOgiqxibuFBoC64hlnspwalYCvobF9w" alt=""><figcaption><p>Output of https://x.com/nginx_status</p></figcaption></figure>

<figure><img src="/files/ll7lbzUMpFu9hyyXhfPZ" alt=""><figcaption><p>Code Block from my instance conf</p></figcaption></figure>

* TODO: Investigate the Nginx virtual host traffic status module\
  <https://github.com/vozlt/nginx-module-vts>


# PostgreSQL 14 Tuning

## Install PGBouncer

ff


# Web (Puma) Tuning

Puma tuning comes down to&#x20;

## Advice from Puma Dev

<figure><img src="/files/VgMc7mb7ZFlIPCII3AoE" alt=""><figcaption><p><a href="https://ruby.social/@evanphx/109365852402052564">https://ruby.social/@evanphx/109365852402052564</a></p></figcaption></figure>


# Sidekiq Tuning


# ElasticSearch Tuning


# Redis Tuning

## Recommendations

### /etc/sysctl.conf Tweaks

from <https://redis.io/docs/management/admin/> and others

`vm.overcommit_memory = 1`

`vm.swappiness = 1`

`net.core.somaxconn = 65365`

### Disable Transparent Huge Pages

from <https://redis.io/docs/management/admin/>

`echo never > /sys/kernel/mm/transparent_hugepage/enabled`

More: <https://www.mongodb.com/docs/manual/tutorial/transparent-huge-pages/>

### /etc/redis/redis.conf Tweaks

from <https://redis.io/docs/management/admin/>

`maxmemory 3gb`<br>

from&#x20;

`timeout 300`\
`tcp-keepalive 0`\
`tcp-backlog 65536`

from <https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-22-04>

```
rename-command FLUSHDB ""
rename-command FLUSHALL ""
rename-command DEBUG ""
```


# Reading Material

## Random

<https://hub.sunny.garden/2022/11/19/notes-on-installing-mastodon-from-source/>\
<https://vitobotta.com/2022/11/07/setting-up-a-personal-mastodon-instance/>\
<https://rixx.de/blog/on-running-a-mastodon-instance/>\
<https://aus.social/@nathan@mastodon.lol/109379096132183347>\
<https://denise.dreamwidth.org/91757.html>\
<https://ipng.ch/s/articles/2022/11/20/mastodon-1.html>\
<https://hub.sunny.garden/page/i-signed-up-now-what/>\
\
<https://www.elastic.co/guide/en/elasticsearch/reference/8.5/system-config.html><br>

## Blogs about scaling

{% embed url="<https://nora.codes/post/scaling-mastodon-in-the-face-of-an-exodus/>" %}

{% embed url="<https://lond.com.br/2018/07/13/running-a-mastodon-instance-using-archlinux.html>" %}


# OS Monitoring

## Investigate context switches per second

Run vmstat 1 for 5-10 samples

![](https://lh5.googleusercontent.com/QW5_Zv1-UujuLxYV8-inLquV15bvEnfU2GwVXJWV-HTet2cx_iPetKlM2VBzPxR33OadRF2J_8pMqu4qwzWrven5tY_D-ROy3SwX9-w7WB0YcAYv4sMW7G2KwQxxwmYal8YXKULa_kvPy9Sz1GstH-6x16E1LWZyjH0Gvd9bKMu0tAiAIxUTPzOdEpMKqQ)