
Despite the growing popularity of static hosting solutions like Cloudflare Pages
and Netlify, an S3 bucket with a CDN in front of it is still a good choice for
hosting a static website.

Deploying to S3 is very simple and effectively boils down to a variation of the
following AWS CLI command:

```shell
aws s3 sync <website> s3://<bucket>
```

Unfortunately this is not an atomic operation, so if something breaks during the
sync or if the uploads are taking a long time, you will be left in an inconsistent
state.

For example a new version of an html page might have been uploaded but the
corresponding styles, javascript or images haven't.

Luckily this can be fixed easily - we just need to upload the assets first and
the pages after that.

For a React Vite SPA the commands would look something like this:

```shell
aws s3 cp dist s3://<bucket_name> --recursive --exclude index.html
aws s3 sync dist s3://<bucket_name> --delete
```

The same principle applies to any type of website as long as you can separate the
pages from the rest of the assets.

For websites with multiple pages you can still encounter issues like broken links
due to referencing pages that haven't been uploaded yet, but you won't have broken pages.


Atomic Sync For Website Deployments to S3


Recently I had to insert data from JSON files stored in S3 into ClickHouse.
The files were gzipped, but didn't contain the `.gz` suffix, so ClickHouse kept
treating them as regular JSON files and throwing an error when trying to parse them.
When I looked at the signature of the [s3 table function](https://clickhouse.com/docs/en/sql-reference/table-functions/s3)
I noticed a problem.

```sql
s3(url [, NOSIGN | access_key_id, secret_access_key, [session_token]] [,format] [,structure] [,compression_method])
```

The `compression_method` argument I wanted to specify was behind two other arguments,
one of which I really didn't want to specify.

First I tried specifying the argument as you would in a programming language like Python.
Then I tried using Snowflake-style `=>` but it didn't work either. So I gave up
on named arguments and tried finding an alternative approach.

After  a decent amount of googling I stumbled upon this GitHub issue [comment](https://github.com/ClickHouse/ClickHouse/issues/54631#issuecomment-1719488328)
that looked promising, and indeed it was what I needed.
In my case the query looked something like this:

```sql
SELECT * FROM s3(
    '<bucket_url>',
    '<aws_access_key_id>',
    '<aws_secret_access_key>',
    auto,
    auto,
    'gz'
)
```

Since I couldn't find any details on `auto` in ClickHouse documentation other than
a passing reference in the [schema inference docs](https://clickhouse.com/docs/en/interfaces/schema-inference#table-functions-file-s3-url-hdfs),
I was wondering if it worked for all functions or just table functions.

I decided to test it on the first function with more than one optional argument I
come by. I ended up trying with [`makeDateTime64`](https://clickhouse.com/docs/en/sql-reference/functions/date-time-functions#makedatetime64),
but it didn't work.

```sql
SELECT makeDateTime64(2023, 2, 28, 17, 12, 33, auto, auto, 'Europe/Amsterdam');
```

So I guess, at least for now, we only get to use `auto` with table functions.


Skipping Optional Arguments in ClickHouse Table Functions


Recently I inherited a ClickHouse cluster and needed to look into the query backing
a materialized view. Since the view wasn't defined in the code I had to obtain it
from ClickHouse itself. However I wasn't able to find in the ClickHouse cloud UI
and a quick google search wasn't helpful either.

Then I remembered that relational databases hold that information inside the
information schema tables, so I checked to see if something like that is available
in ClickHouse. Indeed it is, and here's the query to get the query behind a view:

```sql
SELECT
  table_schema, table_name, view_definition
FROM
  information_schema.views;
```

**UPDATE**: There is an even better way to do what we want by using [`SHOW CREATE`](https://clickhouse.com/docs/en/sql-reference/statements/show#show-create-table--dictionary--view--database),
which returns the create query that could be used to create the view:

```sql
SHOW CREATE VIEW <view_name>;
```


Getting Query Backing a View in ClickHouse


Official nginx [docs on proxy caching](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache)
as well as most of the guides on the topic don't mention that it only works if
[proxy buffering](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering)
is enabled. This may be due to the fact that it's enabled by default.

However from my experience proxy caching has been mostly useless and I usually turn
it off to get the extra bit of performance, and only enable it for locations where
I use caching.

In addition to that, it is also disabled by default in ingress nginx so it also
needs to be enabled for the rules where caching is enabled.


Proxy Cache Requires Proxy Buffering in Nginx


When it comes to managing resources in kubernetes it's often recommended to disable
CPU limit enforcement. This will allow you to set requests=limits so your pods
have [`Guaranteed`](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#guaranteed)
QoS class without being unnecessarily throttled.

If you're running relatively well-behaved workloads, with some leeway when it comes
to available resources and that can scale out using HPA and cluster autoscaler
if need be, the risk of disabling CPU limit enforcement should be low and should
be outweighed by the benefits of your pods being able to handle short load spikes
without losing performance.

A lot of online resources suggest using `--cpu-cfs-quota` kubelet command line flag
to disable CPU limit enforcement. However this flag has been deprecated along
with most other flags. Even though it doesn't seem like these flags will be
removed soon (especially on EKS since it's usually a couple releases behind) it's
safer to use the [config file](https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/).

The config file parameter we're looking to change is `cpuCFSQuota`. Since the managed
node AMIs come with existing kubelet config we need to modify it. To do so we need
to use user data of the launch template used for the node groups. To modify the
the config file we can use `jq`:

```text
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "$(jq ".cpuCFSQuota=false" /etc/kubernetes/kubelet/kubelet-config.json)" \
    > /etc/kubernetes/kubelet/kubelet-config.json

--==MYBOUNDARY==--
```

[Bottlerocket](https://github.com/bottlerocket-os/bottlerocket) uses toml for user data.
Since I've never used bottlerocket I haven't had a chance to test it but it should
look something like this:

```toml
[settings.kubernetes]
cpuCFSQuota = false
```

Note that depending on what you're using to set the user data you may need to
base64-encode it first (e.g. using `base64encode` function in terraform).

For more info see EKS managed node [user data docs](https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html#launch-template-user-data)
as well as [user guide](https://github.com/awslabs/amazon-eks-ami/blob/master/doc/USER_GUIDE.md#customizing-kubelet-config)
for the official managed node AMIs.


Disabling CPU Limit Enforcement On EKS Managed Nodes


When it comes to resource management in kubernetes one of the most common pieces
of advice you come across is that memory limit for a container should be equal to
its memory request. If you're like me and like learning things the hard way, you
probably lost a node due to a spike in memory usage by one of the pods. Setting
requests and limits this way is safer but is a delicate act of balancing resource
utilization and stability of your pods. If you set too high values your actual
resource utilization can be low and if you set them too low your application will
keep getting killed as it tries to consume more memory than you specified.

If you're frugal when it comes to handing out resources to your workloads or your
workloads are poorly coded or handle highly unpredictable loads, you've probably
seen containers being restarted with reason `OOMKilled`. This means that one of
the processes inside the container tried to allocate amount of memory that would
put the total memory usage above the limit specified for the container. In this
case the process that got killed was the main process inside the container, which
caused the restart.

However many apps run multiple processes and there is no guarantee that the main
process will get killed. This means that if your app uses multiple processes and
doesn't monitor them properly one of them might get killed without you ever knowing.

If you're running kubernetes 1.24 or higher and using prometheus you can collect and
create alerts on cAdvisor metric `container_oom_events_total`. If you're on an older
version of kubernetes you can use node exporter to collect `node_vmstat_oom_kill`
and set up alerts to let you know when a process gets OOM killed. However this will
only tell you on which nodes it happened and you'd need to figure out in which
container it happened (you can use `container_processes` cAdvisor metric for that).

In either case, a naive implementation of the alerts could add noise to your
container restart alerts in situations where the process that gets killed is the
main process of the container.


OOM Killer is Silent in Kubernetes


I'll illustrate this example with python apps but a similar project structure and
way of specifying dependencies can be found in other languages.

```text
my-project
├── apps
│   └── my-app
│       ├── Dockerfile
│       └── pyproject.toml
└── lib
    └── my-lib
```

`pyproject.toml` contains something along the lines of:

```toml
[tool.poetry.dependencies]
my-lib = { path = "../../lib/my-lib", develop = false }
```
To build a container image for `my-app` we would need to specify the entire
`my-project` as build context or use ugly scripts to temporarily put together the
things we need in the same build context. This can impact build performance if
the project grows very large or `.dockerignore` files aren't used to exclude stuff
like virtual environments. It can also be error-prone as code from other apps can
easily end up in our image if we aren't careful when copying the files.

Fortunately this can all be avoided by using [buildx](https://docs.docker.com/engine/reference/commandline/buildx_build/)
for building images. `buildx` allows us to use multiple build contexts and
mount the files from them while building the image. In our example `Dockerfile`
would contain something like this:

```dockerfile
# Workdir needs to be at least two levels deep here. In this case it also needs to
# be 3 levels deep from the root so it doesn't override /lib.
WORKDIR /my-project/my-app/deps

RUN --mount=target=../../lib,from=lib poetry install
```

Now we can run something like this:

```shell
docker buildx build --build-context lib=../../lib .
```

and voilà, we have our image!


Building Container Images With Local Dependencies Outside Build Context


Recently I was doing some weird shit with docker and docker compose that required
a weird DNS setup. While debugging a problem with DNS resolution in nginx I came across
a strange issue: I could reach a host by curl and ping but wasn't able to resolve
it using nslookup and dig.

A quick Google search wasn't really helpful so I decided to run `strace` on both
tools and see if there's an obvious difference in how they handle name resolution.
After looking at the trace I finally spotted the difference: curl was opening
`/etc/hosts` and dig wasn't!

Turns out [`extra_hosts`](https://docs.docker.com/compose/compose-file/compose-file-v3/#extra_hosts)
is adding hostnames to the `/etc/hosts` file which isn't checked by the DNS tools or
nginx resolver.


Host Reachable Through cURL but Cannot Be Resolved by nslookup


You may often find yourself accidentally installing packages to your global python
interpreter because you forgot to activate a virtual environment. Thankfully you
can make pip require virtualenv to be activated in order to perform the install.

There are two ways to do this: either by using an environment variable `PIP_REQUIRE_VIRTUALENV`
or by enabling `require-virtualenv` pip config option.

To make this a default globally either add the following to your `.bashrc`
(or `.zshrc` or whatever other rc you are using)


```bash
echo "export PIP_REQUIRE_VIRTUALENV=true" >> ~/.bashrc
```

or run the following command


```shell
python -m pip config set global.require-virtualenv true
```

Which will add something along the lines of

```ini
[global]
require-virtualenv = true
```

to your `~/.config/pip/pip.conf` (may be a different location on your system).

I prefer the second option as that way I don't clutter my `.bashrc` and if I add
new pip config entries they will all be in the same file. The second option
also works on Windows.

---

Setting this option globally will be useful most of the time but you may still
need to run commands for the global interpreter.

```console
$ pip freeze
ERROR: Could not find an activated virtualenv (required).
```

This can be easily achieved by adding `PIP_REQUIRE_VIRTUALENV=false` in front
of the command that's failing:


```console
$ PIP_REQUIRE_VIRTUALENV=false pip freeze
```


Today I Learned