Reduce the size of Akri's Containers

[kate-goldenring]

Akri's containers are heavily bloated and can be slimmed down in size. An investigation should be done into Akri's Docker files and build system to make this reduction.
For example, the Akri agent (v0.8.16) can be slimmed down from 159MB to 32MB using docker-slim. This shows that there should be a way to optimize our builds:

kagold@nuc-ubuntu2:~$ docker image  ls | grep akri/agent
ghcr.io/project-akri/akri/agent.slim   latest        7d732d1c1bf4   29 minutes ago   32.6MB
ghcr.io/project-akri/akri/agent        v0.8.16-dev   a3ab8bcae40f   5 days ago       159MB

Walk through of how dockerslim reduces Akri container image sizes:

Note: I am not explicitly recommending that we use dockerslim; rather using this analysis to illustrate that our containers have a lot of room for optimization

1. Install docker-slim

curl -L -o ds.tar.gz https://downloads.dockerslim.com/releases/1.37.6/dist_linux.tar.gz
tar -xvf ds.tar.gz
mv  dist_linux/docker-slim /usr/local/bin/
sudo mv  dist_linux/docker-slim /usr/local/bin/
sudo mv  dist_linux/docker-slim-sensor /usr/local/bin/

2. Pull Akri's agent container docker pull ghcr.io/project-akri/akri/agent:v0.8.16-dev
3. Note the image's current size with docker image ls | grep akri/agent
4. Use docker-slim to create a slimmer version of the container: docker-slim build --http-probe=false ghcr.io/project-akri/akri/agent:v0.8.16-dev
5. Note the image's reduced size with docker ls | grep akri/agent
6. Verify the Agent still works as expected by installing Akri with its debugEcho discovery Handler. Be sure to specify the slim image and set the image pull policy to Never so it uses the local image. If using containerd instead of Docker, you may need to load the image into the containerd namespace. See the note on K3s for more details.

helm install akri akri-helm-charts/akri-dev  \
   $AKRI_HELM_CRICTL_CONFIGURATION   \
   --set agent.allowDebugEcho=true   \
   --set debugEcho.discovery.enabled=true     \
   --set debugEcho.configuration.enabled=true \
   --set debugEcho.configuration.shared=false \
   --set agent.image.repository="ghcr.io/project-akri/akri/agent.slim" \
   --set agent.image.tag="latest" \
   --set agent.image.pullPolicy="Never"

Note: If using K3s, load the image into the K3s containerd namespace as follows:

docker save ghcr.io/project-akri/akri/agent.slim:latest -o slim-agent.tar
sudo k3s ctr images import --no-unpack slim-agent.tar

For non-k3s containerd, such as with standard K8s, use sudo ctr -n=k8s.io images import --no-unpack slim-agent.tar

[adithyaj]

@kate-goldenring I'd be happy to start taking a look at this!

[kate-goldenring]

Great @adithyaj, I'll go ahead and assign you to it. Thes docs should help with understanding how to build the containers. I think it may be good to look at what our cross-build containers are doing. @bfjelds, you have more background than me on our builds if you have any pointers to add.

[adithyaj]

I'm looking to see if there are any preferable alternatives to docker-slim as it does an analysis of what we do with the agent and tries to minify the image. The docker image it produced on my end uses a tarball and unpacks the specific directories before running which raises two potential issues:

1. It might not be reliable

   • If there is a scenario that docker-slim did not monitor us go through it won't work properly on the minified image. Unclear how we can make sure we've run through everything.

2. It becomes less readable

   • Right now we have our dependencies laid out and the dockerfile is easy to understand; with the dockerfile after using docker-slim because it has an arbitrary tarball it makes it less readable and much harder to figure out the dependencies.

We are using debian:buster-slim as the base image which is about 70MB so maybe an alternative to minimize that further could work (ex: use debian:stable-slim to reduce it to 35MB base and continue)

[agracey]

What is docker-slim doing to get down to 32MB? It looks like the crictl and agent binaries are 27MB and 26MB respectively.

[kate-goldenring]

We could rerun the dockerslim scenario and try to inspect the layers of the resultant image

[kate-goldenring]

@adithyaj any updates on this?

[adithyaj]

Sorry - slowed down on this while working I was on a few other things. I'll prioritize this more now.

@agracey - That's a good question, last I checked it just compressed the binaries and had a few other dependencies running, I'll bring up that image and see what else it does.

[kate-goldenring]

@adithyaj any chance we'd have this in for the next release? Would be nice to have smaller containers for it

[jbpaux]

would be great to update from buster (Debian 10) to at least bullseye (Debian 11) as bullseye is now the stable release (since 1 year and a half) and bookworm (Debian 12) is coming in the coming weeks

[ag17sep]

@kate-goldenring I am currently looking into the issue. I was thinking that maybe using alpine as the base image might help reducing the image size. Just wanted to check if you tried using alpine and faced any challenges. If not, then I can do a small POC to validate.

[jbpaux]

From my experience, it's never a good idea to go the Alpine way for these kind of images, poor package management, no DNS over TCP support, poor ecosystem.

[agracey]

FWIW, SUSE maintains full open source, free to use and distribute base images: https://registry.suse.com/

We have a few different sizes of "base" image depending on how minimized you need. We also keep them up to date and are on the embargo list for CVEs to be able to release patches along with CVE publication.

[kate-goldenring]

@ag17sep thanks for looking into this. I am not sure what we have tried but we could try one of SUSE's minimized bases @agracey. Feel free to explore multiple paths

[diconico07]

While trying other ways of building akri (i.e ways to not use cross, cf slack), I created some images, I got the agent's image down to 57MB (might be possible to get it even slimmer).

For the details I got OBS to build RPMs for akri, then I created an image containing that RPM and its dependencies.

[github-actions]

Issue has been automatically marked as stale due to inactivity for 90 days. Update the issue to remove label, otherwise it will be automatically closed.