Experiences with an anti-pattern: multiple processes — one container

Imagine this situation. You made a software on-premise. There will be a situation for some reason when you need to use containers and not VMs or servers. This situation can be an acquisition or just a requirement from a customer or you just want to be a bit more modern. You are not an infra guy/gal/whatever, you are someone who just made a bunch of services. What would you do? Well I guess an average person would just put everything into a container. Then they realize there is no such thing as initd or systemd and they look for a solution like tini, supervisord and entrypoint.d.

In the past days I faced the results of this kind of strategy, and this is also a lift-and-shift situation. As wiktionary says:

Lift and shift: (computing) The practice of deploying existing on-premises software to the cloud, with minimal changes to its architecture.

Sounds casual. But what’s the matter? Why is it a problem? Why is it not recommended to do so and still have a solution for running more than one service in a container?

In the past days I faced the following situations, one-by-one, and these are perfect reasons to forget the multiple services — one container line-up in production.

Logging

On one of the first days, we had some issues. Well what do you do then? Read the logs. We used Splunk as a central logging service, which is to be honest not so bad as others stated.

So, the requirement says they need to produce their logs to the stdout. However, if you have multiple services in a container, and all of those produce their logs to stdout, one will cut in the other’s log. That results in an unparsable stdout, which means I couldn’t find some logs in Splunk. Guess what, since it ran in an auto-scaling group with a minimum size of 8, we had to open 8 terminal windows and watch the logs that way.

Debugging

When you open a terminal window, you expect to see the output of just one service. This is true for the last example too, since if you want to take a look for the logs straight from a container for some reason, it’s a bit fuzzy to see all of the services’ logs. But also testing service is easier when you have just one service, and you don’t have to put your hand into a container.

Scaling

Scaling is much harder. For example you have an authentication application, a web application which is the user interface application and an application that is responsible for routing between them. Isn’t it easier to cut these up? (Also you can use a load balancing rule instead of a full app for routing.) But wouldn’t it be easier to use 2 auto-scaling groups instead of one with unnecessarily running applications? They eat up the resources and maybe you don’t even need one more authentication application just another web application.

Health

What if your application does not reply? It’s just frozen somehow. You know, infinite loop, long-running tasks, etc. You know, what casual people would call frozen. Or maybe zombie processes. (These are not zombie processes, because zombie processes are processes that are not properly exited.) In this case there are 2 options. You don’t even check the health of the different apps, there are no health check endpoints, which lead to an improper health check, and you just check if a process runs or not. The other option is you have a health check endpoint and you have to restart a whole container, if you have any problems, so you restart not just one application if one is frozen.

But why do we have this option even though?

When you just want to try something fast and you don’t want to spend much time on it, also you want to have a disposable system, it’s obvious to use docker because installing a whole virtual machine takes time and it’s too heavy for just trying a few things, but you want to run more than one processes.

However, maybe it’s still smarter to use a docker compose, because writing a docker compose file is not a big deal even for an average person. An advantage of docker compose is that the system is reproducible anytime. And still, you can experience all of the above-mentioned advantages.

All in all,

I think everyone should think twice if they want to use a container as a VM, cause it has many disadvantages. And sometimes these failures remain in the system because it becomes an ordinary thing. And well, if someone breaks a house’s window and it’s not gonna be fixed, in a few weeks there gonna be some other broken windows and graffiti too, because no one will care anymore. So take care of your things and do a qualitative job.