The 12th factor in a 12-factor app is that we should run background jobs as one-off processes. This enables us to scale them independently and ensure they have the correct resources assigned to them.
This is the 12th video in our 12-factor Application Modernisation series. In this video, Marc Firth (Managing Director at Firney) explains software engineering best practices for building scalable, reliable web services that are efficient to work with.
See our Application Modernisation with 12-Factor App series and the full App Modernisation playlist on YouTube.
A transcript of the video above is included below.
Why we should run background hobs as one-off processes
Marc: Today, we’re talking about why we should run background jobs as one-off processes.
These can be things such as database migrations, background reporting or anything, working with a queue in the background.
Application modernisation
This video is part of our 12 Factor app series, where we’re talking about how to make applications more reliable, scalable and efficient to work with.
Foreground and background processes
Your application is likely to have both foreground and background processes. The foreground processes will be something like serving a web request and background processes will be your background jobs. These can be things such as database migrations, clean-up tasks and reporting, admin jobs, tasks working with queues, console commands and any other background jobs that your program needs to run.
Run background jobs in an identical environment
Now, when you’re running these background jobs, they should be run in an environment that’s identical to the long-running processes of the application. They should run against the same release of that code base using the same config as any other process running against that release.
Your background jobs’ code should ship with the application code to avoid any synchronisation issues. So it should be part of the same release.
Otherwise, you can run into issues such as the background job, trying to change the name of a column in the database whilst the application tries to use the old name for that column in the database.
Running jobs locally: The REPL shell
When it comes to building those background jobs, 12-factor favours any language which provides a REPL shell out of the box. If you are unfamiliar with the REPL shell, it’s essentially a terminal where you have your Read-Evaluate-Print-Loop and you can run your scripts through that terminal.
When developers are working with these commands locally, they would just run the command on their terminal in order to get it to do whatever it needs to do.
Running jobs in production
For the production environment, you could SSH into that environment and run your command that way. But even better, if you can set it up within Kubernetes config as a scheduled task, that’s going to be a lot more secure and scalable for your application.
Best practices for running background jobs
I’ve got a few tips for you for running background jobs, and the first is targeted at startups.
Tip 1: Use a framework
If you’re a startup, it’s quite likely that you’re favouring a monolith-based architecture as opposed to a microservice-based architecture.
For that, I’d suggest using a web framework that allows both application processes and command line commands to be built under the same framework, because this then allows you to deploy the application as a whole and also run those command line commands.
When you want to scale up your processes, you can then move to Kubernetes and have multiple instances of your application with different entry points to run the command or the application process.
So with Laravel, for instance, you would have your application being served using a web service such as Nginx, and then you can also create your own Artisan commands, which you can run on the command line locally and you can run via SSH or schedule them in Kubernetes on the production environment.
There’s also a couple of extra tools in the Laravel ecosystem which allow you to monitor those jobs using Horizon and also to build any admin functionality using a tool called “Laravel Nova”. But if you’re able to build in a microservice-based approach that will be more lightweight and scalable.
Tip 2: Use a single release to update app, jobs & APIs
Those dedicated microservices need to ship together or be able to communicate using different versions of the APIs to communicate between services.
An easy way to do that is to update all the versions of your microservices in a single release and make sure you’re testing that as a snapshot across the board.
Tip 3: Capture log output centrally
Of course, you want to make sure that all those background jobs are writing their logs to STDOUT so that we can capture all that, log output. For more tips on that see our video about capturing log output, which I’ll leave a link to down in the comments.
Closing thoughts
So I hope that helps you organise your background jobs more effectively. So forget a lot of this video if you did, follow us on LinkedIn, subscribe to our YouTube channel and I’ll see you in the next one.