DevSecOps applied: Setting up automated security auditing of a node.js app with Bitbucket pipelines

Before I start I want to get a bit Zen and make something real clear: Just like with DevOps, Agile, and any other buzzword ways of working in our field — the essence of DevSecOps is philosophical and organizational. The practices and tools we superficially see when we look at a DevSecOps team (devs running pentests, conducting threat analysis, vuln scanning in CI/CD) are the results of these underlying characteristics.

These are symptoms of the more important, fundamental aspects of the team.

I went to a great talk by Igor Goulko from DevOps Talks last night who made it very clear, that a common failing when seeking to implement DevOps is simply cargo-culting the artifacts and changing Sysadmin job titles to “DevOps”, without understanding the fundamental reasons for DevOps.

Great, got it.

Once we understand that though, the next question is — and developers understandably ask — is how do you actually DO it? Where’s the CODE? Without working code, this is all just words and philosophy. Well, here is some code.

The aspect we’re looking at here is static analysis of third-party libraries in a node.js framework — namely express. Reasons being:

• available and well-known library
• static code analysis relatively quick and simple to set up and run
• out of the box npm now provides excellent 3rd party dependency auditing (formerly Node Security Platform)

On that third point — these days almost nobody writes everything from scratch; we all stand on the shoulders of code giants, and 99% of the time we’re not intimately familiar with their inner workings or vulnerabilities.

This is no indictment of noob or lazy developers, but a reality that in order to be productive we don’t have time to fundamentally understand all nuances and potential security flaws in external tools.

The result of this, is the for the Bad Guys it’s a goldmine when a commonly-used library (e.g. express …. or … openssl) has vulnerabilities, because the number of potential victims is huge.

You could guess how simple it sometimes is, for Bad Guys to write an exploit for a certain library, search the web for systems using that library, and exploit them.

The goal here is to avoid becoming one of their casualties.

So on to the implementation.

I’m a fan of bitbucket, and their pipelines for CICD — Simple YAML config, hosted CI.CD.

It’s pretty awesome.

Clone the express repo and push to your personal repo

git clone git@github.com:expressjs/express.git
git remote add bitbucket git@bitbucket.org:nick_doyle/express.git
git push -u bitbucket master

We now have our own copy of the express code on our personal Bitbucket account.

Now, in the repository settings, create the pipeline:

The template they provide for node.js is pretty good.

We just add in “npm audit”:

image: node:6.9.4
pipelines:
  default:
    - step:
        caches:
          - node
        script:
         - npm install
         - npm test
        - npm audit

Save the change, and watch the pipeline run.

Except — oh no:

It looks like npm doesn’t recognize “audit”.

That’s because the docker image we’re using in the pipeline (node:6.9.4) uses npm v3.10.10, which doesn’t yet include “audit”.

So, we need to find a newer node.js docker image to use.

Looking in docker hub to find better options: https://hub.docker.com/r/library/node/tags/

We find many candidates. I’d try alpine first, because it’s so much smaller (24mb vs 266mb) hence likely faster to execute our pipeline (though really, bitbucket is gonna have it cached).

Next we need to update our pipeline to use it.

From that pipeline page above, click “View Configuration” link (around the middle of the screen), and change the image property:

Save and re-run:

Excellent, our pipeline has run, and failed because vulnerable dependencies were found — just what we want.
We can also run npm audit locally:

nick@minerva:~/ws/lab/devsecops/express$ npm audit
=== npm audit security report ===
# Run npm install — save-dev mocha@5.2.0 to resolve 2 vulnerabilities
SEMVER WARNING: Recommended action is a potentially breaking change
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Low │ Regular Expression Denial of Service │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ debug │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ mocha [dev] │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ mocha > debug │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://nodesecurity.io/advisories/534 │
└───────────────┴──────────────────────────────────────────────────────────────┘
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Critical │ Command Injection │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ growl │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ mocha [dev] │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ mocha > growl │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://nodesecurity.io/advisories/146 │
└───────────────┴──────────────────────────────────────────────────────────────┘
# Run npm install — save-dev supertest@3.1.0 to resolve 2 vulnerabilities
SEMVER WARNING: Recommended action is a potentially breaking change
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Moderate │ Regular Expression Denial of Service │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ mime │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ supertest [dev] │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ supertest > superagent > mime │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://nodesecurity.io/advisories/535 │
└───────────────┴──────────────────────────────────────────────────────────────┘
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Low │ Large gzip Denial of Service │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ superagent │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ supertest [dev] │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ supertest > superagent │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://nodesecurity.io/advisories/479 │
└───────────────┴──────────────────────────────────────────────────────────────┘
# Run npm install — save-dev connect-redis@3.3.3 to resolve 1 vulnerability
SEMVER WARNING: Recommended action is a potentially breaking change
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Low │ Regular Expression Denial of Service │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ debug │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ connect-redis [dev] │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ connect-redis > debug │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://nodesecurity.io/advisories/534 │
└───────────────┴──────────────────────────────────────────────────────────────┘
found 5 vulnerabilities (3 low, 1 moderate, 1 critical) in 821 scanned packages
5 vulnerabilities require semver-major dependency updates.
nick@minerva:~/ws/lab/devsecops/express$

Pretty cool how each vuln even links to more info.

An example for the mime one: https://nodesecurity.io/advisories/535

You’ll notice that a couple of those vulns are “SEMVER WARNING”s.

This just means the package’s major version differs from the recommended, hence it’s possible the API has changed and you’ll need to update your code in order to use the new version.

You could carefully go check the release notes, and check your code.

In practice you really should.

But here we’re just gonna update to the new versions, run our beautiful unit tests and call it a day. I’ve got a good feeling about this!

OK, retry audit:

Great. Now run our unit tests locally …

…..

All passing, outstanding.

Let’s check the changes that “npm audit fix — force” made:

Sure enough, they’re just later versions in packages.json.

Great, commit this and push!

Back on bitbucket though, we see our tests fail for a different reason:

And this happens despite our tests passing locally.

Now quick quiz — if you had to bet $50, what would be the cause of this?

Pick your answer, then scroll down ….

If you guessed it’s because we’re using the node-alpine docker image instead of the standard node one (which uses debian), you’re right!

Right here my (correct and thorough, though slower) developer instinct is to understand WHY that particular test breaks in the alpine, but not the debian image.

But for now it’s good enough to know that if we change our bitbucket-pipeline.yml to image: node:10.6.0 then it works:

And back on bitbucket:

So there you have it. To review, we:

• Set up a bitbucket repo, with a CI pipeline auditing 3rd party dependencies
• Debugged that pipeline by changing the docker image it used (twice)
• Used the new “npm audit” feature to help update our dependencies and remove known vulnerabilities
• Used the pipeline to verify that our changes removed vulnerabilities
• Once, and for every commit from now on

We now can be much more confident in using and deploying our sweet express app.

Soon I hope to be doing similar for AWS infrastructure changes, possibly using another CI/CD tool. Stay tuned!

References

Npm doco on audit

https://blog.npmjs.org/post/173719309445/npm-audit-identify-and-fix-insecure

https://docs.npmjs.com/getting-started/running-a-security-audit

Bitbucket Pipelines

https://bitbucket.org/product/features/pipelines

The now vulnerability-free-passing-tests-but-potentially-not-100% express repo used in this example:

https://bitbucket.org/nick_doyle/express/src/master/