NGinx Denying all access to folders but PHP scripts are not being affected by rule

I'm trying to set security directives in my server configuration file for NGinx. I've got the following directive:

location /config {
    deny all;
    return 404;
}

All the files in that directory are being restricted but PHP files are not being affected by that directive and my intention is to deny everything. I'm assuming that other directive in my config file is overriding this one but I'm quite novice in NGinx.

This is the full config code for the server:

server{
        listen 80;
        server_name mydomain.com;
        root myrootpath;
        index index.php index.html index.htm;

        include security-directives;

        location ~ \.php$ {
            fastcgi_pass   127.0.0.1:9123;
            fastcgi_index  index.php;
            fastcgi_param  SCRIPT_FILENAME  
            $document_root$fastcgi_script_name;
            include        fastcgi_params;
        }
}

security-directives file contains the directive detailed in the first code block.