ASP.NET, MVC, Entity Framework website slow TTFB

I've created a website that hosted in IIS on a server local to my company, it will only be used by employees of the business. I've been using Chrome dev tools to monitor the network activity and see loading times etc. I've noticed that my TTFB is always really slow, even when I'm loading very little data.

I read an article earlier that stated the TTFB should ideally be less than 500ms, yet mine is routinely over 3 or 4 seconds atleast!

I've included a screenshot of the dev tool page showing what I mean:

The particular page I've loaded is the home page. It connects to an MSSQL DB using EF6 and does one query to check if a person exists in a table with about 200 entries. Here is the code:

using (var db = new DbContext())
{
     var ad = httpContext.User.Identity.GetActiveDirectory();

     var person = db.PERSON.Include(p => p.PERSON_PRIVILEGE_ALLOC).FirstOrDefault(p => p.ACTIVE_DIRECTORY == ad);
}

If I load this same page on my machine using IIS Express the TTFB for the html doc is 35ms. Compared to a TTFB of 3450ms when loaded on our main server.

This is my first proper website to be hosted and so I'm relatively new to this. I understand what TTFB is but I'm unsure what is taking so long? The Db query is small and quick, the html isn't over sized, the CCS and JS is about 1MB when compressed using GZIP.

Does anyone have any info or suggestions to what is causing this and how it could be sped up? Could it potentially just be caused by the server?