Professional advice regarding page loading speed

Okay so i have all my CSS and JS resources loaded in one single base file which contains all my css and javascript files. Now this single base file is INCLUDED in all the web pages.

Also the website contains many other php pages which are using those resources. Which means all the resources are loaded again and again every time i visit a php page.

Please take a look at this diagram here:

So lets says i visit Page1.php...then all the four files get loaded for that page...which are file1.css,file2.css,file3.js,file4.js....

Now lets say i visit Page2.php...then the same thing happens again....that all the four files get loaded for that page...which are file1.css,file2.css,file3.js,file4.js....

So now if we calculate the loading time for the Page1.php it would be equal to the combined loading time of all the following things :

1)- Loading time for HTML and Php written on Page1.php 2)- Loading time for file1.css 3)- Loading time for file2.css 4)- Loading time for file3.js 5)- Loading time for file4.js

So my question here is :

1)- How can i improve the loading time of a page ?

(I know its a good practice to include all the styles and javascript in a single file to make the code look clean....but what if there are some javascript or css files in the base file that are not being used by other pages...lets say Page3.php does not need file2.css and file4.js....and so i,m just wasting my resources and increasing my page load time trying to load file2.css and file4.js when Page3.php loads).

2)- Now lets says we have another scenario where a weird Javascript file processes a million records and then shows us some result on "Page1.php". Now this processing and loading would obviously take some time....lets say 5 seconds.

(Now imagine waiting for 5 seconds every time you visit a page... Lets says you visit Page2.php and have to wait for 5 seconds all for nothing.... and just because you made the mistake of including that weird javascript file in your base file...and since the base file is linked to all other pages you will have to wait for 5 seconds every time you visit a page.)

So is,nt it better to load that weird javascript file only for Page1.php and not for other pages since it has got nothing to do with the other pages ? What is the best practice here....