Embedding data into a coordinate system for the web

I am currently building a web app that returns data based on the x and y mouse coordinates of an image. The app flow is as follows:

1. The user is presented with a single image on screen (e.g. image1.png).
2. The user clicks the mouse on the image in a certain x,y coordinate (e.g. x=10, y=50).
3. The app takes these x,y coordinates and returns some additional data.
4. The format of this data looks like this: { labelNumber: 10, labelName: 'test', r: 55, g: 200, b: 255 }.
5. The precise data that is returned is dependent on the x,y coordinate that the user clicks on within an image. We have thousands of unique images.

The current methodology for retrieving the data is to parse an .npy multidimensional array (nDarray) which returns the data based on the x,y coordinate that is passed to the .npy array. For example, nDarray[10][50] returns { labelNumber: 10, labelName: 'test', r: 55, g: 200, b: 255 }.

The problem is that this method is not performant. The size of the .npy arrays are far too large and increase in relation to the resolution of the images (we have some very hi-resolution images!). In addition, we have thousands of images which means our total asset size is hundreds of gigabytes large.

My question is, does anyone have any ideas for storing the required data and accessing it when the user clicks on a particular x,y coordinate of an image? We require the assets to be as small as possible so that the application runs smoothly.

Thanks in advance.