BufferedReader and Sockets in java

Suppose I'm using a Socket InputStream like this,in order to read the following data:

(first part encoded in ASCII)

    the rest of this message is binaryCRLF  
    01100110010...

code:

InputStreamReader isr = new InputStreamReader(socket.getInputStream()); 
BufferedReader in = new BufferedReader(isr);
String line = in.readLine();

what I expect at this point,is that line contains the text:

the rest of this message is binary

Then I would like to use the original socket InputStream:

 InputStream is = socket.getInputStream();  
    ...

to read the binary part of the message.

Now, what if the BufferedReader has buffered some of the binary data,

this would be a problem because I would miss part of the data.

Am I misssing something ?