mardi 5 novembre 2019

How to write multiple requested files to the server without causing IO blocking

I am having problems using java to write the files requested by the foreground to the server.

Description of the problem: The user uploads the 600M file, slices it to each piece of 100M to make a request, requests the background through the asynchronous request, and writes the file to the server after receiving the request in the background. As a result, the server IO is blocked and the request cannot respond. What kind of strategy should I take? Is it by modifying the asynchronous request as a synchronous request? How to deal with the server side?

For the javascript code:

  piecesize = 1024000*100;//100M

  async handlefiles(files:File[]){
    for (let i = 0; i < files.length; i++){
      this.uploadfile(files[i], i);
    }
  }

  async uploadfile(file:File,index:number){
    let filesize = file.size;
    let total = Math.ceil(filesize / this.piecesize);
    await this.handle(file, file.name, filesize, total);


  }

  async  handle(file:File, filename, filesize, total) {
    let start = 0, end = 0;
    let i = 0;
    while (i < total){
      end = start + this.piecesize;
      if (end >= filesize)
        end = filesize;

      let piece = file.slice(start, end);

      await this.send(filename, i, total, piece);

      start = end;
      i++;
    }
  }

  async send(filename, pieceindex, total, piece) {
    let piecename = 'piece-' + UUID.UUID();

    let formData:FormData = new FormData();
    formData.append('file', piece, piecename);
    this.httpclient.post("/receiveFiles",
      formData,
      {
        params:{
          'filename': filename,
          'piecename': piecename,
          'pieceindex': pieceindex,
          'total': total,
        },
      }).subscribe(data => {
        console.log("upload success");
    });
  }

Server-side java code

public Map receiveFiles(
        String piecename,
        int pieceindex,
        String filename,
        int total,
        MultipartFile file) throws IOException {
    String tmppath = prop.getProperty("codepath") + "/temp/"+ filename + "/";
    File msc = new File(tmppath);
    if (!msc.exists()){
        boolean mkdirs = msc.mkdirs();
        if(!mkdirs){
            throw new FileSystemException("error:The path ["+tmppath+"] cannot create ...");
        }
    }
    Path path = Paths.get(tmppath + pieceindex + "-" + piecename);
    HashMap result = new HashMap();
    result.put("result",true);
    try{
        Files.write(path, file.getBytes());
    }catch (IOException io){
        logger.error(io.getMessage());
        result.put("result",false);
    }
    List<Path> pieces = Files.list(Paths.get(tmppath)).filter(Files::isRegularFile).collect(Collectors.toList());
    result.put("size",pieces.size());
    logger.debug("receiveFiles return {}, filename {}, pieceindex {}", result, filename, pieceindex);
    return result;
}



Aucun commentaire:

Enregistrer un commentaire