Skip to content

[ernie] failed to predict. (data_id=3 log_id=3) [ernie|0] Failed to process(batch: [3]): In user code: #2008

Description

@lisdoo

请问以下报错是什么原因?
[ernie] failed to predict. (data_id=3 log_id=3) [ernie|0] Failed to process(batch: [3]): In user code:

File "/home/aistudio/work/PaddleNLP/legacy/applications/neural_search/recall/in_batch_negative/export_model.py", line 56, in <module>
  paddle.jit.save(model, save_path)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/decorator.py", line 232, in fun
  return caller(func, *(extras + args), **kw)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
  return wrapped_func(*args, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/api.py", line 752, in wrapper
  func(layer, path, input_spec, **configs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/decorator.py", line 232, in fun
  return caller(func, *(extras + args), **kw)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
  return wrapped_func(*args, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/dygraph/base.py", line 75, in __impl__
  return func(*args, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/api.py", line 1043, in save
  static_func.concrete_program_specify_input_spec(
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/dy2static/program_translator.py", line 709, in concrete_program_specify_input_spec
  concrete_program, _ = self.get_concrete_program(
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/dy2static/program_translator.py", line 589, in get_concrete_program
  concrete_program, partial_program_layer = self._program_cache[
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/dy2static/program_translator.py", line 1249, in __getitem__
  self._caches[item_id] = self._build_once(item)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/dy2static/program_translator.py", line 1193, in _build_once
  concrete_program = ConcreteProgram.from_func_spec(
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/decorator.py", line 232, in fun
  return caller(func, *(extras + args), **kw)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
  return wrapped_func(*args, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/dygraph/base.py", line 75, in __impl__
  return func(*args, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/dy2static/program_translator.py", line 1063, in from_func_spec
  outputs = static_func(*inputs)
File "/home/aistudio/work/PaddleNLP/legacy/applications/neural_search/recall/in_batch_negative/base_model.py", line 111, in get_pooled_embedding
  _, cls_embedding = self.ptm(input_ids, token_type_ids, position_ids, attention_mask)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1256, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1235, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddlenlp/transformers/ernie/modeling.py", line 331, in forward
  if attention_mask is None:
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/dy2static/convert_operators.py", line 352, in convert_ifelse
  out = _run_py_ifelse(
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/dy2static/convert_operators.py", line 429, in _run_py_ifelse
  py_outs = true_fn() if pred else false_fn()
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddlenlp/transformers/ernie/modeling.py", line 332, in forward
  attention_mask = paddle.unsqueeze(
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/layers/math_op_patch.py", line 445, in __impl__
  current_block(self).append_op(
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/framework.py", line 4013, in append_op
  op = Operator(
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/framework.py", line 2781, in __init__
  for frame in traceback.extract_stack():

InvalidArgumentError: Axis should be less than 2, but received axis is 2.
  [Hint: Expected axis < max_dim, but received axis:2 >= max_dim:2.] (at /paddle/paddle/phi/kernels/funcs/common_shape.h:53)
  [operator < equal > error]. Please check the input dict and checkout PipelineServingLogs/pipeline.log for more details.

运行的环境是:

python -V

Python 3.8.10

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

pip list | grep paddle

paddle-bfloat 0.1.7
paddle-serving-app 0.9.0
paddle-serving-client 0.9.0
paddle-serving-server-gpu 0.9.0.post1028
paddle2onnx 1.2.11
paddlefsl 1.1.0
paddlenlp 2.4.2
paddlepaddle-gpu 2.4.2

按照这个例子训练了一个召回模型
https://github.com/PaddlePaddle/PaddleNLP/tree/develop/slm/applications/neural_search/recall/in_batch_negative

现在是部署推理阶段报错了

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions