justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269049.107@dunegpschedd01.fnal.gov

Jobsub ID269049.107@dunegpschedd01.fnal.gov
Workflow ID11251
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors4
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 15:20:56
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2025-12-18 17:23:19
From worker nodeHostnameb9p10p3260.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors4
RSS bytes7864320000 (7500 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-18 15:45:14
Input filesvd-protodune-top:3430_21_b_pvdt.test
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-12-18 17:23:19
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

===============================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        44.493        172.73        618.266       70.0913       157.955        30     
----------------------------------------------------------------------------------------------------------------------------------
source:VDColdboxTDERawInput(read)                6.432e-05    0.000121558   0.000193681   0.00011736    3.2151e-05       30     
produce:caldata:DataPrepModule                   0.125197      0.249698      0.611783      0.250183      0.0889226       30     
produce:wclsdatavd:WireCellToolkit                35.6983       57.051        109.946       49.4399       17.731         30     
produce:gaushit:GausHitFinder                     0.35313       2.55378       6.96329       2.19929       1.84062        30     
produce:reco3d:SpacePointSolver                   1.7322        26.681        96.1179       5.0009        31.3219        30     
produce:hitpdune:DisambigFromSpacePoints        0.00989834     0.385408       1.55422      0.0345844     0.487753        30     
produce:pandora:StandardPandora                   1.90917       81.9646       439.011       10.3882       109.884        30     
produce:pandoraTrack:LArPandoraTrackCreation     0.0400163      0.79441       2.91583      0.188248      0.935642        30     
produce:pandoraGnocalo:GnocchiCalorimetry       0.00589347     0.0423891     0.138999      0.0175451     0.0411703       30     
[art]:TriggerResults:TriggerResultInserter       1.249e-05    2.88081e-05    5.639e-05     2.601e-05    1.10071e-05      30     
end_path:out1:RootOutput                         3.16e-06      6.39e-06      2.347e-05      4.3e-06     4.0079e-06       30     
end_path:out1:RootOutput(write)                   1.80782       2.98839       5.01761       2.84834      0.897081        30     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6839.84 MB
  Peak resident set size usage (VmHWM): 4401.94 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 30
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 3 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:    vdcb_adcChannelRawRmsFiller:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.00 sec/event
Art has completed and will exit with status 0.
Output files:
\tReco: 3430_21_b_pvdt_reco_stage1_20251218T171414_reco.root
Forming reco metadata
Successfully opened file 3430_21_b_pvdt_reco_stage1_20251218T171414_reco.root
Traceback (most recent call last):
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x1469fba8af10>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='metacat.fnal.gov', port=9443): Max retries exceeded with url: /dune_meta_prod/app/data/file?with_metadata=yes&with_provenance=no&with_datasets=no&name=3430_21_b_pvdt.test&namespace=vd-protodune-top (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x1469fba8af10>: Failed to establish a new connection: [Errno 110] Connection timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/273045f43d7c86de1848a0b2b33e0d614d73cf83/meta_maker.py", line 48, in <module>
    results = inherit_metadata.inherit(args.parent)
  File "/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/273045f43d7c86de1848a0b2b33e0d614d73cf83/inherit_metadata.py", line 70, in inherit
    'metadata':get_parent_md(parent_name),
  File "/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/273045f43d7c86de1848a0b2b33e0d614d73cf83/inherit_metadata.py", line 25, in get_parent_md
    parent_file = mc.get_file(did=parent_name, with_metadata=True,
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 1259, in get_file
    return self.get_json(url)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 206, in get_json
    return self.unpack_json_data(self.send_request("get", uri_suffix, headers=headers, stream=True))
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 154, in send_request
    self.LastResponse = response = self.retry_request(method, url, headers=headers, **args)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 131, in retry_request
    response = requests.get(url, timeout=self.Timeout, **args)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='metacat.fnal.gov', port=9443): Max retries exceeded with url: /dune_meta_prod/app/data/file?with_metadata=yes&with_provenance=no&with_datasets=no&name=3430_21_b_pvdt.test&namespace=vd-protodune-top (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x1469fba8af10>: Failed to establish a new connection: [Errno 110] Connection timed out'))
Error in reco metadata
justIN time: 2025-12-19 05:14:35 UTC       justIN version: 01.05.03