Jobsub ID 43068.100@dunegpschedd01.fnal.gov
| Jobsub ID | 43068.100@dunegpschedd01.fnal.gov |
| Workflow ID | 2330 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 11:31:56 |
| Site | UK_QMUL |
| Entry | DUNE_UK_London_QMUL_arcce02 |
| Last heartbeat | 2025-09-08 15:55:45 |
| From worker node | Hostname | cn080.htc.esc.qmul |
| cpuinfo | Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-09-08 14:17:25 |
| Input files | vd-protodune:np02vd_raw_run039324_3005_df-s05-d3_dw_0_20250908T083641.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-09-08 15:55:45 |
| Saved logs | justin-logs:43068.100-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
hannelSelector : 0 sec
[16:51:40.992] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:51:40.992] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:51:40.992] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:51:40.992] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:51:40.992] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[16:51:40.992] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[16:51:40.992] I [ timer ] Timer: Total node execution : 58.20999952591956 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 37400 traces tagged "gauss"
FrameSaver: q=7.47611e+06 n=843060 tag=gauss
wclsFrameSaver: saving 46939 traces tagged "wiener"
FrameSaver: q=8.06444e+06 n=805753 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1378 XUs and 1207 XVs -> 59 XUVs
C:0 T:1 39 XUs and 52 XVs -> 2 XUVs
C:0 T:2 1375 XUs and 1026 XVs -> 38 XUVs
C:0 T:3 48 XUs and 60 XVs -> 0 XUVs
C:0 T:4 19 XUs and 16 XVs -> 0 XUVs
C:0 T:5 358 XUs and 279 XVs -> 44 XUVs
C:0 T:6 112 XUs and 321 XVs -> 7 XUVs
C:0 T:7 1941 XUs and 2084 XVs -> 116 XUVs
C:0 T:8 18283 XUs and 42943 XVs -> 4054 XUVs
C:0 T:9 14772 XUs and 24565 XVs -> 2604 XUVs
C:0 T:10 9989 XUs and 17410 XVs -> 5003 XUVs
C:0 T:11 1543 XUs and 787 XVs -> 57 XUVs
C:0 T:12 1618 XUs and 1636 XVs -> 278 XUVs
C:0 T:13 968 XUs and 1096 XVs -> 64 XUVs
C:0 T:14 308 XUs and 214 XVs -> 18 XUVs
C:0 T:15 981 XUs and 843 XVs -> 71 XUVs
12415 XUVs total
1664 collection wire objects
12415 potential space points
Neighbour search...
1934163 tests to find 770448 neighbours
Iterating with no regularization...
Begin: 7.92089e+08
0 6.90697e+08
1 6.83277e+08
2 6.82223e+08
3 6.81904e+08
Now with regularization...
Begin: 6.69583e+08
0 6.69305e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
08-Sep-2025 16:52:55 BST Closed output file "np02vd_raw_run039324_3005_df-s05-d3_dw_0_20250908T083641_reco_stage1_20250908T155255_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 140.48 217.334 334.203 216.622 43.4443 26
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 8.6379e-05 0.000122501 0.000314041 0.000117791 4.07049e-05 26
produce:tpcrawdecoder:PDVDTPCReader 19.1572 31.9969 66.6739 29.0364 11.6159 26
produce:triggerrawdecoder:PDVDTriggerReader4 0.0331616 0.0361339 0.0455743 0.0336223 0.0046948 26
produce:pdvddaphne:DAPHNEReaderPDVD 5.90004 8.56113 13.0686 7.96152 1.92275 26
produce:ophit:OpHitFinder 0.0461442 0.0731621 0.0857589 0.0757524 0.00992507 26
produce:opflash:OpFlashFinderVerticalDrift 0.0103487 0.0197054 0.0282776 0.0207048 0.0044914 26
produce:wclsdatavd:WireCellToolkit 56.0386 76.6615 123.425 74.1752 16.0082 26
produce:gaushit:GausHitFinder 0.936149 1.67149 2.35167 1.66592 0.385305 26
produce:nhitsfilter:NumberOfHitsFilter 0.000580271 0.000806474 0.00144009 0.000715361 0.000198041 26
produce:reco3d:SpacePointSolver 7.30135 12.85 19.5594 12.6177 2.94678 26
produce:hitpdune:DisambigFromSpacePoints 0.179213 0.314872 0.533352 0.295253 0.0909182 26
produce:pandora:StandardPandora 21.3309 77.2042 145.868 72.2617 32.1315 26
produce:pandoraTrack:LArPandoraTrackCreation 1.24822 3.26707 6.25437 3.10329 1.33108 26
produce:pandoraGnocalo:GnocchiCalorimetry 0.0232704 0.0437979 0.0651105 0.0447996 0.0104203 26
[art]:TriggerResults:TriggerResultInserter 4.1199e-05 8.44317e-05 0.000207921 8.23625e-05 3.12534e-05 26
end_path:out1:RootOutput 5.707e-06 1.81241e-05 6.5398e-05 1.6316e-05 1.10268e-05 26
end_path:out1:RootOutput(write) 3.12289 4.54816 7.70467 4.68796 0.95335 26
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4682.41 MB
Peak resident set size usage (VmHWM): 2794.51 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_3005_df-s05-d3_dw_0_20250908T083641_reco_stage1_20250908T155255_keepup.root
\tHists: np02vd_raw_run039324_3005_df-s05-d3_dw_0_20250908T083641_reco_stage1_20250908T155257_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_3005_df-s05-d3_dw_0_20250908T083641_reco_stage1_20250908T155255_keepup.root
Traceback (most recent call last):
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/20943968d0063a8f11ac6a4062b5ad0c8392b138/meta_maker.py", line 48, in <module>
results = inherit_metadata.inherit(args.parent)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/20943968d0063a8f11ac6a4062b5ad0c8392b138/inherit_metadata.py", line 72, in inherit
'metadata':get_parent_md(parent_name),
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/20943968d0063a8f11ac6a4062b5ad0c8392b138/inherit_metadata.py", line 27, in get_parent_md
parent_file = mc.get_file(did=parent_name, with_metadata=True,
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_0_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 1259, in get_file
return self.get_json(url)
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_0_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 206, in get_json
return self.unpack_json_data(self.send_request("get", uri_suffix, headers=headers, stream=True))
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_0_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 168, in send_request
raise WebAPIError(url, response)
metacat.webapi.webapi.WebAPIError: <html><body><h2>Application error</h2>
<h3>connection to server at "ifdb11.fnal.gov" (2620:6a:0:8430:f0:0:240:225), port 5463 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
connection to server at "ifdb11.fnal.gov" (131.225.240.225), port 5463 failed: FATAL: remaining connection slots are reserved for non-replication superuser connections
</h3>
<pre>
Traceback (most recent call last):
File "/home/dbidata/products/webpie/current/webpie/WPApp.py", line 651, in wsgi_call
response = root_handler._handle_request(request, "", path_down, args)
File "/home/dbidata/products/webpie/current/webpie/WPApp.py", line 444, in _handle_request
return subhandler._handle_request(request, path + "/" + word, path_down, args)
File "/home/dbidata/products/metacat_server/dune/server/common_handler.py", line 149, in _handle_request
response = super()._handle_request(request, path, path_down, args)
File "/home/dbidata/products/webpie/current/webpie/WPApp.py", line 442, in _handle_request
return subhandler(request, relpath, **args)
File "/home/dbidata/products/metacat_server/dune/server/common_handler.py", line 94, in decorated
try: out = method(self, *params, **agrs)
File "/home/dbidata/products/metacat_server/dune/server/data_handler.py", line 1251, in file
db = self.App.connect()
File "/home/dbidata/products/metacat_server/dune/lib/metacat/auth/server/base_server.py", line 48, in connect
return self.DB.connect()
File "/home/dbidata/products/pythreader/current/pythreader/core.py", line 22, in smethod
out = method(self, *params, **args)
File "/home/dbidata/products/wsdbtools/current/wsdbtools/ConnectionPool.py", line 285, in connect
use_connection = self.Connector.connect()
File "/home/dbidata/products/wsdbtools/current/wsdbtools/ConnectionPool.py", line 169, in connect
conn = psycopg2.connect(self.Connstr)
File "/home/dbidata/products/metacat_server/venv/lib64/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "ifdb11.fnal.gov" (2620:6a:0:8430:f0:0:240:225), port 5463 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
connection to server at "ifdb11.fnal.gov" (131.225.240.225), port 5463 failed: FATAL: remaining connection slots are reserved for non-replication superuser connections
</pre>
</body>
</html>
Error in reco metadata 