Jobsub ID 262865.45@dunegpschedd01.fnal.gov
| Jobsub ID | 262865.45@dunegpschedd01.fnal.gov |
| Workflow ID | 10978 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 2 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-12-03 20:39:42 |
| Site | NL_SURFsara |
| Entry | DUNE_SurfSARA_arc03 |
| Last heartbeat | 2025-12-03 22:07:30 |
| From worker node | Hostname | wn-da-10.gina.surf.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 2 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-12-03 20:52:36 |
| Input files | vd-protodune:np02vd_raw_run041106_0715_df-s04-d0_dw_0_20251203T202636.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-12-03 22:07:30 |
| Saved logs | justin-logs:262865.45-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
r2":9818 [9818] "wiener3":7501 [7501] ] cmm:[ ]
Retagger: tagging trace set: wiener with 28963 traces, 28963 summary
Retagger: tagging trace set: gauss with 22819 traces, 0 summary
[23:06:30.413] D [ glue ] sink frame: #34968 @2.3e-05 with 86039 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[23:06:30.413] D [ glue ] <FrameFanout:nfsp> call=97: see EOS
[23:06:30.414] D [ glue ] <ChannelSelector:chsel7> see EOS at call=48
[23:06:30.414] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=97 anode=7
[23:06:30.414] D [ glue ] <ChannelSelector:chsel6> see EOS at call=48
[23:06:30.414] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=97 anode=6
[23:06:30.414] D [ glue ] <ChannelSelector:chsel5> see EOS at call=48
[23:06:30.414] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=97 anode=5
[23:06:30.414] D [ glue ] <ChannelSelector:chsel4> see EOS at call=48
[23:06:30.414] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=97 anode=4
[23:06:30.414] D [ glue ] <ChannelSelector:chsel3> see EOS at call=48
[23:06:30.414] D [ aux ] <Resampler:resmp3> EOS at call=97
[23:06:30.414] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=97 anode=3
[23:06:30.414] D [ glue ] <ChannelSelector:chsel2> see EOS at call=48
[23:06:30.414] D [ aux ] <Resampler:resmp2> EOS at call=97
[23:06:30.414] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=97 anode=2
[23:06:30.414] D [ glue ] <ChannelSelector:chsel1> see EOS at call=48
[23:06:30.414] D [ aux ] <Resampler:resmp1> EOS at call=97
[23:06:30.414] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=97 anode=1
[23:06:30.414] D [ glue ] <ChannelSelector:chsel0> see EOS at call=48
[23:06:30.414] D [ aux ] <Resampler:resmp0> EOS at call=97
[23:06:30.414] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=97 anode=0
[23:06:30.414] D [ glue ] <FrameFanin:nfsp> EOS at call=97 with 8
[23:06:30.414] D [ glue ] frame sink sees EOS
[23:06:30.414] D [ pgraph ] <Pgrapher:> graph execution complete
[23:06:30.414] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 17.59 sec
[23:06:30.414] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 17.19 sec
[23:06:30.414] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 16.89 sec
[23:06:30.414] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 16.7 sec
[23:06:30.414] I [ timer ] Timer: WireCell::Aux::Resampler : 0.21 sec
[23:06:30.414] I [ timer ] Timer: WireCell::Aux::Resampler : 0.2 sec
[23:06:30.414] I [ timer ] Timer: WireCell::Aux::Resampler : 0.2 sec
[23:06:30.414] I [ timer ] Timer: WireCell::Aux::Resampler : 0.2 sec
[23:06:30.414] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[23:06:30.414] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[23:06:30.414] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[23:06:30.414] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[23:06:30.414] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:06:30.414] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[23:06:30.414] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:06:30.414] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:06:30.414] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:06:30.414] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[23:06:30.414] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:06:30.414] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[23:06:30.415] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:06:30.415] I [ timer ] Timer: WireCell::Gen::Retagger : 0 sec
[23:06:30.415] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:06:30.415] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[23:06:30.415] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[23:06:30.415] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[23:06:30.415] I [ timer ] Timer: Total node execution : 69.20000084117055 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 22819 traces tagged "gauss"
FrameSaver: q=7.30633e+06 n=508894 tag=gauss
wclsFrameSaver: saving 28963 traces tagged "wiener"
FrameSaver: q=7.80008e+06 n=485492 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 429 XUs and 603 XVs -> 32 XUVs
C:0 T:1 944 XUs and 1065 XVs -> 48 XUVs
C:0 T:2 1382 XUs and 1248 XVs -> 44 XUVs
C:0 T:3 286 XUs and 358 XVs -> 5 XUVs
C:0 T:4 531 XUs and 666 XVs -> 21 XUVs
C:0 T:5 5964 XUs and 7614 XVs -> 401 XUVs
C:0 T:6 2693 XUs and 1372 XVs -> 56 XUVs
C:0 T:7 765 XUs and 857 XVs -> 58 XUVs
665 XUVs total
484 collection wire objects
665 potential space points
Neighbour search...
15225 tests to find 4988 neighbours
Iterating with no regularization...
Begin: 1.00265e+10
0 9.96422e+09
1 9.95964e+09
Now with regularization...
Begin: 9.84225e+09
0 9.84216e+09
03-Dec-2025 23:06:53 CET Closed output file "np02vd_raw_run041106_0715_df-s04-d0_dw_0_20251203T202636_reco_stage1_20251203T220653_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 56.2834 89.0225 228.781 82.0112 30.4271 49
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 5.5004e-05 6.83792e-05 0.00017306 6.295e-05 1.77985e-05 49
produce:tpcrawdecoder:PDVDTPCReader 5.53648 6.27305 9.0208 5.84808 0.857273 49
produce:triggerrawdecoder:PDVDTriggerReader4 0.293964 0.304834 0.343291 0.301033 0.0107345 49
produce:pdvddaphne:DAPHNEReaderPDVD 0.000261958 0.000299973 0.000650859 0.000283119 7.36622e-05 49
produce:ophit:OpHitFinder 5.2881e-05 7.06505e-05 0.00045704 5.6508e-05 5.79292e-05 49
produce:opflash:OpFlashFinderVerticalDrift 3.8373e-05 5.05337e-05 0.000284191 4.0817e-05 3.73371e-05 49
produce:wclsdatavd:WireCellToolkit 34.1061 52.5296 96.392 46.5238 14.581 49
produce:gaushit:GausHitFinder 0.383006 0.606479 0.973056 0.587976 0.142993 49
produce:nhitsfilter:NumberOfHitsFilter 8.7096e-05 0.000160378 0.000402135 0.000145988 6.37886e-05 49
produce:reco3d:SpacePointSolver 3.01232 5.9932 12.0477 5.6603 2.09672 49
produce:hitpdune:DisambigFromSpacePoints 0.0226982 0.0692951 0.203171 0.0605306 0.0368819 49
produce:pandora:StandardPandora 6.20044 21.0197 110.022 14.079 19.0599 49
produce:pandoraTrack:LArPandoraTrackCreation 0.106431 0.278502 0.679398 0.221872 0.138428 49
produce:pandoraGnocalo:GnocchiCalorimetry 0.00600524 0.0135725 0.0236021 0.0135793 0.00363126 49
[art]:TriggerResults:TriggerResultInserter 1.4047e-05 2.0098e-05 5.0947e-05 1.8315e-05 6.80732e-06 49
end_path:out1:RootOutput 3.657e-06 5.00596e-06 3.6719e-05 4.339e-06 4.58754e-06 49
end_path:out1:RootOutput(write) 1.71295 1.90044 2.18067 1.8891 0.112487 49
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4576.83 MB
Peak resident set size usage (VmHWM): 2546.15 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run041106_0715_df-s04-d0_dw_0_20251203T202636_reco_stage1_20251203T220653_keepup.root
\tHists: np02vd_raw_run041106_0715_df-s04-d0_dw_0_20251203T202636_reco_stage1_20251203T220653_keepup_hists.root
Forming reco metadata
Traceback (most recent call last):
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/ff18e11900bfcc2b467fc81eba35d7e7de647a80/meta_maker.py", line 1, in <module>
import place_metadata, get_events_for_md, inherit_metadata, add_origins
File "/cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e26-p3915-prof/lib/ROOT/_facade.py", line 154, in _importhook
return _orig_ihook(name, *args, **kwds)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/ff18e11900bfcc2b467fc81eba35d7e7de647a80/inherit_metadata.py", line 1, in <module>
from metacat.webapi import MetaCatClient
File "/cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e26-p3915-prof/lib/ROOT/_facade.py", line 154, in _importhook
return _orig_ihook(name, *args, **kwds)
ModuleNotFoundError: No module named 'metacat'
Error in reco metadata 