justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 32473.0@dunegpschedd02.fnal.gov

Jobsub ID32473.0@dunegpschedd02.fnal.gov
Workflow ID2135
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-02 21:47:08
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce02
Last heartbeat2025-09-02 22:59:14
From worker nodeHostnamedune001-7965187.0-lcg2699.gridpp.rl.ac.uk
cpuinfoAMD EPYC 9654 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-02 21:50:58
Input filesvd-protodune:np02vd_raw_run039275_0448_df-s05-d4_dw_0_20250902T052912.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-02 22:59:14
Saved logsjustin-logs:32473.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

samples within desired window for channel 8681
wclsFrameSaver: no samples within desired window for channel 8682
wclsFrameSaver: no samples within desired window for channel 8863
wclsFrameSaver: no samples within desired window for channel 8865
wclsFrameSaver: no samples within desired window for channel 9065
wclsFrameSaver: no samples within desired window for channel 9066
wclsFrameSaver: no samples within desired window for channel 9067
wclsFrameSaver: no samples within desired window for channel 9068
wclsFrameSaver: no samples within desired window for channel 9069
wclsFrameSaver: no samples within desired window for channel 9070
wclsFrameSaver: no samples within desired window for channel 9070
wclsFrameSaver: no samples within desired window for channel 9071
wclsFrameSaver: no samples within desired window for channel 9072
wclsFrameSaver: no samples within desired window for channel 9073
wclsFrameSaver: no samples within desired window for channel 9073
wclsFrameSaver: no samples within desired window for channel 9073
wclsFrameSaver: no samples within desired window for channel 9074
wclsFrameSaver: no samples within desired window for channel 9074
wclsFrameSaver: no samples within desired window for channel 9075
wclsFrameSaver: no samples within desired window for channel 9075
wclsFrameSaver: no samples within desired window for channel 9075
wclsFrameSaver: no samples within desired window for channel 9076
wclsFrameSaver: no samples within desired window for channel 9076
wclsFrameSaver: no samples within desired window for channel 9077
wclsFrameSaver: no samples within desired window for channel 9077
wclsFrameSaver: no samples within desired window for channel 9078
wclsFrameSaver: no samples within desired window for channel 9078
wclsFrameSaver: no samples within desired window for channel 9079
wclsFrameSaver: no samples within desired window for channel 9079
wclsFrameSaver: no samples within desired window for channel 9080
wclsFrameSaver: no samples within desired window for channel 9081
wclsFrameSaver: no samples within desired window for channel 9081
wclsFrameSaver: no samples within desired window for channel 9082
wclsFrameSaver: no samples within desired window for channel 9083
wclsFrameSaver: no samples within desired window for channel 9084
wclsFrameSaver: no samples within desired window for channel 9084
wclsFrameSaver: no samples within desired window for channel 9085
wclsFrameSaver: no samples within desired window for channel 9085
wclsFrameSaver: no samples within desired window for channel 9086
wclsFrameSaver: no samples within desired window for channel 9086
wclsFrameSaver: no samples within desired window for channel 9087
wclsFrameSaver: no samples within desired window for channel 9087
wclsFrameSaver: no samples within desired window for channel 9338
wclsFrameSaver: no samples within desired window for channel 9406
wclsFrameSaver: no samples within desired window for channel 9408
wclsFrameSaver: no samples within desired window for channel 9532
wclsFrameSaver: no samples within desired window for channel 9689
wclsFrameSaver: no samples within desired window for channel 9969
wclsFrameSaver: no samples within desired window for channel 9971
wclsFrameSaver: no samples within desired window for channel 9972
wclsFrameSaver: no samples within desired window for channel 10308
wclsFrameSaver: no samples within desired window for channel 10309
wclsFrameSaver: no samples within desired window for channel 10310
wclsFrameSaver: no samples within desired window for channel 10311
wclsFrameSaver: no samples within desired window for channel 10312
wclsFrameSaver: no samples within desired window for channel 10312
wclsFrameSaver: no samples within desired window for channel 10377
wclsFrameSaver: no samples within desired window for channel 10859
wclsFrameSaver: no samples within desired window for channel 10859
wclsFrameSaver: no samples within desired window for channel 10896
wclsFrameSaver: no samples within desired window for channel 10897
wclsFrameSaver: no samples within desired window for channel 10939
wclsFrameSaver: no samples within desired window for channel 11335
wclsFrameSaver: no samples within desired window for channel 11335
wclsFrameSaver: no samples within desired window for channel 11336
wclsFrameSaver: no samples within desired window for channel 11336
wclsFrameSaver: no samples within desired window for channel 11337
wclsFrameSaver: no samples within desired window for channel 11337
wclsFrameSaver: no samples within desired window for channel 11338
wclsFrameSaver: no samples within desired window for channel 11339
wclsFrameSaver: no samples within desired window for channel 11340
wclsFrameSaver: no samples within desired window for channel 11732
wclsFrameSaver: no samples within desired window for channel 11732
wclsFrameSaver: no samples within desired window for channel 12232
wclsFrameSaver: no samples within desired window for channel 12232
wclsFrameSaver: no samples within desired window for channel 12233
FrameSaver: q=9.37255e+06 n=1095378 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1111 XUs and 1805 XVs -> 77 XUVs
C:0 T:1 279 XUs and 281 XVs -> 13 XUVs
C:0 T:2 1314 XUs and 1850 XVs -> 62 XUVs
C:0 T:3 599 XUs and 717 XVs -> 22 XUVs
C:0 T:4 1061 XUs and 1894 XVs -> 64 XUVs
C:0 T:5 1248 XUs and 1290 XVs -> 72 XUVs
C:0 T:6 1086 XUs and 1194 XVs -> 50 XUVs
C:0 T:7 631 XUs and 490 XVs -> 30 XUVs
C:0 T:8 1815 XUs and 1438 XVs -> 73 XUVs
C:0 T:9 3586 XUs and 4440 XVs -> 296 XUVs
C:0 T:10 1625 XUs and 1726 XVs -> 127 XUVs
C:0 T:11 1503 XUs and 1959 XVs -> 123 XUVs
C:0 T:12 2117 XUs and 2366 XVs -> 189 XUVs
C:0 T:13 1762 XUs and 1325 XVs -> 109 XUVs
C:0 T:14 1533 XUs and 1780 XVs -> 99 XUVs
C:0 T:15 1948 XUs and 1496 XVs -> 77 XUVs
1483 XUVs total
1277 collection wire objects
1483 potential space points
Neighbour search...
9053 tests to find 4534 neighbours
Iterating with no regularization...
Begin: 5.58497e+08
0 5.50529e+08
1 5.50466e+08
Now with regularization...
Begin: 5.42298e+08
0 5.42295e+08
02-Sep-2025 22:58:51 UTC  Closed output file "np02vd_raw_run039275_0448_df-s05-d4_dw_0_20250902T052912_reco_stage1_20250902T225851_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        150.53        223.216       343.89        220.286       44.2592        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      9.1958e-05    0.000129622   0.000282415   0.000109735   4.57966e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               17.2877       18.366        19.6035       18.1702      0.672387        18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0458574     0.0484484     0.0667966     0.0464642    0.00532077       18     
produce:pdvddaphne:DAPHNEReaderPDVD               4.53415       5.33364       6.97873       5.35777      0.679869        18     
produce:ophit:OpHitFinder                        0.0370135     0.0672982     0.0791658     0.0689919    0.00996763       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00543978     0.0169229     0.0306847     0.0162803    0.00545527       18     
produce:wclsdatavd:WireCellToolkit                72.6256       85.3453       101.898       85.0411       7.33039        18     
produce:gaushit:GausHitFinder                     1.17658       1.86607       2.45565       1.82259      0.349333        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000271048   0.000453487    0.0011973    0.000391469   0.000202313      18     
produce:reco3d:SpacePointSolver                   9.88099       18.9722       26.275        18.6757       4.56922        18     
produce:hitpdune:DisambigFromSpacePoints         0.167472      0.329758      0.531424       0.32494      0.103478        18     
produce:pandora:StandardPandora                   25.0443       84.4891       181.65        84.0905       35.603         18     
produce:pandoraTrack:LArPandoraTrackCreation     0.611271       1.60799       2.74254       1.53623      0.564015        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0326055     0.0448385     0.0631412     0.0450568    0.00808768       18     
[art]:TriggerResults:TriggerResultInserter      4.1142e-05    5.20379e-05   0.000103616   4.80325e-05   1.37171e-05      18     
end_path:out1:RootOutput                         8.333e-06    1.42475e-05   3.4372e-05    1.16375e-05   6.87185e-06      18     
end_path:out1:RootOutput(write)                   6.26138       6.64903       7.06952       6.64061      0.220906        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5637.48 MB
  Peak resident set size usage (VmHWM): 3516.04 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039275_0448_df-s05-d4_dw_0_20250902T052912_reco_stage1_20250902T225851_keepup.root
\tHists: np02vd_raw_run039275_0448_df-s05-d4_dw_0_20250902T052912_reco_stage1_20250902T225851_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2025-11-05 14:41:12 UTC       justIN version: 01.05.01