justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 32473.1@dunegpschedd02.fnal.gov

Jobsub ID32473.1@dunegpschedd02.fnal.gov
Workflow ID2135
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-02 21:47:08
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2025-09-02 23:05:55
From worker nodeHostnamecomp05-24
cpuinfoIntel(R) Xeon(R) Platinum 8352Y CPU @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-02 21:50:26
Input filesvd-protodune:np02vd_raw_run039275_0445_df-s02-d2_dw_0_20250902T052200.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-02 23:05:55
Saved logsjustin-logs:32473.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

window for channel 10881
wclsFrameSaver: no samples within desired window for channel 10886
wclsFrameSaver: no samples within desired window for channel 11341
wclsFrameSaver: no samples within desired window for channel 11341
wclsFrameSaver: no samples within desired window for channel 11342
wclsFrameSaver: no samples within desired window for channel 11343
wclsFrameSaver: no samples within desired window for channel 11345
wclsFrameSaver: no samples within desired window for channel 11778
wclsFrameSaver: no samples within desired window for channel 11788
wclsFrameSaver: no samples within desired window for channel 11798
wclsFrameSaver: no samples within desired window for channel 11798
wclsFrameSaver: no samples within desired window for channel 11799
wclsFrameSaver: no samples within desired window for channel 11800
wclsFrameSaver: no samples within desired window for channel 11801
wclsFrameSaver: no samples within desired window for channel 11809
wclsFrameSaver: no samples within desired window for channel 11813
wclsFrameSaver: no samples within desired window for channel 11813
FrameSaver: q=4.72368e+06 n=668136 tag=gauss
wclsFrameSaver: saving 38455 traces tagged "wiener"
wclsFrameSaver: no samples within desired window for channel 6194
wclsFrameSaver: no samples within desired window for channel 6832
wclsFrameSaver: no samples within desired window for channel 7033
wclsFrameSaver: no samples within desired window for channel 7246
wclsFrameSaver: no samples within desired window for channel 7444
wclsFrameSaver: no samples within desired window for channel 7448
wclsFrameSaver: no samples within desired window for channel 7579
wclsFrameSaver: no samples within desired window for channel 7595
wclsFrameSaver: no samples within desired window for channel 7601
wclsFrameSaver: no samples within desired window for channel 7873
wclsFrameSaver: no samples within desired window for channel 7874
wclsFrameSaver: no samples within desired window for channel 7899
wclsFrameSaver: no samples within desired window for channel 9425
wclsFrameSaver: no samples within desired window for channel 9535
wclsFrameSaver: no samples within desired window for channel 9580
wclsFrameSaver: no samples within desired window for channel 9646
wclsFrameSaver: no samples within desired window for channel 9790
wclsFrameSaver: no samples within desired window for channel 9793
wclsFrameSaver: no samples within desired window for channel 9828
wclsFrameSaver: no samples within desired window for channel 9828
wclsFrameSaver: no samples within desired window for channel 9881
wclsFrameSaver: no samples within desired window for channel 9908
wclsFrameSaver: no samples within desired window for channel 9909
wclsFrameSaver: no samples within desired window for channel 9909
wclsFrameSaver: no samples within desired window for channel 9911
wclsFrameSaver: no samples within desired window for channel 10001
wclsFrameSaver: no samples within desired window for channel 10327
wclsFrameSaver: no samples within desired window for channel 10584
wclsFrameSaver: no samples within desired window for channel 10711
wclsFrameSaver: no samples within desired window for channel 10713
wclsFrameSaver: no samples within desired window for channel 10779
wclsFrameSaver: no samples within desired window for channel 10779
wclsFrameSaver: no samples within desired window for channel 10782
wclsFrameSaver: no samples within desired window for channel 10782
wclsFrameSaver: no samples within desired window for channel 10783
wclsFrameSaver: no samples within desired window for channel 10832
wclsFrameSaver: no samples within desired window for channel 10832
wclsFrameSaver: no samples within desired window for channel 10844
wclsFrameSaver: no samples within desired window for channel 10881
wclsFrameSaver: no samples within desired window for channel 10886
wclsFrameSaver: no samples within desired window for channel 11341
wclsFrameSaver: no samples within desired window for channel 11341
wclsFrameSaver: no samples within desired window for channel 11342
wclsFrameSaver: no samples within desired window for channel 11343
wclsFrameSaver: no samples within desired window for channel 11345
wclsFrameSaver: no samples within desired window for channel 11778
wclsFrameSaver: no samples within desired window for channel 11788
wclsFrameSaver: no samples within desired window for channel 11798
wclsFrameSaver: no samples within desired window for channel 11798
wclsFrameSaver: no samples within desired window for channel 11798
wclsFrameSaver: no samples within desired window for channel 11799
wclsFrameSaver: no samples within desired window for channel 11800
wclsFrameSaver: no samples within desired window for channel 11800
wclsFrameSaver: no samples within desired window for channel 11801
wclsFrameSaver: no samples within desired window for channel 11809
wclsFrameSaver: no samples within desired window for channel 11813
wclsFrameSaver: no samples within desired window for channel 11813
wclsFrameSaver: no samples within desired window for channel 11813
FrameSaver: q=4.96215e+06 n=651394 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 79 XUs and 72 XVs -> 4 XUVs
C:0 T:1 334 XUs and 317 XVs -> 27 XUVs
C:0 T:2 32 XUs and 36 XVs -> 0 XUVs
C:0 T:3 371 XUs and 349 XVs -> 13 XUVs
C:0 T:4 525 XUs and 795 XVs -> 34 XUVs
C:0 T:5 173 XUs and 230 XVs -> 10 XUVs
C:0 T:6 246 XUs and 309 XVs -> 21 XUVs
C:0 T:7 649 XUs and 901 XVs -> 52 XUVs
C:0 T:8 1145 XUs and 1075 XVs -> 76 XUVs
C:0 T:9 872 XUs and 744 XVs -> 36 XUVs
C:0 T:10 375 XUs and 358 XVs -> 30 XUVs
C:0 T:11 1951 XUs and 1585 XVs -> 105 XUVs
C:0 T:12 1808 XUs and 1909 XVs -> 105 XUVs
C:0 T:13 193 XUs and 162 XVs -> 23 XUVs
C:0 T:14 1206 XUs and 979 XVs -> 45 XUVs
C:0 T:15 491 XUs and 1001 XVs -> 47 XUVs
628 XUVs total
538 collection wire objects
628 potential space points
Neighbour search...
3418 tests to find 2042 neighbours
Iterating with no regularization...
Begin: 1.4807e+08
0 1.45069e+08
1 1.44992e+08
Now with regularization...
Begin: 1.42284e+08
0 1.42271e+08
03-Sep-2025 00:05:32 BST  Closed output file "np02vd_raw_run039275_0445_df-s02-d2_dw_0_20250902T052200_reco_stage1_20250902T230532_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        164.66        248.269       371.333       238.918       50.0751        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.2298e-05    7.94773e-05   0.000151265   7.0039e-05    2.34301e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               19.2951       32.3075       59.372        29.1791       11.494         18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0492432     0.0522243     0.0821863     0.0501272    0.00734848       18     
produce:pdvddaphne:DAPHNEReaderPDVD               8.4747        9.95242       12.8224       9.46951       1.30654        18     
produce:ophit:OpHitFinder                        0.0400948     0.0508539     0.0593572     0.0533465    0.00627063       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00718195     0.0139444     0.0213512     0.0149409    0.00395667       18     
produce:wclsdatavd:WireCellToolkit                74.8651       92.4257       104.476       92.9448       7.2628         18     
produce:gaushit:GausHitFinder                     1.03739       1.84413       2.51888       1.87696      0.388438        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000241896   0.000447484   0.000619961   0.000449615   9.59525e-05      18     
produce:reco3d:SpacePointSolver                   7.85059       21.3815       32.6234       21.8762       6.48749        18     
produce:hitpdune:DisambigFromSpacePoints         0.0982791     0.300615      0.493315      0.298773      0.109503        18     
produce:pandora:StandardPandora                   23.2105       82.7266       192.793       76.5801       40.8024        18     
produce:pandoraTrack:LArPandoraTrackCreation      0.59893       1.59887       3.17099       1.53776      0.620451        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0177697     0.0328452     0.0454017     0.0317434    0.00716506       18     
[art]:TriggerResults:TriggerResultInserter      2.2454e-05    3.05306e-05   6.5126e-05    2.79585e-05   9.22153e-06      18     
end_path:out1:RootOutput                         7.252e-06    9.60978e-06    2.03e-05      8.368e-06    3.37937e-06      18     
end_path:out1:RootOutput(write)                   5.1423        5.54121       5.97423       5.5267       0.234781        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5113.14 MB
  Peak resident set size usage (VmHWM): 3216.56 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039275_0445_df-s02-d2_dw_0_20250902T052200_reco_stage1_20250902T230532_keepup.root
\tHists: np02vd_raw_run039275_0445_df-s02-d2_dw_0_20250902T052200_reco_stage1_20250902T230533_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2025-11-05 14:41:12 UTC       justIN version: 01.05.01