justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 298752.2@dunegpschedd01.fnal.gov

Jobsub ID298752.2@dunegpschedd01.fnal.gov
Workflow ID12206
Stage ID1
User namemsiden@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2096103424 (1999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-21 21:36:13
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw6
Last heartbeat2026-01-22 03:12:12
From worker nodeHostnamemh-7763-18.t2.ucsd.edu
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-21 21:37:26
Input fileshd-protodune:np04hd_raw_run027308_0018_dataflow2_datawriter_0_20240619T183931.hdf5
JobscriptExit code0
Real time5h (20058s)
CPU time1h (6728s = 33%)
Max RSS bytes2120761344 (2022 MiB)
Outputting started2026-01-22 03:11:46
Output files
Finished2026-01-22 03:12:12
Saved logsjustin-logs:298752.2-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

for channel 8316
wclsFrameSaver: no samples within desired window for channel 8394
wclsFrameSaver: no samples within desired window for channel 8397
wclsFrameSaver: no samples within desired window for channel 8402
wclsFrameSaver: no samples within desired window for channel 8410
wclsFrameSaver: no samples within desired window for channel 8443
wclsFrameSaver: no samples within desired window for channel 8455
wclsFrameSaver: no samples within desired window for channel 8484
wclsFrameSaver: no samples within desired window for channel 8499
wclsFrameSaver: no samples within desired window for channel 8547
wclsFrameSaver: no samples within desired window for channel 8568
wclsFrameSaver: no samples within desired window for channel 8588
wclsFrameSaver: no samples within desired window for channel 8600
wclsFrameSaver: no samples within desired window for channel 8602
wclsFrameSaver: no samples within desired window for channel 8602
wclsFrameSaver: no samples within desired window for channel 8603
wclsFrameSaver: no samples within desired window for channel 8603
wclsFrameSaver: no samples within desired window for channel 8603
wclsFrameSaver: no samples within desired window for channel 8617
wclsFrameSaver: no samples within desired window for channel 8665
wclsFrameSaver: no samples within desired window for channel 8689
wclsFrameSaver: no samples within desired window for channel 8699
wclsFrameSaver: no samples within desired window for channel 8717
wclsFrameSaver: no samples within desired window for channel 8768
wclsFrameSaver: no samples within desired window for channel 8768
wclsFrameSaver: no samples within desired window for channel 8771
wclsFrameSaver: no samples within desired window for channel 8773
wclsFrameSaver: no samples within desired window for channel 8780
wclsFrameSaver: no samples within desired window for channel 8826
wclsFrameSaver: no samples within desired window for channel 8847
wclsFrameSaver: no samples within desired window for channel 8852
wclsFrameSaver: no samples within desired window for channel 8861
wclsFrameSaver: no samples within desired window for channel 8883
wclsFrameSaver: no samples within desired window for channel 8950
wclsFrameSaver: no samples within desired window for channel 8951
wclsFrameSaver: no samples within desired window for channel 8986
wclsFrameSaver: no samples within desired window for channel 9023
wclsFrameSaver: no samples within desired window for channel 9030
wclsFrameSaver: no samples within desired window for channel 9030
wclsFrameSaver: no samples within desired window for channel 9031
wclsFrameSaver: no samples within desired window for channel 9031
wclsFrameSaver: no samples within desired window for channel 9031
wclsFrameSaver: no samples within desired window for channel 9036
wclsFrameSaver: no samples within desired window for channel 9037
wclsFrameSaver: no samples within desired window for channel 9062
wclsFrameSaver: no samples within desired window for channel 9062
wclsFrameSaver: no samples within desired window for channel 9063
wclsFrameSaver: no samples within desired window for channel 9063
wclsFrameSaver: no samples within desired window for channel 9063
wclsFrameSaver: no samples within desired window for channel 9064
wclsFrameSaver: no samples within desired window for channel 9064
wclsFrameSaver: no samples within desired window for channel 9064
wclsFrameSaver: no samples within desired window for channel 9074
wclsFrameSaver: no samples within desired window for channel 9103
wclsFrameSaver: no samples within desired window for channel 9167
wclsFrameSaver: no samples within desired window for channel 9180
wclsFrameSaver: no samples within desired window for channel 9181
wclsFrameSaver: no samples within desired window for channel 9182
wclsFrameSaver: no samples within desired window for channel 9192
wclsFrameSaver: no samples within desired window for channel 9200
wclsFrameSaver: no samples within desired window for channel 9267
wclsFrameSaver: no samples within desired window for channel 9421
wclsFrameSaver: no samples within desired window for channel 9608
wclsFrameSaver: no samples within desired window for channel 9618
wclsFrameSaver: no samples within desired window for channel 9619
wclsFrameSaver: no samples within desired window for channel 9619
wclsFrameSaver: no samples within desired window for channel 9632
wclsFrameSaver: no samples within desired window for channel 9655
wclsFrameSaver: no samples within desired window for channel 9656
wclsFrameSaver: no samples within desired window for channel 9675
wclsFrameSaver: no samples within desired window for channel 9676
FrameSaver: q=3.39518e+06 n=1441203 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 5406 XUs and 5230 XVs -> 62 XUVs
C:0 T:1 74696 XUs and 66049 XVs -> 2194 XUVs
C:0 T:2 67796 XUs and 85466 XVs -> 2967 XUVs
C:0 T:3 289 XUs and 226 XVs -> 7 XUVs
C:0 T:4 290 XUs and 173 XVs -> 2 XUVs
C:0 T:5 109744 XUs and 101241 XVs -> 4401 XUVs
C:0 T:6 38600 XUs and 39564 XVs -> 1491 XUVs
C:0 T:7 145 XUs and 224 XVs -> 9 XUVs
11133 XUVs total
4566 collection wire objects
11133 potential space points
Neighbour search...
908799 tests to find 395174 neighbours
Iterating with no regularization...
Begin: 7.00056e+07
0 6.28722e+07
1 6.23039e+07
2 6.22634e+07
Now with regularization...
Begin: 5.86437e+07
0 5.86291e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
21-Jan-2026 19:11:35 PST  Closed output file "np04hd_raw_run027308_0018_dataflow2_datawriter_0_20240619T183931_20260122T031135_keepup.root"

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          357.871       569.457       868.305       562.321       149.917        35     
------------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        8.3879e-05    0.000153783   0.000295887   0.000119469   6.01036e-05      35     
produce:tpcrawdecoder:PDHDTPCReader                 229.83        380.513       731.722       348.203       119.52         35     
produce:triggerrawdecoder:PDHDTriggerReader3       0.133033      0.167114      0.412128      0.135433      0.0862976       35     
produce:timingrawdecoder:PDHDTimingRawDecoder      0.0664619     0.0896641     0.404498      0.0672743     0.0738098       35     
produce:ctbrawdecoder:PDHDCTBRawDecoder             0.33183       0.37022      0.879628      0.337079      0.108103        35     
produce:beamevent:BeamEvent                       0.00709996      1.81742       62.7418      0.0167831      10.4486        35     
produce:fembfilter:PDHDFEMBFilter                 9.4499e-05    0.000160622   0.000635304   0.000133659   8.94568e-05      35     
produce:wclsdatahd:WireCellToolkit                  68.6489       86.8823       104.259       87.0864       7.61107        35     
produce:gaushit:GausHitFinder                      0.558173       1.35067       3.35475       1.25149      0.528354        35     
produce:nhitsfilter:NumberOfHitsFilter            0.000267407   0.000505697   0.00109039    0.000488015   0.000165107      35     
produce:reco3d:SpacePointSolver                    0.514414       2.71769       6.71311       2.31762       1.60093        33     
produce:hitpdune:DisambigFromSpacePoints           0.325483       1.62094       3.91485       1.34251      0.883249        33     
produce:pandora:StandardPandora                     10.6753       89.6993       469.759       63.6978       92.3649        33     
produce:pandoraWriter:StandardPandora              0.140576      0.274616       0.48503      0.264695      0.0687335       33     
produce:pandoraTrack:LArPandoraTrackCreation        1.22379       4.11095       8.81948       3.63708       1.93107        33     
produce:pandoraShower:LArPandoraShowerCreation      1.20954       4.20329       9.40178       4.36732       2.06993        33     
[art]:TriggerResults:TriggerResultInserter        2.9129e-05    6.47225e-05   0.000115209   6.2009e-05    2.33103e-05      35     
end_path:out1:RootOutput                            4.5e-06     1.46639e-05    4.676e-05     1.434e-05     6.604e-06       35     
end_path:out1:RootOutput(write)                    0.694133       1.2661        1.99453       1.31428      0.346738        35     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38269.9 MB
  Peak resident set size usage (VmHWM): 2120.76 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
lsing
total 732M
-rw-r--r-- 1 cuser cuser 529M Jan 21 19:11 np04hd_raw_run027308_0018_dataflow2_datawriter_0_20240619T183931_20260122T031135_keepup.root
-rw-r--r-- 1 cuser cuser 193M Jan 21 19:11 Pandora_Events.pndr
-rw-r--r-- 1 cuser cuser 8.4M Jan 21 19:11 jobscript.log
-rw-r--r-- 1 cuser cuser 268K Jan 21 19:11 mem.db
-rw-r--r-- 1 cuser cuser  52K Jan 21 19:11 time.db
-rw-r--r-- 1 cuser cuser  18K Jan 21 19:11 np04hd_raw_run027308_0018_dataflow2_datawriter_0_20240619T183931.hdf5_calibana.root
-rw-r--r-- 1 cuser cuser 3.5K Jan 21 19:11 Pandora_Geometry.xml
-rw-r--r-- 1 cuser cuser   83 Jan 21 13:37 all-input-dids.txt
-rw-r--r-- 1 cuser cuser    0 Jan 21 13:37 debugprod.log
disk usage
732M	.
justIN time: 2026-02-05 11:17:59 UTC       justIN version: 01.06.00