justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 298724.16@dunegpschedd01.fnal.gov

Jobsub ID298724.16@dunegpschedd01.fnal.gov
Workflow ID12195
Stage ID1
User namemsiden@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2096103424 (1999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-21 19:54:08
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw6
Last heartbeat2026-01-21 23:27:43
From worker nodeHostnamemh-7763-18.t2.ucsd.edu
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-21 19:55:29
Input fileshd-protodune:np04hd_raw_run027306_0001_dataflow1_datawriter_0_20240619T175445.hdf5
JobscriptExit code0
Real time3h (12710s)
CPU time32m (1968s = 15%)
Max RSS bytes1992970240 (1900 MiB)
Outputting started2026-01-21 23:27:20
Output files
Finished2026-01-21 23:27:43
Saved logsjustin-logs:298724.16-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

er: no samples within desired window for channel 9341
wclsFrameSaver: no samples within desired window for channel 9341
wclsFrameSaver: no samples within desired window for channel 9342
wclsFrameSaver: no samples within desired window for channel 9342
wclsFrameSaver: no samples within desired window for channel 9343
wclsFrameSaver: no samples within desired window for channel 9344
wclsFrameSaver: no samples within desired window for channel 9344
wclsFrameSaver: no samples within desired window for channel 9345
wclsFrameSaver: no samples within desired window for channel 9346
wclsFrameSaver: no samples within desired window for channel 9347
wclsFrameSaver: no samples within desired window for channel 9347
wclsFrameSaver: no samples within desired window for channel 9348
wclsFrameSaver: no samples within desired window for channel 9348
wclsFrameSaver: no samples within desired window for channel 9349
wclsFrameSaver: no samples within desired window for channel 9350
wclsFrameSaver: no samples within desired window for channel 9351
wclsFrameSaver: no samples within desired window for channel 9352
wclsFrameSaver: no samples within desired window for channel 9352
wclsFrameSaver: no samples within desired window for channel 9353
wclsFrameSaver: no samples within desired window for channel 9353
wclsFrameSaver: no samples within desired window for channel 9354
wclsFrameSaver: no samples within desired window for channel 9354
wclsFrameSaver: no samples within desired window for channel 9355
wclsFrameSaver: no samples within desired window for channel 9356
wclsFrameSaver: no samples within desired window for channel 9357
wclsFrameSaver: no samples within desired window for channel 9357
wclsFrameSaver: no samples within desired window for channel 9358
wclsFrameSaver: no samples within desired window for channel 9359
wclsFrameSaver: no samples within desired window for channel 9360
wclsFrameSaver: no samples within desired window for channel 9360
wclsFrameSaver: no samples within desired window for channel 9361
wclsFrameSaver: no samples within desired window for channel 9362
wclsFrameSaver: no samples within desired window for channel 9363
wclsFrameSaver: no samples within desired window for channel 9363
wclsFrameSaver: no samples within desired window for channel 9364
wclsFrameSaver: no samples within desired window for channel 9535
wclsFrameSaver: no samples within desired window for channel 9713
wclsFrameSaver: no samples within desired window for channel 10237
FrameSaver: q=2.61116e+06 n=1169613 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 7384 XUs and 6360 XVs -> 222 XUVs
C:0 T:1 109003 XUs and 107850 XVs -> 4997 XUVs
C:0 T:2 29484 XUs and 36049 XVs -> 585 XUVs
C:0 T:3 866 XUs and 839 XVs -> 39 XUVs
C:0 T:4 64 XUs and 32 XVs -> 0 XUVs
C:0 T:5 31496 XUs and 29090 XVs -> 437 XUVs
C:0 T:6 35802 XUs and 39234 XVs -> 679 XUVs
C:0 T:7 163 XUs and 119 XVs -> 3 XUVs
6962 XUVs total
2889 collection wire objects
6962 potential space points
Neighbour search...
185382 tests to find 90644 neighbours
Iterating with no regularization...
Begin: 3.57215e+07
0 3.28507e+07
1 3.26877e+07
2 3.26715e+07
Now with regularization...
Begin: 3.15083e+07
0 3.15042e+07
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.000343382 0.000343353 0.000343382 0.000343353 0.000343382 0.000343382 0.000343353 0.000343382 0.000343382 0.000343353 0.000343382 
----view 1: 0.0260731 0.0256488 0.0339708 0.0264502 0.0334994 0.0337627 0.0299417 0.0264176 0.027823 0.0262809 0.0247842 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.000343353 0.000343382 0.000343353 0.000343382 0.000343382 0.000343353 0.000343382 0.000343382 0.000343353 0.000343382 0.000343353 
----view 1: 0.0256488 0.0339708 0.0264502 0.0334994 0.0337627 0.0299417 0.0264176 0.027823 0.0262809 0.0247842 0.0317961 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.000343382 0.000343353 0.000343382 0.000343382 0.000343353 0.000343382 0.000343382 0.000343353 0.000343382 0.000343353 0.000343382 
----view 1: 0.0339708 0.0264502 0.0334994 0.0337627 0.0299417 0.0264176 0.027823 0.0262809 0.0247842 0.0317961 0.0341287 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.000343353 0.000343382 0.000343382 0.000343353 0.000343382 0.000343382 0.000343353 0.000343382 0.000343353 0.000343382 0.000343382 
----view 1: 0.0264502 0.0334994 0.0337627 0.0299417 0.0264176 0.027823 0.0262809 0.0247842 0.0317961 0.0341287 0.0442816 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.000343382 0.000343382 0.000343353 0.000343382 0.000343382 0.000343353 0.000343382 0.000343353 0.000343382 0.000343382 0.000343353 
----view 1: 0.0334994 0.0337627 0.0299417 0.0264176 0.027823 0.0262809 0.0247842 0.0317961 0.0341287 0.0442816 0.0278648 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.000343382 0.000343353 0.000343382 0.000343382 0.000343353 0.000343382 0.000343353 0.000343382 0.000343382 0.000343353 0.000343382 
----view 1: 0.0337627 0.0299417 0.0264176 0.027823 0.0262809 0.0247842 0.0317961 0.0341287 0.0442816 0.0278648 0.025852 
21-Jan-2026 15:27:14 PST  Closed output file "np04hd_raw_run027306_0001_dataflow1_datawriter_0_20240619T175445_20260121T232714_keepup.root"

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          205.153       350.028       602.569       309.845       103.668        36     
------------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        8.5659e-05    0.000111755   0.000204118   9.8304e-05    2.98156e-05      36     
produce:tpcrawdecoder:PDHDTPCReader                 204.577       296.456       445.415       289.624       53.6168        36     
produce:triggerrawdecoder:PDHDTriggerReader3       0.132357      0.142011      0.404307      0.133812      0.0444509       36     
produce:timingrawdecoder:PDHDTimingRawDecoder      0.0664118     0.0763837     0.399589      0.0669478     0.0546362       36     
produce:ctbrawdecoder:PDHDCTBRawDecoder            0.330496      0.357255      0.878067       0.33414      0.0985022       36     
produce:beamevent:BeamEvent                       0.000775522     2.9047        104.034     0.00269461      17.094         36     
produce:fembfilter:PDHDFEMBFilter                  3.817e-05    0.000100075   0.000661523   7.0199e-05    0.000101511      36     
produce:wclsdatahd:WireCellToolkit                  60.1805       76.9651       94.8259       73.1399       10.4435        15     
produce:gaushit:GausHitFinder                       0.44944      0.848051       1.47088      0.766463       0.24589        15     
produce:nhitsfilter:NumberOfHitsFilter            0.000202448   0.00032603    0.000581925   0.000292988   9.70726e-05      15     
produce:reco3d:SpacePointSolver                    0.455953       1.5686        5.4339       0.951659       1.30175        15     
produce:hitpdune:DisambigFromSpacePoints            0.3524       0.777146       1.34565      0.716992      0.286244        15     
produce:pandora:StandardPandora                     9.65015       34.5489       85.2466       24.5104       20.2238        15     
produce:pandoraWriter:StandardPandora              0.114985      0.187523      0.306238      0.153217      0.0608202       15     
produce:pandoraTrack:LArPandoraTrackCreation        1.09453       2.03268       5.09966       1.83886      0.995882        15     
produce:pandoraShower:LArPandoraShowerCreation     0.795608       2.10946       4.00292       2.13067      0.852983        15     
[art]:TriggerResults:TriggerResultInserter         2.466e-05    4.61518e-05   9.9069e-05    4.1885e-05    1.94509e-05      36     
end_path:out1:RootOutput                           4.44e-06     8.12606e-06    3.71e-05      6.37e-06     5.54527e-06      36     
end_path:out1:RootOutput(write)                    0.0109486     0.415168       1.42164      0.0149128     0.500334        36     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38173.1 MB
  Peak resident set size usage (VmHWM): 1992.97 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
lsing
total 262M
-rw-r--r-- 1 cuser cuser 188M Jan 21 15:27 np04hd_raw_run027306_0001_dataflow1_datawriter_0_20240619T175445_20260121T232714_keepup.root
-rw-r--r-- 1 cuser cuser  71M Jan 21 15:27 Pandora_Events.pndr
-rw-r--r-- 1 cuser cuser 2.8M Jan 21 15:27 jobscript.log
-rw-r--r-- 1 cuser cuser 212K Jan 21 15:27 mem.db
-rw-r--r-- 1 cuser cuser  44K Jan 21 15:27 time.db
-rw-r--r-- 1 cuser cuser  18K Jan 21 15:27 np04hd_raw_run027306_0001_dataflow1_datawriter_0_20240619T175445.hdf5_calibana.root
-rw-r--r-- 1 cuser cuser 3.5K Jan 21 15:27 Pandora_Geometry.xml
-rw-r--r-- 1 cuser cuser   83 Jan 21 11:55 all-input-dids.txt
-rw-r--r-- 1 cuser cuser    0 Jan 21 11:55 debugprod.log
disk usage
262M	.
justIN time: 2026-02-05 11:22:05 UTC       justIN version: 01.06.00