justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 298724.17@dunegpschedd01.fnal.gov

Jobsub ID298724.17@dunegpschedd01.fnal.gov
Workflow ID12195
Stage ID1
User namemsiden@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2096103424 (1999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-21 19:54:08
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw6
Last heartbeat2026-01-21 21:38:12
From worker nodeHostnamemh-7763-12.t2.ucsd.edu
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-21 19:55:07
Input fileshd-protodune:np04hd_raw_run027306_0003_dataflow0_datawriter_0_20240619T175630.hdf5
JobscriptExit code0
Real time1h (6162s)
CPU time6m (375s = 6%)
Max RSS bytes1856741376 (1770 MiB)
Outputting started2026-01-21 21:37:50
Output files
Finished2026-01-21 21:38:12
Saved logsjustin-logs:298724.17-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

9058
wclsFrameSaver: no samples within desired window for channel 9069
wclsFrameSaver: no samples within desired window for channel 9091
wclsFrameSaver: no samples within desired window for channel 9094
wclsFrameSaver: no samples within desired window for channel 9096
wclsFrameSaver: no samples within desired window for channel 9102
wclsFrameSaver: no samples within desired window for channel 9109
wclsFrameSaver: no samples within desired window for channel 9113
wclsFrameSaver: no samples within desired window for channel 9114
wclsFrameSaver: no samples within desired window for channel 9115
wclsFrameSaver: no samples within desired window for channel 9115
wclsFrameSaver: no samples within desired window for channel 9115
wclsFrameSaver: no samples within desired window for channel 9115
wclsFrameSaver: no samples within desired window for channel 9116
wclsFrameSaver: no samples within desired window for channel 9116
wclsFrameSaver: no samples within desired window for channel 9118
wclsFrameSaver: no samples within desired window for channel 9118
wclsFrameSaver: no samples within desired window for channel 9119
wclsFrameSaver: no samples within desired window for channel 9119
wclsFrameSaver: no samples within desired window for channel 9120
wclsFrameSaver: no samples within desired window for channel 9124
wclsFrameSaver: no samples within desired window for channel 9138
wclsFrameSaver: no samples within desired window for channel 9139
wclsFrameSaver: no samples within desired window for channel 9146
wclsFrameSaver: no samples within desired window for channel 9186
wclsFrameSaver: no samples within desired window for channel 9189
wclsFrameSaver: no samples within desired window for channel 9237
wclsFrameSaver: no samples within desired window for channel 9254
wclsFrameSaver: no samples within desired window for channel 9262
wclsFrameSaver: no samples within desired window for channel 9264
wclsFrameSaver: no samples within desired window for channel 9264
wclsFrameSaver: no samples within desired window for channel 9337
wclsFrameSaver: no samples within desired window for channel 9357
wclsFrameSaver: no samples within desired window for channel 9357
wclsFrameSaver: no samples within desired window for channel 9358
wclsFrameSaver: no samples within desired window for channel 9358
wclsFrameSaver: no samples within desired window for channel 9359
wclsFrameSaver: no samples within desired window for channel 9360
wclsFrameSaver: no samples within desired window for channel 9360
wclsFrameSaver: no samples within desired window for channel 9360
wclsFrameSaver: no samples within desired window for channel 9361
wclsFrameSaver: no samples within desired window for channel 9361
wclsFrameSaver: no samples within desired window for channel 9361
wclsFrameSaver: no samples within desired window for channel 9362
wclsFrameSaver: no samples within desired window for channel 9362
wclsFrameSaver: no samples within desired window for channel 9362
wclsFrameSaver: no samples within desired window for channel 9362
wclsFrameSaver: no samples within desired window for channel 9362
wclsFrameSaver: no samples within desired window for channel 9363
wclsFrameSaver: no samples within desired window for channel 9363
wclsFrameSaver: no samples within desired window for channel 9363
wclsFrameSaver: no samples within desired window for channel 9363
wclsFrameSaver: no samples within desired window for channel 9364
wclsFrameSaver: no samples within desired window for channel 9364
wclsFrameSaver: no samples within desired window for channel 9364
wclsFrameSaver: no samples within desired window for channel 9364
wclsFrameSaver: no samples within desired window for channel 9365
wclsFrameSaver: no samples within desired window for channel 9365
wclsFrameSaver: no samples within desired window for channel 9365
wclsFrameSaver: no samples within desired window for channel 9366
wclsFrameSaver: no samples within desired window for channel 9366
wclsFrameSaver: no samples within desired window for channel 9367
wclsFrameSaver: no samples within desired window for channel 9416
wclsFrameSaver: no samples within desired window for channel 9430
wclsFrameSaver: no samples within desired window for channel 9431
wclsFrameSaver: no samples within desired window for channel 9632
wclsFrameSaver: no samples within desired window for channel 9633
wclsFrameSaver: no samples within desired window for channel 9633
wclsFrameSaver: no samples within desired window for channel 9638
wclsFrameSaver: no samples within desired window for channel 9658
wclsFrameSaver: no samples within desired window for channel 9739
wclsFrameSaver: no samples within desired window for channel 9765
FrameSaver: q=2.1939e+06 n=883221 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 5490 XUs and 5592 XVs -> 49 XUVs
C:0 T:1 57414 XUs and 74254 XVs -> 1846 XUVs
C:0 T:2 6547 XUs and 5340 XVs -> 392 XUVs
C:0 T:3 23 XUs and 41 XVs -> 0 XUVs
C:0 T:4 301 XUs and 567 XVs -> 59 XUVs
C:0 T:5 11951 XUs and 13430 XVs -> 479 XUVs
C:0 T:6 6507 XUs and 6369 XVs -> 1012 XUVs
C:0 T:7 164 XUs and 143 XVs -> 2 XUVs
3839 XUVs total
1691 collection wire objects
3839 potential space points
Neighbour search...
77619 tests to find 44878 neighbours
Iterating with no regularization...
Begin: 3.87156e+07
0 3.61346e+07
1 3.59697e+07
2 3.59519e+07
Now with regularization...
Begin: 3.41188e+07
0 3.41138e+07
98 104 108
0.0025 0.0085 0.01105
21-Jan-2026 13:37:46 PST  Closed output file "np04hd_raw_run027306_0003_dataflow0_datawriter_0_20240619T175630_20260121T213746_keepup.root"

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          141.372       303.605       499.652       304.791       92.7324        20     
------------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        6.9819e-05    0.000111887   0.000279127   0.000109624   4.66003e-05      20     
produce:tpcrawdecoder:PDHDTPCReader                  140.8        285.299       470.505       288.983       91.025         20     
produce:triggerrawdecoder:PDHDTriggerReader3       0.132932      0.139764      0.187962      0.134932      0.0143217       20     
produce:timingrawdecoder:PDHDTimingRawDecoder      0.0664995     0.069613      0.0939352     0.0671847    0.00712951       20     
produce:ctbrawdecoder:PDHDCTBRawDecoder             0.33215      0.347245      0.461299      0.334207      0.0338747       20     
produce:beamevent:BeamEvent                       0.00280113      4.82109       96.2857     0.00801426      20.9834        20     
produce:fembfilter:PDHDFEMBFilter                 3.6759e-05    0.00010138    0.000649133   4.7835e-05    0.000146404      20     
[art]:TriggerResults:TriggerResultInserter         1.613e-05    2.78096e-05    7.336e-05    2.3565e-05    1.30096e-05      20     
end_path:out1:RootOutput                           3.51e-06     7.0784e-06    3.0559e-05     5.705e-06    5.64232e-06      20     
end_path:out1:RootOutput(write)                   0.00761186     0.109242       1.00506      0.0132554     0.289746        20     
produce:wclsdatahd:WireCellToolkit                  79.0655       83.9495       88.8334       83.9495       4.88396         2     
produce:gaushit:GausHitFinder                      0.507898       1.01437       1.52084       1.01437      0.506471         2     
produce:nhitsfilter:NumberOfHitsFilter            0.000203928    0.0004998    0.000795672    0.0004998    0.000295872       2     
produce:reco3d:SpacePointSolver                    0.458511       0.88992       1.32133       0.88992      0.431409         2     
produce:hitpdune:DisambigFromSpacePoints           0.318686      0.878503       1.43832      0.878503      0.559817         2     
produce:pandora:StandardPandora                     19.5004       36.1139       52.7274       36.1139       16.6135         2     
produce:pandoraWriter:StandardPandora              0.144442      0.214982      0.285523      0.214982      0.0705407        2     
produce:pandoraTrack:LArPandoraTrackCreation        1.33966       2.45685       3.57405       2.45685       1.1172          2     
produce:pandoraShower:LArPandoraShowerCreation      1.0498        2.55762       4.06543       2.55762       1.50782         2     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38027.3 MB
  Peak resident set size usage (VmHWM): 1856.74 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
lsing
total 38M
-rw-r--r--. 1 cuser cuser  27M Jan 21 13:37 np04hd_raw_run027306_0003_dataflow0_datawriter_0_20240619T175630_20260121T213746_keepup.root
-rw-r--r--. 1 cuser cuser 9.7M Jan 21 13:35 Pandora_Events.pndr
-rw-r--r--. 1 cuser cuser 783K Jan 21 13:37 jobscript.log
-rw-r--r--. 1 cuser cuser 120K Jan 21 13:37 mem.db
-rw-r--r--. 1 cuser cuser  28K Jan 21 13:37 time.db
-rw-r--r--. 1 cuser cuser  13K Jan 21 13:37 np04hd_raw_run027306_0003_dataflow0_datawriter_0_20240619T175630.hdf5_calibana.root
-rw-r--r--. 1 cuser cuser 3.5K Jan 21 13:37 Pandora_Geometry.xml
-rw-r--r--. 1 cuser cuser   83 Jan 21 11:55 all-input-dids.txt
-rw-r--r--. 1 cuser cuser    0 Jan 21 11:55 debugprod.log
disk usage
38M	.
justIN time: 2026-02-05 11:22:05 UTC       justIN version: 01.06.00