justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 298724.5@dunegpschedd01.fnal.gov

Jobsub ID298724.5@dunegpschedd01.fnal.gov
Workflow ID12195
Stage ID1
User namemsiden@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2096103424 (1999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-21 19:54:08
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw6
Last heartbeat2026-01-22 00:10:54
From worker nodeHostnamemh-7763-18.t2.ucsd.edu
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-21 19:55:19
Input fileshd-protodune:np04hd_raw_run027306_0002_dataflow3_datawriter_0_20240619T175558.hdf5
JobscriptExit code0
Real time4h (15308s)
CPU time1h (4311s = 28%)
Max RSS bytes2046541824 (1951 MiB)
Outputting started2026-01-22 00:10:28
Output files
Finished2026-01-22 00:10:54
Saved logsjustin-logs:298724.5-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

es within desired window for channel 8782
wclsFrameSaver: no samples within desired window for channel 8789
wclsFrameSaver: no samples within desired window for channel 8790
wclsFrameSaver: no samples within desired window for channel 8790
wclsFrameSaver: no samples within desired window for channel 8791
wclsFrameSaver: no samples within desired window for channel 8791
wclsFrameSaver: no samples within desired window for channel 8791
wclsFrameSaver: no samples within desired window for channel 8803
wclsFrameSaver: no samples within desired window for channel 8810
wclsFrameSaver: no samples within desired window for channel 8859
wclsFrameSaver: no samples within desired window for channel 8861
wclsFrameSaver: no samples within desired window for channel 8882
wclsFrameSaver: no samples within desired window for channel 8985
wclsFrameSaver: no samples within desired window for channel 9054
wclsFrameSaver: no samples within desired window for channel 9065
wclsFrameSaver: no samples within desired window for channel 9069
wclsFrameSaver: no samples within desired window for channel 9090
wclsFrameSaver: no samples within desired window for channel 9118
wclsFrameSaver: no samples within desired window for channel 9136
wclsFrameSaver: no samples within desired window for channel 9147
wclsFrameSaver: no samples within desired window for channel 9153
wclsFrameSaver: no samples within desired window for channel 9155
wclsFrameSaver: no samples within desired window for channel 9157
wclsFrameSaver: no samples within desired window for channel 9159
wclsFrameSaver: no samples within desired window for channel 9159
wclsFrameSaver: no samples within desired window for channel 9160
wclsFrameSaver: no samples within desired window for channel 9160
wclsFrameSaver: no samples within desired window for channel 9172
wclsFrameSaver: no samples within desired window for channel 9174
wclsFrameSaver: no samples within desired window for channel 9182
wclsFrameSaver: no samples within desired window for channel 9237
wclsFrameSaver: no samples within desired window for channel 9270
wclsFrameSaver: no samples within desired window for channel 9275
wclsFrameSaver: no samples within desired window for channel 9297
wclsFrameSaver: no samples within desired window for channel 9298
wclsFrameSaver: no samples within desired window for channel 9298
wclsFrameSaver: no samples within desired window for channel 9299
wclsFrameSaver: no samples within desired window for channel 9300
wclsFrameSaver: no samples within desired window for channel 9300
wclsFrameSaver: no samples within desired window for channel 9301
wclsFrameSaver: no samples within desired window for channel 9301
wclsFrameSaver: no samples within desired window for channel 9302
wclsFrameSaver: no samples within desired window for channel 9303
wclsFrameSaver: no samples within desired window for channel 9303
wclsFrameSaver: no samples within desired window for channel 9304
wclsFrameSaver: no samples within desired window for channel 9304
wclsFrameSaver: no samples within desired window for channel 9305
wclsFrameSaver: no samples within desired window for channel 9305
wclsFrameSaver: no samples within desired window for channel 9306
wclsFrameSaver: no samples within desired window for channel 9307
wclsFrameSaver: no samples within desired window for channel 9308
wclsFrameSaver: no samples within desired window for channel 9309
wclsFrameSaver: no samples within desired window for channel 9310
wclsFrameSaver: no samples within desired window for channel 9311
wclsFrameSaver: no samples within desired window for channel 9311
wclsFrameSaver: no samples within desired window for channel 9312
wclsFrameSaver: no samples within desired window for channel 9313
wclsFrameSaver: no samples within desired window for channel 9313
wclsFrameSaver: no samples within desired window for channel 9314
wclsFrameSaver: no samples within desired window for channel 9315
wclsFrameSaver: no samples within desired window for channel 9333
wclsFrameSaver: no samples within desired window for channel 9341
wclsFrameSaver: no samples within desired window for channel 9544
wclsFrameSaver: no samples within desired window for channel 9571
wclsFrameSaver: no samples within desired window for channel 9586
wclsFrameSaver: no samples within desired window for channel 9588
wclsFrameSaver: no samples within desired window for channel 9588
wclsFrameSaver: no samples within desired window for channel 9688
wclsFrameSaver: no samples within desired window for channel 9751
FrameSaver: q=3.19997e+06 n=1357136 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 8639 XUs and 6333 XVs -> 104 XUVs
C:0 T:1 88831 XUs and 73057 XVs -> 3213 XUVs
C:0 T:2 19626 XUs and 14944 XVs -> 243 XUVs
C:0 T:3 709 XUs and 578 XVs -> 8 XUVs
C:0 T:4 1319 XUs and 1801 XVs -> 67 XUVs
C:0 T:5 125904 XUs and 114325 XVs -> 4927 XUVs
C:0 T:6 18702 XUs and 17704 XVs -> 579 XUVs
C:0 T:7 63 XUs and 101 XVs -> 0 XUVs
9141 XUVs total
3392 collection wire objects
9141 potential space points
Neighbour search...
328025 tests to find 192650 neighbours
Iterating with no regularization...
Begin: 7.57098e+07
0 7.00323e+07
1 6.96834e+07
2 6.9639e+07
Now with regularization...
Begin: 6.61278e+07
0 6.61092e+07
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
99 113 117
0.0035 0.0175 0.02005
109 99 99
0.014 0.0035 0.00205
21-Jan-2026 16:10:19 PST  Closed output file "np04hd_raw_run027306_0002_dataflow3_datawriter_0_20240619T175558_20260122T001019_keepup.root"

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          267.727       434.128       661.754       446.575       90.1127        35     
------------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        7.3929e-05    0.00014218    0.000280627   0.000125949   5.13394e-05      35     
produce:tpcrawdecoder:PDHDTPCReader                 226.64        312.75        423.594       313.531       48.5753        35     
produce:triggerrawdecoder:PDHDTriggerReader3       0.132884      0.136755      0.206063      0.134716      0.0119454       35     
produce:timingrawdecoder:PDHDTimingRawDecoder      0.0664386     0.0878094     0.411089      0.0672128     0.0791946       35     
produce:ctbrawdecoder:PDHDCTBRawDecoder            0.331935      0.376737      0.610363      0.336273      0.0912276       35     
produce:beamevent:BeamEvent                       0.00362921      2.50733       87.2559     0.00910578      14.5343        35     
produce:fembfilter:PDHDFEMBFilter                  5.528e-05    0.000154706   0.000632304   0.000130899   0.00011082       35     
produce:wclsdatahd:WireCellToolkit                  61.5674       80.6031       97.9722       81.5796       9.25516        27     
produce:gaushit:GausHitFinder                      0.475559       1.15958       1.89162       1.1594       0.292165        27     
produce:nhitsfilter:NumberOfHitsFilter            0.000319947   0.000473926   0.000652704   0.000461005   8.58856e-05      27     
produce:reco3d:SpacePointSolver                    0.801035       2.46506       7.05592       2.10081       1.51219        26     
produce:hitpdune:DisambigFromSpacePoints            0.46628       1.35794       2.71247       1.16575      0.638097        26     
produce:pandora:StandardPandora                     19.0604       62.0494       184.877       50.4801       41.5517        26     
produce:pandoraWriter:StandardPandora              0.133721      0.247119      0.370957      0.238297      0.0709613       26     
produce:pandoraTrack:LArPandoraTrackCreation       0.957043       3.27482       7.60365       2.87676        1.542         26     
produce:pandoraShower:LArPandoraShowerCreation     0.745321       3.51582       7.31196       3.00559       1.6582         26     
[art]:TriggerResults:TriggerResultInserter        2.2249e-05    5.50034e-05   0.000106119    5.318e-05    2.25159e-05      35     
end_path:out1:RootOutput                           4.54e-06     1.16919e-05    7.061e-05     7.69e-06     1.17833e-05      35     
end_path:out1:RootOutput(write)                    0.0118464     0.869924       1.74028      0.892362      0.536553        35     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38216.9 MB
  Peak resident set size usage (VmHWM): 2046.54 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
lsing
total 532M
-rw-r--r-- 1 cuser cuser 382M Jan 21 16:10 np04hd_raw_run027306_0002_dataflow3_datawriter_0_20240619T175558_20260122T001019_keepup.root
-rw-r--r-- 1 cuser cuser 143M Jan 21 15:59 Pandora_Events.pndr
-rw-r--r-- 1 cuser cuser 6.2M Jan 21 16:10 jobscript.log
-rw-r--r-- 1 cuser cuser 244K Jan 21 16:10 mem.db
-rw-r--r-- 1 cuser cuser  48K Jan 21 16:10 time.db
-rw-r--r-- 1 cuser cuser  18K Jan 21 16:10 np04hd_raw_run027306_0002_dataflow3_datawriter_0_20240619T175558.hdf5_calibana.root
-rw-r--r-- 1 cuser cuser 3.5K Jan 21 16:10 Pandora_Geometry.xml
-rw-r--r-- 1 cuser cuser   83 Jan 21 11:55 all-input-dids.txt
-rw-r--r-- 1 cuser cuser    0 Jan 21 11:55 debugprod.log
disk usage
532M	.
justIN time: 2026-02-05 11:23:38 UTC       justIN version: 01.06.00