justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 298724.6@dunegpschedd01.fnal.gov

Jobsub ID298724.6@dunegpschedd01.fnal.gov
Workflow ID12195
Stage ID1
User namemsiden@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2096103424 (1999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-21 19:54:08
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw6
Last heartbeat2026-01-21 21:25:37
From worker nodeHostnamemh-7763-12.t2.ucsd.edu
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-21 19:55:14
Input fileshd-protodune:np04hd_raw_run027306_0003_dataflow2_datawriter_0_20240619T175631.hdf5
JobscriptExit code0
Real time1h (5399s)
CPU time7m (457s = 8%)
Max RSS bytes1855279104 (1769 MiB)
Outputting started2026-01-21 21:25:13
Output files
Finished2026-01-21 21:25:37
Saved logsjustin-logs:298724.6-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

or channel 8700
wclsFrameSaver: no samples within desired window for channel 8701
wclsFrameSaver: no samples within desired window for channel 8713
wclsFrameSaver: no samples within desired window for channel 8723
wclsFrameSaver: no samples within desired window for channel 8774
wclsFrameSaver: no samples within desired window for channel 8787
wclsFrameSaver: no samples within desired window for channel 8866
wclsFrameSaver: no samples within desired window for channel 8874
wclsFrameSaver: no samples within desired window for channel 8891
wclsFrameSaver: no samples within desired window for channel 8892
wclsFrameSaver: no samples within desired window for channel 8893
wclsFrameSaver: no samples within desired window for channel 8894
wclsFrameSaver: no samples within desired window for channel 8894
wclsFrameSaver: no samples within desired window for channel 8895
wclsFrameSaver: no samples within desired window for channel 8895
wclsFrameSaver: no samples within desired window for channel 8896
wclsFrameSaver: no samples within desired window for channel 8896
wclsFrameSaver: no samples within desired window for channel 8896
wclsFrameSaver: no samples within desired window for channel 8897
wclsFrameSaver: no samples within desired window for channel 8898
wclsFrameSaver: no samples within desired window for channel 8898
wclsFrameSaver: no samples within desired window for channel 8898
wclsFrameSaver: no samples within desired window for channel 8899
wclsFrameSaver: no samples within desired window for channel 8899
wclsFrameSaver: no samples within desired window for channel 8899
wclsFrameSaver: no samples within desired window for channel 8900
wclsFrameSaver: no samples within desired window for channel 8900
wclsFrameSaver: no samples within desired window for channel 8901
wclsFrameSaver: no samples within desired window for channel 8901
wclsFrameSaver: no samples within desired window for channel 8902
wclsFrameSaver: no samples within desired window for channel 8903
wclsFrameSaver: no samples within desired window for channel 8903
wclsFrameSaver: no samples within desired window for channel 8903
wclsFrameSaver: no samples within desired window for channel 8904
wclsFrameSaver: no samples within desired window for channel 8904
wclsFrameSaver: no samples within desired window for channel 8904
wclsFrameSaver: no samples within desired window for channel 8905
wclsFrameSaver: no samples within desired window for channel 8907
wclsFrameSaver: no samples within desired window for channel 8929
wclsFrameSaver: no samples within desired window for channel 8937
wclsFrameSaver: no samples within desired window for channel 8978
wclsFrameSaver: no samples within desired window for channel 8984
wclsFrameSaver: no samples within desired window for channel 8988
wclsFrameSaver: no samples within desired window for channel 9004
wclsFrameSaver: no samples within desired window for channel 9013
wclsFrameSaver: no samples within desired window for channel 9019
wclsFrameSaver: no samples within desired window for channel 9019
wclsFrameSaver: no samples within desired window for channel 9108
wclsFrameSaver: no samples within desired window for channel 9156
wclsFrameSaver: no samples within desired window for channel 9167
wclsFrameSaver: no samples within desired window for channel 9175
wclsFrameSaver: no samples within desired window for channel 9177
wclsFrameSaver: no samples within desired window for channel 9196
wclsFrameSaver: no samples within desired window for channel 9204
wclsFrameSaver: no samples within desired window for channel 9225
wclsFrameSaver: no samples within desired window for channel 9243
wclsFrameSaver: no samples within desired window for channel 9250
wclsFrameSaver: no samples within desired window for channel 9260
wclsFrameSaver: no samples within desired window for channel 9390
wclsFrameSaver: no samples within desired window for channel 9390
wclsFrameSaver: no samples within desired window for channel 9391
wclsFrameSaver: no samples within desired window for channel 9392
wclsFrameSaver: no samples within desired window for channel 9393
wclsFrameSaver: no samples within desired window for channel 9393
wclsFrameSaver: no samples within desired window for channel 9394
wclsFrameSaver: no samples within desired window for channel 9395
wclsFrameSaver: no samples within desired window for channel 9395
wclsFrameSaver: no samples within desired window for channel 9396
wclsFrameSaver: no samples within desired window for channel 9397
wclsFrameSaver: no samples within desired window for channel 9397
FrameSaver: q=2.69189e+06 n=1228225 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 8091 XUs and 6757 XVs -> 206 XUVs
C:0 T:1 145095 XUs and 110785 XVs -> 4404 XUVs
C:0 T:2 11632 XUs and 6154 XVs -> 310 XUVs
C:0 T:3 21 XUs and 36 XVs -> 1 XUVs
C:0 T:4 807 XUs and 801 XVs -> 19 XUVs
C:0 T:5 36986 XUs and 32911 XVs -> 950 XUVs
C:0 T:6 19171 XUs and 10696 XVs -> 249 XUVs
C:0 T:7 400 XUs and 455 XVs -> 44 XUVs
6183 XUVs total
3121 collection wire objects
6183 potential space points
Neighbour search...
163915 tests to find 84028 neighbours
Iterating with no regularization...
Begin: 4.81797e+07
0 4.56025e+07
1 4.54396e+07
2 4.54259e+07
Now with regularization...
Begin: 4.33071e+07
0 4.33026e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
74 80 82
-0.0215 -0.0155 -0.01495
92 96 98
-0.0035 0.0005 0.00105
21-Jan-2026 13:25:07 PST  Closed output file "np04hd_raw_run027306_0003_dataflow2_datawriter_0_20240619T175631_20260121T212507_keepup.root"

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          192.451       331.661       470.559       352.681       77.5299        16     
------------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                         7.149e-05    0.000122789   0.000242428   0.000117839   3.73245e-05      16     
produce:tpcrawdecoder:PDHDTPCReader                 191.868       302.782       371.885       308.286       62.9514        16     
produce:triggerrawdecoder:PDHDTriggerReader3       0.132803      0.159106      0.411307      0.134131      0.0697466       16     
produce:timingrawdecoder:PDHDTimingRawDecoder      0.0665912     0.0710677     0.118396      0.0669457     0.0125549       16     
produce:ctbrawdecoder:PDHDCTBRawDecoder             0.33258      0.377143      0.671532       0.33537       0.10074        16     
produce:beamevent:BeamEvent                       0.00321135      6.12196       97.8362     0.00781426      23.6805        16     
produce:fembfilter:PDHDFEMBFilter                 4.5119e-05    0.000109414   0.000529795   5.5929e-05    0.000125406      16     
[art]:TriggerResults:TriggerResultInserter         1.967e-05    3.23829e-05    7.305e-05    3.07595e-05   1.18806e-05      16     
end_path:out1:RootOutput                           3.22e-06     5.87737e-06   2.2459e-05    4.6495e-06    4.41923e-06      16     
end_path:out1:RootOutput(write)                    0.0079372       0.165       0.968613      0.0146033     0.317821        16     
produce:wclsdatahd:WireCellToolkit                  74.5332       76.8944       78.2114       77.9385       1.67331         3     
produce:gaushit:GausHitFinder                      0.566965      0.714195      0.897112      0.678506      0.137124         3     
produce:nhitsfilter:NumberOfHitsFilter            0.000259128   0.000368186   0.000447465   0.000397966   7.97196e-05       3     
produce:reco3d:SpacePointSolver                    0.572442      0.990838       1.30952       1.09056       0.30906         3     
produce:hitpdune:DisambigFromSpacePoints           0.559724      0.894911       1.17435      0.950657      0.253999         3     
produce:pandora:StandardPandora                     14.8097       33.6575       52.818        33.3448       15.5184         3     
produce:pandoraWriter:StandardPandora              0.149446       0.18832      0.224556      0.190958      0.0307201        3     
produce:pandoraTrack:LArPandoraTrackCreation       0.866569       1.76796       2.50552       1.9318       0.679055         3     
produce:pandoraShower:LArPandoraShowerCreation      1.17761       2.03518       2.53117       2.39675      0.608869         3     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 37987.1 MB
  Peak resident set size usage (VmHWM): 1855.28 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
lsing
total 55M
-rw-r--r--. 1 cuser cuser  39M Jan 21 13:25 np04hd_raw_run027306_0003_dataflow2_datawriter_0_20240619T175631_20260121T212507_keepup.root
-rw-r--r--. 1 cuser cuser  15M Jan 21 13:18 Pandora_Events.pndr
-rw-r--r--. 1 cuser cuser 756K Jan 21 13:25 jobscript.log
-rw-r--r--. 1 cuser cuser 108K Jan 21 13:25 mem.db
-rw-r--r--. 1 cuser cuser  28K Jan 21 13:25 time.db
-rw-r--r--. 1 cuser cuser  12K Jan 21 13:25 np04hd_raw_run027306_0003_dataflow2_datawriter_0_20240619T175631.hdf5_calibana.root
-rw-r--r--. 1 cuser cuser 3.5K Jan 21 13:25 Pandora_Geometry.xml
-rw-r--r--. 1 cuser cuser   83 Jan 21 11:55 all-input-dids.txt
-rw-r--r--. 1 cuser cuser    0 Jan 21 11:55 debugprod.log
disk usage
55M	.
justIN time: 2026-02-05 11:23:42 UTC       justIN version: 01.06.00