justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 295528.5@dunegpschedd02.fnal.gov

Jobsub ID295528.5@dunegpschedd02.fnal.gov
Workflow ID12398
Stage ID1
User namemsiden@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2096103424 (1999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-26 11:58:05
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2026-01-26 12:05:24
From worker nodeHostnamedunegli-8436038-0-fnpc17129.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-26 11:59:26
Input fileshd-protodune:np04hd_raw_run027308_0000_dataflow1_datawriter_0_20240619T182030.hdf5
JobscriptExit code0
Real time5m (345s)
CPU time5m (315s = 91%)
Max RSS bytes1605136384 (1530 MiB)
Outputting started2026-01-26 12:05:12
Output files
Finished2026-01-26 12:05:24
Saved logsjustin-logs:295528.5-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

sired window for channel 8563
wclsFrameSaver: no samples within desired window for channel 8567
wclsFrameSaver: no samples within desired window for channel 8577
wclsFrameSaver: no samples within desired window for channel 8577
wclsFrameSaver: no samples within desired window for channel 8580
wclsFrameSaver: no samples within desired window for channel 8587
wclsFrameSaver: no samples within desired window for channel 8615
wclsFrameSaver: no samples within desired window for channel 8635
wclsFrameSaver: no samples within desired window for channel 8661
wclsFrameSaver: no samples within desired window for channel 8661
wclsFrameSaver: no samples within desired window for channel 8667
wclsFrameSaver: no samples within desired window for channel 8725
wclsFrameSaver: no samples within desired window for channel 8743
wclsFrameSaver: no samples within desired window for channel 8748
wclsFrameSaver: no samples within desired window for channel 8764
wclsFrameSaver: no samples within desired window for channel 8791
wclsFrameSaver: no samples within desired window for channel 8811
wclsFrameSaver: no samples within desired window for channel 8812
wclsFrameSaver: no samples within desired window for channel 8822
wclsFrameSaver: no samples within desired window for channel 8850
wclsFrameSaver: no samples within desired window for channel 8863
wclsFrameSaver: no samples within desired window for channel 8863
wclsFrameSaver: no samples within desired window for channel 8864
wclsFrameSaver: no samples within desired window for channel 8864
wclsFrameSaver: no samples within desired window for channel 8864
wclsFrameSaver: no samples within desired window for channel 8865
wclsFrameSaver: no samples within desired window for channel 8865
wclsFrameSaver: no samples within desired window for channel 8865
wclsFrameSaver: no samples within desired window for channel 8879
wclsFrameSaver: no samples within desired window for channel 8933
wclsFrameSaver: no samples within desired window for channel 8943
wclsFrameSaver: no samples within desired window for channel 8948
wclsFrameSaver: no samples within desired window for channel 8949
wclsFrameSaver: no samples within desired window for channel 8950
wclsFrameSaver: no samples within desired window for channel 8952
wclsFrameSaver: no samples within desired window for channel 8997
wclsFrameSaver: no samples within desired window for channel 9009
wclsFrameSaver: no samples within desired window for channel 9019
wclsFrameSaver: no samples within desired window for channel 9019
wclsFrameSaver: no samples within desired window for channel 9019
wclsFrameSaver: no samples within desired window for channel 9020
wclsFrameSaver: no samples within desired window for channel 9020
wclsFrameSaver: no samples within desired window for channel 9028
wclsFrameSaver: no samples within desired window for channel 9055
wclsFrameSaver: no samples within desired window for channel 9067
wclsFrameSaver: no samples within desired window for channel 9076
wclsFrameSaver: no samples within desired window for channel 9086
wclsFrameSaver: no samples within desired window for channel 9087
wclsFrameSaver: no samples within desired window for channel 9101
wclsFrameSaver: no samples within desired window for channel 9106
wclsFrameSaver: no samples within desired window for channel 9160
wclsFrameSaver: no samples within desired window for channel 9171
wclsFrameSaver: no samples within desired window for channel 9181
wclsFrameSaver: no samples within desired window for channel 9183
wclsFrameSaver: no samples within desired window for channel 9198
wclsFrameSaver: no samples within desired window for channel 9198
wclsFrameSaver: no samples within desired window for channel 9213
wclsFrameSaver: no samples within desired window for channel 9214
wclsFrameSaver: no samples within desired window for channel 9225
wclsFrameSaver: no samples within desired window for channel 9227
wclsFrameSaver: no samples within desired window for channel 9250
wclsFrameSaver: no samples within desired window for channel 9262
wclsFrameSaver: no samples within desired window for channel 9266
wclsFrameSaver: no samples within desired window for channel 9407
wclsFrameSaver: no samples within desired window for channel 9481
wclsFrameSaver: no samples within desired window for channel 9650
wclsFrameSaver: no samples within desired window for channel 9689
wclsFrameSaver: no samples within desired window for channel 9698
wclsFrameSaver: no samples within desired window for channel 10167
FrameSaver: q=2.10492e+06 n=960336 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 8121 XUs and 7399 XVs -> 119 XUVs
C:0 T:1 151567 XUs and 139726 XVs -> 5909 XUVs
C:0 T:2 7650 XUs and 6672 XVs -> 1024 XUVs
C:0 T:3 101 XUs and 173 XVs -> 1 XUVs
C:0 T:4 195 XUs and 118 XVs -> 3 XUVs
C:0 T:5 30445 XUs and 18186 XVs -> 542 XUVs
C:0 T:6 12997 XUs and 5674 XVs -> 170 XUVs
C:0 T:7 193 XUs and 267 XVs -> 10 XUVs
7778 XUVs total
3145 collection wire objects
7778 potential space points
Neighbour search...
357654 tests to find 129438 neighbours
Iterating with no regularization...
Begin: 4.85884e+07
0 4.21211e+07
1 4.15542e+07
2 4.15146e+07
Now with regularization...
Begin: 3.97236e+07
0 3.97152e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Jan-2026 12:05:04 UTC  Opened output file with pattern "%ifb_%tc_keepup.root"
26-Jan-2026 12:05:10 UTC  Closed output file "np04hd_raw_run027308_0000_dataflow1_datawriter_0_20240619T182030_20260126T120510_keepup.root"

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          263.673       263.673       263.673       263.673          0            1     
------------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        0.00033356    0.00033356    0.00033356    0.00033356         0            1     
produce:tpcrawdecoder:PDHDTPCReader                 12.7245       12.7245       12.7245       12.7245          0            1     
produce:triggerrawdecoder:PDHDTriggerReader3       0.0548758     0.0548758     0.0548758     0.0548758         0            1     
produce:timingrawdecoder:PDHDTimingRawDecoder      0.0273757     0.0273757     0.0273757     0.0273757         0            1     
produce:ctbrawdecoder:PDHDCTBRawDecoder            0.134006      0.134006      0.134006      0.134006          0            1     
produce:beamevent:BeamEvent                        0.725391      0.725391      0.725391      0.725391          0            1     
produce:fembfilter:PDHDFEMBFilter                 0.00101852    0.00101852    0.00101852    0.00101852         0            1     
produce:wclsdatahd:WireCellToolkit                  173.995       173.995       173.995       173.995          0            1     
produce:gaushit:GausHitFinder                       1.65175       1.65175       1.65175       1.65175          0            1     
produce:nhitsfilter:NumberOfHitsFilter            0.00104026    0.00104026    0.00104026    0.00104026         0            1     
produce:reco3d:SpacePointSolver                     3.17762       3.17762       3.17762       3.17762          0            1     
produce:hitpdune:DisambigFromSpacePoints            1.63152       1.63152       1.63152       1.63152          0            1     
produce:pandora:StandardPandora                     53.6664       53.6664       53.6664       53.6664          0            1     
produce:pandoraWriter:StandardPandora              0.344267      0.344267      0.344267      0.344267          0            1     
produce:pandoraTrack:LArPandoraTrackCreation        6.42245       6.42245       6.42245       6.42245          0            1     
produce:pandoraShower:LArPandoraShowerCreation      7.31544       7.31544       7.31544       7.31544          0            1     
[art]:TriggerResults:TriggerResultInserter        0.000185057   0.000185057   0.000185057   0.000185057        0            1     
end_path:out1:RootOutput                          5.9956e-05    5.9956e-05    5.9956e-05    5.9956e-05         0            1     
end_path:out1:RootOutput(write)                     1.70554       1.70554       1.70554       1.70554          0            1     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 13084.6 MB
  Peak resident set size usage (VmHWM): 1605.14 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
lsing
total 17M
-rw-r--r-- 1 dunegli fnalgrid  12M Jan 26 12:05 np04hd_raw_run027308_0000_dataflow1_datawriter_0_20240619T182030_20260126T120510_keepup.root
-rw-r--r-- 1 dunegli fnalgrid 4.4M Jan 26 12:04 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid 230K Jan 26 12:05 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid  44K Jan 26 12:05 mem.db
-rw-r--r-- 1 dunegli fnalgrid  16K Jan 26 12:05 time.db
-rw-r--r-- 1 dunegli fnalgrid 7.2K Jan 26 12:05 np04hd_raw_run027308_0000_dataflow1_datawriter_0_20240619T182030.hdf5_calibana.root
-rw-r--r-- 1 dunegli fnalgrid 3.5K Jan 26 12:05 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid   83 Jan 26 11:59 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid    0 Jan 26 11:59 debugprod.log
disk usage
17M	.
justIN time: 2026-02-04 20:43:41 UTC       justIN version: 01.06.00