justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 277580.1@dunegpschedd01.fnal.gov

Jobsub ID277580.1@dunegpschedd01.fnal.gov
Workflow ID11880
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-14 06:32:15
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2026-01-14 10:26:42
From worker nodeHostnamehnode80.pic.es
cpuinfoAMD EPYC 7443P 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-14 06:33:10
Input filesmonte-carlo-011880-000001
JobscriptExit code0
Real time3h (13951s)
CPU time2h (8407s = 60%)
Max RSS bytes4354527232 (4152 MiB)
Outputting started2026-01-14 10:25:42
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11880/1/001/reco_stage1.root
Finished2026-01-14 10:26:42
Saved logsjustin-logs:277580.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

cScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 132053 traces, 0 summary
Retagger: tagging trace set: gauss with 99852 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 99852 traces tagged "gauss"
FrameSaver: q=3.37265e+06 n=1738092 tag=gauss
wclsFrameSaver: saving 132053 traces tagged "wiener"
FrameSaver: q=3.58953e+06 n=1531226 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 46255 XUs and 29918 XVs -> 1198 XUVs
C:0 T:1 207002 XUs and 109154 XVs -> 30634 XUVs
C:0 T:2 83593 XUs and 50243 XVs -> 3481 XUVs
C:0 T:3 41474 XUs and 20707 XVs -> 245 XUVs
C:0 T:4 13767 XUs and 10735 XVs -> 176 XUVs
C:0 T:5 64166 XUs and 37756 XVs -> 5995 XUVs
C:0 T:6 46332 XUs and 55080 XVs -> 2045 XUVs
C:0 T:7 21509 XUs and 25381 XVs -> 138 XUVs
C:0 T:8 11531 XUs and 10009 XVs -> 156 XUVs
C:0 T:9 89542 XUs and 95078 XVs -> 5202 XUVs
C:0 T:10 44040 XUs and 28716 XVs -> 6798 XUVs
C:0 T:11 83 XUs and 100 XVs -> 11 XUVs
56079 XUVs total
12350 collection wire objects
50846 potential space points
Neighbour search...
10010784 tests to find 4185978 neighbours
Iterating with no regularization...
Begin: 7.30393e+07
0 5.93214e+07
1 5.84659e+07
2 5.83474e+07
3 5.83095e+07
Now with regularization...
Begin: 4.85029e+07
0 4.84487e+07
1 4.84142e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411730996
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411730996
fLifetime: 30000 [us]
14-Jan-2026 11:25:41 CET  Closed output file "reco_stage1.root"
14-Jan-2026 11:25:41 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          150.128        236.2        310.581       236.298       53.7473        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000629397   0.00267666    0.00378464    0.00310679    0.00107942       10     
reco:ophit:OpHitFinder                             0.173825      0.211333      0.264937      0.214067      0.0290321       10     
reco:opflash:OpFlashFinder                         0.110581      0.159893      0.268358      0.145505      0.0453248       10     
reco:opslicer:OpSlicer                             0.313296       0.46179      0.602095      0.461923      0.0983902       10     
reco:caldata:DataPrepModule                         3.07953       3.19191       3.46447       3.13616      0.115635        10     
reco:wclsdatasp:WireCellToolkit                     20.7945       22.5092       25.3795       22.4194       1.29414        10     
reco:gaushit:GausHitFinder                         0.443963      0.790439      0.981053      0.854389      0.174702        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000242797   0.000371558   0.000486933   0.000386325   7.29682e-05      10     
reco:reco3d:SpacePointSolver                        5.80191       13.9109       35.2275       11.9637       8.13681        10     
reco:hitpdune:DisambigFromSpacePoints              0.492587       1.19109       1.94542       1.24195      0.426086        10     
reco:pandora:StandardPandora                        10.5895       24.7906       37.3132       22.085        9.4458         10     
reco:pandoraWriter:StandardPandora                 0.163572      0.257068      0.323213      0.264713      0.0528559       10     
reco:pandoraTrack:LArPandoraTrackCreation           1.42337       2.35893       3.2876         2.304       0.597923        10     
reco:pandoraShower:LArPandoraShowerCreation          1.073        2.80389       4.37503       2.79469       1.02495        10     
reco:pandoracalo:Calorimetry                       0.790571       1.41663       1.89892       1.42116      0.351059        10     
reco:pandoracalonosce:Calorimetry                  0.727797       1.32103       1.78389       1.31509      0.334901        10     
reco:pandorapid:Chi2ParticleID                    0.00255663    0.00376912    0.00776647    0.00351048    0.00143954       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0209662     0.0341761     0.0634801     0.0318972     0.0111056       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00244465    0.00315348    0.00418901    0.00313453    0.000530115      10     
reco:pandoraShowercalo:ShowerCalorimetry            1.10893       3.58171       8.48047       3.1807        1.96358        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.06036       3.21357       5.31602       3.11722       1.28006        10     
reco:emtrkmichelid:EmTrackMichelId                  84.7917       122.944       151.756       126.109       22.6132        10     
reco:crttag:SingleCRTMatchingProducer             0.00440463    0.00821463     0.0122492    0.00795762    0.00243607       10     
reco:crtreco:TwoCRTMatchingProducer               0.00340179    0.00549999    0.00663688    0.00585525    0.00103956       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.355001      0.611448      0.835749      0.622631      0.138078        10     
reco:pandora2Track:LArPandoraTrackCreation          3.41599       8.47499       14.0743       7.92577       3.25434        10     
reco:pandora2calo:Calorimetry                       1.79312       4.43134       6.42428       4.46962       1.40123        10     
reco:pandora2calonosce:Calorimetry                  1.69929       4.18819       6.2524        4.22895       1.35954        10     
reco:pandora2pid:Chi2ParticleID                   0.00540704    0.00807336     0.0104705    0.00815003    0.00152621       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0268582     0.0398139     0.0501261     0.0404495    0.00706009       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00522379    0.00780115    0.00993334    0.00796129    0.00143044       10     
reco:pandora2Shower:LArPandoraShowerCreation        1.36663       3.09254        4.376        3.03228      0.994892        10     
reco:pandora2Showercalo:ShowerCalorimetry           1.59883       4.13249       5.59657       4.62583       1.32652        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      1.50663       4.02972        7.404        4.00835       1.58156        10     
[art]:TriggerResults:TriggerResultInserter         1.696e-05    2.44027e-05   7.7362e-05    1.8395e-05    1.7677e-05       10     
end_path:out1:RootOutput                           3.67e-06     4.4382e-06     7.98e-06     4.0455e-06    1.1987e-06       10     
end_path:out1:RootOutput(write)                     1.31783       1.94758       2.49005       1.92475      0.366893        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5158.67 MB
  Peak resident set size usage (VmHWM): 4354.53 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.02 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   2.05 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.02 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10869616
-rw-r--r-- 1 dune001 dune 2521397153 Jan 14 10:44 detsim_stage2.root
-rw-r--r-- 1 dune001 dune 2386862116 Jan 14 10:37 detsim_stage1.root
-rw-r--r-- 1 dune001 dune 2259458444 Jan 14 11:25 reco_stage1.root
-rw-r--r-- 1 dune001 dune 1972760034 Jan 14 08:18 g4_stage2.root
-rw-r--r-- 1 dune001 dune 1902084788 Jan 14 07:41 g4_stage1.root
-rw-r--r-- 1 dune001 dune   87293270 Jan 14 11:23 Pandora_Events.pndr
-rw-r--r-- 1 dune001 dune     243376 Jan 14 11:25 jobscript.log
-rw-r--r-- 1 dune001 dune     217088 Jan 14 11:25 mem.db
-rw-r--r-- 1 dune001 dune      62376 Jan 14 11:25 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 dune      36864 Jan 14 11:25 time.db
-rw-r--r-- 1 dune001 dune        519 Jan 14 10:44 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Jan 14 08:18 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Jan 14 11:25 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune001 dune         52 Jan 14 07:33 all-input-dids.txt
-rw-r--r-- 1 dune001 dune          7 Jan 14 11:25 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune          0 Jan 14 10:49 ProtoDUNE.log
-rw-r--r-- 1 dune001 dune          0 Jan 14 10:44 debugprod.log
justIN time: 2026-02-06 03:23:59 UTC       justIN version: 01.06.00