justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268366.0@dunegpschedd02.fnal.gov

Jobsub ID268366.0@dunegpschedd02.fnal.gov
Workflow ID11709
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-11 05:48:26
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce03
Last heartbeat2026-01-11 10:14:21
From worker nodeHostnamecn502.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-11 05:49:27
Input filesmonte-carlo-011709-000001
JobscriptExit code0
Real time4h (15674s)
CPU time4h (15468s = 98%)
Max RSS bytes4340842496 (4139 MiB)
Outputting started2026-01-11 10:10:42
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11709/1/001/reco_stage1.root
Finished2026-01-11 10:14:21
Saved logsjustin-logs:268366.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

leKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 169500 traces, 0 summary
Retagger: tagging trace set: gauss with 127018 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 127018 traces tagged "gauss"
FrameSaver: q=4.51995e+06 n=2232995 tag=gauss
wclsFrameSaver: saving 169500 traces tagged "wiener"
FrameSaver: q=4.80285e+06 n=1970315 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 60727 XUs and 36707 XVs -> 837 XUVs
C:0 T:1 135814 XUs and 95262 XVs -> 8504 XUVs
C:0 T:2 127851 XUs and 177419 XVs -> 15439 XUVs
C:0 T:3 57538 XUs and 60233 XVs -> 439 XUVs
C:0 T:4 19131 XUs and 17329 XVs -> 154 XUVs
C:0 T:5 103571 XUs and 65783 XVs -> 12079 XUVs
C:0 T:6 108310 XUs and 113386 XVs -> 3916 XUVs
C:0 T:7 44385 XUs and 46846 XVs -> 507 XUVs
C:0 T:8 7372 XUs and 6927 XVs -> 129 XUVs
C:0 T:9 63715 XUs and 70406 XVs -> 9254 XUVs
C:0 T:10 63376 XUs and 38725 XVs -> 6907 XUVs
C:0 T:11 108 XUs and 81 XVs -> 9 XUVs
58174 XUVs total
15055 collection wire objects
53799 potential space points
Neighbour search...
7519587 tests to find 3503248 neighbours
Iterating with no regularization...
Begin: 9.8485e+07
0 7.99946e+07
1 7.86e+07
2 7.84287e+07
3 7.83797e+07
Now with regularization...
Begin: 6.49076e+07
0 6.48391e+07
1 6.47935e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411670251
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411670251
fLifetime: 30000 [us]
11-Jan-2026 10:10:39 GMT  Closed output file "reco_stage1.root"
11-Jan-2026 10:10:39 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          489.326       645.323       851.057       619.884       102.224        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00142532    0.00420041    0.00651536    0.00418775    0.00163799       10     
reco:ophit:OpHitFinder                             0.362762      0.415545      0.553273      0.405446      0.0530464       10     
reco:opflash:OpFlashFinder                         0.302885      0.354208      0.417745      0.351449      0.0299264       10     
reco:opslicer:OpSlicer                             0.798589       1.04191       1.47149       1.01275      0.176355        10     
reco:caldata:DataPrepModule                         6.95805       7.27485       7.61296       7.29247      0.174003        10     
reco:wclsdatasp:WireCellToolkit                     53.4885       58.1676       61.7343       58.3066       2.33587        10     
reco:gaushit:GausHitFinder                          1.50304       1.96257       2.42585       1.91053       0.28239        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000766089   0.000911904   0.00111137    0.000911769   9.90341e-05      10     
reco:reco3d:SpacePointSolver                        14.6904       21.9392       37.8419       20.3809       6.40248        10     
reco:hitpdune:DisambigFromSpacePoints               2.2792        3.34309       5.36495       3.25311      0.930721        10     
reco:pandora:StandardPandora                        37.1639       72.1693       152.279       61.2202       33.5139        10     
reco:pandoraWriter:StandardPandora                 0.574734      0.672187      0.833525      0.645247      0.0801198       10     
reco:pandoraTrack:LArPandoraTrackCreation           10.1172       14.0637       18.4331       14.8818       3.0844         10     
reco:pandoraShower:LArPandoraShowerCreation         8.54173       15.4804       27.0847       15.0333       5.00516        10     
reco:pandoracalo:Calorimetry                        4.46769       6.24348       7.75482       6.4746        1.32004        10     
reco:pandoracalonosce:Calorimetry                   4.11822       5.82544       7.27427       6.1224        1.17799        10     
reco:pandorapid:Chi2ParticleID                    0.00694615    0.00834549     0.0104028     0.0084346     0.0010526       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0810681     0.0928872     0.111075      0.0906886    0.00927768       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00689695    0.00818997     0.0101167    0.00812867    0.00103813       10     
reco:pandoraShowercalo:ShowerCalorimetry            8.42299       13.4136       23.109        12.9145       4.07231        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       8.5873        13.3231       21.9009       13.1501       3.97029        10     
reco:emtrkmichelid:EmTrackMichelId                  225.197       273.344       335.784       262.273       33.7632        10     
reco:crttag:SingleCRTMatchingProducer              0.0222797     0.0342683     0.0425706     0.0350729    0.00535387       10     
reco:crtreco:TwoCRTMatchingProducer                0.0168446     0.0284332     0.0339606     0.0293295    0.00479353       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.40722       1.84301       2.3301        1.8843       0.264823        10     
reco:pandora2Track:LArPandoraTrackCreation          30.9198       47.1246       69.8991       46.5346       10.1062        10     
reco:pandora2calo:Calorimetry                       13.5294       20.8823       28.7986       20.9544       3.88434        10     
reco:pandora2calonosce:Calorimetry                  11.2619       18.3709       25.4124       18.5676       3.38187        10     
reco:pandora2pid:Chi2ParticleID                    0.0169869     0.0233276     0.0279303     0.0228321    0.00318509       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.103692      0.128296      0.156273      0.126525      0.0172549       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0171627     0.0235796     0.0298488     0.0229355    0.00383576       10     
reco:pandora2Shower:LArPandoraShowerCreation        9.62386       13.6942       20.9381       13.6777       3.0488         10     
reco:pandora2Showercalo:ShowerCalorimetry           10.5452       15.7281       22.0615       15.8129       3.54013        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      9.18986       13.6762       17.495        13.455        2.76992        10     
[art]:TriggerResults:TriggerResultInserter        5.4847e-05    7.86377e-05   0.000197912   6.50785e-05   4.07718e-05      10     
end_path:out1:RootOutput                           8.05e-06     9.9814e-06    1.8511e-05    8.6375e-06    3.03862e-06      10     
end_path:out1:RootOutput(write)                     3.32386       4.42452       5.68658       4.25973      0.789841        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5145.33 MB
  Peak resident set size usage (VmHWM): 4340.84 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.26 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.57 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.25 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11354980
-rw-r--r-- 1 pildune32 pildune 2624390481 Jan 11 08:20 detsim_stage2.root
-rw-r--r-- 1 pildune32 pildune 2485669208 Jan 11 08:17 detsim_stage1.root
-rw-r--r-- 1 pildune32 pildune 2361549669 Jan 11 10:10 reco_stage1.root
-rw-r--r-- 1 pildune32 pildune 2066161149 Jan 11 06:18 g4_stage2.root
-rw-r--r-- 1 pildune32 pildune 2002345862 Jan 11 05:57 g4_stage1.root
-rw-r--r-- 1 pildune32 pildune   86771540 Jan 11 10:01 Pandora_Events.pndr
-rw-r--r-- 1 pildune32 pildune     242753 Jan 11 10:10 jobscript.log
-rw-r--r-- 1 pildune32 pildune     217088 Jan 11 10:10 mem.db
-rw-r--r-- 1 pildune32 pildune      62376 Jan 11 10:10 Pandora_Geometry.xml
-rw-r--r-- 1 pildune32 pildune      36864 Jan 11 10:10 time.db
-rw-r--r-- 1 pildune32 pildune        519 Jan 11 08:20 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 pildune32 pildune        519 Jan 11 06:18 g4_protoDUNE_hist.root
-rw-r--r-- 1 pildune32 pildune        519 Jan 11 10:10 protoDUNE_reco1_hist.root
-rw-r--r-- 1 pildune32 pildune         52 Jan 11 05:49 all-input-dids.txt
-rw-r--r-- 1 pildune32 pildune          7 Jan 11 10:10 justin-processed-pfns.txt
-rw-r--r-- 1 pildune32 pildune          0 Jan 11 08:28 ProtoDUNE.log
-rw-r--r-- 1 pildune32 pildune          0 Jan 11 08:20 debugprod.log
justIN time: 2026-02-06 11:44:10 UTC       justIN version: 01.06.00