justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 270889.0@dunegpschedd01.fnal.gov

Jobsub ID270889.0@dunegpschedd01.fnal.gov
Workflow ID11415
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-30 06:20:18
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce16-multicore
Last heartbeat2025-12-30 09:38:45
From worker nodeHostnamegpu01.pic.es
cpuinfoIntel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-30 06:30:11
Input filesmonte-carlo-011415-000001
JobscriptExit code0
Real time3h (11278s)
CPU time2h (10674s = 94%)
Max RSS bytes4116946944 (3926 MiB)
Outputting started2025-12-30 09:38:10
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11415/1/001/reco_stage1.root
Finished2025-12-30 09:38:45
Saved logsjustin-logs:270889.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ol adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 131040 traces, 0 summary
Retagger: tagging trace set: gauss with 102651 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 102651 traces tagged "gauss"
FrameSaver: q=3.64429e+06 n=1868007 tag=gauss
wclsFrameSaver: saving 131040 traces tagged "wiener"
FrameSaver: q=3.84606e+06 n=1672823 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 36392 XUs and 25686 XVs -> 327 XUVs
C:0 T:1 99637 XUs and 68512 XVs -> 8800 XUVs
C:0 T:2 55665 XUs and 88039 XVs -> 2916 XUVs
C:0 T:3 29343 XUs and 38595 XVs -> 140 XUVs
C:0 T:4 14934 XUs and 14977 XVs -> 142 XUVs
C:0 T:5 88069 XUs and 50787 XVs -> 7605 XUVs
C:0 T:6 82099 XUs and 121364 XVs -> 4198 XUVs
C:0 T:7 34768 XUs and 52572 XVs -> 379 XUVs
C:0 T:8 10226 XUs and 11660 XVs -> 105 XUVs
C:0 T:9 67072 XUs and 89601 XVs -> 4331 XUVs
C:0 T:10 28684 XUs and 21465 XVs -> 3315 XUVs
C:0 T:11 134 XUs and 116 XVs -> 19 XUVs
32277 XUVs total
11961 collection wire objects
29918 potential space points
Neighbour search...
2556084 tests to find 1210088 neighbours
Iterating with no regularization...
Begin: 7.63436e+07
0 6.18763e+07
1 6.08407e+07
2 6.06982e+07
3 6.06514e+07
Now with regularization...
Begin: 4.95149e+07
0 4.94601e+07
1 4.94268e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411429277
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411429277
fLifetime: 30000 [us]
30-Dec-2025 10:38:08 CET  Closed output file "reco_stage1.root"
30-Dec-2025 10:38:08 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          311.873       427.73        496.994       439.919       54.3869        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000926283   0.00329181    0.00526264    0.00352112    0.00119438       10     
reco:ophit:OpHitFinder                             0.226994      0.272662      0.337111      0.271809      0.0284464       10     
reco:opflash:OpFlashFinder                         0.147456      0.212328       0.31514      0.206839      0.0433878       10     
reco:opslicer:OpSlicer                             0.549295      0.672737      0.810442      0.689684      0.084586        10     
reco:caldata:DataPrepModule                         5.26961       5.55473       6.19692       5.4201        0.28397        10     
reco:wclsdatasp:WireCellToolkit                     43.4599       47.3018       49.2893       47.8028       1.79391        10     
reco:gaushit:GausHitFinder                          1.27762       1.52964       1.69152       1.58996      0.147283        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000587336   0.000679854   0.000922293   0.000671387   9.1442e-05       10     
reco:reco3d:SpacePointSolver                        9.51667       15.3453       24.8528       14.0091       4.38496        10     
reco:hitpdune:DisambigFromSpacePoints               1.47662       2.23427       2.71097       2.26213      0.332524        10     
reco:pandora:StandardPandora                        24.9007       37.5968       47.6203       37.7699       6.45429        10     
reco:pandoraWriter:StandardPandora                  0.4125       0.736228       1.23096      0.492263      0.341738        10     
reco:pandoraTrack:LArPandoraTrackCreation           5.24965       8.56215       10.7727       9.13514       1.50028        10     
reco:pandoraShower:LArPandoraShowerCreation         5.09131       9.44583       11.5127       10.1216       2.04417        10     
reco:pandoracalo:Calorimetry                        2.25658       3.81931       4.90048       3.91012      0.648428        10     
reco:pandoracalonosce:Calorimetry                   2.04365       3.6647        4.59121       3.76832      0.646294        10     
reco:pandorapid:Chi2ParticleID                    0.00475998     0.0059083    0.00753536    0.00598032    0.000732553      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0534679     0.0676893     0.0796108     0.0690052    0.00873041       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00469641    0.00573168    0.00641438    0.00577471    0.000524538      10     
reco:pandoraShowercalo:ShowerCalorimetry            4.39079       8.63145       10.4719       9.59299       2.0194         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.98698       8.40328       10.2551       9.39788       2.05402        10     
reco:emtrkmichelid:EmTrackMichelId                  149.637       190.804       221.495       194.924       20.1977        10     
reco:crttag:SingleCRTMatchingProducer              0.0174275     0.0254513     0.033323      0.0259752    0.00454536       10     
reco:crtreco:TwoCRTMatchingProducer                0.0141991     0.0207275      0.02671      0.0222301    0.00416173       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.02568       1.28941       1.59032       1.24831      0.170777        10     
reco:pandora2Track:LArPandoraTrackCreation          14.2289       27.9829       36.3404       29.8437       6.49442        10     
reco:pandora2calo:Calorimetry                       5.69546       11.853        15.4052       12.7007       2.86841        10     
reco:pandora2calonosce:Calorimetry                  5.48947       11.2628       15.3866       12.0068       2.68598        10     
reco:pandora2pid:Chi2ParticleID                    0.011167      0.0143869     0.0168857     0.0144832    0.00158945       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.070547      0.0863244     0.102701      0.090556      0.0103532       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0111731     0.0145108     0.0162559     0.0150513    0.00150023       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.67817       8.60521       10.9873       9.07364       1.9082         10     
reco:pandora2Showercalo:ShowerCalorimetry           4.7981        9.67955       12.263        10.3694       2.15648        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.0764        8.33917       10.4855       8.91787       1.87323        10     
[art]:TriggerResults:TriggerResultInserter        4.7512e-05    6.85176e-05   0.000177881    5.757e-05    3.67421e-05      10     
end_path:out1:RootOutput                           8.924e-06    1.19625e-05   2.2898e-05    9.8795e-06    4.36315e-06      10     
end_path:out1:RootOutput(write)                     2.45567       3.54407       6.32039       3.11368       1.26577        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4927.11 MB
  Peak resident set size usage (VmHWM): 4116.95 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.43 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9809688
-rw-r--r-- 1 dune001 dune 2317663078 Dec 30 09:24 detsim_stage2.root
-rw-r--r-- 1 dune001 dune 2198156989 Dec 30 09:22 detsim_stage1.root
-rw-r--r-- 1 dune001 dune 2010797813 Dec 30 10:38 reco_stage1.root
-rw-r--r-- 1 dune001 dune 1753609897 Dec 30 07:51 g4_stage2.root
-rw-r--r-- 1 dune001 dune 1687802670 Dec 30 07:37 g4_stage1.root
-rw-r--r-- 1 dune001 dune   76446060 Dec 30 10:32 Pandora_Events.pndr
-rw-r--r-- 1 dune001 dune     242928 Dec 30 10:38 jobscript.log
-rw-r--r-- 1 dune001 dune     217088 Dec 30 10:38 mem.db
-rw-r--r-- 1 dune001 dune      62376 Dec 30 10:38 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 dune      36864 Dec 30 10:38 time.db
-rw-r--r-- 1 dune001 dune        519 Dec 30 09:24 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Dec 30 07:51 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Dec 30 10:38 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune001 dune         52 Dec 30 07:30 all-input-dids.txt
-rw-r--r-- 1 dune001 dune          7 Dec 30 10:38 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune          0 Dec 30 09:31 ProtoDUNE.log
-rw-r--r-- 1 dune001 dune          0 Dec 30 09:24 debugprod.log
justIN time: 2026-02-07 00:50:28 UTC       justIN version: 01.06.00