justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 247214.0@dunegpschedd01.fnal.gov

Jobsub ID247214.0@dunegpschedd01.fnal.gov
Workflow ID9948
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-11 18:26:27
SiteIT_CNAF
EntryCMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat2025-11-11 20:47:59
From worker nodeHostnamewn-s04-05-05-03.cr.cnaf.infn.it
cpuinfoAMD EPYC 7313 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statestalled
Started2025-11-11 18:27:50
Input filesmonte-carlo-009948-000001
Outputting started2025-11-11 20:35:27
Output files
Finished2025-11-11 21:34:38
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

dc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 81005 traces, 0 summary
Retagger: tagging trace set: gauss with 65770 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 65770 traces tagged "gauss"
FrameSaver: q=2.21914e+06 n=1210224 tag=gauss
wclsFrameSaver: saving 81005 traces tagged "wiener"
FrameSaver: q=2.31438e+06 n=1085369 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 34930 XUs and 26013 XVs -> 309 XUVs
C:0 T:1 47343 XUs and 45350 XVs -> 4102 XUVs
C:0 T:2 84872 XUs and 97566 XVs -> 4082 XUVs
C:0 T:3 39434 XUs and 43783 XVs -> 203 XUVs
C:0 T:4 7202 XUs and 5463 XVs -> 35 XUVs
C:0 T:5 18636 XUs and 10329 XVs -> 1556 XUVs
C:0 T:6 40645 XUs and 40561 XVs -> 2627 XUVs
C:0 T:7 18140 XUs and 17242 XVs -> 179 XUVs
C:0 T:8 2743 XUs and 2244 XVs -> 15 XUVs
C:0 T:9 14334 XUs and 17441 XVs -> 1034 XUVs
C:0 T:10 30779 XUs and 24158 XVs -> 5223 XUVs
C:0 T:11 17 XUs and 21 XVs -> 3 XUVs
19368 XUVs total
7338 collection wire objects
18183 potential space points
Neighbour search...
1427893 tests to find 763172 neighbours
Iterating with no regularization...
Begin: 4.76442e+07
0 4.36341e+07
1 4.33254e+07
2 4.3283e+07
Now with regularization...
Begin: 3.70264e+07
0 3.70055e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410452215
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410452215
fLifetime: 30000 [us]
11-Nov-2025 21:35:26 CET  Closed output file "reco_stage1.root"
11-Nov-2025 21:35:26 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          139.102       202.835       281.313       208.02        38.528         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00061744    0.00311274    0.00531931     0.003392     0.00151312       10     
reco:ophit:OpHitFinder                             0.174308      0.220059      0.290196      0.215722      0.0305414       10     
reco:opflash:OpFlashFinder                         0.0860182     0.129315      0.158479      0.131886      0.0190623       10     
reco:opslicer:OpSlicer                             0.328631      0.464781      0.727179      0.457273      0.0998014       10     
reco:caldata:DataPrepModule                         3.50801       3.82429       4.29613       3.7072       0.256943        10     
reco:wclsdatasp:WireCellToolkit                     23.4542       24.7414       25.701        24.9606      0.771297        10     
reco:gaushit:GausHitFinder                         0.608005      0.749291       1.08328      0.697044      0.148594        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000237366   0.000335907   0.000566515   0.000297274   0.000109996      10     
reco:reco3d:SpacePointSolver                        4.4506        9.71483       19.0487       8.7686        4.41384        10     
reco:hitpdune:DisambigFromSpacePoints              0.526539      0.973525       1.63745       0.97956      0.312525        10     
reco:pandora:StandardPandora                        8.9538        19.0981       32.7873       19.2901       6.77296        10     
reco:pandoraWriter:StandardPandora                 0.187688       0.25866      0.353144      0.272581      0.0504609       10     
reco:pandoraTrack:LArPandoraTrackCreation           1.11195       1.85864       2.87233       1.96836       0.50862        10     
reco:pandoraShower:LArPandoraShowerCreation        0.979716       2.12491       3.69844       2.1013       0.836119        10     
reco:pandoracalo:Calorimetry                       0.648747       1.16149       1.76455       1.26965      0.338433        10     
reco:pandoracalonosce:Calorimetry                  0.596336       1.05052       1.52782       1.14084      0.321248        10     
reco:pandorapid:Chi2ParticleID                    0.00251757    0.00354918    0.00577922    0.00322795    0.000901214      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.024526      0.0328526     0.0516068     0.0305184    0.00840102       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00228863    0.00299255     0.0037769     0.0029769    0.00043255       10     
reco:pandoraShowercalo:ShowerCalorimetry            1.02638       2.39243       4.27664       2.40251       1.03493        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.01897       2.34325       4.0922        2.33883       1.01929        10     
reco:emtrkmichelid:EmTrackMichelId                  78.9335       108.656       142.763       110.718        18.67         10     
reco:crttag:SingleCRTMatchingProducer             0.00544278    0.00887331     0.0142251    0.00900883    0.00237671       10     
reco:crtreco:TwoCRTMatchingProducer               0.00321188    0.00458268    0.00604921    0.00446922    0.000884005      10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.457747      0.611947       0.73096       0.63961      0.0905988       10     
reco:pandora2Track:LArPandoraTrackCreation          3.27766       6.19051       10.9797       6.1263        2.29489        10     
reco:pandora2calo:Calorimetry                       1.6705        3.3553        5.92072       3.5064        1.30842        10     
reco:pandora2calonosce:Calorimetry                  1.55404       3.1398        5.38463       3.24685       1.21135        10     
reco:pandora2pid:Chi2ParticleID                    0.0058728    0.00818137     0.0129227    0.00782137    0.00225813       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0278692     0.0383477     0.0547546     0.0382698    0.00756877       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.005304     0.00744724     0.0101215    0.00753913    0.00144139       10     
reco:pandora2Shower:LArPandoraShowerCreation        1.26153       2.22881       3.78677       2.23942      0.737504        10     
reco:pandora2Showercalo:ShowerCalorimetry           1.42882       2.89581       4.98091       2.97666       1.10306        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      1.35328       2.63463       4.2656        2.78817      0.958907        10     
[art]:TriggerResults:TriggerResultInserter        3.6268e-05    6.98193e-05   0.000188184    4.807e-05    4.68875e-05      10     
end_path:out1:RootOutput                           5.941e-06     9.129e-06    1.8234e-05     7.434e-06    3.51439e-06      10     
end_path:out1:RootOutput(write)                     1.25195       1.84645       2.72491       1.83782       0.41429        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4968.68 MB
  Peak resident set size usage (VmHWM): 4162.16 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   2.28 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10067924
-rw-r--r-- 1 neutrino002 neutrino 2360110227 Nov 11 21:00 detsim_stage2.root
-rw-r--r-- 1 neutrino002 neutrino 2241370755 Nov 11 20:58 detsim_stage1.root
-rw-r--r-- 1 neutrino002 neutrino 2026260639 Nov 11 21:35 reco_stage1.root
-rw-r--r-- 1 neutrino002 neutrino 1845155180 Nov 11 19:43 g4_stage2.root
-rw-r--r-- 1 neutrino002 neutrino 1766219839 Nov 11 19:34 g4_stage1.root
-rw-r--r-- 1 neutrino002 neutrino   69804590 Nov 11 21:33 Pandora_Events.pndr
-rw-r--r-- 1 neutrino002 neutrino     242704 Nov 11 21:35 jobscript.log
-rw-r--r-- 1 neutrino002 neutrino     217088 Nov 11 21:35 mem.db
-rw-r--r-- 1 neutrino002 neutrino      62376 Nov 11 21:35 Pandora_Geometry.xml
-rw-r--r-- 1 neutrino002 neutrino      36864 Nov 11 21:35 time.db
-rw-r--r-- 1 neutrino002 neutrino        519 Nov 11 21:00 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 neutrino002 neutrino        519 Nov 11 19:43 g4_protoDUNE_hist.root
-rw-r--r-- 1 neutrino002 neutrino        519 Nov 11 21:35 protoDUNE_reco1_hist.root
-rw-r--r-- 1 neutrino002 neutrino         52 Nov 11 19:27 all-input-dids.txt
-rw-r--r-- 1 neutrino002 neutrino          7 Nov 11 21:35 justin-processed-pfns.txt
-rw-r--r-- 1 neutrino002 neutrino          0 Nov 11 21:04 ProtoDUNE.log
-rw-r--r-- 1 neutrino002 neutrino          0 Nov 11 21:00 debugprod.log
justIN time: 2026-02-10 08:03:05 UTC       justIN version: 01.06.00