justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 247056.1@dunegpschedd02.fnal.gov

Jobsub ID247056.1@dunegpschedd02.fnal.gov
Workflow ID10251
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-16 10:29:46
SiteIT_CNAF
EntryCMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat2025-11-16 13:18:00
From worker nodeHostnamewn-s04-05-03-03.cr.cnaf.infn.it
cpuinfoAMD EPYC 7313 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-16 10:30:49
Input filesmonte-carlo-010251-000001
JobscriptExit code0
Real time2h (9694s)
CPU time2h (9283s = 95%)
Max RSS bytes4084690944 (3895 MiB)
Outputting started2025-11-16 13:12:25
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10251/1/001/reco_stage1.root
Finished2025-11-16 13:18:00
Saved logsjustin-logs:247056.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

ice:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 132454 traces, 0 summary
Retagger: tagging trace set: gauss with 104074 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 104074 traces tagged "gauss"
FrameSaver: q=3.4798e+06 n=1844270 tag=gauss
wclsFrameSaver: saving 132454 traces tagged "wiener"
FrameSaver: q=3.64398e+06 n=1624860 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 39044 XUs and 40004 XVs -> 425 XUVs
C:0 T:1 64914 XUs and 62673 XVs -> 3137 XUVs
C:0 T:2 87864 XUs and 82474 XVs -> 3286 XUVs
C:0 T:3 42176 XUs and 32817 XVs -> 263 XUVs
C:0 T:4 13159 XUs and 16316 XVs -> 129 XUVs
C:0 T:5 58663 XUs and 47694 XVs -> 7730 XUVs
C:0 T:6 112815 XUs and 86389 XVs -> 5119 XUVs
C:0 T:7 43120 XUs and 31330 XVs -> 296 XUVs
C:0 T:8 9676 XUs and 7305 XVs -> 177 XUVs
C:0 T:9 119449 XUs and 78093 XVs -> 13687 XUVs
C:0 T:10 28711 XUs and 11846 XVs -> 2729 XUVs
C:0 T:11 124 XUs and 93 XVs -> 11 XUVs
36989 XUVs total
11503 collection wire objects
34191 potential space points
Neighbour search...
4620419 tests to find 2030036 neighbours
Iterating with no regularization...
Begin: 6.77377e+07
0 5.96343e+07
1 5.91858e+07
2 5.91246e+07
3 5.91076e+07
Now with regularization...
Begin: 5.02131e+07
0 5.02056e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410547390
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410547390
fLifetime: 30000 [us]
16-Nov-2025 14:12:23 CET  Closed output file "reco_stage1.root"
16-Nov-2025 14:12:23 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          205.146       245.395       303.315       246.288       32.2481        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000997117   0.00285505    0.00664875    0.00262953    0.00153566       10     
reco:ophit:OpHitFinder                             0.170576      0.225442      0.315675      0.211026      0.0454839       10     
reco:opflash:OpFlashFinder                         0.0930873      0.11989      0.162621       0.12055      0.0215684       10     
reco:opslicer:OpSlicer                              0.34796      0.475965      0.730564       0.47354      0.115769        10     
reco:caldata:DataPrepModule                         3.54373       4.04171       6.15401       3.63228      0.850006        10     
reco:wclsdatasp:WireCellToolkit                     26.2226       28.5244       31.4866       28.2427       1.60716        10     
reco:gaushit:GausHitFinder                         0.602968       0.8797        1.21599      0.877289      0.184814        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000272523   0.000433687   0.000866641   0.000406666   0.000156583      10     
reco:reco3d:SpacePointSolver                        5.92145       12.1861       18.2847       12.3118       4.81274        10     
reco:hitpdune:DisambigFromSpacePoints              0.616854       1.09234        1.542        1.05873      0.279174        10     
reco:pandora:StandardPandora                        11.7205       22.3687       33.5812       21.4134       7.14682        10     
reco:pandoraWriter:StandardPandora                 0.219367       0.33061      0.466329      0.318669      0.0745609       10     
reco:pandoraTrack:LArPandoraTrackCreation           1.52529       2.30852       3.16095       2.34811      0.510828        10     
reco:pandoraShower:LArPandoraShowerCreation         1.56851       2.56988       3.24132       2.61268      0.603336        10     
reco:pandoracalo:Calorimetry                       0.843822       1.43744       2.04246       1.45087      0.344703        10     
reco:pandoracalonosce:Calorimetry                  0.769172       1.30387       1.85027       1.30948      0.314253        10     
reco:pandorapid:Chi2ParticleID                    0.00276693    0.00409632    0.00781927    0.00360667    0.00147443       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0245095     0.0401785     0.0648541     0.0360365     0.0126387       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00269392    0.00365599    0.00529356    0.00338105    0.000933419      10     
reco:pandoraShowercalo:ShowerCalorimetry            1.51094       2.91814       3.8112        3.02938      0.779638        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.64547       2.86108       3.75954       3.02102      0.719131        10     
reco:emtrkmichelid:EmTrackMichelId                  119.335       133.793       166.345       127.566       14.6355        10     
reco:crttag:SingleCRTMatchingProducer             0.00731504     0.0106087     0.014915      0.0102978    0.00256986       10     
reco:crtreco:TwoCRTMatchingProducer               0.00384079    0.00551035    0.00751581    0.00554326    0.000924984      10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.462285      0.657425       0.84288      0.653481      0.128552        10     
reco:pandora2Track:LArPandoraTrackCreation          4.64679       7.42641       9.91798       7.12781       1.78502        10     
reco:pandora2calo:Calorimetry                       2.26749       3.98033       5.36204       3.84863      0.984998        10     
reco:pandora2calonosce:Calorimetry                  2.1524        3.77577       5.04294       3.65813      0.937954        10     
reco:pandora2pid:Chi2ParticleID                    0.0066534    0.00893104     0.0140187    0.00847674    0.00235076       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0313367     0.0485097     0.0852768     0.042168      0.0159405       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0064376     0.0086651     0.0135456    0.00819097    0.00213792       10     
reco:pandora2Shower:LArPandoraShowerCreation        1.85862       2.78768       3.83751       2.67055      0.601837        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.00738       3.6512        6.81035       3.39425       1.2267         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      1.86555       3.33778       6.3409        3.05193       1.13534        10     
[art]:TriggerResults:TriggerResultInserter        3.4846e-05    8.88793e-05   0.000290447   5.4728e-05    7.92771e-05      10     
end_path:out1:RootOutput                           8.025e-06    1.66373e-05   3.5086e-05    1.3701e-05    7.72317e-06      10     
end_path:out1:RootOutput(write)                     1.47535       2.13177       3.54781       1.96068      0.518647        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4891.52 MB
  Peak resident set size usage (VmHWM): 4084.69 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.18 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   2.46 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9681856
-rw-r--r-- 1 neutrino002 neutrino 2296370880 Nov 16 13:29 detsim_stage2.root
-rw-r--r-- 1 neutrino002 neutrino 2181932018 Nov 16 13:28 detsim_stage1.root
-rw-r--r-- 1 neutrino002 neutrino 1987199714 Nov 16 14:12 reco_stage1.root
-rw-r--r-- 1 neutrino002 neutrino 1714902795 Nov 16 11:49 g4_stage2.root
-rw-r--r-- 1 neutrino002 neutrino 1657379457 Nov 16 11:39 g4_stage1.root
-rw-r--r-- 1 neutrino002 neutrino   75804430 Nov 16 14:09 Pandora_Events.pndr
-rw-r--r-- 1 neutrino002 neutrino     242806 Nov 16 14:12 jobscript.log
-rw-r--r-- 1 neutrino002 neutrino     217088 Nov 16 14:12 mem.db
-rw-r--r-- 1 neutrino002 neutrino      62376 Nov 16 14:12 Pandora_Geometry.xml
-rw-r--r-- 1 neutrino002 neutrino      36864 Nov 16 14:12 time.db
-rw-r--r-- 1 neutrino002 neutrino        519 Nov 16 13:30 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 neutrino002 neutrino        519 Nov 16 11:49 g4_protoDUNE_hist.root
-rw-r--r-- 1 neutrino002 neutrino        519 Nov 16 14:12 protoDUNE_reco1_hist.root
-rw-r--r-- 1 neutrino002 neutrino         52 Nov 16 11:30 all-input-dids.txt
-rw-r--r-- 1 neutrino002 neutrino          7 Nov 16 14:12 justin-processed-pfns.txt
-rw-r--r-- 1 neutrino002 neutrino          0 Nov 16 13:34 ProtoDUNE.log
-rw-r--r-- 1 neutrino002 neutrino          0 Nov 16 13:30 debugprod.log
justIN time: 2026-02-09 10:52:21 UTC       justIN version: 01.06.00