justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 264652.0@dunegpschedd02.fnal.gov

Jobsub ID264652.0@dunegpschedd02.fnal.gov
Workflow ID11372
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-28 08:47:46
SiteUK_Bristol
EntryCMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat2025-12-28 15:00:28
From worker nodeHostnamehd79.dice.priv
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-28 08:57:09
Input filesmonte-carlo-011372-000001
JobscriptExit code0
Real time6h (21697s)
CPU time5h (21162s = 97%)
Max RSS bytes4641910784 (4426 MiB)
Outputting started2025-12-28 14:58:47
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11372/1/001/reco_stage1.root
Finished2025-12-28 15:00:28
Saved logsjustin-logs:264652.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

dRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 136262 traces, 0 summary
Retagger: tagging trace set: gauss with 109436 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 109436 traces tagged "gauss"
FrameSaver: q=3.69439e+06 n=2010863 tag=gauss
wclsFrameSaver: saving 136262 traces tagged "wiener"
FrameSaver: q=3.87279e+06 n=1806557 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 44001 XUs and 49135 XVs -> 528 XUVs
C:0 T:1 90564 XUs and 91198 XVs -> 5456 XUVs
C:0 T:2 43297 XUs and 85646 XVs -> 2921 XUVs
C:0 T:3 20033 XUs and 30055 XVs -> 160 XUVs
C:0 T:4 8889 XUs and 10974 XVs -> 141 XUVs
C:0 T:5 39406 XUs and 47942 XVs -> 5346 XUVs
C:0 T:6 137525 XUs and 133061 XVs -> 4266 XUVs
C:0 T:7 59959 XUs and 54556 XVs -> 426 XUVs
C:0 T:8 12150 XUs and 12810 XVs -> 202 XUVs
C:0 T:9 87657 XUs and 109071 XVs -> 5688 XUVs
C:0 T:10 27237 XUs and 22963 XVs -> 3642 XUVs
C:0 T:11 143 XUs and 123 XVs -> 22 XUVs
28798 XUVs total
12735 collection wire objects
26237 potential space points
Neighbour search...
1315329 tests to find 642080 neighbours
Iterating with no regularization...
Begin: 6.67084e+07
0 5.54096e+07
1 5.48192e+07
2 5.47523e+07
3 5.47351e+07
Now with regularization...
Begin: 4.51266e+07
0 4.51256e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411391136
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411391136
fLifetime: 30000 [us]
28-Dec-2025 14:58:45 UTC  Closed output file "reco_stage1.root"
28-Dec-2025 14:58:45 UTC  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          394.643       608.224       884.103       585.799       123.697        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00121102    0.00416319    0.00618291    0.00465763     0.0017645       10     
reco:ophit:OpHitFinder                             0.347429      0.392449      0.456756      0.390099      0.0298446       10     
reco:opflash:OpFlashFinder                         0.210515      0.289193      0.353002      0.300454      0.0443393       10     
reco:opslicer:OpSlicer                             0.993266       1.20857       1.42094       1.21698      0.119612        10     
reco:caldata:DataPrepModule                         6.95553       7.15355       7.52659       7.12498      0.151737        10     
reco:wclsdatasp:WireCellToolkit                     43.1046       47.4146       50.9096       48.2454       2.58594        10     
reco:gaushit:GausHitFinder                          1.47759        2.121        2.79898       2.18098      0.341844        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000501436   0.000737613   0.00100041    0.00074014    0.000132412      10     
reco:reco3d:SpacePointSolver                        11.0587       20.5969       31.8964       21.2216       6.04682        10     
reco:hitpdune:DisambigFromSpacePoints               1.60713       3.02361       4.98192       2.9218       0.847498        10     
reco:pandora:StandardPandora                        32.2029       69.6761       134.572       67.8504       27.5146        10     
reco:pandoraWriter:StandardPandora                 0.485576      0.666689      0.889293      0.653869      0.106215        10     
reco:pandoraTrack:LArPandoraTrackCreation           4.66678       8.38568       13.4983       7.93948       2.28313        10     
reco:pandoraShower:LArPandoraShowerCreation         4.52765       10.2188       19.102        9.82358       3.64674        10     
reco:pandoracalo:Calorimetry                        2.2147        4.4616        6.75235       4.3277        1.15016        10     
reco:pandoracalonosce:Calorimetry                   2.04108       4.10332       6.3875        4.0073        1.08981        10     
reco:pandorapid:Chi2ParticleID                    0.00835865    0.00979948     0.0125138    0.00962654    0.00132937       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0681467     0.0961815     0.123883      0.0956111     0.0153322       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00796901    0.00940705     0.0116838    0.00938909    0.00111268       10     
reco:pandoraShowercalo:ShowerCalorimetry            4.0089        10.2303       18.2624       10.1137       3.46511        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.90394       10.0513       17.9857       10.0894       3.44488        10     
reco:emtrkmichelid:EmTrackMichelId                  225.204       317.186       417.404       314.387       49.8311        10     
reco:crttag:SingleCRTMatchingProducer              0.0121548     0.0206928     0.0266222     0.0205145    0.00438316       10     
reco:crtreco:TwoCRTMatchingProducer               0.00962766     0.0160331     0.0204133     0.0161383    0.00312052       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.13894       1.54901       1.96622       1.58743      0.217044        10     
reco:pandora2Track:LArPandoraTrackCreation          14.5391       28.5377       50.4202       27.0524       9.13881        10     
reco:pandora2calo:Calorimetry                       5.8161        13.2236       22.7532       12.5909       4.2246         10     
reco:pandora2calonosce:Calorimetry                  5.45082       12.512        21.4125       11.9355       3.99566        10     
reco:pandora2pid:Chi2ParticleID                    0.0196843     0.0255232     0.0357834     0.0249028    0.00438196       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0859956     0.123664      0.174637      0.120508      0.0223327       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0198847     0.0252556     0.0352385     0.0246502    0.00420979       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.09574       10.1665       16.972        9.72507       3.03648        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.91958       10.7749       18.0726       10.3057       3.34214        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.31526       9.39854       15.0888       9.08835       2.76128        10     
[art]:TriggerResults:TriggerResultInserter        5.2319e-05    7.30436e-05   0.000191601   5.9918e-05    3.97541e-05      10     
end_path:out1:RootOutput                           9.148e-06    1.12263e-05   1.6982e-05     1.057e-05    2.26354e-06      10     
end_path:out1:RootOutput(write)                     3.05308       4.40003       5.86681       4.37688      0.748534        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5441.79 MB
  Peak resident set size usage (VmHWM): 4641.91 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.26 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.71 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.14 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11502308
-rw-r--r-- 1 dnpil001 users 2655949468 Dec 28 13:14 detsim_stage2.root
-rw-r--r-- 1 dnpil001 users 2518171372 Dec 28 13:10 detsim_stage1.root
-rw-r--r-- 1 dnpil001 users 2393416988 Dec 28 14:58 reco_stage1.root
-rw-r--r-- 1 dnpil001 users 2104852586 Dec 28 09:28 g4_stage2.root
-rw-r--r-- 1 dnpil001 users 2018497239 Dec 28 09:05 g4_stage1.root
-rw-r--r-- 1 dnpil001 users   86793210 Dec 28 14:51 Pandora_Events.pndr
-rw-r--r-- 1 dnpil001 users     242992 Dec 28 14:58 jobscript.log
-rw-r--r-- 1 dnpil001 users     217088 Dec 28 14:58 mem.db
-rw-r--r-- 1 dnpil001 users      62376 Dec 28 14:58 Pandora_Geometry.xml
-rw-r--r-- 1 dnpil001 users      36864 Dec 28 14:58 time.db
-rw-r--r-- 1 dnpil001 users        519 Dec 28 13:14 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dnpil001 users        519 Dec 28 09:28 g4_protoDUNE_hist.root
-rw-r--r-- 1 dnpil001 users        519 Dec 28 14:58 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dnpil001 users         52 Dec 28 08:57 all-input-dids.txt
-rw-r--r-- 1 dnpil001 users          7 Dec 28 14:58 justin-processed-pfns.txt
-rw-r--r-- 1 dnpil001 users          0 Dec 28 13:24 ProtoDUNE.log
-rw-r--r-- 1 dnpil001 users          0 Dec 28 13:14 debugprod.log
justIN time: 2026-02-07 03:32:38 UTC       justIN version: 01.06.00