justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 43352.1@dunegpschedd01.fnal.gov

Jobsub ID43352.1@dunegpschedd01.fnal.gov
Workflow ID2356
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-09-08 20:39:15
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2025-09-08 22:37:30
From worker nodeHostnamedunegli-45666-0-cmswn4014.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-08 20:42:15
Input filesmonte-carlo-002356-000001
JobscriptExit code0
Real time1h (6899s)
CPU time1h (6846s = 99%)
Max RSS bytes4277116928 (4078 MiB)
Outputting started2025-09-08 22:37:14
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/02356/1/001/reco_stage1.root
Finished2025-09-08 22:37:30
Saved logsjustin-logs:43352.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

KeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 138252 traces, 0 summary
Retagger: tagging trace set: gauss with 102687 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 102687 traces tagged "gauss"
FrameSaver: q=3.51685e+06 n=1823800 tag=gauss
wclsFrameSaver: saving 138252 traces tagged "wiener"
FrameSaver: q=3.77048e+06 n=1618491 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 44304 XUs and 40032 XVs -> 593 XUVs
C:0 T:1 129894 XUs and 104573 XVs -> 7519 XUVs
C:0 T:2 73365 XUs and 120665 XVs -> 5203 XUVs
C:0 T:3 31932 XUs and 54615 XVs -> 659 XUVs
C:0 T:4 12012 XUs and 16179 XVs -> 143 XUVs
C:0 T:5 56852 XUs and 72884 XVs -> 4129 XUVs
C:0 T:6 121892 XUs and 104844 XVs -> 13190 XUVs
C:0 T:7 32050 XUs and 35927 XVs -> 466 XUVs
C:0 T:8 9433 XUs and 11391 XVs -> 348 XUVs
C:0 T:9 143667 XUs and 123636 XVs -> 20174 XUVs
C:0 T:10 28256 XUs and 53991 XVs -> 5759 XUVs
C:0 T:11 50 XUs and 34 XVs -> 5 XUVs
58188 XUVs total
12821 collection wire objects
52277 potential space points
Neighbour search...
8450119 tests to find 3869472 neighbours
Iterating with no regularization...
Begin: 6.03523e+07
0 4.71031e+07
1 4.61396e+07
2 4.59847e+07
3 4.59383e+07
4 4.59172e+07
Now with regularization...
Begin: 3.72887e+07
0 3.72442e+07
1 3.72075e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 409168142
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 409168142
fLifetime: 30000 [us]
08-Sep-2025 22:37:13 UTC  Closed output file "reco_stage1.root"
08-Sep-2025 22:37:13 UTC  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          173.09        201.532       241.531       198.279       21.5681        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00072751    0.00317267    0.00559881    0.00317411    0.00141321       10     
reco:ophit:OpHitFinder                              0.1672       0.192655      0.216381      0.191285      0.0139813       10     
reco:opflash:OpFlashFinder                         0.0865864     0.112075      0.139916       0.11243      0.0145175       10     
reco:opslicer:OpSlicer                             0.359726      0.411456      0.472186      0.412724      0.0371785       10     
reco:caldata:DataPrepModule                         3.3593        3.51217       3.85438       3.47345      0.129616        10     
reco:wclsdatasp:WireCellToolkit                     22.3258       23.1593       24.5271       23.1462      0.576195        10     
reco:gaushit:GausHitFinder                         0.630548      0.758272      0.982613      0.753641      0.106636        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000295019   0.000522239   0.00178763    0.000348576   0.00043115       10     
reco:reco3d:SpacePointSolver                        5.8907        9.43671       14.2816       9.17271       2.74632        10     
reco:hitpdune:DisambigFromSpacePoints              0.775486       1.00132       1.22984      0.983178      0.142254        10     
reco:pandora:StandardPandora                        11.0538       18.6903       25.6843       17.9348       4.25531        10     
reco:pandoraWriter:StandardPandora                 0.212619      0.250643       0.29945      0.246368      0.025438        10     
reco:pandoraTrack:LArPandoraTrackCreation           1.32975       1.86154       2.59958       1.77918       0.34956        10     
reco:pandoraShower:LArPandoraShowerCreation         1.52814       2.1998        3.05217       2.08138      0.442215        10     
reco:pandoracalo:Calorimetry                       0.877829       1.17407       1.60861       1.13502      0.201981        10     
reco:pandoracalonosce:Calorimetry                  0.786319       1.07981       1.5022        1.03699      0.196115        10     
reco:pandorapid:Chi2ParticleID                    0.00255944    0.00331717    0.00432035    0.00329942    0.00046686       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0285076     0.0320188     0.037238      0.0314133    0.00287872       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0023724    0.00292502    0.00342067    0.00296526    0.000318258      10     
reco:pandoraShowercalo:ShowerCalorimetry            1.71957       2.53971       3.54024       2.40589      0.543163        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.66635       2.47783       3.46297       2.36095      0.536089        10     
reco:emtrkmichelid:EmTrackMichelId                  95.5416       108.431       125.629       105.944       9.66339        10     
reco:crttag:SingleCRTMatchingProducer              0.0077392     0.0104044     0.0185747    0.00912731    0.00322124       10     
reco:crtreco:TwoCRTMatchingProducer               0.00431383    0.00485668    0.00572646    0.00481959    0.000418298      10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.456232       0.58757      0.672984      0.597076      0.0658638       10     
reco:pandora2Track:LArPandoraTrackCreation          4.8983        6.35562       8.5359        6.14698       1.10413        10     
reco:pandora2calo:Calorimetry                       2.58033       3.55262       4.81748       3.4311       0.659265        10     
reco:pandora2calonosce:Calorimetry                  2.43153       3.38224       4.54697       3.27304      0.641186        10     
reco:pandora2pid:Chi2ParticleID                   0.00661527     0.0080467     0.0109964    0.00761001     0.0012416       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0342262     0.0415375     0.0590491     0.0395575     0.0065908       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00637159    0.00763068     0.0100888    0.00731294    0.00119854       10     
reco:pandora2Shower:LArPandoraShowerCreation        1.69432       2.34501       3.10969       2.32068      0.399182        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.2436        3.09925       4.07404       3.05885      0.545978        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.07721       2.85734       3.67904       2.82842      0.471968        10     
[art]:TriggerResults:TriggerResultInserter        4.1349e-05    6.26852e-05   0.00013692    5.1097e-05    2.97976e-05      10     
end_path:out1:RootOutput                           8.205e-06    1.25455e-05    2.109e-05    1.21125e-05   3.83598e-06      10     
end_path:out1:RootOutput(write)                     1.57058       1.89171       2.49508       1.7818       0.256975        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5101.31 MB
  Peak resident set size usage (VmHWM): 4277.12 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.15 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   2.15 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10000268
-rw-r--r-- 1 dunegli fnalgrid 2358390195 Sep  8 22:02 detsim_stage2.root
-rw-r--r-- 1 dunegli fnalgrid 2246310106 Sep  8 22:01 detsim_stage1.root
-rw-r--r-- 1 dunegli fnalgrid 2043913989 Sep  8 22:37 reco_stage1.root
-rw-r--r-- 1 dunegli fnalgrid 1792717364 Sep  8 20:52 g4_stage2.root
-rw-r--r-- 1 dunegli fnalgrid 1723606872 Sep  8 20:45 g4_stage1.root
-rw-r--r-- 1 dunegli fnalgrid   74711690 Sep  8 22:34 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid     242847 Sep  8 22:37 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid     217088 Sep  8 22:37 mem.db
-rw-r--r-- 1 dunegli fnalgrid      62376 Sep  8 22:37 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid      36864 Sep  8 22:37 time.db
-rw-r--r-- 1 dunegli fnalgrid        519 Sep  8 22:02 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dunegli fnalgrid        519 Sep  8 20:52 g4_protoDUNE_hist.root
-rw-r--r-- 1 dunegli fnalgrid        519 Sep  8 22:37 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dunegli fnalgrid         52 Sep  8 20:42 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid          7 Sep  8 22:37 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid          0 Sep  8 22:05 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Sep  8 22:02 debugprod.log
justIN time: 2025-09-18 20:54:34 UTC       justIN version: 01.05.00