justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 270292.0@dunegpschedd01.fnal.gov

Jobsub ID270292.0@dunegpschedd01.fnal.gov
Workflow ID11348
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-26 18:47:47
SiteUK_Bristol
EntryCMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat2025-12-27 00:31:44
From worker nodeHostnamehd71.dice.priv
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-26 18:49:19
Input filesmonte-carlo-011348-000001
JobscriptExit code0
Real time5h (20440s)
CPU time5h (19937s = 97%)
Max RSS bytes4382793728 (4179 MiB)
Outputting started2025-12-27 00:30:00
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11348/1/001/reco_stage1.root
Finished2025-12-27 00:31:44
Saved logsjustin-logs:270292.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

dcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 150631 traces, 0 summary
Retagger: tagging trace set: gauss with 112387 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 112387 traces tagged "gauss"
FrameSaver: q=3.77522e+06 n=1975021 tag=gauss
wclsFrameSaver: saving 150631 traces tagged "wiener"
FrameSaver: q=4.03692e+06 n=1734140 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 26299 XUs and 31593 XVs -> 338 XUVs
C:0 T:1 55633 XUs and 60971 XVs -> 4262 XUVs
C:0 T:2 141546 XUs and 176014 XVs -> 22701 XUVs
C:0 T:3 32671 XUs and 41499 XVs -> 635 XUVs
C:0 T:4 14197 XUs and 12705 XVs -> 94 XUVs
C:0 T:5 60941 XUs and 73218 XVs -> 5957 XUVs
C:0 T:6 111477 XUs and 205739 XVs -> 19636 XUVs
C:0 T:7 24515 XUs and 33140 XVs -> 410 XUVs
C:0 T:8 8042 XUs and 9082 XVs -> 84 XUVs
C:0 T:9 45026 XUs and 168978 XVs -> 7125 XUVs
C:0 T:10 128819 XUs and 87821 XVs -> 19234 XUVs
C:0 T:11 251 XUs and 230 XVs -> 16 XUVs
80492 XUVs total
14818 collection wire objects
74156 potential space points
Neighbour search...
15259994 tests to find 6697676 neighbours
Iterating with no regularization...
Begin: 9.04539e+07
0 6.05807e+07
1 5.91841e+07
2 5.89349e+07
3 5.88439e+07
4 5.87985e+07
Now with regularization...
Begin: 4.8607e+07
0 4.8509e+07
1 4.84403e+07
2 4.83863e+07
3 4.83408e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411359241
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411359241
fLifetime: 30000 [us]
27-Dec-2025 00:29:58 UTC  Closed output file "reco_stage1.root"
27-Dec-2025 00:29:58 UTC  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          330.157       572.498       881.083       559.107       135.666        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00147192    0.00397428    0.00781983    0.00391004    0.00188403       10     
reco:ophit:OpHitFinder                             0.305952      0.385042      0.501772      0.380821      0.0521581       10     
reco:opflash:OpFlashFinder                          0.19713      0.283713      0.423421      0.281051      0.0635744       10     
reco:opslicer:OpSlicer                             0.820448       1.14909       1.50533       1.15222      0.174271        10     
reco:caldata:DataPrepModule                         6.89394       7.14919       7.47392       7.13965      0.185029        10     
reco:wclsdatasp:WireCellToolkit                     42.6208       45.3498       50.8102       44.5554       2.64849        10     
reco:gaushit:GausHitFinder                          1.20464       1.97059       2.84922       1.93094      0.411293        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000496258   0.000683892   0.000875204   0.000664537   0.000120517      10     
reco:reco3d:SpacePointSolver                        8.2221        21.7938       38.1397       22.2697       7.24991        10     
reco:hitpdune:DisambigFromSpacePoints               1.34695       2.85847       4.52891       2.71753      0.813697        10     
reco:pandora:StandardPandora                        25.1685       66.2218       159.364       60.315        33.5686        10     
reco:pandoraWriter:StandardPandora                 0.399802      0.637215      0.869538      0.612692       0.1188         10     
reco:pandoraTrack:LArPandoraTrackCreation           2.99941       7.54973       13.2931       7.08737       2.7733         10     
reco:pandoraShower:LArPandoraShowerCreation         2.11091       8.59382       17.3855       8.05266       3.75012        10     
reco:pandoracalo:Calorimetry                        1.61698       4.05333       6.77294       4.02282       1.37996        10     
reco:pandoracalonosce:Calorimetry                   1.40953       3.73936       6.23138       3.71322       1.29884        10     
reco:pandorapid:Chi2ParticleID                    0.00640118    0.00929724     0.0119397    0.00900559    0.00163752       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0625188     0.0917875     0.113148      0.0940564     0.0139291       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00616511    0.00876238     0.0117145    0.00867246    0.00151707       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.00044       8.60411       17.0767       8.11091       3.63126        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.95218       8.25493        16.3         7.58312       3.5136         10     
reco:emtrkmichelid:EmTrackMichelId                  199.202       304.063       408.369       297.897       51.7247        10     
reco:crttag:SingleCRTMatchingProducer              0.0141114     0.0215408     0.0278932     0.0221144    0.00422438       10     
reco:crtreco:TwoCRTMatchingProducer               0.00829252     0.014726      0.0175046     0.0153124    0.00254987       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.933571       1.44405       1.69044       1.49264      0.212181        10     
reco:pandora2Track:LArPandoraTrackCreation          7.74749       24.1608       44.8297       22.8648       9.34889        10     
reco:pandora2calo:Calorimetry                       3.65636       11.6008       21.8657       10.9462       4.59826        10     
reco:pandora2calonosce:Calorimetry                  3.44886       10.8921       19.6873       10.4808       4.11789        10     
reco:pandora2pid:Chi2ParticleID                    0.0133199     0.0233551     0.0359045     0.0225921    0.00544124       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0747576     0.117047      0.164884      0.117999      0.0218648       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0132429     0.0229674     0.0348395     0.0222023    0.00521112       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.02602       8.87206       15.803        8.63038       3.19271        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.24583       9.62507       16.5466       9.45707       3.39972        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.98004       8.56304       14.2892       8.18729       2.86476        10     
[art]:TriggerResults:TriggerResultInserter        6.3691e-05    8.26373e-05   0.000167857   7.31985e-05   3.01855e-05      10     
end_path:out1:RootOutput                           8.746e-06    1.27331e-05   3.1871e-05    1.02445e-05   6.66067e-06      10     
end_path:out1:RootOutput(write)                     2.68139       4.23515       5.68422       4.18678      0.766083        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5187.42 MB
  Peak resident set size usage (VmHWM): 4382.79 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.26 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.67 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.14 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11043532
-rw-r--r-- 1 dnpil001 users 2561593643 Dec 26 22:51 detsim_stage2.root
-rw-r--r-- 1 dnpil001 users 2427752090 Dec 26 22:48 detsim_stage1.root
-rw-r--r-- 1 dnpil001 users 2280593192 Dec 27 00:29 reco_stage1.root
-rw-r--r-- 1 dnpil001 users 2020467509 Dec 26 19:18 g4_stage2.root
-rw-r--r-- 1 dnpil001 users 1934477401 Dec 26 18:58 g4_stage1.root
-rw-r--r-- 1 dnpil001 users   83073560 Dec 27 00:22 Pandora_Events.pndr
-rw-r--r-- 1 dnpil001 users     243114 Dec 27 00:29 jobscript.log
-rw-r--r-- 1 dnpil001 users     217088 Dec 27 00:29 mem.db
-rw-r--r-- 1 dnpil001 users      62376 Dec 27 00:29 Pandora_Geometry.xml
-rw-r--r-- 1 dnpil001 users      36864 Dec 27 00:29 time.db
-rw-r--r-- 1 dnpil001 users        519 Dec 26 22:51 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dnpil001 users        519 Dec 26 19:18 g4_protoDUNE_hist.root
-rw-r--r-- 1 dnpil001 users        519 Dec 27 00:29 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dnpil001 users         52 Dec 26 18:49 all-input-dids.txt
-rw-r--r-- 1 dnpil001 users          7 Dec 27 00:29 justin-processed-pfns.txt
-rw-r--r-- 1 dnpil001 users          0 Dec 26 23:00 ProtoDUNE.log
-rw-r--r-- 1 dnpil001 users          0 Dec 26 22:51 debugprod.log
justIN time: 2026-02-07 03:36:51 UTC       justIN version: 01.06.00