Jobsub ID 265804.0@dunegpschedd02.fnal.gov
| Jobsub ID | 265804.0@dunegpschedd02.fnal.gov |
| Workflow ID | 11507 |
| Stage ID | 1 |
| User name | pmelas@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 8388608000 (8000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-01-04 09:48:54 |
| Site | CZ_FZU |
| Entry | Nova_CZ_FZU_osgce2 |
| Last heartbeat | 2026-01-04 15:50:02 |
| From worker node | Hostname | milan29.farm.particle.cz |
| cpuinfo | AMD EPYC 7543 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 10485760000 (10000 MiB) |
| Wall seconds limit | 172800 (48 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2026-01-04 09:49:24 |
| Input files | monte-carlo-011507-000001
|
| Jobscript | Exit code | 0 |
| Real time | 3h (13508s) |
| CPU time | 3h (13110s = 97%) |
| Max RSS bytes | 4250796032 (4053 MiB) |
| Outputting started | 2026-01-04 13:34:33 |
| Output files | |
| Finished | 2026-01-04 15:50:02 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
ToolBasedRawDigitPrepService:prepare: Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare: Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 131469 traces, 0 summary
Retagger: tagging trace set: gauss with 98897 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 98897 traces tagged "gauss"
FrameSaver: q=3.5509e+06 n=1742141 tag=gauss
wclsFrameSaver: saving 131469 traces tagged "wiener"
FrameSaver: q=3.77851e+06 n=1547502 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 41150 XUs and 41939 XVs -> 896 XUVs
C:0 T:1 90848 XUs and 70239 XVs -> 7510 XUVs
C:0 T:2 162592 XUs and 157950 XVs -> 25734 XUVs
C:0 T:3 51695 XUs and 43836 XVs -> 581 XUVs
C:0 T:4 7479 XUs and 6896 XVs -> 51 XUVs
C:0 T:5 20762 XUs and 16030 XVs -> 1579 XUVs
C:0 T:6 86708 XUs and 58931 XVs -> 3037 XUVs
C:0 T:7 37783 XUs and 23061 XVs -> 138 XUVs
C:0 T:8 15431 XUs and 15971 XVs -> 232 XUVs
C:0 T:9 126210 XUs and 144920 XVs -> 12347 XUVs
C:0 T:10 9340 XUs and 3637 XVs -> 1240 XUVs
C:0 T:11 30 XUs and 29 XVs -> 5 XUVs
53350 XUVs total
11895 collection wire objects
47698 potential space points
Neighbour search...
8112956 tests to find 3466058 neighbours
Iterating with no regularization...
Begin: 7.2631e+07
0 5.90209e+07
1 5.80546e+07
2 5.79014e+07
3 5.78544e+07
Now with regularization...
Begin: 4.7151e+07
0 4.70801e+07
1 4.70388e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411532806
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411532806
fLifetime: 30000 [us]
04-Jan-2026 14:34:31 CET Closed output file "reco_stage1.root"
04-Jan-2026 14:34:31 CET Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob: # events skipped: 0
====================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
====================================================================================================================================
Full event 324.948 422.313 577.217 408.205 71.951 10
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000841486 0.00391191 0.0066923 0.00404704 0.00170168 10
reco:ophit:OpHitFinder 0.270678 0.347513 0.413115 0.358472 0.0397026 10
reco:opflash:OpFlashFinder 0.149127 0.180517 0.224998 0.180588 0.0191489 10
reco:opslicer:OpSlicer 0.601251 0.835785 1.23874 0.829729 0.165531 10
reco:caldata:DataPrepModule 4.68523 6.79706 7.57195 6.95665 0.750679 10
reco:wclsdatasp:WireCellToolkit 33.1879 39.0112 42.9277 38.8898 2.50879 10
reco:gaushit:GausHitFinder 1.24235 1.57176 1.87141 1.55942 0.200617 10
reco:nhitsfilter:NumberOfHitsFilter 0.000377609 0.000631638 0.000899976 0.000630702 0.000133794 10
reco:reco3d:SpacePointSolver 14.2276 25.5753 44.2197 26.2753 8.87229 10
reco:hitpdune:DisambigFromSpacePoints 1.53769 2.17118 3.10839 2.09051 0.499057 10
reco:pandora:StandardPandora 24.8051 46.8234 97.2002 39.8495 20.0982 10
reco:pandoraWriter:StandardPandora 0.493723 0.57439 0.745871 0.558291 0.0717983 10
reco:pandoraTrack:LArPandoraTrackCreation 2.83116 3.89506 6.39275 3.57929 1.04111 10
reco:pandoraShower:LArPandoraShowerCreation 2.99999 4.75089 7.0654 4.54755 1.33669 10
reco:pandoracalo:Calorimetry 1.75065 2.40069 3.41312 2.22512 0.539341 10
reco:pandoracalonosce:Calorimetry 1.58258 2.12706 3.19272 1.99385 0.505837 10
reco:pandorapid:Chi2ParticleID 0.00517565 0.00703402 0.00890318 0.00703373 0.00104833 10
reco:pandoracalinoxyzt:CalibrationdEdXPDSP 0.0637748 0.0757948 0.100822 0.0719665 0.010301 10
reco:pandoracalipidnoxyzt:Chi2ParticleID 0.00389711 0.00631596 0.00893369 0.00637747 0.00123799 10
reco:pandoraShowercalo:ShowerCalorimetry 3.21244 5.18214 7.79864 4.99396 1.47561 10
reco:pandoraShowercalonosce:ShowerCalorimetry 3.10771 4.98366 7.64534 4.70114 1.47379 10
reco:emtrkmichelid:EmTrackMichelId 185.172 223.602 273.242 217.659 26.6886 10
reco:crttag:SingleCRTMatchingProducer 0.0173456 0.0247353 0.0359744 0.0247064 0.00619229 10
reco:crtreco:TwoCRTMatchingProducer 0.00773355 0.0114055 0.0222272 0.0095476 0.00416549 10
reco:anodepiercerst0:T0RecoAnodePiercers 0.803544 1.0511 1.37486 1.02706 0.151456 10
reco:pandora2Track:LArPandoraTrackCreation 9.01605 14.377 21.0032 13.1386 3.92076 10
reco:pandora2calo:Calorimetry 4.82839 7.48232 10.3174 6.94827 1.85443 10
reco:pandora2calonosce:Calorimetry 4.68529 6.97204 9.87935 6.40641 1.73295 10
reco:pandora2pid:Chi2ParticleID 0.0145327 0.0175227 0.0227796 0.0168903 0.00247664 10
reco:pandora2calinoxyzt:CalibrationdEdXPDSP 0.082954 0.099988 0.114685 0.102179 0.0112078 10
reco:pandora2calipidnoxyzt:Chi2ParticleID 0.014681 0.0174905 0.0209359 0.0175594 0.00204705 10
reco:pandora2Shower:LArPandoraShowerCreation 3.39889 5.13403 7.14978 4.72845 1.25534 10
reco:pandora2Showercalo:ShowerCalorimetry 4.24718 6.35409 9.14396 5.89274 1.57548 10
reco:pandora2Showercalonosce:ShowerCalorimetry 4.01262 5.98596 8.34681 5.49288 1.50416 10
[art]:TriggerResults:TriggerResultInserter 5.523e-05 7.91986e-05 0.00018111 6.97895e-05 3.46155e-05 10
end_path:out1:RootOutput 9.06e-06 1.0943e-05 1.845e-05 1.0125e-05 2.58475e-06 10
end_path:out1:RootOutput(write) 2.96087 3.74272 4.66543 3.61069 0.53573 10
====================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5078.59 MB
Peak resident set size usage (VmHWM): 4250.8 MB
Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor: Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor: digitReader: 0.09 sec/event
ToolBasedRawDigitPrepService:dtor: adcSampleFiller: 0.31 sec/event
ToolBasedRawDigitPrepService:dtor: adcScaleAdcToKe: 0.05 sec/event
ToolBasedRawDigitPrepService:dtor: pdspTailPedRemovalZKe: 4.25 sec/event
ToolBasedRawDigitPrepService:dtor: adcKeepAllSignalFinder: 0.18 sec/event
ToolBasedRawDigitPrepService:dtor: adcScaleKeToAdc: 0.06 sec/event
ToolBasedRawDigitPrepService:dtor: pdsp_RemoveBadChannels: 0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10164328
-rw-r--r-- 1 fermilab users 2388480523 Jan 4 13:22 detsim_stage2.root
-rw-r--r-- 1 fermilab users 2268591153 Jan 4 13:20 detsim_stage1.root
-rw-r--r-- 1 fermilab users 2099195453 Jan 4 14:34 reco_stage1.root
-rw-r--r-- 1 fermilab users 1823265121 Jan 4 11:11 g4_stage2.root
-rw-r--r-- 1 fermilab users 1748179944 Jan 4 10:56 g4_stage1.root
-rw-r--r-- 1 fermilab users 79927560 Jan 4 14:29 Pandora_Events.pndr
-rw-r--r-- 1 fermilab users 242938 Jan 4 14:34 jobscript.log
-rw-r--r-- 1 fermilab users 217088 Jan 4 14:34 mem.db
-rw-r--r-- 1 fermilab users 62376 Jan 4 14:34 Pandora_Geometry.xml
-rw-r--r-- 1 fermilab users 36864 Jan 4 14:34 time.db
-rw-r--r-- 1 fermilab users 519 Jan 4 13:22 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 fermilab users 519 Jan 4 11:11 g4_protoDUNE_hist.root
-rw-r--r-- 1 fermilab users 519 Jan 4 14:34 protoDUNE_reco1_hist.root
-rw-r--r-- 1 fermilab users 52 Jan 4 10:49 all-input-dids.txt
-rw-r--r-- 1 fermilab users 7 Jan 4 14:34 justin-processed-pfns.txt
-rw-r--r-- 1 fermilab users 0 Jan 4 13:28 ProtoDUNE.log
-rw-r--r-- 1 fermilab users 0 Jan 4 13:22 debugprod.log
