Jobsub ID 264193.0@dunegpschedd02.fnal.gov
| Jobsub ID | 264193.0@dunegpschedd02.fnal.gov |
| Workflow ID | 11332 |
| Stage ID | 1 |
| User name | pmelas@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 8388608000 (8000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2025-12-24 21:41:30 |
| Site | ES_PIC |
| Entry | DUNE_T1_ES_PIC_ce15-multicore |
| Last heartbeat | 2025-12-25 03:08:01 |
| From worker node | Hostname | tds441.pic.es |
| cpuinfo | AMD EPYC 7502 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 10485760000 (10000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-12-24 21:45:24 |
| Input files | monte-carlo-011332-000001
|
| Jobscript | Exit code | 0 |
| Real time | 5h (19328s) |
| CPU time | 5h (18515s = 95%) |
| Max RSS bytes | 4425936896 (4220 MiB) |
| Outputting started | 2025-12-25 03:07:33 |
| Output files | |
| Finished | 2025-12-25 03:08:01 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
ol adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare: Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare: Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 136768 traces, 0 summary
Retagger: tagging trace set: gauss with 100524 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 100524 traces tagged "gauss"
FrameSaver: q=3.39609e+06 n=1703581 tag=gauss
wclsFrameSaver: saving 136768 traces tagged "wiener"
FrameSaver: q=3.61735e+06 n=1482551 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 38242 XUs and 34983 XVs -> 954 XUVs
C:0 T:1 88598 XUs and 97345 XVs -> 15991 XUVs
C:0 T:2 50783 XUs and 78494 XVs -> 3381 XUVs
C:0 T:3 26414 XUs and 32024 XVs -> 174 XUVs
C:0 T:4 12118 XUs and 8442 XVs -> 127 XUVs
C:0 T:5 49141 XUs and 31285 XVs -> 4015 XUVs
C:0 T:6 132437 XUs and 145059 XVs -> 8307 XUVs
C:0 T:7 40193 XUs and 49855 XVs -> 639 XUVs
C:0 T:8 8032 XUs and 5965 XVs -> 85 XUVs
C:0 T:9 46622 XUs and 45928 XVs -> 3521 XUVs
C:0 T:10 46224 XUs and 29541 XVs -> 6053 XUVs
C:0 T:11 1844 XUs and 1842 XVs -> 349 XUVs
43596 XUVs total
13018 collection wire objects
39135 potential space points
Neighbour search...
3583967 tests to find 1929196 neighbours
Iterating with no regularization...
Begin: 5.68035e+07
0 4.81178e+07
1 4.74448e+07
2 4.73327e+07
3 4.7295e+07
Now with regularization...
Begin: 3.7667e+07
0 3.76202e+07
1 3.75892e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411320504
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411320504
fLifetime: 30000 [us]
25-Dec-2025 04:07:30 CET Closed output file "reco_stage1.root"
25-Dec-2025 04:07:30 CET Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob: # events skipped: 0
====================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
====================================================================================================================================
Full event 546.275 707.576 876.143 687.306 107.909 10
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00144947 0.00478109 0.00780033 0.00542406 0.00205816 10
reco:ophit:OpHitFinder 0.310076 0.430327 0.478599 0.434234 0.0472411 10
reco:opflash:OpFlashFinder 0.235965 0.276059 0.34945 0.26996 0.0334685 10
reco:opslicer:OpSlicer 0.727399 1.15565 1.46258 1.18916 0.206258 10
reco:caldata:DataPrepModule 6.60424 7.75879 8.24698 7.89809 0.481419 10
reco:wclsdatasp:WireCellToolkit 40.2049 51.4803 56.4029 52.0959 4.67348 10
reco:gaushit:GausHitFinder 1.24571 2.16213 2.48677 2.36341 0.381503 10
reco:nhitsfilter:NumberOfHitsFilter 0.000648767 0.00097792 0.00121836 0.0010187 0.000154824 10
reco:reco3d:SpacePointSolver 13.992 27.9787 40.9825 28.8951 8.01548 10
reco:hitpdune:DisambigFromSpacePoints 2.0261 3.01126 3.93287 3.12583 0.640323 10
reco:pandora:StandardPandora 37.9042 71.2907 125.152 63.5725 23.2709 10
reco:pandoraWriter:StandardPandora 0.498403 0.736308 0.963419 0.690966 0.139785 10
reco:pandoraTrack:LArPandoraTrackCreation 5.84146 10.3482 15.8395 10.085 3.03152 10
reco:pandoraShower:LArPandoraShowerCreation 7.64806 11.5485 16.7374 11.0406 2.70433 10
reco:pandoracalo:Calorimetry 3.4206 5.68558 8.44656 5.84274 1.82907 10
reco:pandoracalonosce:Calorimetry 2.84977 5.22762 7.22689 4.97031 1.43333 10
reco:pandorapid:Chi2ParticleID 0.00561578 0.0110589 0.0133322 0.0122448 0.00252479 10
reco:pandoracalinoxyzt:CalibrationdEdXPDSP 0.0549432 0.111434 0.181382 0.113026 0.0340915 10
reco:pandoracalipidnoxyzt:Chi2ParticleID 0.00551133 0.0109811 0.0175534 0.0108968 0.00315106 10
reco:pandoraShowercalo:ShowerCalorimetry 5.98236 11.926 15.2342 11.9315 2.47851 10
reco:pandoraShowercalonosce:ShowerCalorimetry 8.26676 11.1987 14.6831 10.4283 2.14353 10
reco:emtrkmichelid:EmTrackMichelId 305.76 380.812 453.408 388.624 47.6863 10
reco:crttag:SingleCRTMatchingProducer 0.0167006 0.0318455 0.0532792 0.0303012 0.0100078 10
reco:crtreco:TwoCRTMatchingProducer 0.0095394 0.0208684 0.0282146 0.0224036 0.00534184 10
reco:anodepiercerst0:T0RecoAnodePiercers 0.95502 1.41364 1.6104 1.46005 0.183324 10
reco:pandora2Track:LArPandoraTrackCreation 19.4858 31.4825 44.0818 32.0949 8.07284 10
reco:pandora2calo:Calorimetry 7.98047 15.6449 24.3845 14.5588 4.60861 10
reco:pandora2calonosce:Calorimetry 9.61162 14.8273 24.6647 12.9628 4.56317 10
reco:pandora2pid:Chi2ParticleID 0.0188241 0.0271387 0.0391166 0.0264481 0.00558209 10
reco:pandora2calinoxyzt:CalibrationdEdXPDSP 0.0887626 0.133512 0.172883 0.133349 0.0251339 10
reco:pandora2calipidnoxyzt:Chi2ParticleID 0.018639 0.0258484 0.0341626 0.0254864 0.00460749 10
reco:pandora2Shower:LArPandoraShowerCreation 7.28027 11.3347 16.5074 11.3046 2.81886 10
reco:pandora2Showercalo:ShowerCalorimetry 7.38948 13.1682 19.867 12.9657 4.11055 10
reco:pandora2Showercalonosce:ShowerCalorimetry 7.19613 11.2545 15.1694 10.8629 2.66499 10
[art]:TriggerResults:TriggerResultInserter 5.1021e-05 8.08642e-05 0.000168815 7.5482e-05 3.08725e-05 10
end_path:out1:RootOutput 1.0861e-05 1.38906e-05 2.062e-05 1.3816e-05 2.52124e-06 10
end_path:out1:RootOutput(write) 3.60224 4.82697 6.01345 4.82605 0.685962 10
====================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5262.74 MB
Peak resident set size usage (VmHWM): 4425.94 MB
Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor: Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor: digitReader: 0.08 sec/event
ToolBasedRawDigitPrepService:dtor: adcSampleFiller: 0.32 sec/event
ToolBasedRawDigitPrepService:dtor: adcScaleAdcToKe: 0.06 sec/event
ToolBasedRawDigitPrepService:dtor: pdspTailPedRemovalZKe: 4.97 sec/event
ToolBasedRawDigitPrepService:dtor: adcKeepAllSignalFinder: 0.19 sec/event
ToolBasedRawDigitPrepService:dtor: adcScaleKeToAdc: 0.06 sec/event
ToolBasedRawDigitPrepService:dtor: pdsp_RemoveBadChannels: 0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11107864
-rw-r--r-- 1 dune001 dune 2580221334 Dec 25 02:05 detsim_stage2.root
-rw-r--r-- 1 dune001 dune 2448449220 Dec 25 02:02 detsim_stage1.root
-rw-r--r-- 1 dune001 dune 2315220485 Dec 25 04:07 reco_stage1.root
-rw-r--r-- 1 dune001 dune 2011778319 Dec 24 23:16 g4_stage2.root
-rw-r--r-- 1 dune001 dune 1931329032 Dec 24 22:56 g4_stage1.root
-rw-r--r-- 1 dune001 dune 86828080 Dec 25 03:59 Pandora_Events.pndr
-rw-r--r-- 1 dune001 dune 243099 Dec 25 04:07 jobscript.log
-rw-r--r-- 1 dune001 dune 217088 Dec 25 04:07 mem.db
-rw-r--r-- 1 dune001 dune 62376 Dec 25 04:07 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 dune 36864 Dec 25 04:07 time.db
-rw-r--r-- 1 dune001 dune 519 Dec 25 02:05 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune 519 Dec 24 23:16 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune 519 Dec 25 04:07 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune001 dune 52 Dec 24 22:45 all-input-dids.txt
-rw-r--r-- 1 dune001 dune 7 Dec 25 04:07 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune 0 Dec 25 02:16 ProtoDUNE.log
-rw-r--r-- 1 dune001 dune 0 Dec 25 02:06 debugprod.log
