Jobsub ID 269342.1@dunegpschedd01.fnal.gov
| Jobsub ID | 269342.1@dunegpschedd01.fnal.gov |
| Workflow ID | 11291 |
| Stage ID | 1 |
| User name | pmelas@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 8388608000 (8000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2025-12-20 03:57:41 |
| Site | US_FNAL-FermiGrid |
| Entry | FNAL_GPGrid_ce04_mcore_op_duneonly |
| Last heartbeat | 2025-12-20 10:39:53 |
| From worker node | Hostname | dunegli-8113788-0-fnpc18003.fnal.gov |
| cpuinfo | AMD EPYC 7551 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 10485760000 (10000 MiB) |
| Wall seconds limit | 172800 (48 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-12-20 03:58:38 |
| Input files | monte-carlo-011291-000001
|
| Jobscript | Exit code | 0 |
| Real time | 3h (13258s) |
| CPU time | 3h (12978s = 97%) |
| Max RSS bytes | 4350070784 (4148 MiB) |
| Outputting started | 2025-12-20 07:39:37 |
| Output files | |
| Finished | 2025-12-20 10:39:53 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
oolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 139406 traces, 0 summary
Retagger: tagging trace set: gauss with 105649 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 105649 traces tagged "gauss"
FrameSaver: q=3.56298e+06 n=1896006 tag=gauss
wclsFrameSaver: saving 139406 traces tagged "wiener"
FrameSaver: q=3.77963e+06 n=1675909 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 31297 XUs and 41274 XVs -> 628 XUVs
C:0 T:1 65671 XUs and 74889 XVs -> 3938 XUVs
C:0 T:2 107317 XUs and 121905 XVs -> 7232 XUVs
C:0 T:3 53718 XUs and 47244 XVs -> 276 XUVs
C:0 T:4 14778 XUs and 11845 XVs -> 190 XUVs
C:0 T:5 87852 XUs and 66333 XVs -> 15326 XUVs
C:0 T:6 92245 XUs and 107782 XVs -> 3531 XUVs
C:0 T:7 39159 XUs and 42400 XVs -> 318 XUVs
C:0 T:8 15069 XUs and 8791 XVs -> 547 XUVs
C:0 T:9 310781 XUs and 143681 XVs -> 44446 XUVs
C:0 T:10 34542 XUs and 26570 XVs -> 5770 XUVs
C:0 T:11 337 XUs and 268 XVs -> 33 XUVs
82235 XUVs total
14145 collection wire objects
75507 potential space points
Neighbour search...
17424107 tests to find 7439252 neighbours
Iterating with no regularization...
Begin: 6.95662e+07
0 4.37606e+07
1 4.21315e+07
2 4.19164e+07
3 4.18391e+07
4 4.18014e+07
Now with regularization...
Begin: 3.27119e+07
0 3.26318e+07
1 3.25671e+07
2 3.25119e+07
3 3.24634e+07
4 3.2421e+07
5 3.23834e+07
6 3.23503e+07
7 3.23203e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411226013
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411226013
fLifetime: 30000 [us]
20-Dec-2025 07:39:36 UTC Closed output file "reco_stage1.root"
20-Dec-2025 07:39:36 UTC Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob: # events skipped: 0
====================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
====================================================================================================================================
Full event 339.66 402.989 507.288 387.866 47.5883 10
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000785657 0.0031627 0.00538996 0.00335515 0.00141572 10
reco:ophit:OpHitFinder 0.295736 0.32807 0.354676 0.324795 0.0215995 10
reco:opflash:OpFlashFinder 0.188622 0.235263 0.296021 0.225365 0.0292624 10
reco:opslicer:OpSlicer 0.797601 0.858857 0.969247 0.851 0.0494656 10
reco:caldata:DataPrepModule 5.34805 5.5138 5.80462 5.50818 0.135774 10
reco:wclsdatasp:WireCellToolkit 31.8976 36.2337 39.1056 37.3112 2.54893 10
reco:gaushit:GausHitFinder 1.12574 1.36041 1.64245 1.33451 0.149836 10
reco:nhitsfilter:NumberOfHitsFilter 0.000522989 0.000576827 0.000665258 0.000550472 5.22738e-05 10
reco:reco3d:SpacePointSolver 13.3832 18.304 37.8564 16.0242 6.9633 10
reco:hitpdune:DisambigFromSpacePoints 1.599 2.13428 3.2587 1.94527 0.479425 10
reco:pandora:StandardPandora 27.8009 42.1518 53.5684 42.3367 7.55676 10
reco:pandoraWriter:StandardPandora 0.420626 0.610443 1.05653 0.504895 0.223927 10
reco:pandoraTrack:LArPandoraTrackCreation 4.93729 5.95295 7.7128 5.31055 1.06586 10
reco:pandoraShower:LArPandoraShowerCreation 5.86141 7.6021 11.2543 6.94813 1.59948 10
reco:pandoracalo:Calorimetry 2.31148 2.89318 3.94262 2.72997 0.523735 10
reco:pandoracalonosce:Calorimetry 2.12449 2.61128 3.4272 2.48866 0.398247 10
reco:pandorapid:Chi2ParticleID 0.00576432 0.00703333 0.0092788 0.00697991 0.000998907 10
reco:pandoracalinoxyzt:CalibrationdEdXPDSP 0.052058 0.0568911 0.069221 0.0542228 0.00552812 10
reco:pandoracalipidnoxyzt:Chi2ParticleID 0.0056897 0.00626268 0.00727533 0.00604343 0.000608963 10
reco:pandoraShowercalo:ShowerCalorimetry 5.60674 7.23643 10.7748 6.66885 1.53447 10
reco:pandoraShowercalonosce:ShowerCalorimetry 5.66227 6.90143 10.4258 6.46506 1.39788 10
reco:emtrkmichelid:EmTrackMichelId 171.406 195.961 241.79 188.904 20.4794 10
reco:crttag:SingleCRTMatchingProducer 0.0136698 0.018712 0.0269049 0.0178031 0.00363894 10
reco:crtreco:TwoCRTMatchingProducer 0.00981588 0.0115995 0.0149071 0.0110326 0.001568 10
reco:anodepiercerst0:T0RecoAnodePiercers 0.825163 1.01886 1.37012 0.999189 0.161196 10
reco:pandora2Track:LArPandoraTrackCreation 17.2222 21.3838 30.0201 19.4166 4.00586 10
reco:pandora2calo:Calorimetry 7.62921 9.42863 13.7222 8.47387 1.83979 10
reco:pandora2calonosce:Calorimetry 7.07192 8.9674 13.3061 8.41074 1.86475 10
reco:pandora2pid:Chi2ParticleID 0.0163733 0.0186002 0.0217362 0.0187624 0.00176754 10
reco:pandora2calinoxyzt:CalibrationdEdXPDSP 0.0674245 0.074703 0.0893576 0.0721425 0.00641975 10
reco:pandora2calipidnoxyzt:Chi2ParticleID 0.0151865 0.0170689 0.0196372 0.0164947 0.00151915 10
reco:pandora2Shower:LArPandoraShowerCreation 6.16175 7.09712 9.93894 6.70798 1.11967 10
reco:pandora2Showercalo:ShowerCalorimetry 6.10684 7.6044 11.5079 7.04373 1.65672 10
reco:pandora2Showercalonosce:ShowerCalorimetry 4.83129 6.40475 10.2978 5.80699 1.61984 10
[art]:TriggerResults:TriggerResultInserter 3.5337e-05 5.34312e-05 0.000108947 4.50805e-05 2.11285e-05 10
end_path:out1:RootOutput 6.883e-06 7.8541e-06 1.2073e-05 7.3795e-06 1.4623e-06 10
end_path:out1:RootOutput(write) 3.14577 3.86188 4.68253 3.86043 0.530396 10
====================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5171.81 MB
Peak resident set size usage (VmHWM): 4350.07 MB
Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor: Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor: digitReader: 0.07 sec/event
ToolBasedRawDigitPrepService:dtor: adcSampleFiller: 0.21 sec/event
ToolBasedRawDigitPrepService:dtor: adcScaleAdcToKe: 0.05 sec/event
ToolBasedRawDigitPrepService:dtor: pdspTailPedRemovalZKe: 3.56 sec/event
ToolBasedRawDigitPrepService:dtor: adcKeepAllSignalFinder: 0.11 sec/event
ToolBasedRawDigitPrepService:dtor: adcScaleKeToAdc: 0.05 sec/event
ToolBasedRawDigitPrepService:dtor: pdsp_RemoveBadChannels: 0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11008968
-rw-r--r-- 1 dunegli fnalgrid 2547252156 Dec 20 06:30 detsim_stage2.root
-rw-r--r-- 1 dunegli fnalgrid 2408723943 Dec 20 06:27 detsim_stage1.root
-rw-r--r-- 1 dunegli fnalgrid 2277519904 Dec 20 07:39 reco_stage1.root
-rw-r--r-- 1 dunegli fnalgrid 2010033468 Dec 20 04:20 g4_stage2.root
-rw-r--r-- 1 dunegli fnalgrid 1943927516 Dec 20 04:04 g4_stage1.root
-rw-r--r-- 1 dunegli fnalgrid 85102400 Dec 20 07:34 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid 242891 Dec 20 07:39 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid 217088 Dec 20 07:39 mem.db
-rw-r--r-- 1 dunegli fnalgrid 62376 Dec 20 07:39 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid 36864 Dec 20 07:39 time.db
-rw-r--r-- 1 dunegli fnalgrid 519 Dec 20 06:30 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dunegli fnalgrid 519 Dec 20 04:20 g4_protoDUNE_hist.root
-rw-r--r-- 1 dunegli fnalgrid 519 Dec 20 07:39 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dunegli fnalgrid 52 Dec 20 03:58 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid 7 Dec 20 07:39 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid 0 Dec 20 06:35 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid 0 Dec 20 06:30 debugprod.log
