Jobsub ID 271459.1@dunegpschedd01.fnal.gov
| Jobsub ID | 271459.1@dunegpschedd01.fnal.gov |
| Workflow ID | 11493 |
| Stage ID | 1 |
| User name | pmelas@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 8388608000 (8000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2026-01-03 19:22:08 |
| Site | CZ_FZU |
| Entry | Nova_CZ_FZU_osgce2 |
| Last heartbeat | 2026-01-04 01:21:38 |
| From worker node | Hostname | milan29.farm.particle.cz |
| cpuinfo | AMD EPYC 7543 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 10485760000 (10000 MiB) |
| Wall seconds limit | 172800 (48 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2026-01-03 19:23:05 |
| Input files | monte-carlo-011493-000001
|
| Jobscript | Exit code | 0 |
| Real time | 3h (13414s) |
| CPU time | 3h (12958s = 96%) |
| Max RSS bytes | 4135284736 (3943 MiB) |
| Outputting started | 2026-01-03 23:06:39 |
| Output files | |
| Finished | 2026-01-04 01:21:38 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
ice:prepare: Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare: Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 143150 traces, 0 summary
Retagger: tagging trace set: gauss with 104751 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 104751 traces tagged "gauss"
FrameSaver: q=3.53961e+06 n=1829893 tag=gauss
wclsFrameSaver: saving 143150 traces tagged "wiener"
FrameSaver: q=3.77551e+06 n=1600666 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 38401 XUs and 23957 XVs -> 225 XUVs
C:0 T:1 85100 XUs and 60914 XVs -> 4913 XUVs
C:0 T:2 61296 XUs and 190161 XVs -> 13782 XUVs
C:0 T:3 24342 XUs and 51063 XVs -> 362 XUVs
C:0 T:4 18764 XUs and 16649 XVs -> 160 XUVs
C:0 T:5 85226 XUs and 57776 XVs -> 6199 XUVs
C:0 T:6 114011 XUs and 146590 XVs -> 6794 XUVs
C:0 T:7 43494 XUs and 49390 XVs -> 323 XUVs
C:0 T:8 11684 XUs and 6659 XVs -> 102 XUVs
C:0 T:9 95169 XUs and 69403 XVs -> 8879 XUVs
C:0 T:10 18191 XUs and 11189 XVs -> 2496 XUVs
C:0 T:11 450 XUs and 254 XVs -> 54 XUVs
44289 XUVs total
15094 collection wire objects
40865 potential space points
Neighbour search...
4650535 tests to find 2144398 neighbours
Iterating with no regularization...
Begin: 5.44426e+07
0 4.23856e+07
1 4.14289e+07
2 4.12719e+07
3 4.12252e+07
4 4.12056e+07
Now with regularization...
Begin: 3.14535e+07
0 3.14094e+07
1 3.13819e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411520585
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411520585
fLifetime: 30000 [us]
04-Jan-2026 00:06:36 CET Closed output file "reco_stage1.root"
04-Jan-2026 00:06:36 CET Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob: # events skipped: 0
====================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
====================================================================================================================================
Full event 291.966 423.055 549.212 433.099 88.0041 10
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00110582 0.0033695 0.00462365 0.00363069 0.00107404 10
reco:ophit:OpHitFinder 0.343311 0.384871 0.457894 0.377879 0.0331246 10
reco:opflash:OpFlashFinder 0.163825 0.205792 0.260171 0.202344 0.0286031 10
reco:opslicer:OpSlicer 0.760367 0.921938 1.13347 0.903234 0.106882 10
reco:caldata:DataPrepModule 6.65398 7.28064 7.75302 7.34071 0.328878 10
reco:wclsdatasp:WireCellToolkit 39.2257 41.6875 47.1183 40.9935 2.47742 10
reco:gaushit:GausHitFinder 1.23888 1.66072 2.2855 1.66356 0.321179 10
reco:nhitsfilter:NumberOfHitsFilter 0.000483599 0.000658456 0.000738657 0.000689193 8.06162e-05 10
reco:reco3d:SpacePointSolver 10.074 22.3851 39.8645 20.1036 9.89483 10
reco:hitpdune:DisambigFromSpacePoints 1.24142 2.16375 3.46116 2.18413 0.695789 10
reco:pandora:StandardPandora 21.0858 41.9116 70.6775 43.3318 16.1745 10
reco:pandoraWriter:StandardPandora 0.352108 0.604823 0.787795 0.618302 0.13684 10
reco:pandoraTrack:LArPandoraTrackCreation 2.55187 4.22739 5.94659 4.30378 1.20027 10
reco:pandoraShower:LArPandoraShowerCreation 2.55433 4.81479 7.82409 4.78821 1.83289 10
reco:pandoracalo:Calorimetry 1.59628 2.62698 3.5514 2.74065 0.695537 10
reco:pandoracalonosce:Calorimetry 1.38962 2.37942 3.20823 2.5832 0.641626 10
reco:pandorapid:Chi2ParticleID 0.00686815 0.00782397 0.00903141 0.0079275 0.000682364 10
reco:pandoracalinoxyzt:CalibrationdEdXPDSP 0.0682832 0.0822443 0.0949336 0.0852368 0.00927142 10
reco:pandoracalipidnoxyzt:Chi2ParticleID 0.00620779 0.0076849 0.00865996 0.0079984 0.000825561 10
reco:pandoraShowercalo:ShowerCalorimetry 2.51493 5.24397 8.55702 5.46896 2.02549 10
reco:pandoraShowercalonosce:ShowerCalorimetry 2.49501 5.05352 8.27814 5.2454 1.94363 10
reco:emtrkmichelid:EmTrackMichelId 168.311 227.216 279.85 229.832 36.8985 10
reco:crttag:SingleCRTMatchingProducer 0.0108452 0.0164238 0.0233571 0.0164874 0.00373074 10
reco:crtreco:TwoCRTMatchingProducer 0.00696252 0.00954393 0.0114531 0.00960533 0.00140624 10
reco:anodepiercerst0:T0RecoAnodePiercers 0.81418 1.07837 1.55923 1.04618 0.213852 10
reco:pandora2Track:LArPandoraTrackCreation 7.8519 14.5635 23.4572 15.2393 5.10642 10
reco:pandora2calo:Calorimetry 3.93878 7.50256 11.1434 8.23826 2.4887 10
reco:pandora2calonosce:Calorimetry 3.69541 7.0839 10.5622 7.74739 2.4118 10
reco:pandora2pid:Chi2ParticleID 0.014942 0.0197455 0.0254908 0.0195008 0.00348787 10
reco:pandora2calinoxyzt:CalibrationdEdXPDSP 0.0784876 0.10017 0.11773 0.10499 0.0139166 10
reco:pandora2calipidnoxyzt:Chi2ParticleID 0.0124435 0.0188072 0.0256972 0.0195154 0.003864 10
reco:pandora2Shower:LArPandoraShowerCreation 2.82166 5.24574 7.77093 5.67674 1.70673 10
reco:pandora2Showercalo:ShowerCalorimetry 3.41356 6.57872 9.54113 7.21511 2.17739 10
reco:pandora2Showercalonosce:ShowerCalorimetry 3.05095 5.97852 8.52043 6.58211 1.93204 10
[art]:TriggerResults:TriggerResultInserter 5.9939e-05 9.45995e-05 0.000193859 7.80545e-05 3.94819e-05 10
end_path:out1:RootOutput 8.99e-06 1.5686e-05 3.597e-05 1.0705e-05 8.48502e-06 10
end_path:out1:RootOutput(write) 2.65665 3.88135 4.76468 4.07857 0.703412 10
====================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4965.07 MB
Peak resident set size usage (VmHWM): 4135.28 MB
Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor: Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor: digitReader: 0.10 sec/event
ToolBasedRawDigitPrepService:dtor: adcSampleFiller: 0.33 sec/event
ToolBasedRawDigitPrepService:dtor: adcScaleAdcToKe: 0.05 sec/event
ToolBasedRawDigitPrepService:dtor: pdspTailPedRemovalZKe: 4.55 sec/event
ToolBasedRawDigitPrepService:dtor: adcKeepAllSignalFinder: 0.20 sec/event
ToolBasedRawDigitPrepService:dtor: adcScaleKeToAdc: 0.07 sec/event
ToolBasedRawDigitPrepService:dtor: pdsp_RemoveBadChannels: 0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10188632
-rw-r--r-- 1 fermilab users 2387676887 Jan 3 22:53 detsim_stage2.root
-rw-r--r-- 1 fermilab users 2264440836 Jan 3 22:51 detsim_stage1.root
-rw-r--r-- 1 fermilab users 2086204193 Jan 4 00:06 reco_stage1.root
-rw-r--r-- 1 fermilab users 1846151073 Jan 3 20:44 g4_stage2.root
-rw-r--r-- 1 fermilab users 1770814519 Jan 3 20:30 g4_stage1.root
-rw-r--r-- 1 fermilab users 77239930 Jan 4 00:01 Pandora_Events.pndr
-rw-r--r-- 1 fermilab users 242830 Jan 4 00:06 jobscript.log
-rw-r--r-- 1 fermilab users 217088 Jan 4 00:06 mem.db
-rw-r--r-- 1 fermilab users 62376 Jan 4 00:06 Pandora_Geometry.xml
-rw-r--r-- 1 fermilab users 36864 Jan 4 00:06 time.db
-rw-r--r-- 1 fermilab users 519 Jan 3 22:53 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 fermilab users 519 Jan 3 20:44 g4_protoDUNE_hist.root
-rw-r--r-- 1 fermilab users 519 Jan 4 00:06 protoDUNE_reco1_hist.root
-rw-r--r-- 1 fermilab users 52 Jan 3 20:23 all-input-dids.txt
-rw-r--r-- 1 fermilab users 7 Jan 4 00:06 justin-processed-pfns.txt
-rw-r--r-- 1 fermilab users 0 Jan 3 22:59 ProtoDUNE.log
-rw-r--r-- 1 fermilab users 0 Jan 3 22:53 debugprod.log
