justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269772.1@dunegpschedd01.fnal.gov

Jobsub ID269772.1@dunegpschedd01.fnal.gov
Workflow ID11311
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-23 09:31:37
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2025-12-23 13:59:57
From worker nodeHostnameb9p10p0963.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-23 09:32:42
Input filesmonte-carlo-011311-000001
JobscriptExit code0
Real time4h (15579s)
CPU time4h (14419s = 92%)
Max RSS bytes4214206464 (4018 MiB)
Outputting started2025-12-23 13:52:23
Output files
Finished2025-12-23 13:59:57
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

are:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 170694 traces, 0 summary
Retagger: tagging trace set: gauss with 125584 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 125584 traces tagged "gauss"
FrameSaver: q=4.26922e+06 n=2172418 tag=gauss
wclsFrameSaver: saving 170694 traces tagged "wiener"
FrameSaver: q=4.52806e+06 n=1890873 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 41053 XUs and 37180 XVs -> 524 XUVs
C:0 T:1 100585 XUs and 93504 XVs -> 5837 XUVs
C:0 T:2 127811 XUs and 182635 XVs -> 11585 XUVs
C:0 T:3 46968 XUs and 57900 XVs -> 528 XUVs
C:0 T:4 14251 XUs and 14611 XVs -> 153 XUVs
C:0 T:5 57260 XUs and 45000 XVs -> 4100 XUVs
C:0 T:6 170338 XUs and 194534 XVs -> 17626 XUVs
C:0 T:7 48140 XUs and 52599 XVs -> 474 XUVs
C:0 T:8 6555 XUs and 9565 XVs -> 104 XUVs
C:0 T:9 49784 XUs and 67644 XVs -> 4709 XUVs
C:0 T:10 79704 XUs and 74150 XVs -> 10744 XUVs
C:0 T:11 531 XUs and 368 XVs -> 57 XUVs
56441 XUVs total
16118 collection wire objects
50925 potential space points
Neighbour search...
5811605 tests to find 2793560 neighbours
Iterating with no regularization...
Begin: 8.91715e+07
0 6.77923e+07
1 6.66034e+07
2 6.64679e+07
3 6.64304e+07
Now with regularization...
Begin: 5.4343e+07
0 5.43048e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411290260
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411290260
fLifetime: 30000 [us]
23-Dec-2025 14:52:20 CET  Closed output file "reco_stage1.root"
23-Dec-2025 14:52:20 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          329.325       427.356       531.694       433.181       64.7329        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00121595    0.00448521    0.00755611    0.00474682    0.00189698       10     
reco:ophit:OpHitFinder                             0.285025      0.370291      0.449634       0.36956      0.0533245       10     
reco:opflash:OpFlashFinder                         0.162153      0.221868      0.283243      0.220859      0.0329866       10     
reco:opslicer:OpSlicer                             0.645402      0.848739       1.05656      0.862863      0.128185        10     
reco:caldata:DataPrepModule                         4.54491       6.34633       7.41022       6.97336       1.05392        10     
reco:wclsdatasp:WireCellToolkit                     35.9057       40.3526       48.8112       40.0454       3.63717        10     
reco:gaushit:GausHitFinder                          1.05456       1.52364       2.17232       1.44619      0.312913        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000495015   0.000657792   0.00106666    0.000635051   0.000157589      10     
reco:reco3d:SpacePointSolver                        15.0763       22.2012       29.945        20.7449       5.03769        10     
reco:hitpdune:DisambigFromSpacePoints               1.42161       2.19929       3.15558       2.05454      0.609587        10     
reco:pandora:StandardPandora                        24.7996       42.5533       67.4919       43.4204       14.2708        10     
reco:pandoraWriter:StandardPandora                 0.283128      0.501064      0.705967      0.512968      0.123136        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.49329       4.54754       7.17299       4.57662       1.25577        10     
reco:pandoraShower:LArPandoraShowerCreation         3.12275       4.61593       5.59882       4.87677       0.78666        10     
reco:pandoracalo:Calorimetry                        1.9549        2.71224       4.35497       2.53944      0.614598        10     
reco:pandoracalonosce:Calorimetry                   1.70906       2.56142       3.97139       2.47935      0.578516        10     
reco:pandorapid:Chi2ParticleID                    0.00457218    0.00680784    0.00843476    0.00681767    0.00133277       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0457804     0.0677777     0.0895072     0.0691899     0.0153699       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00454555    0.00659957    0.00843574    0.00657287    0.00142192       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.28541       5.32604       7.11175       5.09976       1.36933        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.16955       5.52304       7.55498       5.61155       1.47531        10     
reco:emtrkmichelid:EmTrackMichelId                  190.256       233.335       280.468       225.689       31.0112        10     
reco:crttag:SingleCRTMatchingProducer              0.0150769     0.0188974     0.0239202     0.0191196    0.00265773       10     
reco:crtreco:TwoCRTMatchingProducer               0.00904959      0.01111      0.0137072     0.0106449    0.00154378       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.845096       1.08327       1.35479       1.13944      0.162208        10     
reco:pandora2Track:LArPandoraTrackCreation          8.61587       14.939        19.7658       14.9992       3.77687        10     
reco:pandora2calo:Calorimetry                       3.58983       7.38308       11.5586       7.16435       2.17115        10     
reco:pandora2calonosce:Calorimetry                  3.30601       6.65241       8.45129       7.02684       1.54505        10     
reco:pandora2pid:Chi2ParticleID                   0.00829082     0.0146739     0.0203985     0.0147968    0.00376903       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0435348     0.0856336     0.162516      0.0825442     0.0332579       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00827189     0.0156041     0.0312472     0.0131371     0.0061961       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.59507       4.78244       7.35543       4.89776       1.14793        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.43695       6.58537       9.25752       6.47784       1.73212        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.18435       6.23034       8.42887       6.46528       1.8069         10     
[art]:TriggerResults:TriggerResultInserter         4.702e-05    8.82908e-05   0.000223342   7.4776e-05    4.69435e-05      10     
end_path:out1:RootOutput                           1.192e-05    1.57632e-05    2.268e-05     1.57e-05     3.14793e-06      10     
end_path:out1:RootOutput(write)                     2.10634       3.59559       4.49334       3.61706      0.732466        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5066.14 MB
  Peak resident set size usage (VmHWM): 4214.21 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.28 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.91 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10630192
-rw-r--r--. 1 duneprd np-comp 2483217812 Dec 23 13:36 detsim_stage2.root
-rw-r--r--. 1 duneprd np-comp 2348846959 Dec 23 13:34 detsim_stage1.root
-rw-r--r--. 1 duneprd np-comp 2196524742 Dec 23 14:52 reco_stage1.root
-rw-r--r--. 1 duneprd np-comp 1926502512 Dec 23 11:02 g4_stage2.root
-rw-r--r--. 1 duneprd np-comp 1848701539 Dec 23 10:42 g4_stage1.root
-rw-r--r--. 1 duneprd np-comp   80909310 Dec 23 14:45 Pandora_Events.pndr
-rw-r--r--. 1 duneprd np-comp     243055 Dec 23 14:52 jobscript.log
-rw-r--r--. 1 duneprd np-comp     217088 Dec 23 14:52 mem.db
-rw-r--r--. 1 duneprd np-comp      62376 Dec 23 14:52 Pandora_Geometry.xml
-rw-r--r--. 1 duneprd np-comp      36864 Dec 23 14:52 time.db
-rw-r--r--. 1 duneprd np-comp        519 Dec 23 13:36 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 duneprd np-comp        519 Dec 23 11:02 g4_protoDUNE_hist.root
-rw-r--r--. 1 duneprd np-comp        519 Dec 23 14:52 protoDUNE_reco1_hist.root
-rw-r--r--. 1 duneprd np-comp         52 Dec 23 10:32 all-input-dids.txt
-rw-r--r--. 1 duneprd np-comp          7 Dec 23 14:52 justin-processed-pfns.txt
-rw-r--r--. 1 duneprd np-comp          0 Dec 23 13:46 ProtoDUNE.log
-rw-r--r--. 1 duneprd np-comp          0 Dec 23 13:36 debugprod.log
justIN time: 2026-02-04 14:21:00 UTC       justIN version: 01.06.00