justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269086.1@dunegpschedd01.fnal.gov

Jobsub ID269086.1@dunegpschedd01.fnal.gov
Workflow ID11260
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-18 16:44:58
SiteCZ_FZU
EntryNova_CZ_FZU_osgce1
Last heartbeat2025-12-18 20:43:29
From worker nodeHostnamemilan09.farm.particle.cz
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 16:45:57
Input filesmonte-carlo-011260-000001
JobscriptExit code0
Real time3h (14203s)
CPU time3h (13736s = 96%)
Max RSS bytes4572123136 (4360 MiB)
Outputting started2025-12-18 20:42:42
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11260/1/001/reco_stage1.root
Finished2025-12-18 20:43:29
Saved logsjustin-logs:269086.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

BasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 165696 traces, 0 summary
Retagger: tagging trace set: gauss with 127746 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 127746 traces tagged "gauss"
FrameSaver: q=4.30296e+06 n=2292837 tag=gauss
wclsFrameSaver: saving 165696 traces tagged "wiener"
FrameSaver: q=4.5415e+06 n=2033747 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 43439 XUs and 37381 XVs -> 498 XUVs
C:0 T:1 90494 XUs and 95752 XVs -> 8821 XUVs
C:0 T:2 95051 XUs and 136732 XVs -> 6523 XUVs
C:0 T:3 44429 XUs and 49274 XVs -> 528 XUVs
C:0 T:4 16585 XUs and 12839 XVs -> 82 XUVs
C:0 T:5 54638 XUs and 33666 XVs -> 3351 XUVs
C:0 T:6 216184 XUs and 255248 XVs -> 22068 XUVs
C:0 T:7 72624 XUs and 73853 XVs -> 1343 XUVs
C:0 T:8 8665 XUs and 7587 XVs -> 72 XUVs
C:0 T:9 56116 XUs and 67303 XVs -> 4794 XUVs
C:0 T:10 73388 XUs and 63800 XVs -> 8482 XUVs
C:0 T:11 71 XUs and 98 XVs -> 3 XUVs
56565 XUVs total
15692 collection wire objects
51237 potential space points
Neighbour search...
6741077 tests to find 3189554 neighbours
Iterating with no regularization...
Begin: 7.59033e+07
0 6.37019e+07
1 6.28249e+07
2 6.26967e+07
3 6.26602e+07
Now with regularization...
Begin: 5.07605e+07
0 5.07086e+07
1 5.0664e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411196507
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411196507
fLifetime: 30000 [us]
18-Dec-2025 21:42:40 CET  Closed output file "reco_stage1.root"
18-Dec-2025 21:42:40 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          377.302       430.742       513.986       425.877       41.086         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00105906    0.00458223    0.00881713    0.00402965     0.0023404       10     
reco:ophit:OpHitFinder                             0.361999      0.391024      0.447611      0.386553      0.0266244       10     
reco:opflash:OpFlashFinder                          0.18492      0.240023      0.286736       0.23741      0.0294462       10     
reco:opslicer:OpSlicer                               0.822       0.923857       1.13199      0.896676      0.0844582       10     
reco:caldata:DataPrepModule                         6.0905        6.36974       6.60425       6.35492      0.173631        10     
reco:wclsdatasp:WireCellToolkit                     34.4871       36.0346       37.4855       36.0435      0.892942        10     
reco:gaushit:GausHitFinder                          1.25427       1.5416        1.80174       1.52862      0.148349        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000550711   0.000630976   0.000827491   0.000611501   7.79855e-05      10     
reco:reco3d:SpacePointSolver                        18.5176       25.0062       33.1912       23.1821       5.4372         10     
reco:hitpdune:DisambigFromSpacePoints               1.93288       2.36553       3.04045       2.28953      0.340101        10     
reco:pandora:StandardPandora                        31.8345       47.5218       71.3057       44.2456       10.7819        10     
reco:pandoraWriter:StandardPandora                 0.589375      0.693129      0.848403      0.664822      0.0917117       10     
reco:pandoraTrack:LArPandoraTrackCreation           3.77536       4.23989       4.80166       4.24973      0.324412        10     
reco:pandoraShower:LArPandoraShowerCreation         3.61867       4.93473       6.47625       4.69056      0.901788        10     
reco:pandoracalo:Calorimetry                        2.15966       2.41692       2.66166       2.43667      0.139158        10     
reco:pandoracalonosce:Calorimetry                   1.91784       2.21206       2.42181       2.2341       0.139796        10     
reco:pandorapid:Chi2ParticleID                    0.00476687    0.00675925    0.00811244    0.00701477    0.000853389      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.045459      0.0666859     0.0780323     0.0685869    0.00843777       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00514329    0.00630928    0.00702285    0.00641104    0.000597443      10     
reco:pandoraShowercalo:ShowerCalorimetry            3.81153       5.30824       6.76317       5.2951       0.847954        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.72407       5.25143       6.64189       5.24646      0.914926        10     
reco:emtrkmichelid:EmTrackMichelId                  212.49        231.685       268.344       228.34        18.4513        10     
reco:crttag:SingleCRTMatchingProducer              0.0134085     0.0188653     0.0271267     0.0166171    0.00491723       10     
reco:crtreco:TwoCRTMatchingProducer               0.00859922    0.00940548     0.0117901    0.00904189    0.000937765      10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.892992       1.05843       1.21039       1.06848      0.0922019       10     
reco:pandora2Track:LArPandoraTrackCreation          11.6882       14.9831       20.6965       14.3194       2.66187        10     
reco:pandora2calo:Calorimetry                       6.05368       7.49537       9.33949       7.28948       1.03632        10     
reco:pandora2calonosce:Calorimetry                  5.79047       7.07336       9.38035       6.83571       1.07119        10     
reco:pandora2pid:Chi2ParticleID                    0.012539      0.0172479     0.0205037     0.0177895    0.00233335       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0569283     0.0883281     0.113321      0.0870239     0.0137682       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0116861     0.016719      0.0200006     0.0172469    0.00240752       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.44873       5.47657       7.4908        5.15951      0.917537        10     
reco:pandora2Showercalo:ShowerCalorimetry           5.55251       6.74258       8.72495       6.45307      0.991408        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.23486       6.31368       8.31536       5.96815      0.963222        10     
[art]:TriggerResults:TriggerResultInserter         4.41e-05     6.02321e-05   0.00012604      4.9e-05     2.48888e-05      10     
end_path:out1:RootOutput                            8.1e-06      9.772e-06     1.425e-05     9.425e-06    1.59971e-06      10     
end_path:out1:RootOutput(write)                     3.53749       4.14065       5.07296       3.99149      0.423793        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5382.08 MB
  Peak resident set size usage (VmHWM): 4572.12 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.04 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11502704
-rw-r--r-- 1 fermilab users 2641200987 Dec 18 20:27 detsim_stage2.root
-rw-r--r-- 1 fermilab users 2500874452 Dec 18 20:24 detsim_stage1.root
-rw-r--r-- 1 fermilab users 2376514839 Dec 18 21:42 reco_stage1.root
-rw-r--r-- 1 fermilab users 2126812846 Dec 18 18:11 g4_stage2.root
-rw-r--r-- 1 fermilab users 2046517448 Dec 18 17:54 g4_stage1.root
-rw-r--r-- 1 fermilab users   86214060 Dec 18 21:36 Pandora_Events.pndr
-rw-r--r-- 1 fermilab users     243203 Dec 18 21:42 jobscript.log
-rw-r--r-- 1 fermilab users     217088 Dec 18 21:42 mem.db
-rw-r--r-- 1 fermilab users      62376 Dec 18 21:42 Pandora_Geometry.xml
-rw-r--r-- 1 fermilab users      36864 Dec 18 21:42 time.db
-rw-r--r-- 1 fermilab users        519 Dec 18 20:27 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 fermilab users        519 Dec 18 18:11 g4_protoDUNE_hist.root
-rw-r--r-- 1 fermilab users        519 Dec 18 21:42 protoDUNE_reco1_hist.root
-rw-r--r-- 1 fermilab users         52 Dec 18 17:45 all-input-dids.txt
-rw-r--r-- 1 fermilab users          7 Dec 18 21:42 justin-processed-pfns.txt
-rw-r--r-- 1 fermilab users          0 Dec 18 20:36 ProtoDUNE.log
-rw-r--r-- 1 fermilab users          0 Dec 18 20:27 debugprod.log
justIN time: 2025-12-19 07:23:03 UTC       justIN version: 01.05.03