justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 266984.1@dunegpschedd01.fnal.gov

Jobsub ID266984.1@dunegpschedd01.fnal.gov
Workflow ID11166
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-11 20:10:41
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-12 23:41:43
From worker nodeHostnamedunegli-7890379-0-fnpc17119.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-12 11:33:29
Input filesmonte-carlo-011166-000001
JobscriptExit code0
Real time12h (43640s)
CPU time11h (43191s = 98%)
Max RSS bytes4537311232 (4327 MiB)
Outputting started2025-12-12 23:40:49
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11166/1/001/reco_stage1.root
Finished2025-12-12 23:41:43
Saved logsjustin-logs:266984.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

d processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 159074 traces, 0 summary
Retagger: tagging trace set: gauss with 122103 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 122103 traces tagged "gauss"
FrameSaver: q=4.32432e+06 n=2195737 tag=gauss
wclsFrameSaver: saving 159074 traces tagged "wiener"
FrameSaver: q=4.59367e+06 n=1957117 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 45044 XUs and 40320 XVs -> 603 XUVs
C:0 T:1 106596 XUs and 86388 XVs -> 7292 XUVs
C:0 T:2 138977 XUs and 155331 XVs -> 9585 XUVs
C:0 T:3 55655 XUs and 57331 XVs -> 731 XUVs
C:0 T:4 14059 XUs and 14936 XVs -> 281 XUVs
C:0 T:5 73206 XUs and 57359 XVs -> 8276 XUVs
C:0 T:6 150594 XUs and 148549 XVs -> 7481 XUVs
C:0 T:7 49626 XUs and 51880 XVs -> 484 XUVs
C:0 T:8 9368 XUs and 7743 XVs -> 81 XUVs
C:0 T:9 46918 XUs and 93547 XVs -> 5486 XUVs
C:0 T:10 71295 XUs and 54818 XVs -> 7071 XUVs
C:0 T:11 106 XUs and 62 XVs -> 14 XUVs
47385 XUVs total
15191 collection wire objects
42280 potential space points
Neighbour search...
2959634 tests to find 1540590 neighbours
Iterating with no regularization...
Begin: 1.04042e+08
0 8.8185e+07
1 8.69862e+07
2 8.67721e+07
3 8.66989e+07
Now with regularization...
Begin: 7.41283e+07
0 7.40725e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411058666
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411058666
fLifetime: 30000 [us]
12-Dec-2025 23:40:40 UTC  Closed output file "reco_stage1.root"
12-Dec-2025 23:40:40 UTC  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          1327.72       2140.9        3560.27       2017.02       626.166        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00539306     0.0241343     0.059356      0.0194594     0.0152912       10     
reco:ophit:OpHitFinder                             0.345853      0.607932      0.870703      0.594751       0.13529        10     
reco:opflash:OpFlashFinder                         0.474508       1.17035       1.84976       1.13871      0.428813        10     
reco:opslicer:OpSlicer                              0.71153       1.47053       2.11545       1.41315      0.353426        10     
reco:caldata:DataPrepModule                         12.3687       13.2266       14.7748       12.9192      0.715717        10     
reco:wclsdatasp:WireCellToolkit                     162.031       190.593       210.582       197.49        16.7835        10     
reco:gaushit:GausHitFinder                          2.18445       3.86984       5.33433       3.90774      0.866722        10     
reco:nhitsfilter:NumberOfHitsFilter                0.0033651    0.00503704    0.00693608    0.00488796    0.00109163       10     
reco:reco3d:SpacePointSolver                        31.4316       67.4157       172.354       55.1147       39.5384        10     
reco:hitpdune:DisambigFromSpacePoints               7.98423       19.018        50.1339       14.712        11.8949        10     
reco:pandora:StandardPandora                        123.656       252.473       463.741       230.799       111.793        10     
reco:pandoraWriter:StandardPandora                  1.37009       1.84836       2.52988       1.76375      0.358829        10     
reco:pandoraTrack:LArPandoraTrackCreation           42.961        59.4212       85.4026       56.5501       11.8574        10     
reco:pandoraShower:LArPandoraShowerCreation         36.9715       73.1647       152.95        67.5798       31.3914        10     
reco:pandoracalo:Calorimetry                        22.5711       32.469        47.0601       31.593        7.06148        10     
reco:pandoracalonosce:Calorimetry                   20.5416       30.5571       38.373        30.8006       5.71667        10     
reco:pandorapid:Chi2ParticleID                     0.0165148     0.0251242     0.0359353     0.0238739    0.00501793       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.151506      0.184626      0.227347      0.181915      0.0242455       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0139361     0.0205753     0.0245381     0.0216499     0.0031568       10     
reco:pandoraShowercalo:ShowerCalorimetry            37.283        85.2094       173.567       81.471        36.3475        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       35.9614       84.3348       176.117       81.0724       37.6604        10     
reco:emtrkmichelid:EmTrackMichelId                  392.039       548.35        769.07        518.066       110.861        10     
reco:crttag:SingleCRTMatchingProducer              0.0960793     0.170856      0.414536       0.14708      0.0863509       10     
reco:crtreco:TwoCRTMatchingProducer                0.0781044     0.126208       0.16334      0.131456      0.0256807       10     
reco:anodepiercerst0:T0RecoAnodePiercers            3.6144        8.48612       11.5825       9.11167       2.29339        10     
reco:pandora2Track:LArPandoraTrackCreation          101.21        200.882       375.33        194.534       75.861         10     
reco:pandora2calo:Calorimetry                       53.8671       113.178       211.324       108.146       44.1864        10     
reco:pandora2calonosce:Calorimetry                  52.4678       107.066       183.503       108.857       37.6998        10     
reco:pandora2pid:Chi2ParticleID                    0.0363605     0.0569826     0.0819202     0.0567613     0.0144435       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.182426      0.268443       0.36181      0.270694      0.0594583       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0318897     0.0490046     0.0673516     0.048201      0.0109658       10     
reco:pandora2Shower:LArPandoraShowerCreation        34.8082       58.0712       88.7422       57.296        15.2842        10     
reco:pandora2Showercalo:ShowerCalorimetry           48.3195       95.733        166.797       95.1582       32.6551        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      43.5764       83.6912       146.178       80.0289       27.4772        10     
[art]:TriggerResults:TriggerResultInserter        0.000221058   0.00036047    0.00061364    0.000330959   0.000105504      10     
end_path:out1:RootOutput                          1.8804e-05    4.37735e-05   7.9335e-05    4.2595e-05    1.79956e-05      10     
end_path:out1:RootOutput(write)                     4.6087        6.81098       10.3133       6.36131       1.74704        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5355.45 MB
  Peak resident set size usage (VmHWM): 4537.31 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.40 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.61 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.23 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   7.49 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.34 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.24 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11548180
-rw-r--r-- 1 dunegli fnalgrid 2666777882 Dec 12 17:38 detsim_stage2.root
-rw-r--r-- 1 dunegli fnalgrid 2530361075 Dec 12 17:32 detsim_stage1.root
-rw-r--r-- 1 dunegli fnalgrid 2398149429 Dec 12 23:40 reco_stage1.root
-rw-r--r-- 1 dunegli fnalgrid 2118594790 Dec 12 12:05 g4_stage2.root
-rw-r--r-- 1 dunegli fnalgrid 2024986661 Dec 12 11:40 g4_stage1.root
-rw-r--r-- 1 dunegli fnalgrid   85835880 Dec 12 23:07 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid     243185 Dec 12 23:40 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid     217088 Dec 12 23:40 mem.db
-rw-r--r-- 1 dunegli fnalgrid      62376 Dec 12 23:40 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid      36864 Dec 12 23:40 time.db
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 12 17:38 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 12 12:06 g4_protoDUNE_hist.root
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 12 23:40 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dunegli fnalgrid         52 Dec 12 11:33 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid          7 Dec 12 23:40 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 12 18:08 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 12 17:38 debugprod.log
justIN time: 2026-02-07 12:20:30 UTC       justIN version: 01.06.00