justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 20690.0@dunegpschedd01.fnal.gov

Jobsub ID20690.0@dunegpschedd01.fnal.gov
Workflow ID277
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-08-03 19:32:55
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce03
Last heartbeat2025-08-04 00:44:08
From worker nodeHostnamecn504.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-03 19:34:03
Input filesmonte-carlo-000277-000001
JobscriptExit code0
Real time5h (18405s)
CPU time4h (17645s = 95%)
Max RSS bytes4099547136 (3909 MiB)
Outputting started2025-08-04 00:40:49
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/00277/1/001/reco_stage1.root
Finished2025-08-04 00:44:08
Saved logsjustin-logs:20690.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 139836 traces, 0 summary
Retagger: tagging trace set: gauss with 100855 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 100855 traces tagged "gauss"
FrameSaver: q=3.51157e+06 n=1695550 tag=gauss
wclsFrameSaver: saving 139836 traces tagged "wiener"
FrameSaver: q=3.7868e+06 n=1479730 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 26642 XUs and 30546 XVs -> 547 XUVs
C:0 T:1 66912 XUs and 59130 XVs -> 7976 XUVs
C:0 T:2 58107 XUs and 158376 XVs -> 9907 XUVs
C:0 T:3 25581 XUs and 41765 XVs -> 583 XUVs
C:0 T:4 15118 XUs and 15075 XVs -> 239 XUVs
C:0 T:5 122646 XUs and 89429 XVs -> 22350 XUVs
C:0 T:6 97737 XUs and 130381 XVs -> 6890 XUVs
C:0 T:7 39620 XUs and 50727 XVs -> 304 XUVs
C:0 T:8 11015 XUs and 10242 XVs -> 199 XUVs
C:0 T:9 158226 XUs and 117366 XVs -> 24555 XUVs
C:0 T:10 33022 XUs and 20748 XVs -> 3696 XUVs
C:0 T:11 214 XUs and 125 XVs -> 13 XUVs
77259 XUVs total
12440 collection wire objects
71209 potential space points
Neighbour search...
15935507 tests to find 6748332 neighbours
Iterating with no regularization...
Begin: 5.83921e+07
0 4.75811e+07
1 4.64051e+07
2 4.62004e+07
3 4.61363e+07
4 4.61081e+07
Now with regularization...
Begin: 3.54183e+07
0 3.53408e+07
1 3.52901e+07
2 3.5251e+07
3 3.52194e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 408442964
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 408442964
fLifetime: 30000 [us]
04-Aug-2025 01:40:46 BST  Closed output file "reco_stage1.root"
04-Aug-2025 01:40:46 BST  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          642.415       860.193       1065.28       849.873       152.465        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00197842    0.00540303    0.00988748     0.0049385    0.00254603       10     
reco:ophit:OpHitFinder                             0.265991      0.474821      0.614949      0.526983      0.127413        10     
reco:opflash:OpFlashFinder                         0.162981      0.282997      0.381239      0.321068      0.0804251       10     
reco:opslicer:OpSlicer                             0.520389       1.17575       1.66302       1.21968      0.342367        10     
reco:caldata:DataPrepModule                         6.33258       10.4642       12.0405       11.451        1.78701        10     
reco:wclsdatasp:WireCellToolkit                     55.7045       81.8325       87.9693       85.9616       9.14283        10     
reco:gaushit:GausHitFinder                          1.52392       2.57695       3.38787       2.47367      0.546258        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00069593    0.00109632     0.001681     0.00102785    0.000277643      10     
reco:reco3d:SpacePointSolver                        11.2054       31.5402       50.8192       33.8075       12.7478        10     
reco:hitpdune:DisambigFromSpacePoints               2.08825       4.17847       5.46373       4.48396       1.04222        10     
reco:pandora:StandardPandora                        41.4045       74.3996       127.416       61.7907       28.1046        10     
reco:pandoraWriter:StandardPandora                 0.512939      0.891464       1.27464      0.873694      0.226024        10     
reco:pandoraTrack:LArPandoraTrackCreation           9.07296       14.8838       22.1812       14.7613       4.67683        10     
reco:pandoraShower:LArPandoraShowerCreation         9.73119       16.9551       23.0247       17.0179       4.26546        10     
reco:pandoracalo:Calorimetry                        4.12927       7.57191       11.4764       7.65559       2.51119        10     
reco:pandoracalonosce:Calorimetry                   3.88782       7.10002       10.9855       6.89832       2.43563        10     
reco:pandorapid:Chi2ParticleID                    0.00549688     0.0112794     0.0160804     0.0110315    0.00280265       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0620999     0.110655      0.149996      0.110985      0.0217786       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00544441     0.0111198     0.0168588     0.0106754    0.00303992       10     
reco:pandoraShowercalo:ShowerCalorimetry            8.70475       17.2943       24.1872       16.6521       5.23165        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       8.66013       17.0596       24.8001       17.5483       5.64355        10     
reco:emtrkmichelid:EmTrackMichelId                  346.784       404.798       473.811       397.465       46.4065        10     
reco:crttag:SingleCRTMatchingProducer              0.0262053     0.038884      0.050303      0.0379015    0.00816681       10     
reco:crtreco:TwoCRTMatchingProducer                0.0245754      0.03433      0.0455744     0.0341904    0.00699218       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.46016       2.23626       3.3719        1.99886      0.630332        10     
reco:pandora2Track:LArPandoraTrackCreation          34.161        51.8939       73.7114       49.5416       13.2787        10     
reco:pandora2calo:Calorimetry                       14.4631       24.4367       35.0438       23.3296       6.81039        10     
reco:pandora2calonosce:Calorimetry                  13.7403       23.7184       34.3547       22.6429       6.83906        10     
reco:pandora2pid:Chi2ParticleID                    0.0208662     0.0292645     0.0390284     0.0292817    0.00559404       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.106103      0.146211      0.208461      0.138986      0.0280781       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0189822     0.0277994     0.0375195     0.0277054    0.00532226       10     
reco:pandora2Shower:LArPandoraShowerCreation        12.8776       17.3733       23.5518       16.8287       3.77637        10     
reco:pandora2Showercalo:ShowerCalorimetry           13.0136       21.8565       30.5115       19.9659       6.19162        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      11.9025       19.5262       28.2711       18.7412       5.50305        10     
[art]:TriggerResults:TriggerResultInserter        6.6846e-05    0.000117632   0.000182285   0.000115761   2.9744e-05       10     
end_path:out1:RootOutput                          1.1902e-05    1.80475e-05   2.2724e-05    1.81465e-05   3.35386e-06      10     
end_path:out1:RootOutput(write)                     3.16622       5.03803       6.2019        4.94586      0.824619        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4902.85 MB
  Peak resident set size usage (VmHWM): 4099.55 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.40 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   6.66 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.37 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9616572
-rw-r--r-- 1 pildune32 pildune 2279537675 Aug  3 23:13 detsim_stage2.root
-rw-r--r-- 1 pildune32 pildune 2164347742 Aug  3 23:11 detsim_stage1.root
-rw-r--r-- 1 pildune32 pildune 1949078635 Aug  4 01:40 reco_stage1.root
-rw-r--r-- 1 pildune32 pildune 1728331190 Aug  3 21:13 g4_stage2.root
-rw-r--r-- 1 pildune32 pildune 1654328315 Aug  3 20:42 g4_stage1.root
-rw-r--r-- 1 pildune32 pildune   71129980 Aug  4 01:27 Pandora_Events.pndr
-rw-r--r-- 1 pildune32 pildune     242942 Aug  4 01:40 jobscript.log
-rw-r--r-- 1 pildune32 pildune     217088 Aug  4 01:40 mem.db
-rw-r--r-- 1 pildune32 pildune      62376 Aug  4 01:40 Pandora_Geometry.xml
-rw-r--r-- 1 pildune32 pildune      36864 Aug  4 01:40 time.db
-rw-r--r-- 1 pildune32 pildune        519 Aug  3 23:13 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 pildune32 pildune        519 Aug  3 21:13 g4_protoDUNE_hist.root
-rw-r--r-- 1 pildune32 pildune        519 Aug  4 01:40 protoDUNE_reco1_hist.root
-rw-r--r-- 1 pildune32 pildune         52 Aug  3 20:34 all-input-dids.txt
-rw-r--r-- 1 pildune32 pildune          7 Aug  4 01:40 justin-processed-pfns.txt
-rw-r--r-- 1 pildune32 pildune          0 Aug  3 23:28 ProtoDUNE.log
-rw-r--r-- 1 pildune32 pildune          0 Aug  3 23:13 debugprod.log
justIN time: 2025-08-04 16:15:50 UTC       justIN version: 01.04.00