justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 14109.1@dunegpschedd02.fnal.gov

Jobsub ID14109.1@dunegpschedd02.fnal.gov
Workflow ID262
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-08-03 08:14:16
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-08-03 11:45:20
From worker nodeHostnamewn2204241.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-03 08:15:28
Input filesmonte-carlo-000262-000001
JobscriptExit code0
Real time3h (12486s)
CPU time3h (12090s = 96%)
Max RSS bytes3988742144 (3803 MiB)
Outputting started2025-08-03 11:43:35
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/00262/1/001/reco_stage1.root
Finished2025-08-03 11:45:20
Saved logsjustin-logs:14109.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

lSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 125846 traces, 0 summary
Retagger: tagging trace set: gauss with 91683 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 91683 traces tagged "gauss"
FrameSaver: q=3.08906e+06 n=1549130 tag=gauss
wclsFrameSaver: saving 125846 traces tagged "wiener"
FrameSaver: q=3.29566e+06 n=1343364 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 29184 XUs and 26818 XVs -> 692 XUVs
C:0 T:1 67918 XUs and 65503 XVs -> 4219 XUVs
C:0 T:2 41741 XUs and 50022 XVs -> 2479 XUVs
C:0 T:3 22536 XUs and 22845 XVs -> 92 XUVs
C:0 T:4 10029 XUs and 7367 XVs -> 46 XUVs
C:0 T:5 39825 XUs and 22487 XVs -> 3540 XUVs
C:0 T:6 109935 XUs and 88372 XVs -> 7630 XUVs
C:0 T:7 36640 XUs and 29711 XVs -> 524 XUVs
C:0 T:8 11384 XUs and 11001 XVs -> 174 XUVs
C:0 T:9 84978 XUs and 97764 XVs -> 11163 XUVs
C:0 T:10 44539 XUs and 28405 XVs -> 6747 XUVs
C:0 T:11 322 XUs and 223 XVs -> 22 XUVs
37328 XUVs total
11180 collection wire objects
33699 potential space points
Neighbour search...
3386733 tests to find 1670580 neighbours
Iterating with no regularization...
Begin: 5.34925e+07
0 4.19548e+07
1 4.10519e+07
2 4.09342e+07
3 4.09014e+07
Now with regularization...
Begin: 3.20446e+07
0 3.20044e+07
1 3.19814e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 408432816
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 408432816
fLifetime: 30000 [us]
03-Aug-2025 12:43:32 BST  Closed output file "reco_stage1.root"
03-Aug-2025 12:43:32 BST  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          296.26        352.658       415.053       354.65        35.4488        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00155456    0.00399347    0.00816497     0.0035584    0.00185573       10     
reco:ophit:OpHitFinder                             0.227612      0.289327      0.351854      0.282341      0.0386177       10     
reco:opflash:OpFlashFinder                          0.13695      0.157409      0.183628       0.15617      0.0155309       10     
reco:opslicer:OpSlicer                               0.415       0.685047      0.935475      0.697918      0.149754        10     
reco:caldata:DataPrepModule                         5.65636       6.9352        7.84952       7.15113      0.812026        10     
reco:wclsdatasp:WireCellToolkit                     38.524        42.4298       50.0558       41.3569       3.68167        10     
reco:gaushit:GausHitFinder                         0.816973       1.30026       1.5173        1.34729      0.183846        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00042091    0.000735667    0.0013534    0.00062403    0.000308517      10     
reco:reco3d:SpacePointSolver                        7.71458       15.2862       28.078        13.5703       6.80433        10     
reco:hitpdune:DisambigFromSpacePoints              0.766779       1.52941       2.02163       1.67265      0.370893        10     
reco:pandora:StandardPandora                        15.3557       25.0597       29.8244       26.123        4.37835        10     
reco:pandoraWriter:StandardPandora                 0.385855       0.52387      0.907054      0.497236      0.142621        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.49839       4.01277       9.7574        3.50058       1.98467        10     
reco:pandoraShower:LArPandoraShowerCreation         1.92544       3.89183       9.93027       3.50279       2.12527        10     
reco:pandoracalo:Calorimetry                        1.42448       2.26295       4.73961       2.10098      0.854321        10     
reco:pandoracalonosce:Calorimetry                   1.24698       2.02781       5.02063       1.62637       1.05202        10     
reco:pandorapid:Chi2ParticleID                    0.00355054    0.00587743     0.0104717    0.00599655    0.00211744       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.031933      0.0573826     0.091695      0.0527376      0.02085        10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00346964     0.0055171    0.00896001    0.00510943    0.00191608       10     
reco:pandoraShowercalo:ShowerCalorimetry            1.83233       4.07573       11.7173       3.42328       2.66011        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.78771       4.2029        12.2334       3.45763       2.82027        10     
reco:emtrkmichelid:EmTrackMichelId                  171.677       198.789       218.584       201.807       13.9556        10     
reco:crttag:SingleCRTMatchingProducer             0.00986933     0.0186442     0.0467548     0.016241     0.00989285       10     
reco:crtreco:TwoCRTMatchingProducer               0.00623031     0.0121294     0.0349955     0.0104026    0.00781622       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.698053      0.908753       1.10671      0.917449      0.117358        10     
reco:pandora2Track:LArPandoraTrackCreation          7.80064       10.898        14.5411       10.466        1.99631        10     
reco:pandora2calo:Calorimetry                       4.0256        5.60116       8.12462       5.39396       1.26749        10     
reco:pandora2calonosce:Calorimetry                  3.82468       5.53533       8.23862       5.33071       1.2298         10     
reco:pandora2pid:Chi2ParticleID                    0.0103615     0.0156147     0.0205613     0.0157448    0.00264829       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0645354     0.0850989     0.0979317     0.0874621     0.0104495       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0128933     0.0156293     0.0206089     0.0151957    0.00231627       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.70144       4.02447       5.56901       3.95138      0.856948        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.65922       4.55352       6.95467       4.30484       1.18306        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.60085       4.37016       6.21173       4.24297       1.0714         10     
[art]:TriggerResults:TriggerResultInserter        4.6989e-05    8.63529e-05   0.000183213   8.3056e-05    3.48863e-05      10     
end_path:out1:RootOutput                           8.907e-06    1.47598e-05   2.2091e-05    1.40565e-05   3.8451e-06       10     
end_path:out1:RootOutput(write)                     2.43759       2.93936       3.62432        2.913       0.350147        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4802.1 MB
  Peak resident set size usage (VmHWM): 3988.74 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.32 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.29 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.18 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9007320
-rw-r--r-- 1 dune01 dune01 2159333422 Aug  3 11:42 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2053734131 Aug  3 11:40 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1806177399 Aug  3 12:43 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1601829071 Aug  3 09:39 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1536664682 Aug  3 09:21 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   65122550 Aug  3 12:38 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242608 Aug  3 12:43 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Aug  3 12:43 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Aug  3 12:43 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Aug  3 12:43 time.db
-rw-r--r-- 1 dune01 dune01        519 Aug  3 11:42 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Aug  3 09:39 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Aug  3 12:43 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Aug  3 09:15 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Aug  3 12:43 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Aug  3 11:47 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Aug  3 11:42 debugprod.log
justIN time: 2025-08-04 14:09:06 UTC       justIN version: 01.04.00