justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263665.0@dunegpschedd02.fnal.gov

Jobsub ID263665.0@dunegpschedd02.fnal.gov
Workflow ID11297
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-21 09:33:10
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce03
Last heartbeat2025-12-21 13:26:13
From worker nodeHostnamecn104.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-21 09:44:26
Input filesmonte-carlo-011297-000001
JobscriptExit code0
Real time3h (13100s)
CPU time3h (12921s = 98%)
Max RSS bytes4513357824 (4304 MiB)
Outputting started2025-12-21 13:22:47
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11297/1/001/reco_stage1.root
Finished2025-12-21 13:26:13
Saved logsjustin-logs:263665.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

vice:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 111956 traces, 0 summary
Retagger: tagging trace set: gauss with 90636 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 90636 traces tagged "gauss"
FrameSaver: q=3.00565e+06 n=1635107 tag=gauss
wclsFrameSaver: saving 111956 traces tagged "wiener"
FrameSaver: q=3.1475e+06 n=1463998 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 42040 XUs and 27209 XVs -> 330 XUVs
C:0 T:1 70922 XUs and 56771 XVs -> 5079 XUVs
C:0 T:2 75296 XUs and 107267 XVs -> 2926 XUVs
C:0 T:3 39711 XUs and 50809 XVs -> 453 XUVs
C:0 T:4 11705 XUs and 10912 XVs -> 71 XUVs
C:0 T:5 34708 XUs and 26346 XVs -> 3036 XUVs
C:0 T:6 54956 XUs and 54629 XVs -> 1714 XUVs
C:0 T:7 25234 XUs and 21937 XVs -> 114 XUVs
C:0 T:8 8914 XUs and 8250 XVs -> 184 XUVs
C:0 T:9 51528 XUs and 67072 XVs -> 5808 XUVs
C:0 T:10 25635 XUs and 20037 XVs -> 3058 XUVs
C:0 T:11 119 XUs and 77 XVs -> 22 XUVs
22795 XUVs total
10077 collection wire objects
21207 potential space points
Neighbour search...
1186049 tests to find 653844 neighbours
Iterating with no regularization...
Begin: 5.95093e+07
0 5.24184e+07
1 5.20034e+07
2 5.19564e+07
Now with regularization...
Begin: 4.42752e+07
0 4.42538e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411250909
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411250909
fLifetime: 30000 [us]
21-Dec-2025 13:22:43 GMT  Closed output file "reco_stage1.root"
21-Dec-2025 13:22:43 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          412.424       534.583       735.143       526.664       99.8949        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000921394   0.00364258    0.00647324    0.00342171    0.00174919       10     
reco:ophit:OpHitFinder                             0.199039      0.331004      0.412638      0.342713      0.0743424       10     
reco:opflash:OpFlashFinder                         0.163853      0.254904       0.37734      0.252555      0.0549785       10     
reco:opslicer:OpSlicer                             0.418166      0.750011       1.13888       0.72804      0.213465        10     
reco:caldata:DataPrepModule                         4.99257       6.94655       8.54052       6.97683       1.25886        10     
reco:wclsdatasp:WireCellToolkit                     41.3344       49.8173       61.805        49.5681       5.55656        10     
reco:gaushit:GausHitFinder                         0.962822       1.45949       1.84838       1.45558      0.258375        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00064473    0.000943345   0.00120664    0.000967288   0.000164796      10     
reco:reco3d:SpacePointSolver                        7.23412       14.9764       24.7906       15.9448       5.24338        10     
reco:hitpdune:DisambigFromSpacePoints               1.11455       2.16195       3.72517       2.13575      0.679947        10     
reco:pandora:StandardPandora                        18.8119       47.7161       84.9693       40.7376       20.011         10     
reco:pandoraWriter:StandardPandora                  0.35508       0.56092       1.01145      0.524466      0.179845        10     
reco:pandoraTrack:LArPandoraTrackCreation           5.99668       10.6089       18.0454       9.34839       3.52116        10     
reco:pandoraShower:LArPandoraShowerCreation         4.37037        10.75        24.6804       9.28971       5.38413        10     
reco:pandoracalo:Calorimetry                        3.63538       5.08735       7.92265       4.95501       1.16076        10     
reco:pandoracalonosce:Calorimetry                   3.07585       4.65545       7.34266       4.52541       1.18415        10     
reco:pandorapid:Chi2ParticleID                    0.00552297    0.00824208     0.011773     0.00815247    0.00195893       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0547999     0.079673      0.133072      0.0719384     0.0215407       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00561656    0.00812869     0.0114206    0.00832608    0.00176983       10     
reco:pandoraShowercalo:ShowerCalorimetry            5.1664        10.4891       23.7879       9.22868       5.08223        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.52728       10.6744       22.1965       10.1398       4.81369        10     
reco:emtrkmichelid:EmTrackMichelId                  186.837       265.773       344.738       259.339       53.6937        10     
reco:crttag:SingleCRTMatchingProducer              0.0259814     0.0344938     0.0472877     0.0339377    0.00630899       10     
reco:crtreco:TwoCRTMatchingProducer                0.0167086     0.0251103     0.037165      0.0264695    0.00601284       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.13982       1.46573       2.06015       1.39486      0.253747        10     
reco:pandora2Track:LArPandoraTrackCreation          20.2248       32.9985       46.7619       32.4355       8.78577        10     
reco:pandora2calo:Calorimetry                       5.0015        13.6406       19.258        13.7154       4.69153        10     
reco:pandora2calonosce:Calorimetry                  4.74504       12.2906       18.1515       11.4211       3.66009        10     
reco:pandora2pid:Chi2ParticleID                    0.0106075     0.0183324     0.023149      0.0193535    0.00394551       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.056993      0.0955176     0.124301      0.0963939     0.018343        10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0106253     0.0180609     0.0216624     0.0195953    0.00366462       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.68857       8.73354       13.3755       8.23807       2.30673        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.61039       10.1667       15.0768       9.50989       2.99779        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.00824       8.71816       13.2622       8.08338       2.9342         10     
[art]:TriggerResults:TriggerResultInserter        3.7808e-05    6.2568e-05    9.8414e-05    5.9419e-05    2.10212e-05      10     
end_path:out1:RootOutput                           6.375e-06    9.8001e-06    1.6042e-05    9.4175e-06    3.05408e-06      10     
end_path:out1:RootOutput(write)                     1.98159        3.113        4.19216       2.97832      0.655518        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5322.97 MB
  Peak resident set size usage (VmHWM): 4513.36 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.27 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.37 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.25 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10829804
-rw-r--r-- 1 pildune32 pildune 2526278855 Dec 21 11:51 detsim_stage2.root
-rw-r--r-- 1 pildune32 pildune 2399458598 Dec 21 11:49 detsim_stage1.root
-rw-r--r-- 1 pildune32 pildune 2224631445 Dec 21 13:22 reco_stage1.root
-rw-r--r-- 1 pildune32 pildune 1973652771 Dec 21 10:06 g4_stage2.root
-rw-r--r-- 1 pildune32 pildune 1887022473 Dec 21 09:50 g4_stage1.root
-rw-r--r-- 1 pildune32 pildune   78062510 Dec 21 13:17 Pandora_Events.pndr
-rw-r--r-- 1 pildune32 pildune     243158 Dec 21 13:22 jobscript.log
-rw-r--r-- 1 pildune32 pildune     217088 Dec 21 13:22 mem.db
-rw-r--r-- 1 pildune32 pildune      62376 Dec 21 13:22 Pandora_Geometry.xml
-rw-r--r-- 1 pildune32 pildune      36864 Dec 21 13:22 time.db
-rw-r--r-- 1 pildune32 pildune        519 Dec 21 11:51 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 pildune32 pildune        519 Dec 21 10:06 g4_protoDUNE_hist.root
-rw-r--r-- 1 pildune32 pildune        519 Dec 21 13:22 protoDUNE_reco1_hist.root
-rw-r--r-- 1 pildune32 pildune         52 Dec 21 09:44 all-input-dids.txt
-rw-r--r-- 1 pildune32 pildune          7 Dec 21 13:22 justin-processed-pfns.txt
-rw-r--r-- 1 pildune32 pildune          0 Dec 21 11:58 ProtoDUNE.log
-rw-r--r-- 1 pildune32 pildune          0 Dec 21 11:51 debugprod.log
justIN time: 2026-02-07 06:32:39 UTC       justIN version: 01.06.00