justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 248814.0@dunegpschedd02.fnal.gov

Jobsub ID248814.0@dunegpschedd02.fnal.gov
Workflow ID10394
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-20 10:00:58
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce03
Last heartbeat2025-11-20 13:38:39
From worker nodeHostnamecn089.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-20 10:01:31
Input filesmonte-carlo-010394-000001
JobscriptExit code0
Real time3h (12816s)
CPU time3h (12601s = 98%)
Max RSS bytes4404584448 (4200 MiB)
Outputting started2025-11-20 13:35:08
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10394/1/001/reco_stage1.root
Finished2025-11-20 13:38:39
Saved logsjustin-logs:248814.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

vice:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 147192 traces, 0 summary
Retagger: tagging trace set: gauss with 108172 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 108172 traces tagged "gauss"
FrameSaver: q=3.70287e+06 n=1879265 tag=gauss
wclsFrameSaver: saving 147192 traces tagged "wiener"
FrameSaver: q=3.95517e+06 n=1648525 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 31855 XUs and 36430 XVs -> 432 XUVs
C:0 T:1 56581 XUs and 60080 XVs -> 4962 XUVs
C:0 T:2 127870 XUs and 180892 XVs -> 9279 XUVs
C:0 T:3 58049 XUs and 71666 XVs -> 551 XUVs
C:0 T:4 11993 XUs and 12293 XVs -> 187 XUVs
C:0 T:5 51298 XUs and 82942 XVs -> 6370 XUVs
C:0 T:6 118419 XUs and 129002 XVs -> 16016 XUVs
C:0 T:7 45432 XUs and 42522 XVs -> 1520 XUVs
C:0 T:8 10223 XUs and 11665 XVs -> 169 XUVs
C:0 T:9 80949 XUs and 128661 XVs -> 9417 XUVs
C:0 T:10 32042 XUs and 31639 XVs -> 6050 XUVs
C:0 T:11 137 XUs and 123 XVs -> 32 XUVs
54985 XUVs total
13299 collection wire objects
47419 potential space points
Neighbour search...
5248853 tests to find 2529814 neighbours
Iterating with no regularization...
Begin: 6.84372e+07
0 4.87748e+07
1 4.74082e+07
2 4.71639e+07
3 4.7084e+07
4 4.70487e+07
Now with regularization...
Begin: 3.66186e+07
0 3.65494e+07
1 3.65118e+07
2 3.64856e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410627709
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410627709
fLifetime: 30000 [us]
20-Nov-2025 13:35:05 GMT  Closed output file "reco_stage1.root"
20-Nov-2025 13:35:05 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          459.035       574.628       722.501       574.592       80.7498        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00114782    0.00378537     0.0057815    0.00370754    0.00141273       10     
reco:ophit:OpHitFinder                             0.222184      0.276832      0.475602      0.263921      0.0695241       10     
reco:opflash:OpFlashFinder                         0.145949      0.201681      0.283814      0.197008      0.0381321       10     
reco:opslicer:OpSlicer                             0.518859      0.666141       1.02944      0.647361      0.140142        10     
reco:caldata:DataPrepModule                         5.10364       5.64413       6.83315       5.53287      0.467622        10     
reco:wclsdatasp:WireCellToolkit                     48.4914       50.4372       52.7161       50.3496       1.35578        10     
reco:gaushit:GausHitFinder                          1.34194        1.519        1.72939       1.47468      0.135667        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000804494   0.000933329   0.00114912    0.000905335   0.000110248      10     
reco:reco3d:SpacePointSolver                        9.66426       18.1627       35.8051       17.0865       6.89757        10     
reco:hitpdune:DisambigFromSpacePoints               1.89418       2.41495       3.05383       2.44502      0.328631        10     
reco:pandora:StandardPandora                        32.3321       52.3239       86.5498       49.0137       15.6308        10     
reco:pandoraWriter:StandardPandora                 0.475929      0.556083      0.690547      0.549753      0.058693        10     
reco:pandoraTrack:LArPandoraTrackCreation           9.63194       14.2958       19.3808       14.5079       2.94446        10     
reco:pandoraShower:LArPandoraShowerCreation         13.911        18.1486       24.483        17.1952       3.21339        10     
reco:pandoracalo:Calorimetry                        4.70891       6.92276       9.20403       6.67929       1.34596        10     
reco:pandoracalonosce:Calorimetry                   4.60697       6.75006       9.55728       6.66756       1.53167        10     
reco:pandorapid:Chi2ParticleID                    0.00504719    0.00679834     0.0103933    0.00649111     0.0014635       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0596448     0.0712191     0.103522      0.0681453     0.0120091       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00503629    0.00646156     0.0101002    0.00613541    0.00136517       10     
reco:pandoraShowercalo:ShowerCalorimetry            13.4484       17.7842       24.7053       16.2567       3.75647        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       13.2302       17.601        24.5857       16.0068       3.89436        10     
reco:emtrkmichelid:EmTrackMichelId                  174.373       211.573       269.209       204.773       30.2688        10     
reco:crttag:SingleCRTMatchingProducer              0.0325159     0.0402723     0.052998      0.0380742    0.00676035       10     
reco:crtreco:TwoCRTMatchingProducer                0.0268868     0.0348134     0.0425031     0.0335418    0.00536585       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.23884       1.66595       2.4272        1.50645      0.425418        10     
reco:pandora2Track:LArPandoraTrackCreation          38.6414       49.8866       66.0304       45.8979       9.99998        10     
reco:pandora2calo:Calorimetry                       17.7987       22.4466       31.0603       20.9855       3.75064        10     
reco:pandora2calonosce:Calorimetry                  16.3883       21.2812       29.2568       20.0843       3.51651        10     
reco:pandora2pid:Chi2ParticleID                    0.0125703     0.0149593     0.0172831     0.0144981    0.00134763       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0711702     0.0854057     0.0980751     0.0847236    0.00766397       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0117747     0.0144239     0.0165383     0.0142398     0.0013153       10     
reco:pandora2Shower:LArPandoraShowerCreation        12.5169       15.2024       20.4314       14.6649       2.40431        10     
reco:pandora2Showercalo:ShowerCalorimetry           14.9217       18.8695       25.6031       17.8872       3.16495        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      11.3422       16.5126       21.7652       16.5197       2.92155        10     
[art]:TriggerResults:TriggerResultInserter        5.0652e-05    7.15576e-05   0.000128164   5.8985e-05    2.37712e-05      10     
end_path:out1:RootOutput                           8.036e-06    1.01184e-05    1.551e-05     9.36e-06     2.10489e-06      10     
end_path:out1:RootOutput(write)                     2.18517       2.98327       4.25937       2.83273      0.544495        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5208.33 MB
  Peak resident set size usage (VmHWM): 4404.58 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.45 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10968752
-rw-r--r-- 1 pildune32 pildune 2548174520 Nov 20 11:56 detsim_stage2.root
-rw-r--r-- 1 pildune32 pildune 2426586669 Nov 20 11:54 detsim_stage1.root
-rw-r--r-- 1 pildune32 pildune 2267939346 Nov 20 13:35 reco_stage1.root
-rw-r--r-- 1 pildune32 pildune 1995890937 Nov 20 10:25 g4_stage2.root
-rw-r--r-- 1 pildune32 pildune 1910354074 Nov 20 10:07 g4_stage1.root
-rw-r--r-- 1 pildune32 pildune   82436110 Nov 20 13:26 Pandora_Events.pndr
-rw-r--r-- 1 pildune32 pildune     243345 Nov 20 13:35 jobscript.log
-rw-r--r-- 1 pildune32 pildune     217088 Nov 20 13:35 mem.db
-rw-r--r-- 1 pildune32 pildune      62376 Nov 20 13:35 Pandora_Geometry.xml
-rw-r--r-- 1 pildune32 pildune      36864 Nov 20 13:35 time.db
-rw-r--r-- 1 pildune32 pildune        519 Nov 20 11:56 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 pildune32 pildune        519 Nov 20 10:25 g4_protoDUNE_hist.root
-rw-r--r-- 1 pildune32 pildune        519 Nov 20 13:35 protoDUNE_reco1_hist.root
-rw-r--r-- 1 pildune32 pildune         52 Nov 20 10:01 all-input-dids.txt
-rw-r--r-- 1 pildune32 pildune          7 Nov 20 13:35 justin-processed-pfns.txt
-rw-r--r-- 1 pildune32 pildune          0 Nov 20 12:05 ProtoDUNE.log
-rw-r--r-- 1 pildune32 pildune          0 Nov 20 11:56 debugprod.log
justIN time: 2026-02-09 03:29:05 UTC       justIN version: 01.06.00