justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 250366.0@dunegpschedd02.fnal.gov

Jobsub ID250366.0@dunegpschedd02.fnal.gov
Workflow ID10560
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-24 04:18:37
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce03
Last heartbeat2025-11-24 07:14:48
From worker nodeHostnamecn005.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-24 04:19:30
Input filesmonte-carlo-010560-000001
JobscriptExit code0
Real time2h (8866s)
CPU time2h (8709s = 98%)
Max RSS bytes4056952832 (3869 MiB)
Outputting started2025-11-24 06:47:17
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10560/1/001/reco_stage1.root
Finished2025-11-24 07:14:48
Saved logsjustin-logs:250366.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

nels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 121798 traces, 0 summary
Retagger: tagging trace set: gauss with 96429 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 96429 traces tagged "gauss"
FrameSaver: q=3.46863e+06 n=1754183 tag=gauss
wclsFrameSaver: saving 121798 traces tagged "wiener"
FrameSaver: q=3.64031e+06 n=1578898 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 31351 XUs and 21688 XVs -> 220 XUVs
C:0 T:1 49492 XUs and 40374 XVs -> 2441 XUVs
C:0 T:2 73702 XUs and 127251 XVs -> 5591 XUVs
C:0 T:3 35664 XUs and 53771 XVs -> 273 XUVs
C:0 T:4 14269 XUs and 15524 XVs -> 105 XUVs
C:0 T:5 36251 XUs and 32563 XVs -> 3399 XUVs
C:0 T:6 94147 XUs and 85901 XVs -> 3259 XUVs
C:0 T:7 43940 XUs and 33564 XVs -> 296 XUVs
C:0 T:8 10043 XUs and 9139 XVs -> 136 XUVs
C:0 T:9 56040 XUs and 61161 XVs -> 4276 XUVs
C:0 T:10 22106 XUs and 10375 XVs -> 2307 XUVs
C:0 T:11 73 XUs and 48 XVs -> 5 XUVs
22308 XUVs total
9173 collection wire objects
20677 potential space points
Neighbour search...
707009 tests to find 393178 neighbours
Iterating with no regularization...
Begin: 7.75135e+07
0 6.88563e+07
1 6.82297e+07
2 6.81438e+07
3 6.81205e+07
Now with regularization...
Begin: 5.7698e+07
0 5.77018e+07
Warning: metric increased
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410698075
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410698075
fLifetime: 30000 [us]
24-Nov-2025 06:47:14 GMT  Closed output file "reco_stage1.root"
24-Nov-2025 06:47:14 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          192.97        372.081       495.244       387.913       82.2085        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000929383   0.00339513    0.00751238    0.00337102    0.00169387       10     
reco:ophit:OpHitFinder                             0.174113      0.213534       0.23736      0.219152      0.0191522       10     
reco:opflash:OpFlashFinder                         0.092629      0.143097      0.169299      0.148642      0.0231466       10     
reco:opslicer:OpSlicer                             0.383209      0.499201      0.590199      0.512131      0.0577574       10     
reco:caldata:DataPrepModule                         4.65993       4.88407       5.08973       4.8784       0.116233        10     
reco:wclsdatasp:WireCellToolkit                     42.2033       44.0858       46.6858       44.2398       1.2968         10     
reco:gaushit:GausHitFinder                         0.665265       1.17121       1.49359       1.1478       0.211786        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00045315    0.00071095    0.00101492    0.000687055   0.00013157       10     
reco:reco3d:SpacePointSolver                        3.92442       12.0448       31.2652       10.4113       6.78457        10     
reco:hitpdune:DisambigFromSpacePoints              0.680202       1.49698       1.96453       1.60963      0.375619        10     
reco:pandora:StandardPandora                        13.0381       32.0508       77.0382       29.2193       17.3389        10     
reco:pandoraWriter:StandardPandora                 0.255264      0.407598      0.495108      0.411656      0.068255        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.8934        8.21975       10.6751       8.58812       1.81479        10     
reco:pandoraShower:LArPandoraShowerCreation         3.36569       10.2982       14.9609       11.3142       3.61943        10     
reco:pandoracalo:Calorimetry                        1.45091       3.72191       5.16268       3.89843       1.01099        10     
reco:pandoracalonosce:Calorimetry                   1.13404       3.52585       4.82272       3.70762       1.02643        10     
reco:pandorapid:Chi2ParticleID                    0.00328123    0.00467785    0.00615851    0.00467466    0.000680768      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0309586     0.0532047     0.0800662     0.0522031     0.0120171       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00330063    0.00438089    0.00488139    0.00447981    0.000431251      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.57919       9.32464       14.6664       9.73068       3.74583        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.50943       9.13933       14.2503       9.67965       3.71421        10     
reco:emtrkmichelid:EmTrackMichelId                  82.8222       142.901       178.704       144.82        26.1335        10     
reco:crttag:SingleCRTMatchingProducer              0.0155954     0.027778      0.0344072     0.0294897     0.0057499       10     
reco:crtreco:TwoCRTMatchingProducer                0.0134754     0.0239265     0.0304079     0.0244836    0.00510178       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.530764       1.04642       1.68256      0.991659      0.303267        10     
reco:pandora2Track:LArPandoraTrackCreation          9.94955       30.0011       43.1604       30.8374       9.21513        10     
reco:pandora2calo:Calorimetry                       3.89959       12.683        17.9343       13.2242       4.1212         10     
reco:pandora2calonosce:Calorimetry                  3.59577       12.1376       17.2209       12.5004       3.92373        10     
reco:pandora2pid:Chi2ParticleID                   0.00686899     0.0118011     0.0219471     0.0108838    0.00373461       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0383654     0.0665433     0.0984448     0.0663578     0.0151871       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00686183     0.0117015     0.0218769     0.0106872    0.00373087       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.58267       9.64549       12.704        10.2461       2.69308        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.87014       10.6388       14.6493       11.4234       3.5373         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.36353       9.26814       12.7264       9.95405       3.20068        10     
[art]:TriggerResults:TriggerResultInserter         4.107e-05    5.9421e-05    0.000112578   4.95415e-05   2.43784e-05      10     
end_path:out1:RootOutput                           7.245e-06    8.8104e-06     1.258e-05     8.008e-06    1.88576e-06      10     
end_path:out1:RootOutput(write)                     1.20257       2.1573        3.23564       2.10918      0.504098        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4865.83 MB
  Peak resident set size usage (VmHWM): 4056.95 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   2.98 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9276976
-rw-r--r-- 1 pildune32 pildune 2211452508 Nov 24 05:43 detsim_stage2.root
-rw-r--r-- 1 pildune32 pildune 2103837912 Nov 24 05:42 detsim_stage1.root
-rw-r--r-- 1 pildune32 pildune 1857550227 Nov 24 06:47 reco_stage1.root
-rw-r--r-- 1 pildune32 pildune 1655958946 Nov 24 04:35 g4_stage2.root
-rw-r--r-- 1 pildune32 pildune 1604853963 Nov 24 04:24 g4_stage1.root
-rw-r--r-- 1 pildune32 pildune   65355200 Nov 24 06:42 Pandora_Events.pndr
-rw-r--r-- 1 pildune32 pildune     243084 Nov 24 06:47 jobscript.log
-rw-r--r-- 1 pildune32 pildune     217088 Nov 24 06:47 mem.db
-rw-r--r-- 1 pildune32 pildune      62376 Nov 24 06:47 Pandora_Geometry.xml
-rw-r--r-- 1 pildune32 pildune      36864 Nov 24 06:47 time.db
-rw-r--r-- 1 pildune32 pildune        519 Nov 24 05:43 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 pildune32 pildune        519 Nov 24 04:35 g4_protoDUNE_hist.root
-rw-r--r-- 1 pildune32 pildune        519 Nov 24 06:47 protoDUNE_reco1_hist.root
-rw-r--r-- 1 pildune32 pildune         52 Nov 24 04:19 all-input-dids.txt
-rw-r--r-- 1 pildune32 pildune          7 Nov 24 06:47 justin-processed-pfns.txt
-rw-r--r-- 1 pildune32 pildune          0 Nov 24 05:50 ProtoDUNE.log
-rw-r--r-- 1 pildune32 pildune          0 Nov 24 05:43 debugprod.log
justIN time: 2026-02-08 20:07:31 UTC       justIN version: 01.06.00