justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 251503.1@dunegpschedd01.fnal.gov

Jobsub ID251503.1@dunegpschedd01.fnal.gov
Workflow ID10212
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-15 08:06:33
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-11-15 14:58:18
From worker nodeHostnamewn1209250.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-15 08:07:17
Input filesmonte-carlo-010212-000001
JobscriptExit code0
Real time6h (24541s)
CPU time6h (23757s = 96%)
Max RSS bytes4102844416 (3912 MiB)
Outputting started2025-11-15 14:56:19
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10212/1/001/reco_stage1.root
Finished2025-11-15 14:58:18
Saved logsjustin-logs:251503.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

AllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 201389 traces, 0 summary
Retagger: tagging trace set: gauss with 150751 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 150751 traces tagged "gauss"
FrameSaver: q=5.35319e+06 n=2661089 tag=gauss
wclsFrameSaver: saving 201389 traces tagged "wiener"
FrameSaver: q=5.66752e+06 n=2341857 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 55208 XUs and 41367 XVs -> 817 XUVs
C:0 T:1 121989 XUs and 88797 XVs -> 9828 XUVs
C:0 T:2 148380 XUs and 250127 XVs -> 11431 XUVs
C:0 T:3 63784 XUs and 81548 XVs -> 805 XUVs
C:0 T:4 17310 XUs and 12164 XVs -> 168 XUVs
C:0 T:5 84236 XUs and 42970 XVs -> 9075 XUVs
C:0 T:6 155481 XUs and 242562 XVs -> 13679 XUVs
C:0 T:7 55887 XUs and 67527 XVs -> 701 XUVs
C:0 T:8 14271 XUs and 9892 XVs -> 172 XUVs
C:0 T:9 121990 XUs and 108388 XVs -> 9569 XUVs
C:0 T:10 74670 XUs and 87774 XVs -> 8088 XUVs
C:0 T:11 207 XUs and 130 XVs -> 17 XUVs
64350 XUVs total
18809 collection wire objects
58519 potential space points
Neighbour search...
6890629 tests to find 3268578 neighbours
Iterating with no regularization...
Begin: 1.12927e+08
0 9.34769e+07
1 9.21257e+07
2 9.19314e+07
3 9.18743e+07
Now with regularization...
Begin: 7.58548e+07
0 7.58079e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410525516
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410525516
fLifetime: 30000 [us]
15-Nov-2025 14:56:12 GMT  Closed output file "reco_stage1.root"
15-Nov-2025 14:56:12 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          546.388       787.322       1416.4        745.832       247.007        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.0022654    0.00507199     0.0087603    0.00477907     0.0017823       10     
reco:ophit:OpHitFinder                             0.365585      0.471931      0.641559      0.463523      0.0713847       10     
reco:opflash:OpFlashFinder                         0.167261      0.271681      0.441922      0.268467       0.07078        10     
reco:opslicer:OpSlicer                              0.7631        1.13867       1.82247       1.06909      0.274898        10     
reco:caldata:DataPrepModule                         10.288        10.8279       11.9967       10.6517      0.496241        10     
reco:wclsdatasp:WireCellToolkit                     77.5295       83.5706       89.9774       84.5408       3.63333        10     
reco:gaushit:GausHitFinder                          1.72583       2.42113       4.10572       2.31416      0.629163        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000698114   0.00127248    0.00422615    0.000954764   0.00100975       10     
reco:reco3d:SpacePointSolver                        12.0976       26.0339       53.2965       23.2776       11.728         10     
reco:hitpdune:DisambigFromSpacePoints               2.6844        4.09162       8.49548       3.55879       1.67682        10     
reco:pandora:StandardPandora                        37.2541       64.0636       142.927       55.2866       30.4025        10     
reco:pandoraWriter:StandardPandora                 0.607116      0.863202       1.35765      0.818372      0.209713        10     
reco:pandoraTrack:LArPandoraTrackCreation           6.95441       12.8958       25.4557       11.2413       5.29131        10     
reco:pandoraShower:LArPandoraShowerCreation         7.15554       14.9602       35.3985       12.7137       8.02405        10     
reco:pandoracalo:Calorimetry                        3.34169       6.10853       12.6324       5.61459       2.65603        10     
reco:pandoracalonosce:Calorimetry                   3.08785       5.74658       11.494        5.14577       2.46091        10     
reco:pandorapid:Chi2ParticleID                    0.00835119     0.0109594     0.0145834     0.0106682    0.00201053       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.086802      0.117151      0.163174      0.123018      0.0247456       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00808021    0.00998741     0.0139064    0.00956029    0.00159809       10     
reco:pandoraShowercalo:ShowerCalorimetry            6.20599       14.6686       37.7891       12.7786       9.00297        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       6.45796       14.4605       36.2252       12.8159       8.56466        10     
reco:emtrkmichelid:EmTrackMichelId                  282.547       383.827       619.315       369.377       93.7712        10     
reco:crttag:SingleCRTMatchingProducer              0.0274854     0.0408694     0.0605408     0.0385175     0.0119906       10     
reco:crtreco:TwoCRTMatchingProducer                0.0207887     0.0312032     0.0474367     0.0300143    0.00847705       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.29514       1.90707       2.67709       1.88697       0.35278        10     
reco:pandora2Track:LArPandoraTrackCreation          24.7548       45.6591       107.185       38.9513       23.8966        10     
reco:pandora2calo:Calorimetry                       9.56593       20.2426       50.1433       18.0759       11.4603        10     
reco:pandora2calonosce:Calorimetry                  9.27081       19.0009       45.6847       17.0508       10.488         10     
reco:pandora2pid:Chi2ParticleID                    0.020397      0.0275504     0.0416357     0.0253889    0.00622201       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0740022     0.136832      0.213209      0.137218      0.0355736       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0199143     0.0255972     0.0391034     0.0245111    0.00536103       10     
reco:pandora2Shower:LArPandoraShowerCreation        8.38863       15.1632       33.4695       12.7037       7.21096        10     
reco:pandora2Showercalo:ShowerCalorimetry           8.47426       17.4832       42.2473       15.3553       9.80076        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      7.39308       15.814        38.0112       14.3789       8.98762        10     
[art]:TriggerResults:TriggerResultInserter        7.8971e-05    0.000134303   0.000245035   0.000124757   4.21986e-05      10     
end_path:out1:RootOutput                          1.3516e-05    1.89822e-05    2.682e-05     1.776e-05    4.17492e-06      10     
end_path:out1:RootOutput(write)                     3.57361          5          7.93742       4.89845       1.24101        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4916.16 MB
  Peak resident set size usage (VmHWM): 4102.84 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.42 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   6.51 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.37 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9707956
-rw-r--r-- 1 dune01 dune01 2285887737 Nov 15 12:41 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2169717719 Nov 15 12:36 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1971264377 Nov 15 14:56 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1753177303 Nov 15 08:43 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1685987618 Nov 15 08:18 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   74282030 Nov 15 14:37 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242609 Nov 15 14:56 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov 15 14:56 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov 15 14:56 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov 15 14:56 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov 15 12:41 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 15 08:43 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 15 14:56 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov 15 08:07 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov 15 14:56 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov 15 12:53 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov 15 12:41 debugprod.log
justIN time: 2026-02-09 12:24:32 UTC       justIN version: 01.06.00