justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 249531.0@dunegpschedd02.fnal.gov

Jobsub ID249531.0@dunegpschedd02.fnal.gov
Workflow ID10510
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-22 03:26:07
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-11-22 08:25:43
From worker nodeHostnamewn6219291.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-22 03:27:04
Input filesmonte-carlo-010510-000001
JobscriptExit code0
Real time4h (17647s)
CPU time4h (17077s = 96%)
Max RSS bytes4039499776 (3852 MiB)
Outputting started2025-11-22 08:21:13
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10510/1/001/reco_stage1.root
Finished2025-11-22 08:25:43
Saved logsjustin-logs:249531.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

alFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 105622 traces, 0 summary
Retagger: tagging trace set: gauss with 81138 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 81138 traces tagged "gauss"
FrameSaver: q=2.82817e+06 n=1442447 tag=gauss
wclsFrameSaver: saving 105622 traces tagged "wiener"
FrameSaver: q=2.99337e+06 n=1287228 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 38269 XUs and 41214 XVs -> 491 XUVs
C:0 T:1 73196 XUs and 89014 XVs -> 5042 XUVs
C:0 T:2 84753 XUs and 120729 XVs -> 4368 XUVs
C:0 T:3 46550 XUs and 58380 XVs -> 758 XUVs
C:0 T:4 9012 XUs and 9261 XVs -> 186 XUVs
C:0 T:5 27221 XUs and 27160 XVs -> 2653 XUVs
C:0 T:6 80038 XUs and 79082 XVs -> 3919 XUVs
C:0 T:7 31252 XUs and 29667 XVs -> 335 XUVs
C:0 T:8 4431 XUs and 4487 XVs -> 72 XUVs
C:0 T:9 25869 XUs and 32633 XVs -> 2056 XUVs
C:0 T:10 14861 XUs and 5943 XVs -> 2731 XUVs
C:0 T:11 165 XUs and 222 XVs -> 41 XUVs
22652 XUVs total
8550 collection wire objects
20294 potential space points
Neighbour search...
898356 tests to find 481220 neighbours
Iterating with no regularization...
Begin: 6.26307e+07
0 4.76122e+07
1 4.65053e+07
2 4.63846e+07
3 4.63533e+07
Now with regularization...
Begin: 3.86368e+07
0 3.86408e+07
Warning: metric increased
run: 1 ; subrun: 0 ; event: 10
evttime: 410662401
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410662401
fLifetime: 30000 [us]
22-Nov-2025 08:21:07 GMT  Closed output file "reco_stage1.root"
22-Nov-2025 08:21:07 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          461.962       603.056       710.116       630.247       73.0764        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00170104    0.00504758    0.00853127    0.00503574    0.00239174       10     
reco:ophit:OpHitFinder                             0.286804      0.391231      0.476233      0.399355      0.0579834       10     
reco:opflash:OpFlashFinder                         0.155329       0.22185      0.301782      0.213799      0.0403127       10     
reco:opslicer:OpSlicer                              0.71805       1.03938       1.32967       1.06291      0.217002        10     
reco:caldata:DataPrepModule                         6.00286       7.98232       9.14947       8.30958      0.984265        10     
reco:wclsdatasp:WireCellToolkit                     44.001        54.3452       61.4229       55.8723       6.1219         10     
reco:gaushit:GausHitFinder                          1.50582       1.93763       2.41391       1.89658      0.261587        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00057506    0.000897652   0.00110101    0.000891174   0.000151996      10     
reco:reco3d:SpacePointSolver                        18.2545       23.6508       31.8739       23.4385       4.64738        10     
reco:hitpdune:DisambigFromSpacePoints               2.00782       2.39859       3.17435       2.16014      0.397498        10     
reco:pandora:StandardPandora                        30.6114       47.9924       72.7125       46.736        11.584         10     
reco:pandoraWriter:StandardPandora                 0.447622      0.718148      0.885156      0.741928      0.135301        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.5164        7.29564       9.81077       7.49836       1.95704        10     
reco:pandoraShower:LArPandoraShowerCreation         4.52343       8.01116       12.3503       7.97479       2.09197        10     
reco:pandoracalo:Calorimetry                        2.0246        3.68502       5.17739       3.64677      0.928385        10     
reco:pandoracalonosce:Calorimetry                   2.51039       3.52983       4.68756       3.29345      0.832731        10     
reco:pandorapid:Chi2ParticleID                    0.00614118    0.00924567     0.0131989    0.00969923    0.00197484       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0540072     0.0902901     0.113771      0.0949868     0.0196079       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00603819     0.0090963     0.0115214    0.00931957    0.00151627       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.73399       8.4132        13.3246       7.40225       3.01193        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.64068       9.25113       14.6459       10.452        3.80202        10     
reco:emtrkmichelid:EmTrackMichelId                  283.224       336.647       380.438       344.424       32.6679        10     
reco:crttag:SingleCRTMatchingProducer              0.0107221     0.0210213     0.0276092     0.0219411    0.00488333       10     
reco:crtreco:TwoCRTMatchingProducer                0.0100219     0.0190252     0.0286984     0.0202102    0.00523144       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.850454       1.29513       1.72339       1.2372       0.263635        10     
reco:pandora2Track:LArPandoraTrackCreation          14.7313       24.7736       30.6422       27.1086       5.6785         10     
reco:pandora2calo:Calorimetry                       6.35725       13.0144       18.7408       13.138        4.21716        10     
reco:pandora2calonosce:Calorimetry                  5.98282       12.5126       19.8432       13.6044       4.27742        10     
reco:pandora2pid:Chi2ParticleID                    0.0138545     0.0218034     0.0285225     0.0217664    0.00424271       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0644359     0.111293       0.14223      0.115272      0.0220905       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.014115      0.0228191     0.0286255     0.0239083     0.0044899       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.86573       8.5577        11.9442       9.13994       1.86443        10     
reco:pandora2Showercalo:ShowerCalorimetry           6.18576       10.7956       16.7983       10.1667       3.43064        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.16301       9.57327       12.7428       10.4282       3.08716        10     
[art]:TriggerResults:TriggerResultInserter        4.0596e-05    5.94075e-05   0.000169057   4.64875e-05   3.69886e-05      10     
end_path:out1:RootOutput                           6.462e-06    8.0579e-06    1.6621e-05    7.1635e-06    2.89341e-06      10     
end_path:out1:RootOutput(write)                     3.18959       4.51419       5.46638       4.66542      0.718784        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4879.35 MB
  Peak resident set size usage (VmHWM): 4039.5 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.34 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.04 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9556488
-rw-r--r-- 1 dune01 dune01 2271497804 Nov 22 06:37 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2156726725 Nov 22 06:34 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1943069969 Nov 22 08:21 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1702988167 Nov 22 03:52 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1639644139 Nov 22 03:35 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   71288490 Nov 22 08:14 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242685 Nov 22 08:21 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov 22 08:21 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov 22 08:21 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov 22 08:21 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov 22 06:37 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 22 03:52 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 22 08:21 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov 22 03:27 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov 22 08:21 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov 22 06:47 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov 22 06:37 debugprod.log
justIN time: 2026-02-08 22:55:41 UTC       justIN version: 01.06.00