justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 236982.0@dunegpschedd02.fnal.gov

Jobsub ID236982.0@dunegpschedd02.fnal.gov
Workflow ID9576
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-03 21:49:19
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-11-04 02:45:06
From worker nodeHostnamewn1905250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 21:50:04
Input filesmonte-carlo-009576-000001
JobscriptExit code0
Real time4h (17611s)
CPU time4h (17060s = 96%)
Max RSS bytes4423168000 (4218 MiB)
Outputting started2025-11-04 02:43:36
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09576/1/001/reco_stage1.root
Finished2025-11-04 02:45:06
Saved logsjustin-logs:236982.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

leKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 182951 traces, 0 summary
Retagger: tagging trace set: gauss with 136456 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 136456 traces tagged "gauss"
FrameSaver: q=4.68217e+06 n=2381876 tag=gauss
wclsFrameSaver: saving 182951 traces tagged "wiener"
FrameSaver: q=4.96274e+06 n=2096381 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 46255 XUs and 56047 XVs -> 756 XUVs
C:0 T:1 108052 XUs and 114632 XVs -> 6241 XUVs
C:0 T:2 85700 XUs and 108402 XVs -> 4803 XUVs
C:0 T:3 41680 XUs and 42060 XVs -> 285 XUVs
C:0 T:4 14635 XUs and 20712 XVs -> 170 XUVs
C:0 T:5 76722 XUs and 82081 XVs -> 5006 XUVs
C:0 T:6 131502 XUs and 173622 XVs -> 6457 XUVs
C:0 T:7 50734 XUs and 66466 XVs -> 807 XUVs
C:0 T:8 10677 XUs and 9006 XVs -> 169 XUVs
C:0 T:9 70698 XUs and 81552 XVs -> 6603 XUVs
C:0 T:10 53948 XUs and 76651 XVs -> 11063 XUVs
C:0 T:11 130 XUs and 67 XVs -> 8 XUVs
42368 XUVs total
16090 collection wire objects
38819 potential space points
Neighbour search...
2155363 tests to find 1203472 neighbours
Iterating with no regularization...
Begin: 9.34645e+07
0 7.71733e+07
1 7.62123e+07
2 7.60799e+07
3 7.60422e+07
Now with regularization...
Begin: 6.34121e+07
0 6.33675e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410295566
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410295566
fLifetime: 30000 [us]
04-Nov-2025 02:43:34 GMT  Closed output file "reco_stage1.root"
04-Nov-2025 02:43:34 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          466.289       617.773       748.667       609.699       74.6466        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00151952    0.00606153     0.0109826    0.00545075    0.00262734       10     
reco:ophit:OpHitFinder                             0.305437      0.417901      0.478246      0.431508      0.0477693       10     
reco:opflash:OpFlashFinder                         0.216275      0.276817      0.324084      0.281114      0.0321721       10     
reco:opslicer:OpSlicer                              0.60277      0.952815       1.25171      0.997873      0.186748        10     
reco:caldata:DataPrepModule                         6.91448       7.44582       7.89287       7.56442      0.367265        10     
reco:wclsdatasp:WireCellToolkit                     41.4876       45.8303       49.4642       46.2502       2.61873        10     
reco:gaushit:GausHitFinder                          1.6909        2.00591       2.30376       2.02018      0.215252        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000651858   0.000963772    0.0018079    0.000887741   0.000326994      10     
reco:reco3d:SpacePointSolver                        20.8646       30.6635       42.9642       30.1325       6.99208        10     
reco:hitpdune:DisambigFromSpacePoints               2.16874       2.92501       3.62306       2.96046      0.431399        10     
reco:pandora:StandardPandora                        48.9203       70.6322       102.54        62.5688       17.6379        10     
reco:pandoraWriter:StandardPandora                 0.659767      0.773463      0.937828      0.764636      0.0871183       10     
reco:pandoraTrack:LArPandoraTrackCreation           8.8675        10.5852       12.8563       10.2816       1.27309        10     
reco:pandoraShower:LArPandoraShowerCreation         8.94838       12.9446       16.5665       13.096        2.3657         10     
reco:pandoracalo:Calorimetry                        5.25457       6.25762       7.08863       6.3267       0.582971        10     
reco:pandoracalonosce:Calorimetry                   4.62367       5.95422       7.59645       6.14899      0.839068        10     
reco:pandorapid:Chi2ParticleID                    0.00605676     0.0100829      0.01195      0.0107735    0.00196435       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0762179      0.09373       0.11476      0.0957815     0.0104547       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00519569    0.00903982     0.0119381     0.0094123    0.00202572       10     
reco:pandoraShowercalo:ShowerCalorimetry            7.49229       13.762        18.5627       13.8023       2.78568        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       7.32733       13.2478       17.4338       13.3484       2.65608        10     
reco:emtrkmichelid:EmTrackMichelId                  221.418       279.316       337.446       274.621       31.202         10     
reco:crttag:SingleCRTMatchingProducer              0.0165273     0.0283677     0.0400022     0.0294494    0.00641886       10     
reco:crtreco:TwoCRTMatchingProducer                0.0111477     0.020279      0.025938      0.0213178    0.00459524       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.0158        1.24916       1.60999       1.2506       0.176534        10     
reco:pandora2Track:LArPandoraTrackCreation          21.5974       33.0899       40.9038       34.0213       5.61266        10     
reco:pandora2calo:Calorimetry                       10.9506       18.2297       22.6476       18.424        3.1234         10     
reco:pandora2calonosce:Calorimetry                  12.2651       17.1379       22.1782       17.1097       2.75818        10     
reco:pandora2pid:Chi2ParticleID                    0.0198967     0.0252113     0.030154      0.025273     0.00262619       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0942445     0.121754      0.144203      0.120672      0.0143911       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.011394      0.0229703     0.0299393     0.0237622    0.00489785       10     
reco:pandora2Shower:LArPandoraShowerCreation        8.66439       11.4534       16.0416       10.9086       1.97566        10     
reco:pandora2Showercalo:ShowerCalorimetry           10.1464       14.9275       20.442        14.6206       2.56318        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      7.9576        12.6311       18.2259       12.5073       2.44507        10     
[art]:TriggerResults:TriggerResultInserter        7.1695e-05    0.00010778    0.000220595   9.9708e-05    4.2299e-05       10     
end_path:out1:RootOutput                           7.184e-06    1.12222e-05   1.9106e-05    1.0685e-05    2.89749e-06      10     
end_path:out1:RootOutput(write)                     3.34031       4.48272       5.16696       4.40024      0.490059        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5234.38 MB
  Peak resident set size usage (VmHWM): 4405.95 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.12 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.34 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.66 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11413156
-rw-r--r-- 1 dune01 dune01 2642748410 Nov  4 00:57 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2502777702 Nov  4 00:54 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2396056034 Nov  4 02:43 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 2079409533 Nov  3 22:18 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1974868688 Nov  3 21:58 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   90580070 Nov  4 02:34 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     243105 Nov  4 02:43 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov  4 02:43 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov  4 02:43 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov  4 02:43 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov  4 00:57 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov  3 22:18 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov  4 02:43 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov  3 21:50 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov  4 02:43 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov  4 01:06 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov  4 00:57 debugprod.log
justIN time: 2026-02-11 12:01:28 UTC       justIN version: 01.06.00