justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 249530.1@dunegpschedd02.fnal.gov

Jobsub ID249530.1@dunegpschedd02.fnal.gov
Workflow ID10506
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-22 03:26:07
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-11-22 09:02:55
From worker nodeHostnamewn1206241.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-22 03:27:16
Input filesmonte-carlo-010506-000001
JobscriptExit code0
Real time5h (19936s)
CPU time5h (19163s = 96%)
Max RSS bytes3945775104 (3762 MiB)
Outputting started2025-11-22 08:59:33
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10506/1/001/reco_stage1.root
Finished2025-11-22 09:02:55
Saved logsjustin-logs:249530.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

g tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 118122 traces, 0 summary
Retagger: tagging trace set: gauss with 90826 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 90826 traces tagged "gauss"
FrameSaver: q=3.40089e+06 n=1641773 tag=gauss
wclsFrameSaver: saving 118122 traces tagged "wiener"
FrameSaver: q=3.6048e+06 n=1470153 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 27605 XUs and 20378 XVs -> 77 XUVs
C:0 T:1 42741 XUs and 35355 XVs -> 2423 XUVs
C:0 T:2 110295 XUs and 149030 XVs -> 4403 XUVs
C:0 T:3 54764 XUs and 67360 XVs -> 448 XUVs
C:0 T:4 10554 XUs and 12646 XVs -> 58 XUVs
C:0 T:5 35421 XUs and 32748 XVs -> 5191 XUVs
C:0 T:6 132523 XUs and 151686 XVs -> 6728 XUVs
C:0 T:7 50201 XUs and 52519 XVs -> 407 XUVs
C:0 T:8 5265 XUs and 3853 XVs -> 53 XUVs
C:0 T:9 33447 XUs and 31356 XVs -> 5361 XUVs
C:0 T:10 38938 XUs and 25152 XVs -> 4083 XUVs
C:0 T:11 420 XUs and 334 XVs -> 37 XUVs
29269 XUVs total
10658 collection wire objects
27202 potential space points
Neighbour search...
1739966 tests to find 846054 neighbours
Iterating with no regularization...
Begin: 1.08319e+08
0 9.43275e+07
1 9.3201e+07
2 9.30993e+07
3 9.30733e+07
Now with regularization...
Begin: 8.12734e+07
0 8.12435e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410662443
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410662443
fLifetime: 30000 [us]
22-Nov-2025 08:59:29 GMT  Closed output file "reco_stage1.root"
22-Nov-2025 08:59:29 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          563.796       738.082       967.819       761.037       139.776        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00189526    0.00558511    0.00840963    0.00574251    0.00206505       10     
reco:ophit:OpHitFinder                              0.38041      0.420699      0.515643      0.410943      0.0370838       10     
reco:opflash:OpFlashFinder                         0.169955      0.214337      0.280292      0.203985      0.0314492       10     
reco:opslicer:OpSlicer                             0.961893       1.15849       1.35756       1.11131      0.139178        10     
reco:caldata:DataPrepModule                         8.55509       8.9137        9.73254       8.87798      0.301732        10     
reco:wclsdatasp:WireCellToolkit                     61.4319       69.4777       76.4369       68.743        4.29721        10     
reco:gaushit:GausHitFinder                          1.72834       2.09583       2.53889       2.16413      0.295511        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00107826    0.00145383    0.00175942     0.0014178    0.000238886      10     
reco:reco3d:SpacePointSolver                        16.6597       27.7072       45.5522       28.0065       9.21795        10     
reco:hitpdune:DisambigFromSpacePoints               1.96679       2.80922       3.92461       2.89642      0.716209        10     
reco:pandora:StandardPandora                        33.3671       73.4096       148.21        69.9501       32.1442        10     
reco:pandoraWriter:StandardPandora                 0.797084      0.952736       1.19255      0.943119      0.121499        10     
reco:pandoraTrack:LArPandoraTrackCreation           6.66749       10.4024       16.4099       9.4472        3.12718        10     
reco:pandoraShower:LArPandoraShowerCreation         6.55621       12.1245       21.7451       12.6004       4.02618        10     
reco:pandoracalo:Calorimetry                        3.17358       5.79813       9.17451       5.79936       2.0373         10     
reco:pandoracalonosce:Calorimetry                   2.72831       5.37535       9.48573       5.36543       2.02344        10     
reco:pandorapid:Chi2ParticleID                    0.00727419     0.010758      0.0146252     0.010506     0.00210623       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0787689     0.102627      0.128362      0.103919      0.0158475       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00795792     0.0102543     0.0125885     0.0100998    0.00149037       10     
reco:pandoraShowercalo:ShowerCalorimetry            5.74378       14.5424       24.7799       16.1324       6.28441        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.46198       13.9224       26.4913       14.6808       6.37132        10     
reco:emtrkmichelid:EmTrackMichelId                  314.082       370.577       441.932       369.646       47.0382        10     
reco:crttag:SingleCRTMatchingProducer              0.0179721     0.0280406     0.0394621     0.0270579    0.00595673       10     
reco:crtreco:TwoCRTMatchingProducer                0.0180199     0.0256373     0.0323701     0.0241931     0.0050369       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.05785       1.40771       1.69124       1.35329      0.213935        10     
reco:pandora2Track:LArPandoraTrackCreation          21.8468       34.2446       58.5786       34.8941       10.6639        10     
reco:pandora2calo:Calorimetry                       10.2201       18.6749       33.1049       19.2135       7.2173         10     
reco:pandora2calonosce:Calorimetry                  9.35066       17.6408        28.87        19.9436       6.31499        10     
reco:pandora2pid:Chi2ParticleID                    0.0200702     0.0257039     0.0325659     0.025957     0.00367422       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.105091      0.128233      0.157172      0.126454      0.0144477       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.019951      0.0249672     0.032482      0.0254414    0.00400289       10     
reco:pandora2Shower:LArPandoraShowerCreation        6.98816       11.6091       18.6346       12.2859       3.26351        10     
reco:pandora2Showercalo:ShowerCalorimetry           6.8718        15.877        28.9794       16.9806       6.34511        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.23087       13.3509       22.0758       13.6317       5.17189        10     
[art]:TriggerResults:TriggerResultInserter        4.4324e-05    6.88706e-05   0.000176212    5.293e-05    3.77515e-05      10     
end_path:out1:RootOutput                           7.545e-06    8.9821e-06    1.4157e-05     8.531e-06    1.81846e-06      10     
end_path:out1:RootOutput(write)                     3.87743       4.74945       6.05089       4.81862      0.694516        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4792.57 MB
  Peak resident set size usage (VmHWM): 3945.78 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.37 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.58 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9338512
-rw-r--r-- 1 dune01 dune01 2237353490 Nov 22 06:53 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2126128488 Nov 22 06:49 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1908439007 Nov 22 08:59 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1646880378 Nov 22 03:54 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1571032212 Nov 22 03:36 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   72175970 Nov 22 08:50 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242523 Nov 22 08:59 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov 22 08:59 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov 22 08:59 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov 22 08:59 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov 22 06:53 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 22 03:54 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 22 08:59 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov 22 03:27 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov 22 08:59 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov 22 07:06 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov 22 06:53 debugprod.log
justIN time: 2026-02-08 22:54:39 UTC       justIN version: 01.06.00