justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 250226.1@dunegpschedd02.fnal.gov

Jobsub ID250226.1@dunegpschedd02.fnal.gov
Workflow ID10559
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-23 22:57:13
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-11-24 04:01:13
From worker nodeHostnamewn-lb-08.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-23 22:58:57
Input filesmonte-carlo-010559-000001
JobscriptExit code0
Real time4h (17163s)
CPU time4h (16435s = 95%)
Max RSS bytes4151992320 (3959 MiB)
Outputting started2025-11-24 03:45:01
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10559/1/001/reco_stage1.root
Finished2025-11-24 04:01:13
Saved logsjustin-logs:250226.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

caleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 142298 traces, 0 summary
Retagger: tagging trace set: gauss with 108537 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 108537 traces tagged "gauss"
FrameSaver: q=3.89326e+06 n=1974442 tag=gauss
wclsFrameSaver: saving 142298 traces tagged "wiener"
FrameSaver: q=4.12698e+06 n=1761253 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 47016 XUs and 53021 XVs -> 751 XUVs
C:0 T:1 101301 XUs and 105627 XVs -> 9644 XUVs
C:0 T:2 81363 XUs and 111392 XVs -> 4637 XUVs
C:0 T:3 41397 XUs and 42311 XVs -> 249 XUVs
C:0 T:4 13616 XUs and 12680 XVs -> 104 XUVs
C:0 T:5 46994 XUs and 33213 XVs -> 4710 XUVs
C:0 T:6 135599 XUs and 128738 XVs -> 5956 XUVs
C:0 T:7 57287 XUs and 47888 XVs -> 401 XUVs
C:0 T:8 7464 XUs and 9466 XVs -> 56 XUVs
C:0 T:9 45694 XUs and 69303 XVs -> 4356 XUVs
C:0 T:10 55277 XUs and 21264 XVs -> 4135 XUVs
C:0 T:11 145 XUs and 124 XVs -> 19 XUVs
35018 XUVs total
12482 collection wire objects
32373 potential space points
Neighbour search...
1653329 tests to find 886050 neighbours
Iterating with no regularization...
Begin: 8.34807e+07
0 6.89968e+07
1 6.79134e+07
2 6.77632e+07
3 6.77205e+07
Now with regularization...
Begin: 5.64619e+07
0 5.64331e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410698086
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410698086
fLifetime: 30000 [us]
24-Nov-2025 04:44:59 CET  Closed output file "reco_stage1.root"
24-Nov-2025 04:44:59 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          328.772       463.771       549.458       497.786       77.3391        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00128221    0.00628032     0.0116916     0.0067987    0.00283627       10     
reco:ophit:OpHitFinder                             0.337408      0.418831      0.479658      0.421226      0.0390405       10     
reco:opflash:OpFlashFinder                         0.186512      0.277381       0.33903      0.290505      0.0428679       10     
reco:opslicer:OpSlicer                             0.637235      0.846571       1.00563      0.834127      0.111952        10     
reco:caldata:DataPrepModule                         6.20476       6.57729       7.17471       6.49117      0.255013        10     
reco:wclsdatasp:WireCellToolkit                     45.532         63.65        70.8771       67.7884       9.03226        10     
reco:gaushit:GausHitFinder                          1.08351       1.46604       1.87353       1.42388      0.217433        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000379568   0.000747314   0.00119499    0.000684961   0.000242232      10     
reco:reco3d:SpacePointSolver                        8.22897       17.5649       23.0422       20.277        5.41805        10     
reco:hitpdune:DisambigFromSpacePoints               1.08105       1.84269       2.31666       2.05283      0.428889        10     
reco:pandora:StandardPandora                        18.6907       39.9752       60.9121       43.7839       13.1225        10     
reco:pandoraWriter:StandardPandora                 0.393826      0.532622      0.650995      0.559163      0.0766377       10     
reco:pandoraTrack:LArPandoraTrackCreation           3.43162       6.35412       9.39933       6.53253       1.79894        10     
reco:pandoraShower:LArPandoraShowerCreation         3.64081       8.4367        17.1951       8.93297       3.66531        10     
reco:pandoracalo:Calorimetry                        1.83869       3.65273       5.32347       3.67971       1.11027        10     
reco:pandoracalonosce:Calorimetry                   1.67979        3.47         5.00458       3.60916       1.06285        10     
reco:pandorapid:Chi2ParticleID                     0.0054213    0.00622499    0.00705994    0.00617172    0.000476518      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0457405     0.0596983     0.0685473     0.0627499    0.00783024       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00502795    0.00593964    0.00672474    0.00607078    0.000588012      10     
reco:pandoraShowercalo:ShowerCalorimetry            3.02549       8.45053       15.0077       9.27913       3.3309         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.9659        8.10575       13.6131       9.12341       3.01958        10     
reco:emtrkmichelid:EmTrackMichelId                  167.005       219.339       250.187       225.955       28.9542        10     
reco:crttag:SingleCRTMatchingProducer              0.0118714     0.0187673     0.0273182     0.0175021    0.00515629       10     
reco:crtreco:TwoCRTMatchingProducer               0.00944265     0.015689      0.0286341     0.0144914    0.00499024       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.803086       1.13504       1.37491       1.16991      0.177316        10     
reco:pandora2Track:LArPandoraTrackCreation          11.396        22.1574       29.8501       24.4906       6.0707         10     
reco:pandora2calo:Calorimetry                       4.91626       10.9068       15.0675       12.4428       3.3906         10     
reco:pandora2calonosce:Calorimetry                  4.72027       10.388        14.3293       11.881        3.41902        10     
reco:pandora2pid:Chi2ParticleID                    0.0121447     0.0149153     0.0175138     0.0152024    0.00193134       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0591626     0.0751545     0.083834      0.0803406    0.00946737       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0118218     0.0143851     0.0170527     0.0146012    0.00179586       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.05837       7.82429       10.8395       8.13172       2.2647         10     
reco:pandora2Showercalo:ShowerCalorimetry           4.12202       8.78419       11.7128       9.76782       2.7715         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.37285       7.62277       10.5889       8.42523       2.44151        10     
[art]:TriggerResults:TriggerResultInserter         3.988e-05    5.82875e-05   0.000125468   4.98705e-05   2.32948e-05      10     
end_path:out1:RootOutput                           7.632e-06    1.03365e-05   2.2324e-05     8.452e-06    4.4305e-06       10     
end_path:out1:RootOutput(write)                     2.61327       3.53601       3.9424        3.74288      0.464296        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4953.01 MB
  Peak resident set size usage (VmHWM): 4151.99 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.11 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10264580
-rw-r--r--. 1 dune009 dune 2398420340 Nov 24 03:25 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2273322356 Nov 24 03:23 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2099286145 Nov 24 04:44 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1871391668 Nov 24 00:32 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1789477354 Nov 24 00:16 g4_stage1.root
-rw-r--r--. 1 dune009 dune   78415170 Nov 24 04:38 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242805 Nov 24 04:45 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov 24 04:44 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov 24 04:44 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov 24 04:44 time.db
-rw-r--r--. 1 dune009 dune        519 Nov 24 03:25 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 24 00:32 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 24 04:44 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov 24 00:08 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov 24 04:45 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov 24 03:32 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov 24 03:25 debugprod.log
justIN time: 2026-02-08 20:09:27 UTC       justIN version: 01.06.00