justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263475.1@dunegpschedd02.fnal.gov

Jobsub ID263475.1@dunegpschedd02.fnal.gov
Workflow ID11282
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-19 20:07:18
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-12-19 23:42:53
From worker nodeHostnamewn-da-02.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-19 20:08:38
Input filesmonte-carlo-011282-000001
JobscriptExit code0
Real time3h (12769s)
CPU time3h (12632s = 98%)
Max RSS bytes4263288832 (4065 MiB)
Outputting started2025-12-19 23:41:28
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11282/1/001/reco_stage1.root
Finished2025-12-19 23:42:53
Saved logsjustin-logs:263475.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

BasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 98582 traces, 0 summary
Retagger: tagging trace set: gauss with 75412 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 75412 traces tagged "gauss"
FrameSaver: q=2.59723e+06 n=1327411 tag=gauss
wclsFrameSaver: saving 98582 traces tagged "wiener"
FrameSaver: q=2.74723e+06 n=1176175 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 19274 XUs and 19132 XVs -> 123 XUVs
C:0 T:1 29832 XUs and 29256 XVs -> 1833 XUVs
C:0 T:2 55399 XUs and 122353 XVs -> 5669 XUVs
C:0 T:3 25951 XUs and 38405 XVs -> 248 XUVs
C:0 T:4 12450 XUs and 10948 XVs -> 88 XUVs
C:0 T:5 36680 XUs and 30228 XVs -> 3535 XUVs
C:0 T:6 53247 XUs and 116472 XVs -> 6361 XUVs
C:0 T:7 18801 XUs and 32986 XVs -> 308 XUVs
C:0 T:8 4894 XUs and 5101 XVs -> 57 XUVs
C:0 T:9 26375 XUs and 37970 XVs -> 3251 XUVs
C:0 T:10 32464 XUs and 38551 XVs -> 5706 XUVs
C:0 T:11 42 XUs and 33 XVs -> 6 XUVs
27185 XUVs total
8545 collection wire objects
24964 potential space points
Neighbour search...
1918034 tests to find 1007070 neighbours
Iterating with no regularization...
Begin: 9.06192e+07
0 4.53178e+07
1 4.3256e+07
2 4.30705e+07
3 4.30025e+07
4 4.29666e+07
Now with regularization...
Begin: 3.57355e+07
0 3.55978e+07
1 3.55739e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411219585
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411219585
fLifetime: 30000 [us]
20-Dec-2025 00:41:26 CET  Closed output file "reco_stage1.root"
20-Dec-2025 00:41:26 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          280.768       405.033       544.854       406.597       86.974         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000939912   0.00520253    0.00921879    0.00567042     0.0027405       10     
reco:ophit:OpHitFinder                              0.25547      0.302242      0.354096      0.304802      0.0324128       10     
reco:opflash:OpFlashFinder                         0.159953      0.217489      0.272632      0.220094      0.039172        10     
reco:opslicer:OpSlicer                             0.586358      0.718132      0.878739      0.694895      0.0995756       10     
reco:caldata:DataPrepModule                         4.66059       4.87236       5.07201       4.87897      0.107353        10     
reco:wclsdatasp:WireCellToolkit                     33.8638       43.7366       53.3295       44.7055       6.51419        10     
reco:gaushit:GausHitFinder                         0.952011       1.26724       1.55324       1.31854      0.204701        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000547333   0.000794697   0.00174892    0.000656977   0.00034581       10     
reco:reco3d:SpacePointSolver                        5.95723       13.8726       22.2664       13.4224       5.38154        10     
reco:hitpdune:DisambigFromSpacePoints              0.869766       1.57704       2.39844       1.52737      0.488209        10     
reco:pandora:StandardPandora                        24.4094       39.1137       56.9745       38.7207       11.2815        10     
reco:pandoraWriter:StandardPandora                 0.340471      0.445266      0.573554      0.454213      0.0682619       10     
reco:pandoraTrack:LArPandoraTrackCreation           3.62347       6.09342       8.89039       6.47433       1.80512        10     
reco:pandoraShower:LArPandoraShowerCreation         2.69055       6.62763       10.8919       6.48035       2.63796        10     
reco:pandoracalo:Calorimetry                        1.71303       3.28818       5.14273       3.67024       1.1184         10     
reco:pandoracalonosce:Calorimetry                   1.58949        2.903        3.87568       3.14555      0.863752        10     
reco:pandorapid:Chi2ParticleID                    0.00522849    0.00663792    0.00857444     0.0063751    0.00102186       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0372956     0.0497569     0.0634438     0.0511701    0.00845902       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00518689     0.0062794    0.00781052    0.00629933    0.000800287      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.25385       6.30377       8.47951       7.62732       2.36726        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.16853       5.92558       8.3513        6.58249       2.2513         10     
reco:emtrkmichelid:EmTrackMichelId                  153.778       207.446       268.817       206.369       36.3878        10     
reco:crttag:SingleCRTMatchingProducer             0.00907063     0.0195626     0.0262367     0.0187151    0.00559521       10     
reco:crtreco:TwoCRTMatchingProducer               0.00586914     0.0123328     0.0195858     0.0121938    0.00428072       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.560322      0.796339       1.0406       0.773208      0.143456        10     
reco:pandora2Track:LArPandoraTrackCreation          7.38052       19.3297       33.7389       17.4289       8.2857         10     
reco:pandora2calo:Calorimetry                       3.35275       8.91338       15.8178       7.72932       4.07947        10     
reco:pandora2calonosce:Calorimetry                  3.1003        8.45907       14.1734       7.74669       3.60571        10     
reco:pandora2pid:Chi2ParticleID                    0.0118891     0.0155223     0.0200349     0.0154608    0.00261843       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP         0.04701      0.0654486     0.0877722     0.0644384     0.0129528       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0117566     0.0154677     0.0206392     0.0153697    0.00277659       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.09469       6.64787       11.0547       6.1387        2.55412        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.75089       7.1092        13.8819       6.12349       3.41413        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.42689       5.78481       9.78615       5.47384       2.38372        10     
[art]:TriggerResults:TriggerResultInserter        3.1309e-05    4.95355e-05   0.000106112   4.3788e-05    2.02345e-05      10     
end_path:out1:RootOutput                           7.113e-06    8.0293e-06    1.0621e-05    7.7695e-06    9.7016e-07       10     
end_path:out1:RootOutput(write)                     1.98028       2.91534       3.81402       2.85889      0.565148        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5066.94 MB
  Peak resident set size usage (VmHWM): 4263.29 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.21 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.03 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10875744
-rw-r--r--. 1 dune009 dune 2536725231 Dec 19 23:31 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2402511173 Dec 19 23:29 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2229882208 Dec 20 00:41 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1990005884 Dec 19 21:28 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1899829425 Dec 19 21:15 g4_stage1.root
-rw-r--r--. 1 dune009 dune   77194610 Dec 20 00:38 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242835 Dec 20 00:41 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Dec 20 00:41 mem.db
-rw-r--r--. 1 dune009 dune      62376 Dec 20 00:41 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Dec 20 00:41 time.db
-rw-r--r--. 1 dune009 dune        519 Dec 19 23:31 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Dec 19 21:28 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Dec 20 00:41 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Dec 19 21:08 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Dec 20 00:41 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Dec 19 23:38 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Dec 19 23:31 debugprod.log
justIN time: 2026-02-07 07:55:37 UTC       justIN version: 01.06.00