justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 257647.0@dunegpschedd02.fnal.gov

Jobsub ID257647.0@dunegpschedd02.fnal.gov
Workflow ID11001
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-04 16:45:44
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-12-04 21:35:16
From worker nodeHostnamewn-la-05.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-04 16:46:48
Input filesmonte-carlo-011001-000001
JobscriptExit code0
Real time4h (17212s)
CPU time4h (17093s = 99%)
Max RSS bytes4548222976 (4337 MiB)
Outputting started2025-12-04 21:33:41
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11001/1/001/reco_stage1.root
Finished2025-12-04 21:35:16
Saved logsjustin-logs:257647.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

cScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 144559 traces, 0 summary
Retagger: tagging trace set: gauss with 109871 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 109871 traces tagged "gauss"
FrameSaver: q=3.82541e+06 n=1921866 tag=gauss
wclsFrameSaver: saving 144559 traces tagged "wiener"
FrameSaver: q=4.05354e+06 n=1705154 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 53221 XUs and 53695 XVs -> 747 XUVs
C:0 T:1 133003 XUs and 124332 XVs -> 8465 XUVs
C:0 T:2 62654 XUs and 76610 XVs -> 7819 XUVs
C:0 T:3 31601 XUs and 28906 XVs -> 1065 XUVs
C:0 T:4 12089 XUs and 14558 XVs -> 107 XUVs
C:0 T:5 54201 XUs and 60428 XVs -> 4306 XUVs
C:0 T:6 73115 XUs and 86901 XVs -> 5111 XUVs
C:0 T:7 31085 XUs and 35618 XVs -> 304 XUVs
C:0 T:8 6212 XUs and 6608 XVs -> 135 XUVs
C:0 T:9 32955 XUs and 50236 XVs -> 3537 XUVs
C:0 T:10 52367 XUs and 48825 XVs -> 6172 XUVs
C:0 T:11 424 XUs and 419 XVs -> 23 XUVs
37791 XUVs total
12399 collection wire objects
33418 potential space points
Neighbour search...
2862568 tests to find 1364188 neighbours
Iterating with no regularization...
Begin: 7.91632e+07
0 6.40445e+07
1 6.31221e+07
2 6.30108e+07
3 6.29798e+07
Now with regularization...
Begin: 5.218e+07
0 5.21381e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410913724
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410913724
fLifetime: 30000 [us]
04-Dec-2025 22:33:39 CET  Closed output file "reco_stage1.root"
04-Dec-2025 22:33:39 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          317.186       510.227       686.634        504.8        106.646        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00121729    0.00577302     0.010395     0.00597587    0.00265856       10     
reco:ophit:OpHitFinder                             0.338238      0.436291      0.523324      0.433924      0.0549002       10     
reco:opflash:OpFlashFinder                         0.190552      0.310273      0.425087      0.319545      0.0611299       10     
reco:opslicer:OpSlicer                             0.618535      0.902388       1.14325      0.904839       0.15299        10     
reco:caldata:DataPrepModule                         6.21788       6.5317        7.16436       6.48705      0.237041        10     
reco:wclsdatasp:WireCellToolkit                     44.3571       63.0121       69.7412       68.2513       8.92042        10     
reco:gaushit:GausHitFinder                         0.990792       1.54418       1.94621       1.54266      0.285114        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000332886   0.000625472   0.000969283   0.000592663   0.000187447      10     
reco:reco3d:SpacePointSolver                        11.4187       25.2348       45.6789       24.5775       9.77385        10     
reco:hitpdune:DisambigFromSpacePoints              0.881164       2.12537       3.10105       2.22093      0.614168        10     
reco:pandora:StandardPandora                        18.902        51.0674       110.61        44.9624       24.1656        10     
reco:pandoraWriter:StandardPandora                 0.341953      0.531442      0.658312      0.538537      0.092088        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.04437       7.15246       11.7304       7.07609       2.4752         10     
reco:pandoraShower:LArPandoraShowerCreation         2.75261       8.3612        16.3672       7.78625       3.55594        10     
reco:pandoracalo:Calorimetry                        1.57757       3.88659       5.84566       4.0774        1.20334        10     
reco:pandoracalonosce:Calorimetry                   1.43049       3.66092       5.39354       3.84429       1.14032        10     
reco:pandorapid:Chi2ParticleID                    0.00505718    0.00681548    0.00961643    0.00676639    0.00122599       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0386617     0.060836      0.0770201     0.0612605     0.0106082       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00501706    0.00651679    0.00774516    0.00654728    0.000882897      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.57385       8.34488       13.919        8.32649       3.01676        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.49304        8.051        12.2732       8.06455       2.73392        10     
reco:emtrkmichelid:EmTrackMichelId                  161.411       241.451       311.93        238.853       42.9963        10     
reco:crttag:SingleCRTMatchingProducer              0.011077      0.0192638     0.0296615     0.0180321    0.00506898       10     
reco:crtreco:TwoCRTMatchingProducer               0.00926416     0.0170167     0.035772      0.0160039    0.00708756       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.699526       1.17576       1.51295       1.2284       0.227973        10     
reco:pandora2Track:LArPandoraTrackCreation          12.0572       23.602        39.6244       23.2678       7.78337        10     
reco:pandora2calo:Calorimetry                       5.42547       11.7305       18.449        11.5483       3.70622        10     
reco:pandora2calonosce:Calorimetry                  5.34167       11.1767       17.3908       11.0543       3.46373        10     
reco:pandora2pid:Chi2ParticleID                    0.0111904     0.0154957     0.0185064     0.0153632    0.00239786       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0501728     0.0784132     0.101581      0.079197      0.0143055       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0112147     0.0155601     0.0186491     0.0154349    0.00246183       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.47779       8.25505        14.9         8.03778       2.79998        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.55273       9.35851       14.4759       9.41332       2.62591        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.91258       8.07996       12.2599       8.22376       2.20942        10     
[art]:TriggerResults:TriggerResultInserter        3.8408e-05    5.98382e-05   0.000128322   4.9313e-05    2.66812e-05      10     
end_path:out1:RootOutput                           7.381e-06    8.6758e-06    1.2889e-05     8.343e-06    1.45103e-06      10     
end_path:out1:RootOutput(write)                     2.48743       3.79248       4.85777       3.81187       0.68268        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5349.52 MB
  Peak resident set size usage (VmHWM): 4548.22 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.07 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10841004
-rw-r--r--. 1 dune009 dune 2521541604 Dec  4 21:06 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2391989733 Dec  4 21:03 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2243429548 Dec  4 22:33 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1969817593 Dec  4 18:10 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1890362915 Dec  4 17:54 g4_stage1.root
-rw-r--r--. 1 dune009 dune   83435460 Dec  4 22:28 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242960 Dec  4 22:33 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Dec  4 22:33 mem.db
-rw-r--r--. 1 dune009 dune      62376 Dec  4 22:33 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Dec  4 22:33 time.db
-rw-r--r--. 1 dune009 dune        519 Dec  4 21:06 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Dec  4 18:10 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Dec  4 22:33 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Dec  4 17:46 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Dec  4 22:33 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Dec  4 21:15 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Dec  4 21:06 debugprod.log
justIN time: 2025-12-18 18:07:38 UTC       justIN version: 01.05.03