justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.29@dunegpschedd02.fnal.gov

Jobsub ID234970.29@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 20:43:36
From worker nodeHostnamedunegli-7568308-0-fnpc22024.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:37:38
Input filesusertests:pdhd_prod_beam__231691_0_1_20251008T183036Z_gen_g4_IonScintPDExt.root_227038_91_1_20251010T163659Z_PDInt.root
JobscriptExit code0
Real time59m (3555s)
CPU time58m (3482s = 97%)
Max RSS bytes6644948992 (6337 MiB)
Outputting started2025-10-29 20:36:53
Output files
Finished2025-10-29 20:43:36
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

XUs and 13829 XVs -> 236 XUVs
C:0 T:1 179818 XUs and 151718 XVs -> 7298 XUVs
C:0 T:2 32551 XUs and 17340 XVs -> 509 XUVs
C:0 T:3 16 XUs and 14 XVs -> 0 XUVs
C:0 T:4 8 XUs and 17 XVs -> 2 XUVs
C:0 T:5 65719 XUs and 96319 XVs -> 4382 XUVs
C:0 T:6 58426 XUs and 35162 XVs -> 1522 XUVs
13949 XUVs total
4344 collection wire objects
13949 potential space points
Neighbour search...
834761 tests to find 366470 neighbours
Iterating with no regularization...
Begin: 7.58514e+07
0 6.75581e+07
1 6.67709e+07
2 6.66922e+07
3 6.66727e+07
Now with regularization...
Begin: 6.32247e+07
0 6.32112e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:35:08 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 15798!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 105055 traces, 105055 summary
Retagger: tagging trace set: gauss with 80887 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 80887 traces tagged "gauss"
FrameSaver: q=2.64497e+06 n=1282344 tag=gauss
wclsFrameSaver: saving 105055 traces tagged "wiener"
FrameSaver: q=3.47649e+06 n=1233196 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12279 XUs and 9037 XVs -> 151 XUVs
C:0 T:1 94716 XUs and 68058 XVs -> 3128 XUVs
C:0 T:2 30020 XUs and 44663 XVs -> 866 XUVs
C:0 T:3 4 XUs and 9 XVs -> 1 XUVs
C:0 T:4 37 XUs and 41 XVs -> 0 XUVs
C:0 T:5 51106 XUs and 38622 XVs -> 2041 XUVs
C:0 T:6 36829 XUs and 37240 XVs -> 955 XUVs
C:0 T:7 12 XUs and 66 XVs -> 3 XUVs
7145 XUVs total
3033 collection wire objects
7145 potential space points
Neighbour search...
228817 tests to find 109568 neighbours
Iterating with no regularization...
Begin: 4.80704e+07
0 4.46605e+07
1 4.43827e+07
2 4.43535e+07
Now with regularization...
Begin: 4.2318e+07
0 4.23083e+07
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.013311 0.0104628 0.0135195 0.00325334 0.0122033 0.0113156 0.0147327 0.0162133 0.021853 0.0177636 0.00577027 
----view 1: 4.14252e-05 4.14252e-05 4.14252e-05 4.14252e-05 4.14252e-05 4.14252e-05 4.14848e-05 4.14252e-05 4.14252e-05 4.14252e-05 4.14252e-05 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0104628 0.0135195 0.00325334 0.0122033 0.0113156 0.0147327 0.0162133 0.021853 0.0177636 0.00577027 9.53674e-05 
----view 1: 4.14252e-05 4.14252e-05 4.14252e-05 4.14252e-05 4.14252e-05 4.14848e-05 4.14252e-05 4.14252e-05 4.14252e-05 4.14252e-05 4.14252e-05 
29-Oct-2025 20:36:52 UTC  Closed output file "pdhd_prod_beam__231691_0_1_20251008T183036Z_gen_g4_IonScintPDExt.root_227038_91_1_20251010T163659Z_PDInt.root_234970_29_1_20251029T193740Z_detsim_reco1.root"
29-Oct-2025 20:36:52 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            71.5141       84.7776       117.322       80.3264       13.2559        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000655259   0.00371468    0.00693171    0.00338971    0.00189815       10     
reco:opdec:Deconvolution                             0.475601      0.565066      0.674303      0.560596      0.0696391       10     
reco:ophitspe:OpHitFinderDeco                        0.0983169     0.105164      0.120369       0.10343     0.00585332       10     
reco:opflash:OpFlashFinder                           0.0380932     0.0586242     0.0789964     0.0551311     0.0130957       10     
reco:opslicer:OpSlicer                               0.121267      0.149926      0.203462      0.138342      0.0277741       10     
reco:wclsdatahd:WireCellToolkit                       31.6369       32.4837       33.7283       32.2495      0.668265        10     
reco:gaushit:GausHitFinder                           0.375728      0.482842       0.59375      0.475842      0.0670096       10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000186001   0.000268342   0.00047112    0.000241572   8.31757e-05      10     
reco:reco3d:SpacePointSolver                         0.310658      0.612536       1.20582      0.541432      0.259163        10     
reco:hitpdune:DisambigFromSpacePoints                0.255611       0.42004      0.893582      0.366259      0.175163        10     
reco:pandora:StandardPandora                          8.30248       13.9589       25.0642       12.5645       4.7298         10     
reco:pandoraWriter:StandardPandora                    0.11945      0.156191       0.21669      0.148734      0.0277788       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.620276      0.996401       1.46882      0.973944      0.272235        10     
reco:pandoraShower:LArPandoraShowerCreation          0.596451      0.966729       1.77019      0.882414      0.363002        10     
reco:pandoracalo:Calorimetry                         0.354566      0.571376      0.878612      0.531657      0.170187        10     
reco:pandoracalonosce:Calorimetry                    0.346906      0.559428      0.868757       0.52014      0.168038        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.614125       1.01968       1.89236      0.933038      0.414843        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.588977       0.96557       1.78055      0.888608      0.383555        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0205209     0.0310463     0.0454039     0.0287314    0.00694156       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0199119     0.0294722     0.0443978     0.0276345    0.00648205       10     
reco:emtrkmichelid:EmTrackMichelId                    17.3224       21.2382       29.2176       20.0961       3.3594         10     
reco:crttag:SingleCRTMatchingProducer               0.00251113     0.0036607    0.00567152    0.00320263    0.00102301       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00231035    0.00287979    0.00509311    0.00260899    0.00077797       10     
reco:pandora2Track:LArPandoraTrackCreation            1.43367       2.44582       4.45145       2.23581      0.885834        10     
reco:pandora2calo:ShowerCalorimetry                  0.535044      0.904399       1.77881      0.835802      0.393537        10     
reco:pandora2calonosce:ShowerCalorimetry             0.523624       0.88718       1.72853      0.824343      0.383488        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0209991     0.0302917     0.046179      0.0288659    0.00671946       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0196473     0.0288472     0.0440135     0.0276227    0.00653312       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.683736       1.11485       2.06961       1.02673      0.416051        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.726119       1.24856       2.29022       1.15541      0.487947        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.703204       1.22828       2.28749       1.1425       0.486354        10     
[art]:TriggerResults:TriggerResultInserter          1.9206e-05    3.43239e-05   0.000117282   2.61845e-05   2.79107e-05      10     
end_path:out1:RootOutput                             4.689e-06    7.8659e-06    3.3874e-05     5.034e-06    8.67285e-06      10     
end_path:out1:RootOutput(write)                       1.16749       1.44999       1.84061       1.3874       0.207963        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6252.5 MB
  Peak resident set size usage (VmHWM): 4047.9 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11413348
-rw-r--r-- 1 dunegli fnalgrid      40036 Oct 29 20:36 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        199 Oct 29 20:36 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:36 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:36 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:36 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:36 time.db
-rw-r--r-- 1 dunegli fnalgrid 5983029287 Oct 29 20:36 pdhd_prod_beam__231691_0_1_20251008T183036Z_gen_g4_IonScintPDExt.root_227038_91_1_20251010T163659Z_PDInt.root_234970_29_1_20251029T193740Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   51788130 Oct 29 20:36 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:23 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:21 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:21 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5652136113 Oct 29 20:21 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        120 Oct 29 19:37 all-input-dids.txt
justIN time: 2025-11-03 18:32:30 UTC       justIN version: 01.05.01