justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 235845.0@dunegpschedd02.fnal.gov

Jobsub ID235845.0@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-31 20:51:24
SiteUS_UChicago
EntryEngage_US_MWT2_iut2_gk02_condce_mcore
Last heartbeat2025-10-31 23:15:40
From worker nodeHostnameiut2-c320.iu.edu
cpuinfoAMD EPYC 7402 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-31 21:00:44
Input filesusertests:pdhd_prod_beam__226242_0_1_20251008T182622Z_gen_g4_IonScintPDExt.root_227038_35_1_20251010T161124Z_PDInt.root
JobscriptExit code0
Real time2h (7971s)
CPU time2h (7697s = 96%)
Max RSS bytes7003402240 (6678 MiB)
Outputting started2025-10-31 23:13:35
Output filesusertests:pdhd_prod_beam__226242_0_1_20251008T182622Z_gen_g4_IonScintPDExt.root_227038_35_1_20251010T161124Z_PDInt.root_235845_0_1_20251031T210053Z_detsim_reco1.root
Finished2025-10-31 23:15:40
Saved logsjustin-logs:235845.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

6.88622e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 31-Oct-2025 19:10:13 EDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 12075!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 84895 traces, 84895 summary
Retagger: tagging trace set: gauss with 68281 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 68281 traces tagged "gauss"
FrameSaver: q=2.07391e+06 n=1021710 tag=gauss
wclsFrameSaver: saving 84895 traces tagged "wiener"
FrameSaver: q=2.675e+06 n=996665 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11514 XUs and 8958 XVs -> 229 XUVs
C:0 T:1 137628 XUs and 109491 XVs -> 5800 XUVs
C:0 T:2 14075 XUs and 17028 XVs -> 671 XUVs
C:0 T:3 40 XUs and 73 XVs -> 0 XUVs
C:0 T:4 9 XUs and 122 XVs -> 0 XUVs
C:0 T:5 36493 XUs and 24791 XVs -> 474 XUVs
C:0 T:6 4350 XUs and 7282 XVs -> 294 XUVs
7468 XUVs total
2447 collection wire objects
7468 potential space points
Neighbour search...
333558 tests to find 133590 neighbours
Iterating with no regularization...
Begin: 3.59858e+07
0 3.14008e+07
1 3.09581e+07
2 3.09189e+07
3 3.09114e+07
Now with regularization...
Begin: 2.98824e+07
0 2.98777e+07
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00243175 0.00817442 0.00598931 0.00314075 0.010156 0.00443608 0.00170708 0.00146896 0.00141335 0.00311232 0.00182372 0.000498652 0.000515401 
----view 1: 8.29101e-05 8.29697e-05 8.29697e-05 0.000392437 6.6638e-05 6.66976e-05 6.66976e-05 0.000105679 0.000154912 0.000154912 0.000154912 0.000154972 0.000154912 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00243175 0.00817442 0.00598931 0.00314075 0.010156 0.00443608 0.00170708 0.00146896 0.00141335 0.00311232 0.00182372 
----view 1: 8.29101e-05 8.29697e-05 8.29697e-05 0.000392437 6.6638e-05 6.66976e-05 6.66976e-05 0.000105679 0.000154912 0.000154912 0.000154912 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00817442 0.00598931 0.00314075 0.010156 0.00443608 0.00170708 0.00146896 0.00141335 0.00311232 0.00182372 0.000498652 
----view 1: 8.29697e-05 8.29697e-05 0.000392437 6.6638e-05 6.66976e-05 6.66976e-05 0.000105679 0.000154912 0.000154912 0.000154912 0.000154972 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00598931 0.00314075 0.010156 0.00443608 0.00170708 0.00146896 0.00141335 0.00311232 0.00182372 0.000498652 0.000515401 
----view 1: 8.29697e-05 0.000392437 6.6638e-05 6.66976e-05 6.66976e-05 0.000105679 0.000154912 0.000154912 0.000154912 0.000154972 0.000154912 
31-Oct-2025 19:13:33 EDT  Closed output file "pdhd_prod_beam__226242_0_1_20251008T182622Z_gen_g4_IonScintPDExt.root_227038_35_1_20251010T161124Z_PDInt.root_235845_0_1_20251031T210053Z_detsim_reco1.root"
31-Oct-2025 19:13:33 EDT  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            104.84        223.283       322.91        214.178       63.4084        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00185285     0.0093856     0.0159634     0.0101996    0.00457342       10     
reco:opdec:Deconvolution                             0.812068       1.02186       1.35253       1.00989      0.138403        10     
reco:ophitspe:OpHitFinderDeco                        0.186261      0.232854      0.295104      0.236291      0.0299106       10     
reco:opflash:OpFlashFinder                           0.0582369     0.0939304     0.115297      0.0909009     0.0165093       10     
reco:opslicer:OpSlicer                               0.218032      0.297838      0.431304      0.289142      0.0576729       10     
reco:wclsdatahd:WireCellToolkit                        54.13        64.2046       68.6128       66.7753       5.08386        10     
reco:gaushit:GausHitFinder                           0.608228       1.16257       1.57207       1.16852      0.317251        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000576357   0.000837395   0.00118984    0.000808974   0.000177481      10     
reco:reco3d:SpacePointSolver                         0.264908       1.93081       4.10419       1.80174       1.01954        10     
reco:hitpdune:DisambigFromSpacePoints                0.209234       1.12342       1.92899       1.0574       0.539984        10     
reco:pandora:StandardPandora                          13.0695       46.5808       108.697       40.3072       25.9667        10     
reco:pandoraWriter:StandardPandora                   0.283824      0.544573      0.745791      0.582551      0.140423        10     
reco:pandoraTrack:LArPandoraTrackCreation             1.23266       5.01807       8.35272       4.77374       2.37847        10     
reco:pandoraShower:LArPandoraShowerCreation           0.5184        4.65997       7.42557       4.51412       2.17496        10     
reco:pandoracalo:Calorimetry                         0.673078       2.57619       5.07159       2.32002       1.45985        10     
reco:pandoracalonosce:Calorimetry                    0.511899       2.37825       4.92031       2.26759       1.44672        10     
reco:pandoraShowercalo:ShowerCalorimetry              0.39368       4.17775       9.05143       3.64149       2.46713        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.424698       4.11395       8.35284       3.25195       2.4423         10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.050612      0.0868721     0.107935      0.0884333     0.0161535       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0463578     0.0832994     0.112007      0.0828599     0.0164119       10     
reco:emtrkmichelid:EmTrackMichelId                    23.2188       46.8407       62.4852       48.6759       11.7547        10     
reco:crttag:SingleCRTMatchingProducer               0.00869653     0.0134097     0.0237703     0.0124671    0.00462237       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00623303     0.0114988     0.0163069     0.011558     0.00343417       10     
reco:pandora2Track:LArPandoraTrackCreation            2.23967       11.5011       18.6414       10.4179       5.25525        10     
reco:pandora2calo:ShowerCalorimetry                  0.337522       3.7356        7.85882       3.16941       2.14464        10     
reco:pandora2calonosce:ShowerCalorimetry             0.310966       3.61659       7.47792        3.08         2.23208        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0301071     0.0848832     0.128131      0.0902257     0.0250328       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0327708     0.0798913     0.108031      0.0840124     0.0204827       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.558998       4.54684       8.36135       4.66788       2.11921        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.544545       4.66571       9.79959       3.86806       2.70914        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.493334       4.61626       8.88449       3.85117       2.78703        10     
[art]:TriggerResults:TriggerResultInserter          2.9626e-05    8.24244e-05   0.000149823   7.5773e-05     3.66e-05        10     
end_path:out1:RootOutput                            1.0429e-05    2.36754e-05   7.6604e-05    1.7558e-05    1.86199e-05      10     
end_path:out1:RootOutput(write)                       1.86255       2.99443       3.75587       3.00496      0.524764        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6304.21 MB
  Peak resident set size usage (VmHWM): 4065.13 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11740348
-rw-r--r--. 1 dune osgvo      48208 Oct 31 19:13 jobscript.log
-rw-r--r--. 1 dune osgvo        199 Oct 31 19:13 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo        519 Oct 31 19:13 reco_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo       3574 Oct 31 19:13 Pandora_Geometry.xml
-rw-r--r--. 1 dune osgvo     200704 Oct 31 19:13 mem.db
-rw-r--r--. 1 dune osgvo      36864 Oct 31 19:13 time.db
-rw-r--r--. 1 dune osgvo 6155734364 Oct 31 19:13 pdhd_prod_beam__226242_0_1_20251008T182622Z_gen_g4_IonScintPDExt.root_227038_35_1_20251010T161124Z_PDInt.root_235845_0_1_20251031T210053Z_detsim_reco1.root
-rw-r--r--. 1 dune osgvo   55019160 Oct 31 19:12 Pandora_Events.pndr
-rw-r--r--. 1 dune osgvo          0 Oct 31 18:36 ProtoDUNE.log
-rw-r--r--. 1 dune osgvo          0 Oct 31 18:33 debugprod.log
-rw-r--r--. 1 dune osgvo        519 Oct 31 18:33 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo 5811028884 Oct 31 18:33 temp_detsim.root
-rw-r--r--. 1 dune osgvo        120 Oct 31 17:00 all-input-dids.txt
justIN time: 2025-11-05 04:50:29 UTC       justIN version: 01.05.01