Jobsub ID 278268.1@dunegpschedd01.fnal.gov

Jobsub ID	278268.1@dunegpschedd01.fnal.gov
Workflow ID	11924
Stage ID	4
User name	usher@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	8388608000 (8000 MiB)
	Wall seconds limit	28800 (8 hours)
Submitted time	2026-01-14 21:35:02
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce515
Last heartbeat	2026-01-14 21:38:19
From worker node	Hostname	b9p06p2957.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	10485760000 (10000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2026-01-14 21:35:53
Input files	usertests:prod_monte-carlo-011924-000005_gen_g4_detsim.root
Jobscript	Exit code	0
	Real time	1m (81s)
	CPU time	1m (68s = 83%)
	Max RSS bytes	918597632 (876 MiB)
Outputting started	2026-01-14 21:37:15
Output files	usertests:prod_monte-carlo-011924-000005_gen_g4_detsim_reco.root
Finished	2026-01-14 21:38:19
Saved logs	justin-logs:278268.1-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

[94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 18th record. run: 200005 subRun: 0 event: 18 at 14-Jan-2026 22:37:07 CET
++++>>>> total num hits: 458, num free: 458

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2546!
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 81388
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 33902

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 19th record. run: 200005 subRun: 0 event: 19 at 14-Jan-2026 22:37:09 CET
++++>>>> total num hits: 1925, num free: 1925

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2781!
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 95737
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 28259

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 20th record. run: 200005 subRun: 0 event: 20 at 14-Jan-2026 22:37:13 CET
++++>>>> total num hits: 140, num free: 140

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 801!
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 8901
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 331
 ... due to invalid 2d index 0
 ... due to invalid 3d index 2394

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
14-Jan-2026 22:37:14 CET  Closed output file "prod_monte-carlo-011924-000005_gen_g4_detsim_reco.root"
14-Jan-2026 22:37:14 CET  Closed input file "root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/21/ef/prod_monte-carlo-011924-000005_gen_g4_detsim.root"
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_mpv_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing neutrino_mpv_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_mcst_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_sed_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_masked_true2reco_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_index_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_masked_true_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_highE_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_lowE_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_leftover_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_semantics_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_pcluster_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_sed_at_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_chi2_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_occupancy_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_nhits_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_cryo_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_tpc_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge0_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge1_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge2_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time0_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time1_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time2_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key0_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key1_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key2_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_sed_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_semantics_ghost_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_corrected_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_rescaled_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Closing output file

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.369987       1.51256       3.26966       1.3373       0.885945        20     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000201843   0.00232913     0.0388615    0.000356846   0.00838419       20     
reco:gaushit:GausHitFinder                     0.0211667     0.062905      0.141551      0.0612005     0.0310021       20     
reco:cluster3d:Cluster3D                      0.00510607     0.0939692     0.690835      0.0541914      0.15329        20     
reco:opdec:Deconvolution                       0.0723986     0.137901      0.225861      0.130257      0.0474693       20     
reco:ophitspe:OpHitFinderDeco                 0.00459005    0.00777188     0.0116792    0.00772455    0.00226503       20     
reco:opflash:OpFlashFinder                    0.000595959   0.00121589    0.00214431    0.00113674    0.000474546      20     
reco:opslicer:OpSlicer                         0.0449037     0.265025      0.741452      0.160753      0.208273        20     
reco:mcreco:MCReco                             0.0102911     0.253393       0.8422       0.189761      0.240052        20     
[art]:TriggerResults:TriggerResultInserter     1.015e-05    1.97502e-05   3.7721e-05    1.79955e-05   6.91634e-06      20     
end_path:superaMC:LArSoftSuperaDriver          0.125074      0.599162       1.40404      0.592999      0.369538        20     
end_path:out1:RootOutput                       4.66e-06     6.6485e-06     1.379e-05     5.97e-06     2.17615e-06      20     
end_path:out1:RootOutput(write)                0.0433969     0.0882707     0.144494      0.080052      0.027191        20     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 18822 MB
  Peak resident set size usage (VmHWM): 918.598 MB
====================================================================================================
Art has completed and will exit with status 0.
../justin-jobscript: line 88: LARCVNAME: command not found
mv: missing destination file operand after 'larcv.root'
Try 'mv --help' for more information.
-rw-r--r--. 1 duneprd np-comp 130M Jan 14 22:37 prod_monte-carlo-011924-000005_gen_g4_detsim_reco.root
Sandbox folder contains:
total 135M
-rw-r--r--. 1 duneprd np-comp   60 Jan 14 22:35 all-input-dids.txt
-rw-r--r--. 1 duneprd np-comp    0 Jan 14 22:36 debugprod.log
drwxr-xr-x. 2 duneprd np-comp 4.0K Jan 13 21:41 fhicl
-rw-r--r--. 1 duneprd np-comp  12K Jan 14 22:35 fhicl_bundle.tgz
-rw-r--r--. 1 duneprd np-comp  69K Jan 14 22:37 jobscript.log
-rw-r--r--. 1 duneprd np-comp 5.4M Jan 14 22:37 larcv.root
-rw-r--r--. 1 duneprd np-comp 130M Jan 14 22:37 prod_monte-carlo-011924-000005_gen_g4_detsim_reco.root
-rw-r--r--. 1 duneprd np-comp  519 Jan 14 22:37 supera_hist.root
Wrote processed DID: usertests:prod_monte-carlo-011924-000005_gen_g4_detsim.root
=== RECOjob completed successfully ===