Jobsub ID 40705.165@dunegpschedd01.fnal.gov

Jobsub ID	40705.165@dunegpschedd01.fnal.gov
Workflow ID	1980
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-05 06:23:09
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce507
Last heartbeat	2025-09-05 06:28:22
From worker node	Hostname	b9p09p2568.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-05 06:24:56
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_593_20231207T091915Z_gen_g4_detsim_hitreco__20240510T041439Z_reco2.root
Jobscript	Exit code	0
	Real time	3m (182s)
	CPU time	0m (51s = 28%)
	Max RSS bytes	1232072704 (1174 MiB)
Outputting started	2025-09-05 06:28:00
Output files
Finished	2025-09-05 06:28:22
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

6 subRun: 1 event: 59330 at 05-Sep-2025 08:26:06 CEST
Begin processing the 31st record. run: 6483936 subRun: 1 event: 59331 at 05-Sep-2025 08:26:07 CEST
Begin processing the 32nd record. run: 6483936 subRun: 1 event: 59332 at 05-Sep-2025 08:26:09 CEST
Begin processing the 33rd record. run: 6483936 subRun: 1 event: 59333 at 05-Sep-2025 08:26:10 CEST
Begin processing the 34th record. run: 6483936 subRun: 1 event: 59334 at 05-Sep-2025 08:26:11 CEST
Begin processing the 35th record. run: 6483936 subRun: 1 event: 59335 at 05-Sep-2025 08:26:12 CEST
Begin processing the 36th record. run: 6483936 subRun: 1 event: 59336 at 05-Sep-2025 08:26:14 CEST
Begin processing the 37th record. run: 6483936 subRun: 1 event: 59337 at 05-Sep-2025 08:26:15 CEST
Begin processing the 38th record. run: 6483936 subRun: 1 event: 59338 at 05-Sep-2025 08:26:16 CEST
Begin processing the 39th record. run: 6483936 subRun: 1 event: 59339 at 05-Sep-2025 08:26:18 CEST
Begin processing the 40th record. run: 6483936 subRun: 1 event: 59340 at 05-Sep-2025 08:26:19 CEST
Begin processing the 41st record. run: 6483936 subRun: 1 event: 59341 at 05-Sep-2025 08:26:20 CEST
Begin processing the 42nd record. run: 6483936 subRun: 1 event: 59342 at 05-Sep-2025 08:26:21 CEST
Begin processing the 43rd record. run: 6483936 subRun: 1 event: 59343 at 05-Sep-2025 08:26:23 CEST
Begin processing the 44th record. run: 6483936 subRun: 1 event: 59344 at 05-Sep-2025 08:26:24 CEST
Begin processing the 45th record. run: 6483936 subRun: 1 event: 59345 at 05-Sep-2025 08:26:25 CEST
Begin processing the 46th record. run: 6483936 subRun: 1 event: 59346 at 05-Sep-2025 08:26:28 CEST
Begin processing the 47th record. run: 6483936 subRun: 1 event: 59347 at 05-Sep-2025 08:26:29 CEST
Begin processing the 48th record. run: 6483936 subRun: 1 event: 59348 at 05-Sep-2025 08:26:31 CEST
Begin processing the 49th record. run: 6483936 subRun: 1 event: 59349 at 05-Sep-2025 08:26:32 CEST
Begin processing the 50th record. run: 6483936 subRun: 1 event: 59350 at 05-Sep-2025 08:26:36 CEST
Begin processing the 51st record. run: 6483936 subRun: 1 event: 59351 at 05-Sep-2025 08:26:37 CEST
Begin processing the 52nd record. run: 6483936 subRun: 1 event: 59352 at 05-Sep-2025 08:26:38 CEST
Begin processing the 53rd record. run: 6483936 subRun: 1 event: 59353 at 05-Sep-2025 08:26:39 CEST
Begin processing the 54th record. run: 6483936 subRun: 1 event: 59354 at 05-Sep-2025 08:26:40 CEST
Begin processing the 55th record. run: 6483936 subRun: 1 event: 59355 at 05-Sep-2025 08:26:42 CEST
Begin processing the 56th record. run: 6483936 subRun: 1 event: 59356 at 05-Sep-2025 08:26:43 CEST
Begin processing the 57th record. run: 6483936 subRun: 1 event: 59357 at 05-Sep-2025 08:26:44 CEST
Begin processing the 58th record. run: 6483936 subRun: 1 event: 59358 at 05-Sep-2025 08:26:46 CEST
Begin processing the 59th record. run: 6483936 subRun: 1 event: 59359 at 05-Sep-2025 08:26:47 CEST
Begin processing the 60th record. run: 6483936 subRun: 1 event: 59360 at 05-Sep-2025 08:26:48 CEST
Begin processing the 61st record. run: 6483936 subRun: 1 event: 59361 at 05-Sep-2025 08:26:49 CEST
Begin processing the 62nd record. run: 6483936 subRun: 1 event: 59362 at 05-Sep-2025 08:26:50 CEST
Begin processing the 63rd record. run: 6483936 subRun: 1 event: 59363 at 05-Sep-2025 08:26:50 CEST
Begin processing the 64th record. run: 6483936 subRun: 1 event: 59364 at 05-Sep-2025 08:26:52 CEST
Begin processing the 65th record. run: 6483936 subRun: 1 event: 59365 at 05-Sep-2025 08:26:53 CEST
Begin processing the 66th record. run: 6483936 subRun: 1 event: 59366 at 05-Sep-2025 08:26:54 CEST
Begin processing the 67th record. run: 6483936 subRun: 1 event: 59367 at 05-Sep-2025 08:26:58 CEST
Begin processing the 68th record. run: 6483936 subRun: 1 event: 59368 at 05-Sep-2025 08:27:00 CEST
Begin processing the 69th record. run: 6483936 subRun: 1 event: 59369 at 05-Sep-2025 08:27:02 CEST
Begin processing the 70th record. run: 6483936 subRun: 1 event: 59370 at 05-Sep-2025 08:27:03 CEST
Begin processing the 71st record. run: 6483936 subRun: 1 event: 59371 at 05-Sep-2025 08:27:04 CEST
Begin processing the 72nd record. run: 6483936 subRun: 1 event: 59372 at 05-Sep-2025 08:27:05 CEST
Begin processing the 73rd record. run: 6483936 subRun: 1 event: 59373 at 05-Sep-2025 08:27:06 CEST
Begin processing the 74th record. run: 6483936 subRun: 1 event: 59374 at 05-Sep-2025 08:27:07 CEST
Begin processing the 75th record. run: 6483936 subRun: 1 event: 59375 at 05-Sep-2025 08:27:09 CEST
Begin processing the 76th record. run: 6483936 subRun: 1 event: 59376 at 05-Sep-2025 08:27:14 CEST
Begin processing the 77th record. run: 6483936 subRun: 1 event: 59377 at 05-Sep-2025 08:27:15 CEST
Begin processing the 78th record. run: 6483936 subRun: 1 event: 59378 at 05-Sep-2025 08:27:17 CEST
Begin processing the 79th record. run: 6483936 subRun: 1 event: 59379 at 05-Sep-2025 08:27:19 CEST
Begin processing the 80th record. run: 6483936 subRun: 1 event: 59380 at 05-Sep-2025 08:27:21 CEST
Begin processing the 81st record. run: 6483936 subRun: 1 event: 59381 at 05-Sep-2025 08:27:22 CEST
Begin processing the 82nd record. run: 6483936 subRun: 1 event: 59382 at 05-Sep-2025 08:27:26 CEST
Begin processing the 83rd record. run: 6483936 subRun: 1 event: 59383 at 05-Sep-2025 08:27:28 CEST
Begin processing the 84th record. run: 6483936 subRun: 1 event: 59384 at 05-Sep-2025 08:27:29 CEST
Begin processing the 85th record. run: 6483936 subRun: 1 event: 59385 at 05-Sep-2025 08:27:30 CEST
Begin processing the 86th record. run: 6483936 subRun: 1 event: 59386 at 05-Sep-2025 08:27:32 CEST
Begin processing the 87th record. run: 6483936 subRun: 1 event: 59387 at 05-Sep-2025 08:27:33 CEST
Begin processing the 88th record. run: 6483936 subRun: 1 event: 59388 at 05-Sep-2025 08:27:34 CEST
Begin processing the 89th record. run: 6483936 subRun: 1 event: 59389 at 05-Sep-2025 08:27:35 CEST
Begin processing the 90th record. run: 6483936 subRun: 1 event: 59390 at 05-Sep-2025 08:27:37 CEST
Begin processing the 91st record. run: 6483936 subRun: 1 event: 59391 at 05-Sep-2025 08:27:39 CEST
Begin processing the 92nd record. run: 6483936 subRun: 1 event: 59392 at 05-Sep-2025 08:27:40 CEST
Begin processing the 93rd record. run: 6483936 subRun: 1 event: 59393 at 05-Sep-2025 08:27:41 CEST
Begin processing the 94th record. run: 6483936 subRun: 1 event: 59394 at 05-Sep-2025 08:27:43 CEST
Begin processing the 95th record. run: 6483936 subRun: 1 event: 59395 at 05-Sep-2025 08:27:44 CEST
Begin processing the 96th record. run: 6483936 subRun: 1 event: 59396 at 05-Sep-2025 08:27:45 CEST
Begin processing the 97th record. run: 6483936 subRun: 1 event: 59397 at 05-Sep-2025 08:27:50 CEST
Begin processing the 98th record. run: 6483936 subRun: 1 event: 59398 at 05-Sep-2025 08:27:52 CEST
Begin processing the 99th record. run: 6483936 subRun: 1 event: 59399 at 05-Sep-2025 08:27:53 CEST
Begin processing the 100th record. run: 6483936 subRun: 1 event: 59400 at 05-Sep-2025 08:27:55 CEST
05-Sep-2025 08:27:58 CEST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c0/60/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_593_20231207T091915Z_gen_g4_detsim_hitreco__20240510T041439Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.307904       1.16141       3.38223       1.05052      0.505285        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0159356     0.0302819     0.0919142     0.0318669     0.0132301       100    
prod:pandorapid:Chi2ParticleID                 0.0933245      0.14454       0.33526      0.144987      0.0295052       100    
prod:energyreconue:EnergyReco                  0.168519      0.387718      0.827168      0.385571      0.101369        100    
prod:energyreconumu:EnergyReco                0.00175409     0.0798432     0.770525      0.0608699     0.0960453       100    
prod:energyreconc:EnergyReco                   0.0035854     0.0392766     0.187089      0.0328477     0.0337712       100    
[art]:TriggerResults:TriggerResultInserter     1.182e-05    2.05229e-05   9.5061e-05     1.811e-05    1.25242e-05      100    
end_path:ana:PIDASelectionAna                  0.0192942     0.479193       2.41251      0.406627      0.327966        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19611 MB
  Peak resident set size usage (VmHWM): 1232.07 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 56
-rw-r--r--. 1 duneprd np-comp 20222 Sep  5 08:27 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_593_20231207T091915Z_gen_g4_detsim_hitreco__20240510T041439Z_reco2_pida_2025-09-05T_062505Z.root
-rw-r--r--. 1 duneprd np-comp 14216 Sep  5 08:27 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_593_20231207T091915Z_gen_g4_detsim_hitreco__20240510T041439Z_reco2_reco_2025-09-05T_062505Z.log
-rw-r--r--. 1 duneprd np-comp 11970 Sep  5 08:27 jobscript.log
-rw-r--r--. 1 duneprd np-comp   186 Sep  5 08:27 justin-processed-pfns.txt
-rw-r--r--. 1 duneprd np-comp   137 Sep  5 08:24 all-input-dids.txt
-rw-r--r--. 1 duneprd np-comp     0 Sep  5 08:25 debugprod.log