Jobsub ID 40208.141@dunegpschedd01.fnal.gov

Jobsub ID	40208.141@dunegpschedd01.fnal.gov
Workflow ID	1978
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-04 13:46:18
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc03
Last heartbeat	2025-09-04 13:49:00
From worker node	Hostname	wn-da-22.gina.surf.nl
	cpuinfo	AMD EPYC-Rome Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-04 13:47:41
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_272_20231208T002939Z_gen_g4_detsim_hitreco__20240510T040135Z_reco2.root
Jobscript	Exit code	0
	Real time	0m (43s)
	CPU time	0m (34s = 79%)
	Max RSS bytes	850702336 (811 MiB)
Outputting started	2025-09-04 13:48:24
Output files
Finished	2025-09-04 13:49:00
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

un: 74574126 subRun: 1 event: 27231 at 04-Sep-2025 15:48:13 CEST
Begin processing the 32nd record. run: 74574126 subRun: 1 event: 27232 at 04-Sep-2025 15:48:13 CEST
Begin processing the 33rd record. run: 74574126 subRun: 1 event: 27233 at 04-Sep-2025 15:48:13 CEST
Begin processing the 34th record. run: 74574126 subRun: 1 event: 27234 at 04-Sep-2025 15:48:13 CEST
Begin processing the 35th record. run: 74574126 subRun: 1 event: 27235 at 04-Sep-2025 15:48:13 CEST
Begin processing the 36th record. run: 74574126 subRun: 1 event: 27236 at 04-Sep-2025 15:48:14 CEST
Begin processing the 37th record. run: 74574126 subRun: 1 event: 27237 at 04-Sep-2025 15:48:14 CEST
Begin processing the 38th record. run: 74574126 subRun: 1 event: 27238 at 04-Sep-2025 15:48:14 CEST
Begin processing the 39th record. run: 74574126 subRun: 1 event: 27239 at 04-Sep-2025 15:48:14 CEST
Begin processing the 40th record. run: 74574126 subRun: 1 event: 27240 at 04-Sep-2025 15:48:14 CEST
Begin processing the 41st record. run: 74574126 subRun: 1 event: 27241 at 04-Sep-2025 15:48:14 CEST
Begin processing the 42nd record. run: 74574126 subRun: 1 event: 27242 at 04-Sep-2025 15:48:15 CEST
Begin processing the 43rd record. run: 74574126 subRun: 1 event: 27243 at 04-Sep-2025 15:48:15 CEST
Begin processing the 44th record. run: 74574126 subRun: 1 event: 27244 at 04-Sep-2025 15:48:15 CEST
Begin processing the 45th record. run: 74574126 subRun: 1 event: 27245 at 04-Sep-2025 15:48:15 CEST
Begin processing the 46th record. run: 74574126 subRun: 1 event: 27246 at 04-Sep-2025 15:48:15 CEST
Begin processing the 47th record. run: 74574126 subRun: 1 event: 27247 at 04-Sep-2025 15:48:15 CEST
Begin processing the 48th record. run: 74574126 subRun: 1 event: 27248 at 04-Sep-2025 15:48:15 CEST
Begin processing the 49th record. run: 74574126 subRun: 1 event: 27249 at 04-Sep-2025 15:48:15 CEST
Begin processing the 50th record. run: 74574126 subRun: 1 event: 27250 at 04-Sep-2025 15:48:16 CEST
Begin processing the 51st record. run: 74574126 subRun: 1 event: 27251 at 04-Sep-2025 15:48:16 CEST
Begin processing the 52nd record. run: 74574126 subRun: 1 event: 27252 at 04-Sep-2025 15:48:16 CEST
Begin processing the 53rd record. run: 74574126 subRun: 1 event: 27253 at 04-Sep-2025 15:48:16 CEST
Begin processing the 54th record. run: 74574126 subRun: 1 event: 27254 at 04-Sep-2025 15:48:16 CEST
Begin processing the 55th record. run: 74574126 subRun: 1 event: 27255 at 04-Sep-2025 15:48:16 CEST
Begin processing the 56th record. run: 74574126 subRun: 1 event: 27256 at 04-Sep-2025 15:48:16 CEST
Begin processing the 57th record. run: 74574126 subRun: 1 event: 27257 at 04-Sep-2025 15:48:16 CEST
Begin processing the 58th record. run: 74574126 subRun: 1 event: 27258 at 04-Sep-2025 15:48:16 CEST
Begin processing the 59th record. run: 74574126 subRun: 1 event: 27259 at 04-Sep-2025 15:48:17 CEST
Begin processing the 60th record. run: 74574126 subRun: 1 event: 27260 at 04-Sep-2025 15:48:17 CEST
Begin processing the 61st record. run: 74574126 subRun: 1 event: 27261 at 04-Sep-2025 15:48:17 CEST
Begin processing the 62nd record. run: 74574126 subRun: 1 event: 27262 at 04-Sep-2025 15:48:17 CEST
Begin processing the 63rd record. run: 74574126 subRun: 1 event: 27263 at 04-Sep-2025 15:48:17 CEST
Begin processing the 64th record. run: 74574126 subRun: 1 event: 27264 at 04-Sep-2025 15:48:18 CEST
Begin processing the 65th record. run: 74574126 subRun: 1 event: 27265 at 04-Sep-2025 15:48:18 CEST
Begin processing the 66th record. run: 74574126 subRun: 1 event: 27266 at 04-Sep-2025 15:48:18 CEST
Begin processing the 67th record. run: 74574126 subRun: 1 event: 27267 at 04-Sep-2025 15:48:18 CEST
Begin processing the 68th record. run: 74574126 subRun: 1 event: 27268 at 04-Sep-2025 15:48:18 CEST
Begin processing the 69th record. run: 74574126 subRun: 1 event: 27269 at 04-Sep-2025 15:48:18 CEST
Begin processing the 70th record. run: 74574126 subRun: 1 event: 27270 at 04-Sep-2025 15:48:18 CEST
Begin processing the 71st record. run: 74574126 subRun: 1 event: 27271 at 04-Sep-2025 15:48:18 CEST
Begin processing the 72nd record. run: 74574126 subRun: 1 event: 27272 at 04-Sep-2025 15:48:18 CEST
Begin processing the 73rd record. run: 74574126 subRun: 1 event: 27273 at 04-Sep-2025 15:48:19 CEST
Begin processing the 74th record. run: 74574126 subRun: 1 event: 27274 at 04-Sep-2025 15:48:19 CEST
Begin processing the 75th record. run: 74574126 subRun: 1 event: 27275 at 04-Sep-2025 15:48:19 CEST
Begin processing the 76th record. run: 74574126 subRun: 1 event: 27276 at 04-Sep-2025 15:48:19 CEST
Begin processing the 77th record. run: 74574126 subRun: 1 event: 27277 at 04-Sep-2025 15:48:19 CEST
Begin processing the 78th record. run: 74574126 subRun: 1 event: 27278 at 04-Sep-2025 15:48:19 CEST
Begin processing the 79th record. run: 74574126 subRun: 1 event: 27279 at 04-Sep-2025 15:48:19 CEST
Begin processing the 80th record. run: 74574126 subRun: 1 event: 27280 at 04-Sep-2025 15:48:19 CEST
Begin processing the 81st record. run: 74574126 subRun: 1 event: 27281 at 04-Sep-2025 15:48:20 CEST
Begin processing the 82nd record. run: 74574126 subRun: 1 event: 27282 at 04-Sep-2025 15:48:20 CEST
Begin processing the 83rd record. run: 74574126 subRun: 1 event: 27283 at 04-Sep-2025 15:48:20 CEST
Begin processing the 84th record. run: 74574126 subRun: 1 event: 27284 at 04-Sep-2025 15:48:20 CEST
Begin processing the 85th record. run: 74574126 subRun: 1 event: 27285 at 04-Sep-2025 15:48:20 CEST
Begin processing the 86th record. run: 74574126 subRun: 1 event: 27286 at 04-Sep-2025 15:48:20 CEST
Begin processing the 87th record. run: 74574126 subRun: 1 event: 27287 at 04-Sep-2025 15:48:20 CEST
Begin processing the 88th record. run: 74574126 subRun: 1 event: 27288 at 04-Sep-2025 15:48:21 CEST
Begin processing the 89th record. run: 74574126 subRun: 1 event: 27289 at 04-Sep-2025 15:48:21 CEST
Begin processing the 90th record. run: 74574126 subRun: 1 event: 27290 at 04-Sep-2025 15:48:21 CEST
Begin processing the 91st record. run: 74574126 subRun: 1 event: 27291 at 04-Sep-2025 15:48:21 CEST
Begin processing the 92nd record. run: 74574126 subRun: 1 event: 27292 at 04-Sep-2025 15:48:21 CEST
Begin processing the 93rd record. run: 74574126 subRun: 1 event: 27293 at 04-Sep-2025 15:48:21 CEST
Begin processing the 94th record. run: 74574126 subRun: 1 event: 27294 at 04-Sep-2025 15:48:21 CEST
Begin processing the 95th record. run: 74574126 subRun: 1 event: 27295 at 04-Sep-2025 15:48:22 CEST
Begin processing the 96th record. run: 74574126 subRun: 1 event: 27296 at 04-Sep-2025 15:48:22 CEST
Begin processing the 97th record. run: 74574126 subRun: 1 event: 27297 at 04-Sep-2025 15:48:23 CEST
Begin processing the 98th record. run: 74574126 subRun: 1 event: 27298 at 04-Sep-2025 15:48:23 CEST
Begin processing the 99th record. run: 74574126 subRun: 1 event: 27299 at 04-Sep-2025 15:48:23 CEST
Begin processing the 100th record. run: 74574126 subRun: 1 event: 27300 at 04-Sep-2025 15:48:23 CEST
04-Sep-2025 15:48:24 CEST  Closed input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/99/9c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_272_20231208T002939Z_gen_g4_detsim_hitreco__20240510T040135Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0279365     0.0951291     0.408987      0.070962      0.0749961       100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00109029    0.00218019     0.0201674    0.00175741     0.0023265       100    
prod:pandorapid:Chi2ParticleID                0.00247796    0.00701293     0.0435588     0.0058068    0.00493903       100    
prod:energyreconue:EnergyReco                 0.00874282     0.0170826     0.0450653     0.0135539    0.00782981       100    
prod:energyreconumu:EnergyReco                0.00287838     0.0182401     0.263701      0.0106097     0.0314149       100    
prod:energyreconc:EnergyReco                  0.00257712     0.0106871     0.036647      0.0108094     0.0067549       100    
[art]:TriggerResults:TriggerResultInserter     9.558e-06    1.49833e-05   9.6694e-05    1.3646e-05    8.84415e-06      100    
end_path:ana:PIDASelectionAna                 0.00545009     0.0394288     0.289847      0.0273495     0.0417961       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 10067.7 MB
  Peak resident set size usage (VmHWM): 843.706 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 56
-rw-r--r--. 1 dune003 dune 19683 Sep  4 15:48 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_272_20231208T002939Z_gen_g4_detsim_hitreco__20240510T040135Z_reco2_pida_2025-09-04T_134749Z.root
-rw-r--r--. 1 dune003 dune 14417 Sep  4 15:48 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_272_20231208T002939Z_gen_g4_detsim_hitreco__20240510T040135Z_reco2_reco_2025-09-04T_134749Z.log
-rw-r--r--. 1 dune003 dune 12115 Sep  4 15:48 jobscript.log
-rw-r--r--. 1 dune003 dune   219 Sep  4 15:48 justin-processed-pfns.txt
-rw-r--r--. 1 dune003 dune   138 Sep  4 15:47 all-input-dids.txt
-rw-r--r--. 1 dune003 dune     0 Sep  4 15:47 debugprod.log