Jobsub ID 29305.107@dunegpschedd01.fnal.gov

Jobsub ID	29305.107@dunegpschedd01.fnal.gov
Workflow ID	1657
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-20 00:06:32
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod01
Last heartbeat	2025-08-20 00:13:54
From worker node	Hostname	wg62.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-20 00:07:34
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_135_20231207T022452Z_gen_g4_detsim_hitreco__20240510T034357Z_reco2.root
Jobscript	Exit code	0
	Real time	5m (353s)
	CPU time	1m (99s = 28%)
	Max RSS bytes	1389178880 (1324 MiB)
Outputting started	2025-08-20 00:13:29
Output files
Finished	2025-08-20 00:13:54
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

20-Aug-2025 01:10:05 BST
Begin processing the 30th record. run: 6464269 subRun: 1 event: 13530 at 20-Aug-2025 01:10:07 BST
Begin processing the 31st record. run: 6464269 subRun: 1 event: 13531 at 20-Aug-2025 01:10:08 BST
Begin processing the 32nd record. run: 6464269 subRun: 1 event: 13532 at 20-Aug-2025 01:10:10 BST
Begin processing the 33rd record. run: 6464269 subRun: 1 event: 13533 at 20-Aug-2025 01:10:11 BST
Begin processing the 34th record. run: 6464269 subRun: 1 event: 13534 at 20-Aug-2025 01:10:13 BST
Begin processing the 35th record. run: 6464269 subRun: 1 event: 13535 at 20-Aug-2025 01:10:15 BST
Begin processing the 36th record. run: 6464269 subRun: 1 event: 13536 at 20-Aug-2025 01:10:17 BST
Begin processing the 37th record. run: 6464269 subRun: 1 event: 13537 at 20-Aug-2025 01:10:19 BST
Begin processing the 38th record. run: 6464269 subRun: 1 event: 13538 at 20-Aug-2025 01:10:22 BST
Begin processing the 39th record. run: 6464269 subRun: 1 event: 13539 at 20-Aug-2025 01:10:24 BST
Begin processing the 40th record. run: 6464269 subRun: 1 event: 13540 at 20-Aug-2025 01:10:41 BST
Begin processing the 41st record. run: 6464269 subRun: 1 event: 13541 at 20-Aug-2025 01:10:41 BST
Begin processing the 42nd record. run: 6464269 subRun: 1 event: 13542 at 20-Aug-2025 01:10:43 BST
Begin processing the 43rd record. run: 6464269 subRun: 1 event: 13543 at 20-Aug-2025 01:10:44 BST
Begin processing the 44th record. run: 6464269 subRun: 1 event: 13544 at 20-Aug-2025 01:10:45 BST
Begin processing the 45th record. run: 6464269 subRun: 1 event: 13545 at 20-Aug-2025 01:10:47 BST
Begin processing the 46th record. run: 6464269 subRun: 1 event: 13546 at 20-Aug-2025 01:10:48 BST
Begin processing the 47th record. run: 6464269 subRun: 1 event: 13547 at 20-Aug-2025 01:10:50 BST
Begin processing the 48th record. run: 6464269 subRun: 1 event: 13548 at 20-Aug-2025 01:10:53 BST
Begin processing the 49th record. run: 6464269 subRun: 1 event: 13549 at 20-Aug-2025 01:10:55 BST
Begin processing the 50th record. run: 6464269 subRun: 1 event: 13550 at 20-Aug-2025 01:11:15 BST
Begin processing the 51st record. run: 6464269 subRun: 1 event: 13551 at 20-Aug-2025 01:11:17 BST
Begin processing the 52nd record. run: 6464269 subRun: 1 event: 13552 at 20-Aug-2025 01:11:18 BST
Begin processing the 53rd record. run: 6464269 subRun: 1 event: 13553 at 20-Aug-2025 01:11:20 BST
Begin processing the 54th record. run: 6464269 subRun: 1 event: 13554 at 20-Aug-2025 01:11:22 BST
Begin processing the 55th record. run: 6464269 subRun: 1 event: 13555 at 20-Aug-2025 01:11:26 BST
Begin processing the 56th record. run: 6464269 subRun: 1 event: 13556 at 20-Aug-2025 01:11:28 BST
Begin processing the 57th record. run: 6464269 subRun: 1 event: 13557 at 20-Aug-2025 01:11:28 BST
Begin processing the 58th record. run: 6464269 subRun: 1 event: 13558 at 20-Aug-2025 01:11:30 BST
Begin processing the 59th record. run: 6464269 subRun: 1 event: 13559 at 20-Aug-2025 01:11:31 BST
Begin processing the 60th record. run: 6464269 subRun: 1 event: 13560 at 20-Aug-2025 01:11:33 BST
Begin processing the 61st record. run: 6464269 subRun: 1 event: 13561 at 20-Aug-2025 01:11:35 BST
Begin processing the 62nd record. run: 6464269 subRun: 1 event: 13562 at 20-Aug-2025 01:11:38 BST
Begin processing the 63rd record. run: 6464269 subRun: 1 event: 13563 at 20-Aug-2025 01:11:40 BST
Begin processing the 64th record. run: 6464269 subRun: 1 event: 13564 at 20-Aug-2025 01:11:42 BST
Begin processing the 65th record. run: 6464269 subRun: 1 event: 13565 at 20-Aug-2025 01:11:50 BST
Begin processing the 66th record. run: 6464269 subRun: 1 event: 13566 at 20-Aug-2025 01:11:51 BST
Begin processing the 67th record. run: 6464269 subRun: 1 event: 13567 at 20-Aug-2025 01:11:52 BST
Begin processing the 68th record. run: 6464269 subRun: 1 event: 13568 at 20-Aug-2025 01:12:02 BST
Begin processing the 69th record. run: 6464269 subRun: 1 event: 13569 at 20-Aug-2025 01:12:03 BST
Begin processing the 70th record. run: 6464269 subRun: 1 event: 13570 at 20-Aug-2025 01:12:09 BST
Begin processing the 71st record. run: 6464269 subRun: 1 event: 13571 at 20-Aug-2025 01:12:11 BST
Begin processing the 72nd record. run: 6464269 subRun: 1 event: 13572 at 20-Aug-2025 01:12:13 BST
Begin processing the 73rd record. run: 6464269 subRun: 1 event: 13573 at 20-Aug-2025 01:12:15 BST
Begin processing the 74th record. run: 6464269 subRun: 1 event: 13574 at 20-Aug-2025 01:12:16 BST
Begin processing the 75th record. run: 6464269 subRun: 1 event: 13575 at 20-Aug-2025 01:12:18 BST
Begin processing the 76th record. run: 6464269 subRun: 1 event: 13576 at 20-Aug-2025 01:12:20 BST
Begin processing the 77th record. run: 6464269 subRun: 1 event: 13577 at 20-Aug-2025 01:12:21 BST
Begin processing the 78th record. run: 6464269 subRun: 1 event: 13578 at 20-Aug-2025 01:12:26 BST
Begin processing the 79th record. run: 6464269 subRun: 1 event: 13579 at 20-Aug-2025 01:12:33 BST
Begin processing the 80th record. run: 6464269 subRun: 1 event: 13580 at 20-Aug-2025 01:12:35 BST
Begin processing the 81st record. run: 6464269 subRun: 1 event: 13581 at 20-Aug-2025 01:12:37 BST
Begin processing the 82nd record. run: 6464269 subRun: 1 event: 13582 at 20-Aug-2025 01:12:38 BST
Begin processing the 83rd record. run: 6464269 subRun: 1 event: 13583 at 20-Aug-2025 01:12:40 BST
Begin processing the 84th record. run: 6464269 subRun: 1 event: 13584 at 20-Aug-2025 01:12:41 BST
Begin processing the 85th record. run: 6464269 subRun: 1 event: 13585 at 20-Aug-2025 01:12:43 BST
Begin processing the 86th record. run: 6464269 subRun: 1 event: 13586 at 20-Aug-2025 01:12:45 BST
Begin processing the 87th record. run: 6464269 subRun: 1 event: 13587 at 20-Aug-2025 01:12:57 BST
Begin processing the 88th record. run: 6464269 subRun: 1 event: 13588 at 20-Aug-2025 01:12:59 BST
Begin processing the 89th record. run: 6464269 subRun: 1 event: 13589 at 20-Aug-2025 01:13:01 BST
Begin processing the 90th record. run: 6464269 subRun: 1 event: 13590 at 20-Aug-2025 01:13:03 BST
Begin processing the 91st record. run: 6464269 subRun: 1 event: 13591 at 20-Aug-2025 01:13:09 BST
Begin processing the 92nd record. run: 6464269 subRun: 1 event: 13592 at 20-Aug-2025 01:13:10 BST
Begin processing the 93rd record. run: 6464269 subRun: 1 event: 13593 at 20-Aug-2025 01:13:12 BST
Begin processing the 94th record. run: 6464269 subRun: 1 event: 13594 at 20-Aug-2025 01:13:13 BST
Begin processing the 95th record. run: 6464269 subRun: 1 event: 13595 at 20-Aug-2025 01:13:15 BST
Begin processing the 96th record. run: 6464269 subRun: 1 event: 13596 at 20-Aug-2025 01:13:16 BST
Begin processing the 97th record. run: 6464269 subRun: 1 event: 13597 at 20-Aug-2025 01:13:18 BST
Begin processing the 98th record. run: 6464269 subRun: 1 event: 13598 at 20-Aug-2025 01:13:20 BST
Begin processing the 99th record. run: 6464269 subRun: 1 event: 13599 at 20-Aug-2025 01:13:23 BST
Begin processing the 100th record. run: 6464269 subRun: 1 event: 13600 at 20-Aug-2025 01:13:24 BST
20-Aug-2025 01:13:27 BST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0c/26/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_135_20231207T022452Z_gen_g4_detsim_hitreco__20240510T034357Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.416816       2.11538       16.4978       1.3827        2.2779         100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0212645     0.0360807     0.0588913     0.0366029     0.0126187       100    
prod:pandorapid:Chi2ParticleID                 0.123793      0.205027      0.521946      0.188595      0.069473        100    
prod:energyreconue:EnergyReco                  0.227573      0.505893       1.37595      0.441186      0.179072        100    
prod:energyreconumu:EnergyReco                0.00325705     0.253407       4.30533      0.127819      0.596827        100    
prod:energyreconc:EnergyReco                  0.00662335     0.109352      0.908802      0.0640672     0.154792        100    
[art]:TriggerResults:TriggerResultInserter    2.5033e-05    2.86654e-05   0.000109764   2.7183e-05    8.99866e-06      100    
end_path:ana:PIDASelectionAna                  0.0276903      1.00497       9.68258      0.586898       1.40651        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5783.49 MB
  Peak resident set size usage (VmHWM): 1389.18 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 56
-rw-r--r--. 1 job0004 job0004 20209 Aug 20 01:13 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_135_20231207T022452Z_gen_g4_detsim_hitreco__20240510T034357Z_reco2_pida_2025-08-20T_000741Z.root
-rw-r--r--. 1 job0004 job0004 14115 Aug 20 01:13 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_135_20231207T022452Z_gen_g4_detsim_hitreco__20240510T034357Z_reco2_reco_2025-08-20T_000741Z.log
-rw-r--r--. 1 job0004 job0004 11894 Aug 20 01:13 jobscript.log
-rw-r--r--. 1 job0004 job0004   186 Aug 20 01:13 justin-processed-pfns.txt
-rw-r--r--. 1 job0004 job0004   137 Aug 20 01:07 all-input-dids.txt
-rw-r--r--. 1 job0004 job0004     0 Aug 20 01:07 debugprod.log