Jobsub ID 29298.158@dunegpschedd01.fnal.gov

Jobsub ID	29298.158@dunegpschedd01.fnal.gov
Workflow ID	1657
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-19 23:48:32
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod02
Last heartbeat	2025-08-19 23:51:53
From worker node	Hostname	wj52.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-19 23:50:05
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6453326_108_20231206T230245Z_gen_g4_detsim_hitreco__20240510T024251Z_reco2.root
Jobscript	Exit code	0
	Real time	1m (86s)
	CPU time	0m (47s = 54%)
	Max RSS bytes	1091432448 (1040 MiB)
Outputting started	2025-08-19 23:51:32
Output files
Finished	2025-08-19 23:51:53
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

-Aug-2025 00:50:54 BST
Begin processing the 30th record. run: 6453326 subRun: 1 event: 10830 at 20-Aug-2025 00:50:55 BST
Begin processing the 31st record. run: 6453326 subRun: 1 event: 10831 at 20-Aug-2025 00:50:55 BST
Begin processing the 32nd record. run: 6453326 subRun: 1 event: 10832 at 20-Aug-2025 00:50:55 BST
Begin processing the 33rd record. run: 6453326 subRun: 1 event: 10833 at 20-Aug-2025 00:50:55 BST
Begin processing the 34th record. run: 6453326 subRun: 1 event: 10834 at 20-Aug-2025 00:50:56 BST
Begin processing the 35th record. run: 6453326 subRun: 1 event: 10835 at 20-Aug-2025 00:50:56 BST
Begin processing the 36th record. run: 6453326 subRun: 1 event: 10836 at 20-Aug-2025 00:50:56 BST
Begin processing the 37th record. run: 6453326 subRun: 1 event: 10837 at 20-Aug-2025 00:50:56 BST
Begin processing the 38th record. run: 6453326 subRun: 1 event: 10838 at 20-Aug-2025 00:50:57 BST
Begin processing the 39th record. run: 6453326 subRun: 1 event: 10839 at 20-Aug-2025 00:50:57 BST
Begin processing the 40th record. run: 6453326 subRun: 1 event: 10840 at 20-Aug-2025 00:50:58 BST
Begin processing the 41st record. run: 6453326 subRun: 1 event: 10841 at 20-Aug-2025 00:50:58 BST
Begin processing the 42nd record. run: 6453326 subRun: 1 event: 10842 at 20-Aug-2025 00:50:58 BST
Begin processing the 43rd record. run: 6453326 subRun: 1 event: 10843 at 20-Aug-2025 00:50:58 BST
Begin processing the 44th record. run: 6453326 subRun: 1 event: 10844 at 20-Aug-2025 00:50:59 BST
Begin processing the 45th record. run: 6453326 subRun: 1 event: 10845 at 20-Aug-2025 00:51:00 BST
Begin processing the 46th record. run: 6453326 subRun: 1 event: 10846 at 20-Aug-2025 00:51:00 BST
Begin processing the 47th record. run: 6453326 subRun: 1 event: 10847 at 20-Aug-2025 00:51:01 BST
Begin processing the 48th record. run: 6453326 subRun: 1 event: 10848 at 20-Aug-2025 00:51:01 BST
Begin processing the 49th record. run: 6453326 subRun: 1 event: 10849 at 20-Aug-2025 00:51:02 BST
Begin processing the 50th record. run: 6453326 subRun: 1 event: 10850 at 20-Aug-2025 00:51:02 BST
Begin processing the 51st record. run: 6453326 subRun: 1 event: 10851 at 20-Aug-2025 00:51:02 BST
Begin processing the 52nd record. run: 6453326 subRun: 1 event: 10852 at 20-Aug-2025 00:51:03 BST
Begin processing the 53rd record. run: 6453326 subRun: 1 event: 10853 at 20-Aug-2025 00:51:04 BST
Begin processing the 54th record. run: 6453326 subRun: 1 event: 10854 at 20-Aug-2025 00:51:05 BST
Begin processing the 55th record. run: 6453326 subRun: 1 event: 10855 at 20-Aug-2025 00:51:07 BST
Begin processing the 56th record. run: 6453326 subRun: 1 event: 10856 at 20-Aug-2025 00:51:07 BST
Begin processing the 57th record. run: 6453326 subRun: 1 event: 10857 at 20-Aug-2025 00:51:07 BST
Begin processing the 58th record. run: 6453326 subRun: 1 event: 10858 at 20-Aug-2025 00:51:08 BST
Begin processing the 59th record. run: 6453326 subRun: 1 event: 10859 at 20-Aug-2025 00:51:09 BST
Begin processing the 60th record. run: 6453326 subRun: 1 event: 10860 at 20-Aug-2025 00:51:09 BST
Begin processing the 61st record. run: 6453326 subRun: 1 event: 10861 at 20-Aug-2025 00:51:10 BST
Begin processing the 62nd record. run: 6453326 subRun: 1 event: 10862 at 20-Aug-2025 00:51:11 BST
Begin processing the 63rd record. run: 6453326 subRun: 1 event: 10863 at 20-Aug-2025 00:51:12 BST
Begin processing the 64th record. run: 6453326 subRun: 1 event: 10864 at 20-Aug-2025 00:51:12 BST
Begin processing the 65th record. run: 6453326 subRun: 1 event: 10865 at 20-Aug-2025 00:51:12 BST
Begin processing the 66th record. run: 6453326 subRun: 1 event: 10866 at 20-Aug-2025 00:51:13 BST
Begin processing the 67th record. run: 6453326 subRun: 1 event: 10867 at 20-Aug-2025 00:51:13 BST
Begin processing the 68th record. run: 6453326 subRun: 1 event: 10868 at 20-Aug-2025 00:51:14 BST
Begin processing the 69th record. run: 6453326 subRun: 1 event: 10869 at 20-Aug-2025 00:51:14 BST
Begin processing the 70th record. run: 6453326 subRun: 1 event: 10870 at 20-Aug-2025 00:51:14 BST
Begin processing the 71st record. run: 6453326 subRun: 1 event: 10871 at 20-Aug-2025 00:51:15 BST
Begin processing the 72nd record. run: 6453326 subRun: 1 event: 10872 at 20-Aug-2025 00:51:15 BST
Begin processing the 73rd record. run: 6453326 subRun: 1 event: 10873 at 20-Aug-2025 00:51:16 BST
Begin processing the 74th record. run: 6453326 subRun: 1 event: 10874 at 20-Aug-2025 00:51:17 BST
Begin processing the 75th record. run: 6453326 subRun: 1 event: 10875 at 20-Aug-2025 00:51:17 BST
Begin processing the 76th record. run: 6453326 subRun: 1 event: 10876 at 20-Aug-2025 00:51:18 BST
Begin processing the 77th record. run: 6453326 subRun: 1 event: 10877 at 20-Aug-2025 00:51:18 BST
Begin processing the 78th record. run: 6453326 subRun: 1 event: 10878 at 20-Aug-2025 00:51:19 BST
Begin processing the 79th record. run: 6453326 subRun: 1 event: 10879 at 20-Aug-2025 00:51:19 BST
Begin processing the 80th record. run: 6453326 subRun: 1 event: 10880 at 20-Aug-2025 00:51:19 BST
Begin processing the 81st record. run: 6453326 subRun: 1 event: 10881 at 20-Aug-2025 00:51:20 BST
Begin processing the 82nd record. run: 6453326 subRun: 1 event: 10882 at 20-Aug-2025 00:51:20 BST
Begin processing the 83rd record. run: 6453326 subRun: 1 event: 10883 at 20-Aug-2025 00:51:20 BST
Begin processing the 84th record. run: 6453326 subRun: 1 event: 10884 at 20-Aug-2025 00:51:21 BST
Begin processing the 85th record. run: 6453326 subRun: 1 event: 10885 at 20-Aug-2025 00:51:21 BST
Begin processing the 86th record. run: 6453326 subRun: 1 event: 10886 at 20-Aug-2025 00:51:21 BST
Begin processing the 87th record. run: 6453326 subRun: 1 event: 10887 at 20-Aug-2025 00:51:22 BST
Begin processing the 88th record. run: 6453326 subRun: 1 event: 10888 at 20-Aug-2025 00:51:22 BST
Begin processing the 89th record. run: 6453326 subRun: 1 event: 10889 at 20-Aug-2025 00:51:23 BST
Begin processing the 90th record. run: 6453326 subRun: 1 event: 10890 at 20-Aug-2025 00:51:23 BST
Begin processing the 91st record. run: 6453326 subRun: 1 event: 10891 at 20-Aug-2025 00:51:23 BST
Begin processing the 92nd record. run: 6453326 subRun: 1 event: 10892 at 20-Aug-2025 00:51:23 BST
Begin processing the 93rd record. run: 6453326 subRun: 1 event: 10893 at 20-Aug-2025 00:51:24 BST
Begin processing the 94th record. run: 6453326 subRun: 1 event: 10894 at 20-Aug-2025 00:51:25 BST
Begin processing the 95th record. run: 6453326 subRun: 1 event: 10895 at 20-Aug-2025 00:51:25 BST
Begin processing the 96th record. run: 6453326 subRun: 1 event: 10896 at 20-Aug-2025 00:51:30 BST
Begin processing the 97th record. run: 6453326 subRun: 1 event: 10897 at 20-Aug-2025 00:51:30 BST
Begin processing the 98th record. run: 6453326 subRun: 1 event: 10898 at 20-Aug-2025 00:51:30 BST
Begin processing the 99th record. run: 6453326 subRun: 1 event: 10899 at 20-Aug-2025 00:51:30 BST
Begin processing the 100th record. run: 6453326 subRun: 1 event: 10900 at 20-Aug-2025 00:51:31 BST
20-Aug-2025 00:51:31 BST  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/ef/82/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6453326_108_20231206T230245Z_gen_g4_detsim_hitreco__20240510T024251Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0736446      0.3355        2.61311      0.232826       0.34297        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00257859    0.00743332     0.0518627    0.00512816    0.00794609       100    
prod:pandorapid:Chi2ParticleID                 0.0116932      0.04069      0.169217      0.0215633     0.0386036       100    
prod:energyreconue:EnergyReco                  0.0367135     0.0732629     0.361036      0.0566168     0.0474929       100    
prod:energyreconumu:EnergyReco                0.00185678     0.0332188      0.29398      0.0184752     0.0467875       100    
prod:energyreconc:EnergyReco                  0.00386599     0.0311242     0.190005      0.0255165     0.0312414       100    
[art]:TriggerResults:TriggerResultInserter    1.5045e-05    1.96397e-05   9.4721e-05    1.7371e-05    8.56548e-06      100    
end_path:ana:PIDASelectionAna                 0.00913952      0.14927       1.66346      0.0884781     0.220306        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6561.02 MB
  Peak resident set size usage (VmHWM): 1091.43 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 56
-rw-r--r--. 1 job0003 job0003 20444 Aug 20 00:51 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6453326_108_20231206T230245Z_gen_g4_detsim_hitreco__20240510T024251Z_reco2_pida_2025-08-19T_235011Z.root
-rw-r--r--. 1 job0003 job0003 14121 Aug 20 00:51 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6453326_108_20231206T230245Z_gen_g4_detsim_hitreco__20240510T024251Z_reco2_reco_2025-08-19T_235011Z.log
-rw-r--r--. 1 job0003 job0003 11898 Aug 20 00:51 jobscript.log
-rw-r--r--. 1 job0003 job0003   188 Aug 20 00:51 justin-processed-pfns.txt
-rw-r--r--. 1 job0003 job0003   137 Aug 20 00:50 all-input-dids.txt
-rw-r--r--. 1 job0003 job0003     0 Aug 20 00:50 debugprod.log