REPRESENTATIONS OF THE SYMMETRIC GROUP WHICH ARE IRREDUCIBLE OVER SUBGROUPS

JONATHAN BRUNDAN AND ALEXANDER S. KLESHCHEV

Abstract. Let F be an algebraically closed field of characteristic p, and Σ_{n} be the symmetric group on n letters. In this paper we classify all pairs (G, D), where D is an irreducible F Σ_{n}-module of dimension greater than 1 and G is a proper subgroup of Σ_{n}, such that the restriction D↓_{G} is irreducible, provided p > 3.

Introduction

Let F be an algebraically closed field of characteristic p ≥ 0, and Σ_{n} be the symmetric group on n letters. The main result of this paper is a complete solution of the following problem for the case p > 3:

Problem 0.1. Classify all pairs (G, D), where D is an irreducible F Σ_{n}-module of dimension greater than 1 and G is a proper subgroup of Σ_{n} such that the restriction D↓_{G} is irreducible.

Remark 0.2. (i) In the proof we use the classification of 2-transitive groups, which in turn relies on the classification theorem of finite simple groups.

(ii) The case p = 0 was completely solved by Saxl [39]. His work greatly influenced this paper.

(iii) The result in characteristic p is important for the problem of describing maximal subgroups of finite classical groups, see [2], [26].

(iv) Of course, the case dim D = 1, excluded in Problem 0.1, is not interesting from the representation theoretic point of view.

We consider some examples, which will be parts of our main theorem. In what follows we denote by D^{λ} the irreducible F Σ_{n}-module corresponding to a p-regular partition λ = (λ_{1}, λ_{2}, …) of n. Moreover, 1 and sgn will stand for the trivial and the sign representations of F Σ_{n}, respectively. These are the only 1-dimensional representations of Σ_{n}.

Example 0.3. Assume G is the alternating group A_{n} < Σ_{n}. We consider two cases.

(i) p > 2. Using Clifford theory one can easily prove that D^{λ}↓_{A_{n}} is irreducible if and only if D^{λ} ⊗sgn ≅ D̸^{λ}. In other words, D^{λ}↓_{A_{n}} is irreducible if and only if λ^{M} ≠ λ, where λ ↦ λ^{M} is the Mullineux bijection. This bijection is known explicitly, see [30], [14], [4], and section 2 below.

(ii) p = 2. This case has been treated by Benson [3]. He proves that D^{λ}↓_{A_{n}} is irreducible if and only if there is j > 0 such that λ_{2j−1} − λ_{2j} > 2 or λ_{2j−1} + λ_{2j} ≡ 2 (mod 4).

Example 0.4. Assume G = Σ_{n−1} embedded naturally as a subgroup of all permutations in Σ_{n} fixing one point. Gather together the equal parts of λ to represent it in the form λ = (l_{1}^{a_{1}}, …, l_{k}^{a_{k}}) for l_{1} > ⋯ > l_{k}, a_{i} > 0. We say λ is a Jantzen-Seitz partition if

l_{i} − l_{i+1} + a_{i} + a_{i+1} ≡ 0 (mod p) for all 1 ≤ i < k.

Then D^{λ}↓_{Σ_{n−1}} is irreducible if and only if λ is Jantzen-Seitz. This result was conjectured and proved in one direction in [24]. It was first proved in full generality in [27], Theorem D. Somewhat different proofs were found later in [13] and [28].

1991 Mathematics Subject Classification: 20C20, 20C30, 20B35, 20B20.

Both authors acknowledge the support of the NSF (grants DMS-9801442 and DMS-9600124).

Example 0.5. Assume G < Σ_{n} is 2-transitive, and D = D^{(n−1,1)} is the irreducible core of the natural permutation F Σ_{n}-module. We will call D^{(n−1,1)} the natural irreducible F Σ_{n}-module. This case was studied by Mortimer [35]. Note that (modulo the Classification Theorem) the list of 2-transitive groups is known—it can be found for example in [25], see also section 5 of this paper. Mortimer was able to settle all the cases with two little exceptions, one of which (for Ree groups) was later settled by Brandl [7] and the other (for Co_{3}) can be treated using Gap. For brevity, we state Mortimer’s result only for the case p > 3, as follows. If G is 2-transitive and p > 3 then D^{(n−1,1)}↓_{G} is irreducible with the following exceptions:

(i) G ≤ AΓL(m, q), and p divides q;

(ii) G ≤ PΓL(m, q), m ≥ 3, and p divides q;

(iii) G ≤ Sz(q), and p divides q + 1 + m, where m^{2} = 2q;

(iv) G ≤ PΓU(3, q), and p divides q + 1;

(v) G ≤ Re(q), and p divides (q + 1)(q + m + 1), where m^{2} = 3q.

Now we are able to formulate our main result.

Main Theorem. Let n be a positive integer, p > 3, D^{λ} be an irreducible F Σ_{n}-module of dimension > 1, and G < Σ_{n} be a proper subgroup. Then D^{λ}↓_{G} is irreducible if and only if one of the following happens:

(i) λ ≠ λ^{M} and G = A_{n}, see Example 0.3;

(ii) D^{λ} or D^{λ}⊗sgn is D^{(n−1,1)}, and G is a 2-transitive subgroup not in the list of exceptions from Example 0.5.

(iii) D^{λ} or D^{λ} ⊗ sgn is D^{(n−2,2)}, and G is PGL(3, 2) (p = 5), PΓL(2, 8) (p ≠ 7), M_{11} (p ≠ 5), M_{12} (p ≠ 5), M_{23} or M_{24} in their natural permutation representations of degrees n = 7, 9, 11, 12, 23 or 24, respectively;

(iv) D^{λ} or D^{λ} ⊗ sgn is D^{(n−2,1^{2})}, and G is AGL(m, 2) (m ≥ 3), M_{11} (p ≠ 11), M_{11}, M_{12}, 2^{4}.A_{7}, M_{22}, M_{22}.2, M_{23} or M_{24} in their permutation representations of degrees n = 2^{m}, 11, 12, 12, 16, 22, 22, 23 or 24, respectively;

(v) n = 8, p = 5, D^{λ} or D^{λ} ⊗ sgn is D^{(5,3)}, and G is AGL(3, 2) in its permutation representation of degree 8;

(vi) D^{λ} or D^{λ} ⊗ sgn is D^{(21,2,1)} or D^{(21,1^{3})}, and G is M_{24} with n = 24;

(vii) λ is a Jantzen-Seitz partition, and G = Σ_{n−1} embedded naturally, see Example 0.4; (viii) λ is a Jantzen-Seitz partition such that λ^{M} ≠ λ, and G = A_{n−1}; (ix) n ≡ 0 (mod p), D^{λ} or D^{λ} ⊗ sgn is D^{(n−1,1)}, and G < Σ_{n−1} < Σ_{n} is a 2-transitive

subgroup of Σ_{n−1};

(x) D^{λ} or D^{λ} ⊗ sgn is D^{(n−2,2)}, G < Σ_{n−1} < Σ_{n}, and the triple (G, n, p) is one of (M_{11}, 12, 5), (M_{12}, 13, 11), (M_{23}, 24, 11) or (M_{24}, 25, 23);

(xi) D^{λ} or D^{λ} ⊗ sgn is D^{(n−2,1^{2})}, G < Σ_{n−1} < Σ_{n}, and the triple (G, n, p) is one of (AGL(m, 2), 2^{m} + 1, p) with p | (2^{m} + 1), (M_{11}, 13, 13), (M_{12}, 13, 13), (2^{4}.A_{7}, 17, 17), (M_{22}, 23, 23), (M_{22}.2, 23, 23) or (M_{24}, 25, 5);

(xii) n = 9, p = 5, D^{λ} or D^{λ} ⊗ sgn is D^{(6,3)}, and G is AGL(3, 2) embedded into Σ_{8} < Σ_{9} via its permutation representation of degree 8;

(xiii) D^{λ} or D^{λ} ⊗ sgn is D^{(22,1^{3})}, and (G, n, p) = (M_{24}, 25, 5).

Remark 0.6. (i) A nice description of the Jantzen-Seitz partitions λ with λ^{M} = λ in terms of the Mullineux symbols was obtained in [5].

(ii) Unfortunately, we were unable to treat the cases p = 2 and 3 completely using our methods. The main reduction to 2-transitive groups still works for p = 3 (see Theorem 0.7 below) or p = 2 and n odd (see [33]). In these cases there is hope for a complete solution given further analysis of the 2-transitive cases; however the case p = 2 and n even seems to be difficult.

Note that the groups from (i)-(vi) in the theorem above are 2-transitive, while the groups from (vii)-(xiii) are intransitive (even contained in Σ_{n−1}). In other words, if D^{λ}↓_{G} is irreducible for some D^{λ} then either G is 2-transitive or G ≤ Σ_{n−1}. This observation has a direct proof. In characteristic 0 it can be deduced using the Littlewood-Richardson rule, see [39], while in characteristic p it was proved in [33]:

Theorem 0.7. [33, Main Theorem] Let p > 2, n ≥ 4, and G be a subgroup of Σ_{n}. If there exists an irreducible F Σ_{n}-module D, of dimension > 1, such that the restriction D↓_{G} is irreducible then either G ≤ Σ_{n−1} or G is 2-transitive.

Now, we can explain the strategy of the proof of the Main Theorem. By Theorem 0.7, if D^{λ}↓_{G} is irreducible then either G ≤ Σ_{n−1} or G is 2-transitive. Assume first that G ≤ Σ_{n−1}. The case G = Σ_{n−1} is covered by Example 0.4. Otherwise, either G is a 2-transitive subgroup of Σ_{n−1} or is contained in Σ_{n−2}. It turns out that D^{λ}↓_{Σ_{n−2}} is always reducible. So, as a criterion for the irreducibility of D^{λ}↓_{Σ_{n−1}} is known, everything reduces to the case where G < Σ_{n} is 2-transitive. The main tool which allows us to treat the 2-transitive groups is Proposition 3.4 below, as well as some dimension bounds. We note that we used Gap [40] to deal with the sporadic groups. We also appeal to some results on dimensions and characters from the Atlas and modular Atlas [10, 23].

Acknowledgements. We benefited greatly from discussions with Jan Saxl, Gary Seitz, and especially Bill Kantor. We also thank Martin Liebeck and Peter Cameron for some useful remarks and Christine Bessenrodt for spotting a mistake in an earlier version of this paper.

1. Preliminary results

Throughout the paper F is an algebraically closed field of characteristic p > 0. Let G be a group, and M be an F G-module. We denote by M^{G} the space of G-invariants in M. If D_{1}, …, D_{k} are irreducible F G-modules then the notation M = D_{1}| … |D_{k} means that M is a uniserial F G-module with composition factors D_{1}, …, D_{k} counted from bottom to top. Let D_{1}, …, D_{k} be pairwise non-isomorphic. Then M = a_{1}D_{1} +⋯+a_{k}D_{k} means that D_{1}, …, D_{k} are the composition factors of M, with multiplicities a_{1}, …, a_{k}, respectively. Also, we write M ≅ a_{1}D_{1} ⊕ ⋯ ⊕ a_{k}D_{k} if M is completely reducible, and D_{1}, …, D_{k} are the composition factors of M with multiplicities a_{1}, …, a_{k}, respectively. The trivial F G-module is denoted by 1_{G} or simply 1. It is well known that for any irreducible F G-module D,

dim D ≤ \sqrt{|G|}.  (1)

We refer the reader to [17, 19, 20] for the standard facts and notation of the representation theory of the symmetric group Σ_{n}. (However we use the ‘left’ notation while James [17] uses the ‘right’ one—for example, here (1, 2)(2, 3) = (1, 2, 3), and not (1, 3, 2) for the product of transpositions.) In particular, we denote by D^{λ} the irreducible F Σ_{n}-module corresponding to a p-regular partition λ of n. Given any partition μ of n, one associates to it the (standard) Young subgroup Σ_{μ}, the Specht module S^{μ} and the permutation module M^{μ} = (1_{Σ_{μ}})↑^{Σ_{n}}. Sometimes we will need to consider the Specht modules over the field ℂ of complex numbers. To distinguish them from the Specht modules S^{λ} over F we will use the notation S_{ℂ}^{λ}. Let sgn_{Σ_{n}} or simply sgn be the 1-dimensional sign representation of F Σ_{n}. For F Σ_{n}-modules V_{1}, V_{2}, we write V_{1} ∼ V_{2} if V_{1} and V_{2} belong to the same block of the algebra F Σ_{n}.

Let k ≤ n/2. Then a subgroup G < Σ_{n} is called k-homogeneous (resp. k-transitive) if it acts transitively on the unordered (resp. ordered) k-element subsets of {1, 2, …, n}. The following observation will be of great importance.

Lemma 1.1. If G < Σ_{n} is a subgroup then the number of G-orbits on unordered k-element subsets of {1, 2, …, n} is equal to the dimension of the G-invariants space (M^{(n−k,k)})^{G}.

Proof. Note that M^{(n−k,k)}↓_{G} is just the permutation module corresponding to the action of G on the unordered k-element subsets. So it suffices to note that every orbit contributes exactly a 1-dimensional space of invariants.

Now we give various basic lemmas concerning the representation theory of Σ_{n}.

Lemma 1.2. A block component of a self-dual F Σ_{μ}-module is self-dual. In particular, if D^{λ} is an irreducible F Σ_{n}-module then a block component of D^{λ}↓_{Σ_{μ}} is self-dual.

Proof. This follows from the fact that irreducible F Σ_{n}-modules (and hence F Σ_{μ}-modules) are self-dual, see [17], Theorem 11.5.

Lemma 1.3. Let M be a self-dual F Σ_{μ}-module. Then M is irreducible if and only if dim End_{F Σ_{μ}}(M) = 1.

Proof. In one direction the result follows from Schur’s lemma, and in the other direction it follows from the fact that irreducible F Σ_{μ}-modules are self-dual.

Lemma 1.4. Let λ and μ be p-regular partitions of n. Then dim Hom_{F Σ_{n}}(S^{λ}, (S^{μ})^{∗}) = δ_{λμ}.

Proof. The head of S^{λ} is isomorphic to D^{λ} and all other composition factors are of the form D^{ν} with ν ⊳ λ. Similarly, (S^{μ})^{∗} has socle D^{μ} and all other composition factors are of the form D^{ν} with ν ⊳ μ. So if f : S^{λ} → (S^{μ})^{∗} is a non-zero homomorphism we deduce that λ ⊵ μ and μ ⊵ λ. Finally if λ = μ then f can only send the head of S^{λ} to the socle of (S^{λ})^{∗}.

Lemma 1.5. Let p > 2, n ≥ 5, D be an irreducible F Σ_{n}-module of dimension > 1, and c ∈ Σ_{n} be an involution. Then c has both eigenvalues 1 and −1 on D.

Proof. Let ρ : Σ_{n} → GL(D) be the corresponding representation. Then ρ is faithful, and so c must have eigenvalue −1. On the other hand, if c does not have eigenvalue 1 then ρ(c) = − id ∈ Z(ρ(Σ_{n})), which is impossible as Σ_{n} has trivial center.

Lemma 1.6. If p > k then any indecomposable F [Σ_{n−k} × Σ_{k}]-module is of the form I ⊗ D^{μ}, where I is an indecomposable F Σ_{n−k}-module and D^{μ} is an irreducible F Σ_{k}-module. In particular, an F [Σ_{n−k} × Σ_{k}]-module is completely reducible if and only if its restriction to Σ_{n−k} is completely reducible.

Proof. This follows from the fact that the group algebra F Σ_{k} is semisimple.

The following well known fact can be deduced for example from [37], Proposition 2.

Lemma 1.7. Let 1 ≤ j ≤ n − 1. Then ⋀^{j} S^{(n−1,1)} ≅ S^{(n−j,1^{j})}.

We will often use the following characteristic-free version of the Littlewood-Richardson Rule. Let c_{μν}^{λ} denote the classical Littlewood-Richardson coefficient.

Theorem 1.8. [22, 3.1,5.5] Let λ be a partition of m + n. Then the restriction S^{λ}↓_{Σ_{m}×Σ_{n}} has a filtration with factors of the form S^{μ} ⊗ S^{ν}, with S^{μ} ⊗ S^{ν} appearing precisely c_{μν}^{λ} times.

We will also need the following results on decomposition numbers.

Lemma 1.9. [17, 24.1] Let p > 2, 1 ≤ k < p, and n > k + 1. Then S^{(n−k,1^{k})} = D^{(n−k,1^{k})} if p does not divide n, and for p dividing n we have:

S^{(n−k,1^{k})} = D^{(n−k,1^{k})} + D^{(n−k+1,1^{k−1})}.

Lemma 1.10. [17, 24.15] Let r ≤ k and r < p. Then

[S^{(m,k)} : D^{(m+r,k−r)}] = \begin{cases}1 & if p divides m − k + r + 1; \\ 0 & otherwise. \end{cases}

We next record the following curious general result:

Lemma 1.11. Let μ and ν be p-regular partitions of m and n, respectively. Then μ + ν (coordinate-wise sum) is a p-regular partition of m+n, and D^{μ}⊗D^{ν} appears as a composition factor of the restriction D^{μ+ν}↓_{Σ_{m}×Σ_{n}} with multiplicity 1.

Proof. The fact that μ + ν is p-regular is obvious. Moreover, it is proved in [8], Theorem D, that the multiplicity [D^{μ+ν}↓_{Σ_{m}×Σ_{n}} : D^{μ}⊗D^{ν}] is equal to the multiplicity of the tilting module T (μ+ν) over GL_{m+n}(F) as an indecomposable summand of the tensor product T (μ)⊗T (ν). But this multiplicity is 1 as μ + ν is the highest weight of the module T (μ) ⊗ T (ν), and the corresponding weight space is 1-dimensional.

Now we introduce some combinatorial notions concerning partitions. The dominance order on partitions is denoted by ⊳, see [17]. Fix an arbitrary partition λ = (λ_{1} ≥ λ_{2} ≥ …) ⊢ n. The maximal number h with λ_{h} > 0 is called the height of λ and denoted by h(λ). We do not distinguish between λ and its Young diagram

λ = {(i, j) ∈ ℤ_{>0} × ℤ_{>0} | j ≤ λ_{i}}.

Elements (i, j) ∈ ℤ_{>0} × ℤ_{>0} are called nodes. The residue of a node A = (i, j), written res A, is defined to be the residue class (j − i) (mod p). The residue content of the Young diagram λ is defined to be the p-tuple

cont(λ) = (c_{0}, c_{1}, …, c_{p−1})

where

c_{α} = #{nodes in λ of residue α}, α = 0, 1, …, p − 1.

The Nakayama Conjecture [20], Theorems 2.7.41, 6.1.21, claims that D^{λ} and D^{μ} are in the same block (i.e. D^{λ} ∼ D^{μ}) if and only if cont(λ) = cont(μ).

A node (i, λ_{i}) ∈ λ is called removable if λ_{i} > λ_{i+1}. A node (i, λ_{i} + 1) is called addable if i = 1 or i > 1 and λ_{i} < λ_{i−1}. If A = (i, λ_{i}) is a removable node then

λ_{A} := λ \ {A} = (λ_{1}, …, λ_{i−1}, λ_{i} − 1, λ_{i+1}, …)

is a partition of n − 1 obtained from λ by removing A. We say a node (i, j) is above (resp. below) (i′, j′) if i < i′ (resp. i > i′). A removable node A of λ is called normal if for every addable node B above A with res B = res A there exists a removable node C(B) strictly between A and B with res C(B) = res A, and B ≠ B′ implies C(B) ≠ C(B′). A removable node is called good if it is the lowest among the normal nodes of a fixed residue. We remark that there is at most one good node of each residue. The following branching theorem is one of the main results of [29], Theorems 0.5, 0.6, and [32], Theorem 1.4.

Theorem 1.12. [29, 32] Let D^{λ} be an arbitrary irreducible F Σ_{n}-module.

(i) The socle of D^{λ}↓_{Σ_{n−1}} is isomorphic to ⊕D^{λ_{A}}, where the sum is over all good nodes of λ. Moreover, D^{λ}↓_{Σ_{n−1}} is completely reducible (so equal to its socle) if and only if every normal node of λ is good.

(ii) Let A be a removable node of λ such that λ_{A} is p-regular. Then D^{λ_{A}} appears as a composition factor of D^{λ}↓_{Σ_{n−1}} if and only if A is normal. In that case, the multiplicity [D^{λ}↓_{Σ_{n−1}} : D^{λ_{A}}] is equal to the number of normal nodes C above A with res C = res A (counting A itself).

Next we discuss completely splittable representations of Σ_{n}, which will play a role in several different situations. The main reference will be [31] (these modules have also been studied in [43, 34, 38] and several other papers).

Definition 1.13. [31] An irreducible F Σ_{n}-module D is called completely splittable if the restriction D↓_{Σ_{μ}} is completely reducible for any Young subgroup Σ_{μ} of Σ_{n}.

For a partition λ = (λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{s} > 0) define

χ(λ) := λ_{1} − λ_{s} + s.

The next theorem provides necessary and sufficient conditions for an irreducible module to be completely splittable.

Theorem 1.14. [31] Let λ be a p-regular partition of n. Then D^{λ} is completely splittable if and only if χ(λ) ≤ p.

Lemma 1.15. Let p > 2, n ≥ m ≥ p−1, D^{λ} be completely splittable, and λ satisfy h(λ) > 1. Then 1_{Σ_{m}} is not a composition factor of D^{λ}↓_{Σ_{m}}.

Proof. We prove this by downward induction on m = n, n − 1, …, p − 1, the induction base being obvious. Suppose m < n, and the result is known for m + 1. Pick a composition factor D^{μ} of D^{λ}↓_{Σ_{m+1}}. By induction, μ ≠ (m + 1). Note that D^{μ} is completely splittable, so by Theorem 1.12(i), D^{μ}↓_{Σ_{m}} has D^{(m)} as a composition factor only if μ = (m, 1). Moreover, if μ = (m, 1) then χ(μ) = m + 1, so by Theorem 1.14 we also have that m ≤ p − 1 whence m = p − 1. Finally, we have that D^{(p−1,1)}↓_{Σ_{p−1}} ≅ D^{(p−2,1)} using Theorem 1.12(i). So in all cases, D^{μ} ↓_{Σ_{m}} does not have D^{(m)} as a composition factor.

In the special case p = 5, completely splittable modules were considered by Ryba [38]. Denote by f_{n} the nth Fibonacci number, defined from f_{0} = 0, f_{1} = 1, f_{n} = f_{n−1} + f_{n−2} for n ≥ 2. It turns out that the dimensions of the non-trivial completely splittable modules are given by the Fibonacci numbers. The following result deals with the case where h(λ) = 2. However it is general enough, as in characteristic 5 the completely splittable D^{μ} with h(μ) = 3 are obtained as D^{λ} ⊗ sgn for some completely splittable D^{λ} with h(λ) = 2, and completely splittable D^{λ} with h(λ) = 1 or 4 are 1-dimensional.

Theorem 1.16. [38] Let p = 5 and suppose that λ satisfies h(λ) = 2, χ(λ) ≤ p.

(i) If n = 2r +1 is odd then λ = (r +1, r) or (r +2, r −1), in which cases dim D^{(r+1,r)} = f_{n} and dim D^{(r+2,r−1)} = f_{n−1}.

(ii) If n = 2r is even then λ = (r, r) or (r + 1, r − 1), in which cases dim D^{(r,r)} = f_{n−1} and dim D^{(r+1,r−1)} = f_{n}.

We will need to use a well known expression for the Fibonacci numbers:

Lemma 1.17. For any n ≥ 0, we have

f_{n} = \frac{1}{\sqrt{5}} (\frac{1 + \sqrt{5}}{2})^{n} − \frac{1}{\sqrt{5}} (\frac{1 − \sqrt{5}}{2})^{n} .

The remainder of the section is to do with dimensions of the irreducible F Σ_{n}-modules. Most of the results we need were obtained by James [18]. Following [18], set

R_{n}(j) := {D^{λ}, D^{λ} ⊗ sgn | λ = (λ_{1}, λ_{2}, …) with λ_{1} ≥ n − j}.  (2)

Lemma 1.18. [18, Theorem 7 and Table 1] Let p > 3, and D^{λ} be an irreducible F Σ_{n}-module. Assume that n ≥ 9 if p > 5, and n ≥ 11 if p = 5. Then:

(i) either D^{λ} ∈ R_{n}(1) or dim D^{λ} ≥ (n^{2} − 5n + 2)/2;

(ii) either D^{λ} ∈ R_{n}(2) or dim D^{λ} > (n^{2} − 3n + 2)/2.

We need a further result in the spirit of Lemma 1.18. We follow the strategy of James [18]:

Lemma 1.19. [18, Lemma 4] Suppose that N and m are non-negative integers and that f(n) is a function of n satisfying all the following conditions:

(a) 2f(n − 2) > f(n) for all n ≥ N + 2;

(b) for n = N and N + 1, every irreducible F Σ_{n}-module D^{λ} either belongs to R_{n}(m) or has dim D^{λ} > f(n);

(c) for all n ≥ N, if D^{λ} ∈ R_{n}(m + 2) \ R_{n}(m), then dim D^{λ} > f(n).

Then for all n ≥ N, every irreducible F Σ_{n}-module D^{λ} either belongs to R_{n}(m) or has dim D^{λ} > f(n).

Our extension of Lemma 1.18 is as follows.

Lemma 1.20. Let p > 3, n ≥ 13 and D^{λ} be an irreducible F Σ_{n}-module. Then, either D^{λ} ∈ R_{n}(2) or dim D^{λ} ≥ (n^{3} − 9n^{2} + 14n)/6.

Proof. We apply Lemma 1.19 with f(n) = (n^{3} − 9n^{2} + 14n)/6 − 1, m = 2, and N = 13. Condition (a) of the lemma is easy to verify. To check the condition (b) we need to show that the dimensions of irreducible modules over Σ_{13} (resp., Σ_{14}) outside R_{13}(2) (resp. R_{14}(2)) are greater than 142 (resp. 195). We employed the Gap package Specht to do this daunting work. To check the condition (c), we need to prove that the lemma holds for the modules from R_{n}(4) \ R_{n}(2). But this follows from Lemma 1.21.

Remark. The lower bound in Lemma 1.20 above is the best possible in the sense that dim D^{(n−3,3)} = (n^{3} − 9n^{2} + 14n)/6 for infinitely many values of n.

The next result computes the dimensions of irreducible modules D^{λ} ∈ R_{n}(4) \ R_{n}(1).

Lemma 1.21. Let p > 3. Then:

(i) dim D^{(n−2,2)} = \begin{cases}(n^{2} − 5n + 2)/2 & if n ≡ 2 (mod p); \\ (n^{2} − 3n − 2)/2 & if n ≡ 1 (mod p); \\ (n^{2} − 3n)/2 & otherwise. \end{cases}

(ii) dim D^{(n−2,1^{2})} = \begin{cases}(n^{2} − 5n + 6)/2 & if n ≡ 0 (mod p); \\ (n^{2} − 3n + 2)/2 & otherwise. \end{cases}

(iii) dim D^{(n−3,3)} = \begin{cases}(n^{3} − 9n^{2} + 14n)/6 & if n ≡ 4 (mod p); \\ (n^{3} − 6n^{2} − n + 6)/6 & if n ≡ 3 (mod p); \\ (n^{3} − 6n^{2} + 5n − 6)/6 & if n ≡ 2 (mod p); \\ (n^{3} − 6n^{2} + 5n)/6 & otherwise. \end{cases}

(iv) dim D^{(n−3,2,1)} = \begin{cases}(2n^{3} − 15n^{2} + 25n − 6)/6 & if n ≡ 3 (mod p); \\ (2n^{3} − 15n^{2} + 25n + 6)/6 & if n ≡ 1 (mod p); \\ (2n^{3} − 12n^{2} + 16n)/6 & otherwise. \end{cases}

(v) dim D^{(n−3,1^{3})} = \begin{cases}(n^{3} − 9n^{2} + 26n − 24)/6 & if n ≡ 0 (mod p); \\ (n^{3} − 6n^{2} + 11n − 6)/6 & otherwise. \end{cases}

(vi) dim D^{(n−4,4)} = \begin{cases}(n^{4} − 14n^{3} + 47n^{2} − 34n)/24 & if n ≡ 6 (mod p); \\ (n^{4} − 10n^{3} + 11n^{2} + 22n)/24 & if n ≡ 5 (mod p); \\ (n^{4} − 10n^{3} + 23n^{2} − 38n + 24)/24 & if n ≡ 4 (mod p); \\ (n^{4} − 10n^{3} + 23n^{2} − 14n − 24)/24 & if n ≡ 3 (mod p); \\ (n^{4} − 10n^{3} + 23n^{2} − 14n)/24 & otherwise. \end{cases}

(vii) dim D^{(n−4,3,1)} = \begin{cases}(3n^{4} − 38n^{3} + 129n^{2} − 118n)/24 & if n ≡ 5 (mod p); \\ (3n^{4} − 30n^{3} + 69n^{2} − 18n − 24)/24 & if n ≡ 4 (mod p); \\ (3n^{4} − 34n^{3} + 105n^{2} − 74n + 24)/24 & if n ≡ 2 (mod p); \\ (3n^{4} − 30n^{3} + 81n^{2} − 54n)/24 & otherwise. \end{cases}

(viii) dim D^{(n−4,2^{2})} = \begin{cases}(2n^{4} − 28n^{3} + 118n^{2} − 140n + 24)/24 & if n ≡ 3 (mod p); \\ (2n^{4} − 20n^{3} + 46n^{2} + 20n − 24)/24 & if n ≡ 2 (mod p); \\ (2n^{4} − 20n^{3} + 58n^{2} − 40n)/24 & otherwise. \end{cases}

(ix) dim D^{(n−4,2,1^{2})} = \begin{cases}(3n^{4} − 38n^{3} + 153n^{2} − 190n − 24)/24 & if n ≡ 1 (mod p); \\ (3n^{4} − 34n^{3} + 117n^{2} − 134n + 24)/24 & if n ≡ 4 (mod p); \\ (3n^{4} − 30n^{3} + 93n^{2} − 90n)/24 & otherwise. \end{cases}

(x) dim D^{(n−4,1^{4})} = \begin{cases}(n^{4} − 14n^{3} + 71n^{2} − 154n + 120)/24 & if n ≡ 0 (mod p); \\ (n^{4} − 10n^{3} + 35n^{2} − 50n + 24)/24 & otherwise. \end{cases}

Proof. Parts (ii), (v), (x) follow from Lemma 1.9, and (i), (iii), (vi) follow from Lemma 1.10.

(iv) Let λ = (n − 3, 2, 1). We prove the result by induction on n = 5, 6, . . .. For the induction base, the Nakayama Conjecture implies that D^{(2,2,1)} = S^{(2,2,1)} so its dimension is known by the Hook Formula [17], Theorem 20.1. Let n > 5. If n ≢ 1, 3 (mod p) then S^{λ} is irreducible by the Nakayama Conjecture and we can apply the Hook Formula again. The case n ≡ 3 (mod p) is considered in [18], Appendix. Finally, if n ≡ 1 (mod p) then D^{λ}↓_{Σ_{n−1}} ≅ D^{(n−4,2,1)} ⊕ D^{(n−3,1^{2})} by Theorem 1.12(i), and the conclusion follows using (ii) and the inductive hypothesis.

(vii) Let λ = (n − 4, 3, 1). Again, we use induction on n = 7, 8, . . .. If n = 7, the formula follows from the Nakayama Conjecture and the Hook Formula, unless p = 5 which is easily checked directly. So let n > 7. If n ≢ 2, 4, 5 (mod p) then S^{λ} is irreducible by the Nakayama Conjecture. For the case n ≡ 5 (mod p) see [18], Appendix. If n ≡ 4 (mod p) then D^{λ}↓_{Σ_{n−1}} ≅ D^{(n−5,3,1)} ⊕ D^{(n−4,2,1)} ⊕ D^{(n−4,3)} by Theorem 1.12(i), and we apply (iii),

(v) and the inductive hypothesis. If n ≡ 2 (mod p) the result follows similarly since then D^{λ}↓_{Σ_{n−1}} ≅ D^{(n−5,3,1)} ⊕ D^{(n−4,2,1)}.

(viii) Let λ = (n − 4, 2^{2}). Use induction on n ≥ 6. The formula follows using the Hook Formula as before unless n > 6 and n ≡ 2, 3 (mod p). For the case n ≡ 3 (mod p) see [18], Appendix. If n ≡ 2 (mod p) use the fact that D^{λ}↓_{Σ_{n−1}} ≅ D^{(n−5,2^{2})} ⊕ D^{(n−4,2,1)}, (v), and the inductive hypothesis.

(ix) The proof is similar to that of (viii) and uses the decomposition D^{(n−4,2,1^{2})}↓_{Σ_{n−1}} ≅ D^{(n−5,2,1^{2})} ⊕ D^{(n−4,2,1)} for n ≡ 4 (mod p).

Lemma 1.22. Let p > 3, and D^{λ} be an irreducible F Σ_{n}-module with D^{λ} ∉ R_{n}(1). (i) Let n = 5. Then dim D^{λ} ≥ 5. (ii) Let n = 6. Then dim D^{λ} ≥ 8, except for

dim D^{(3^{2})} = dim(D^{(3^{2})} ⊗ sgn) = 5.

(iii) Let n = 7. Then dim D^{λ} ≥ 10, except for

dim D^{(5,2)} = dim(D^{(5,2)} ⊗ sgn) = 8, if p = 5.

(iv) Let n = 8. Then dim D^{λ} ≥ 19, except for

dim D^{(4^{2})} = dim(D^{(4^{2})} ⊗ sgn) = \begin{cases}14, & if p > 5; \\ 13, & if p = 5; \end{cases}

(v) Let n = 9. Then dim D^{λ} ≥ 27, except for

dim D^{(7,2)} = dim(D^{(7,2)} ⊗ sgn) = 19, if p = 7,
dim D^{(6,3)} = dim(D^{(6,3)} ⊗ sgn) = 21, if p = 5.

(vi) Let n = 10. Then dim D^{λ} ≥ 42, except for

dim D^{(8,2)} = dim(D^{(8,2)} ⊗ sgn) = 35,
dim D^{(5^{2})} = dim(D^{(5^{2})} ⊗ sgn) = 34, if p = 5,
dim D^{(8,1^{2})} = dim(D^{(8,1^{2})} ⊗ sgn) = \begin{cases}36, & if p > 5; \\ 28, & if p = 5. \end{cases}

(vii) Let n = 11. Then dim D^{λ} ≥ 66, except for

dim D^{(7,4)} = dim(D^{(7,4)} ⊗ sgn) = 55, if p = 5,
dim D^{(9,2)} = dim(D^{(9,2)} ⊗ sgn) = \begin{cases}44, & if p > 5; \\ 43, & if p = 5; \end{cases}

dim D^{(9,1^{2})} = dim(D^{(9,1^{2})} ⊗ sgn) = \begin{cases}36 & if p = 11; \\ 45 & if p ≠ 11; \end{cases}

(viii) Let n = 12. Then dim D^{λ} ≥ 221, except for

dim D^{(7,5)} = dim(D^{(7,5)} ⊗ sgn) = 144, if p = 5,

dim D^{(10,2)} = dim(D^{(10,2)} ⊗ sgn) = \begin{cases}53, & if p = 11; \\ 43, & if p = 5; \\ 54, & otherwise; \end{cases}

dim D^{(10,1^{2})} = dim(D^{(10,1^{2})} ⊗ sgn) = 55;

dim D^{(6^{2})} = dim(D^{(6^{2})} ⊗ sgn) = \begin{cases}89, & if p = 5; \\ 131, & if p = 7; \\ 132, & otherwise; \end{cases}

dim D^{(9,3)} = dim(D^{(9,3)} ⊗ sgn) = \begin{cases}153, & if p = 5; \\ 154, & otherwise; \end{cases}
dim D^{(9,1^{3})} = dim(D^{(9,1^{3})} ⊗ sgn) = 165.

Proof. This follows from [23, 10] and [17], Tables. The calculation was also checked on Gap, using the package Specht by Andrew Mathas.

The final lemma of the section will be needed when we consider the Mathieu groups.

Lemma 1.23. Let λ be a p-regular partition of n = 22, 23 or 24. Suppose that λ ∉ R_{n}(4), and dim D^{λ} ≤ 10395. Then, p = 2, and λ = (12, 10), (12, 11), or (13, 11).

Proof. This is a calculation involving Gap. All dimensions of all D^{λ} are known (in the Gap library) for n = 14. To obtain a lower bound of the dimension of D^{λ} for larger n, we simply used the branching rule in Theorem 1.12(ii) recursively.

2. Results involving the Mullineux bijection

For any p-regular λ, the module D^{λ} ⊗ sgn is irreducible. The Mullineux bijection λ → λ^{M} on the p-regular partitions of n is defined from

D^{λ} ⊗ sgn ≅ D^{λ^{M}}.

Recall that a partition is called a p-core if none of its hook lengths is divisible by p, see [20], section 2.7. In the special case that λ is a p-core, it is easy to calculate λ^{M} using the following well-known result:

Lemma 2.1. If λ is a p-core then λ^{M} = λ′ (the transpose partition).

Proof. It follows from the Nakayama Conjecture that S^{λ} = D^{λ} and S^{λ′} = D^{λ′}. Now one can use the isomorphism S^{λ} ⊗ sgn ≅ (S^{λ′})^{∗}, see [17], Theorem 8.15.

For more general λ, Theorem 1.12(i) and the Nakayama Conjecture easily imply the following relation between the Mullineux bijection and good nodes:

Lemma 2.2. [30, 4.8] Let λ be a p-regular partition, and A be a good node for λ of residue α. Then there exists a unique good node B for λ^{M} of residue −α such that (λ_{A})^{M} = (λ^{M})_{B}.

The lemma yields an explicit algorithm for computing the bijection λ ↦ λ^{M} in general. However, it is often more convenient to use the description coming from the Mullineux conjecture [36], proved in [14] (see also [4]). To explain the Mullineux algorithm, we first recall some definitions, referring the reader to [14, 4] for more details.

Fix a p-regular Young diagram λ. Its rim is defined to be the set of all nodes (i, j) ∈ λ such that (i + 1, j + 1) ∉ λ; in other words the rim of the Young diagram is its ‘south-east border’. The p-rim (called the p-edge in [14]) is a certain subset of the rim, defined as the union of the p-segments. The first p-segment is simply the first p nodes of the rim, reading along the rim from ‘north-east’ to ‘south-west’. The next p-segment is then obtained by reading off the next p nodes of the rim, but starting from the row immediately below the last node of the first p-segment. The remaining p-segments are obtained by repeating this process. Of course, all but the last p-segment contain exactly p nodes, while the last may contain less. For example, let λ = (6, 4^{2}, 2, 1), p = 5. The nodes of the p-rim (which consists of two p-segments) are coloured in black in the following picture.

[figure]

Set λ^{(1)} = λ, and define λ^{(i)} to be λ^{(i−1)} \ {the p-rim of λ^{(i−1)}}. Let m be the largest number such that λ^{(m)} ≠ ∅. Then the Mullineux symbol of λ is defined to be the array

G(λ) = (\begin{matrix}a_{1} & a_{2} & … & a_{m} \\ r_{1} & r_{2} & … & r_{m} \end{matrix})

where a_{i} is the number of the nodes of the p-rim of λ^{(i)} and r_{i} = h(λ^{(i)}) is the height of λ^{(i)}. The partition can be uniquely reconstructed from its Mullineux symbol, see [36, 14, 4]. For

1 ≤ i ≤ m, set G_{i}(λ) := (\begin{matrix}a_{i} \\ r_{i} \end{matrix}) to be the i-th column of the Mullineux symbol. Also, define

ε_{i} = 0 if p divides a_{i} and ε_{i} = 1 otherwise.

Theorem 2.3. [14, 4] If G(λ) = (\begin{matrix}a_{1} & a_{2} & … & a_{m} \\ r_{1} & r_{2} & … & r_{m} \end{matrix}) then G(λ^{M}) = (\begin{matrix}a_{1} & a_{2} & … & a_{m} \\ s_{1} & s_{2} & … & s_{m} \end{matrix})

where s_{i} = a_{i} + ε_{i} − r_{i}.

A useful consequence of the Mullineux algorithm is the following simple formula for computing the height of the partition λ^{M}:

Corollary 2.4. If a p-regular partition λ has a nodes in its p-rim then the height of λ^{M} is a + ε − h(λ), where ε = 0 if p divides a and ε = 1 otherwise.

Lemma 2.5. Let λ be a p-regular partition, h = h(λ), and A ≠ (h, 1) be a normal node of λ. If A is not the first node of a p-segment of λ then h((λ_{A})^{M}) = h(λ^{M}).

Proof. In view of Corollary 2.4, we need to show that G_{1}(λ) = G_{1}(λ_{A}). If A is not the first and not the last node of a p-segment the statement is clear. Let I_{1}, I_{2}, …, I_{l} be the p-segments of λ counted from top to bottom, and assume A is the last node of I_{j}. We will get a contradiction to the assumption that A is normal. Note that the length of the p-segment I_{j} is p, which is immediate for j < l, while if j = l it is ensured by the assumption that A ≠ (h, 1). Define 1 ≤ k ≤ j to be the maximal index such that I_{k} ∪ I_{k+1} ∪ ⋯ ∪ I_{j} is a connected subset of the rim. Let (x, y) be the first node of I_{k}. As I_{k−1} is disconnected from I_{k}, the node C := (x, y + 1) is addable (if k = 1 this is also true). Moreover it is easy to see that res C = res A. It remains to observe that no removable node between A and C has residue equal to res A, as it would contradict the connectedness of I_{k} ∪ I_{k+1} ∪ ⋯ ∪ I_{j}.

Lemma 2.6. For a p-regular partition λ, set μ := λ^{M} and h := h(μ).

(i) If λ = (s^{r}) for some 1 ≤ s < r < p, then h ≤ s and μ_{h} ≥ r.

(ii) If λ = (s^{r}, u^{t}) for some 1 ≤ u < s < r with r + s − u + t = p, then h ≤ s and μ_{h} ≥ r.

Proof. (i) If s + r − 1 < p then λ is a p-core. So λ^{M} = λ′ by Lemma 2.1, and the result

follows. Otherwise set x := r + s − p. Then the first x columns of G(λ) are of the form (\begin{matrix}p \\ r \end{matrix}),

and λ^{(x)} = ((s − x)^{(r−x)}), which is a p-core. By Lemma 2.1, (λ^{(x)})^{M} = ((r − x)^{(s−x)}). To get λ^{M} we should ‘glue’ x p-rims to (λ^{(x)})^{M}, each consisting of p nodes in p − r = s − x rows, see Theorem 2.3. It follows that h = s − x ≤ s, and μ_{h} ≥ (r − x) + x = r.

(ii) The proof is similar to that of (i). The first u columns of G(λ) have form (\begin{matrix}p \\ r + t \end{matrix}), and

λ^{(u)} = ((s − u)^{(r−u)}), which is a p-core. By Lemma 2.1, (λ^{(u)})^{M} = ((r − u)^{(s−u)}). To get λ^{M} we should ‘glue’ u p-rims to (λ^{(u)})^{M}, each consisting of p nodes in p − r − t = s − u rows. It follows that h = s − u ≤ s, and μ_{h} ≥ (r − u) + u = r.

Lemma 2.7. Let p > k ≥ 2, n > 2k, and λ = (r, 1^{s}) be a p-regular hook partition of n such that h(λ) ≥ k and h(λ^{M}) ≥ k. Then there exists a composition factor D^{μ} of D^{λ}↓_{Σ_{n−1}} with h(μ) ≥ k and h(μ^{M}) ≥ k.

Proof. Let A and B be the bottom and the top removable nodes of λ, respectively. If r > p then G_{1}(λ) = G_{1}(λ_{B}), and so h(λ_{B}), h((λ_{B})^{M}) ≥ k. Moreover, D^{λ_{B}} is a composition factor of D^{λ}↓_{Σ_{n−1}} by Theorem 1.12(ii), as the top removable node is always normal. Now assume r ≤ p. Then the p-rim covers the whole of λ. Moreover, λ is p-regular, so s + r ≤ p − 1 + p = 2p − 1.

If s + r ≠ p, p + 1, then both A and B are good, and λ, λ_{A}, λ_{B} are all p-cores. So the result follows from Lemma 2.1 and Theorem 1.12, as n ≥ 2k. Next, assume s + r = p. Then we have h(λ) = s + 1 = h(λ_{B}), h(λ^{M}) = p − s − 1 = h((λ_{B})^{M}), which proves the result.

Finally, let r + s = p + 1. Then both A and B are normal, so D^{λ_{A}} and D^{λ_{B}} appear in D^{λ}↓_{Σ_{n−1}}. Moreover, λ^{M} = (s + 1, 1^{r−1}). Let C be the top removable node of λ^{M}, and set ν := (λ^{M})_{C}, μ := ν^{M}. Then C is normal for λ^{M}, and so D^{ν} appears in D^{λ^{M}}↓_{Σ_{n−1}}, whence D^{μ} appears in D^{λ}↓_{Σ_{n−1}}. Now, note that h(λ) = s + 1, h(λ^{M}) = r, h(λ_{B}) = s + 1, h((λ_{B})^{M}) = r − 2, h(λ_{A}) = s, h((λ_{A})^{M}) = r − 1, h(μ^{M}) = h(ν) = r, and h(μ) = s − 1. By assumption, s ≥ k−1, r ≥ k, and the result follows, using the assumption n = r+s > 2k.

Now we can prove the main result of the section.

Theorem 2.8. Let p > k ≥ 2, n > 2k, and λ be a p-regular partition of n such that h(λ) ≥ k and h(λ^{M}) ≥ k. Then there exists a composition factor D^{μ} of D^{λ}↓_{Σ_{n−1}} with h(μ) ≥ k and h(μ^{M}) ≥ k.

Proof. Assume for a contradiction that for every composition factor D^{μ} of D^{λ}↓_{Σ_{n−1}} we have h(μ) < k or h(μ^{M}) < k. At least one of h(λ) or h(λ^{M}) must actually equal k, for if both are greater than k then for any good node A, D^{λ_{A}} is a composition factor of D^{λ} ↓_{Σ_{n−1}} by Theorem 1.12(i) but h(λ_{A}), h((λ_{A})^{M}) ≥ k, using Lemma 2.2. Therefore, without loss of generality, we may assume that h(λ) = k. Also by Lemma 2.7, λ is not a hook. We will constantly use the following observation which follows using Theorem 1.12(ii) (observing that λ_{B} is automatically p-regular since p > k = h(λ)) and Lemma 2.5.

(*) If B is a normal node of λ then either h(λ_{B}) < k or h((λ_{B})^{M}) < k. In particular, G_{1}(λ_{B}) ≠ G_{1}(λ).

Let I be the first p-segment of λ. Let A_{1}, A_{2}, …, A_{v} be the removable nodes contained in I other than the last node of I, counted from top to bottom. Since λ is p-regular and not a hook, we certainly have that v ≥ 1. If A := A_{v} is not the first node of I, then A is normal, so by (*) we have G_{1}(λ_{A}) ≠ G_{1}(λ), which contradicts Lemma 2.5. Hence, v = 1 and A = A_{1} is the first node of I. Moreover, the last node of I is not removable, for otherwise G_{1}(λ) = G_{1}(λ_{A}), as λ is not a hook. But this contradicts (*). We deduce in particular that λ has more than one p-segment since it is not a hook. Observe also that the first and second p-segments of λ are connected, for otherwise again G_{1}(λ) = G_{1}(λ_{A}), contradicting (*).

Now let J be the second p-segment of λ and let B_{1}, …, B_{w} be the removable nodes contained in J other than the last node of J, counted from top to bottom. Again, we have w ≥ 1 as λ is p-regular and not a hook. Certainly, B := B_{1} is normal so G_{1}(λ) ≠ G_{1}(λ_{B}) by (*). Hence B must be the first node of J by virtue of Lemma 2.5. Now set μ = λ_{B}, so that h(μ) = k and h(μ^{M}) < k in view of (*). The p-rim of μ is of length at least p + |J| − 1, and |J| > 1 as λ is not a hook. Hence, using Corollary 2.4, h(μ^{M}) ≥ p + |J| − k. We deduce that k > p + |J| − k, hence |J| < p as p ≥ k. So λ in fact has exactly two p-segments, the p-rim of λ coincides with its rim, and has length exactly p + |J|. Also the p-rim of μ has length exactly p + |J| − 1. By Corollary 2.4, h(λ^{M}) = p + |J| − k + 1. But h(λ^{M}) ≥ k and k > p + |J| − k, so h(λ^{M}) = k. As the p-rim of λ coincides with its rim, it follows that the p-rim of λ has length exactly 2k − 1. Applying (*) and Lemma 2.5 once more, we deduce that none of B_{2}, …, B_{w} are normal, so we must have in fact that w = 1 or 2 and λ has one of the following three shapes:

(i)  A B D  (ii)  A γ B C D (iii)  A γ B C

In all diagrams, A is the top node of the first p-segment, B is the top node of the second p-segment. In diagrams (ii) and (iii), γ denotes an addable node having the same residue as C, which ensures that C is not normal. In diagrams (i) and (ii), D is good.

Recall also that we have shown that h(λ) = h(λ^{M}) = k. So we can repeat the above argument for λ^{M} instead of λ, to deduce that λ^{M} must also have one of the above shapes (i)–(iii). The idea now is to show that if λ has any of the shapes (i)–(iii), then such λ does not have Mullineux image of shapes (i)–(iii), giving a contradiction to complete the proof.

Assume first that λ is of shape (i). In particular, λ has just three removable nodes A = (1, k), B = (a, b), D = (k, 1), and k − b + a − 1 = p. As p ≥ k, we get b < a. Moreover, the partition λ^{(1)} obtained from λ by removing the p-rim is λ^{(1)} = ((b − 1)^{a−1}). So we can apply Lemma 2.6(i) to conclude that (λ^{(1)})^{M} = (μ_{1}, …, μ_{h}) with h ≤ b − 1 and μ_{h} ≥ a − 1. Now, to get λ^{M} one needs to adjoin to (λ^{(1)})^{M} two p-segments of lengths 2k − 1 − p and p in k rows. It follows that either the p-rim of λ^{M} is disconnected or the top p-segment of λ^{M} has nodes in the first column. Hence λ^{M} does not have any of the forms (i)–(iii).

Finally assume λ has shapes (ii) or (iii). In particular, λ has removable nodes A = (1, k), B = (a, b), C := (c, d), together with D = (k, 1) in case (ii), and k − b + a − 1 = p. As in the previous paragraph, we get b < a, and the partition λ^{(1)} = ((b − 1)^{a−1}, (d − 1)^{c−a}) satisfies the assumptions of Lemma 2.6(ii) (because C and γ have the same residue). So (λ^{(1)})^{M} = (μ_{1}, …, μ_{h}) with h ≤ b − 1 and μ_{h} ≥ a − 1. Now we repeat the argument from the previous paragraph.

3. Main technical results

In this section we will be interested in dimensions of the spaces of the form

Hom_{F Σ_{n}}(M^{(n−k,k)}, End_{F} (D^{λ})),

where we regard End_{F} (D^{λ})) as an F Σ_{n}-module in the usual way. This is important in view of Proposition 3.4. Note that by Frobenius reciprocity we have

Hom_{F Σ_{n}}(M^{(n−k,k)}, End_{F} (D^{λ})) ≅ End_{F [Σ_{n−k}×Σ_{k}]}(D^{λ}↓_{Σ_{n−k}×Σ_{k}}).  (3)

Recall that M^{(n−k,k)} is a permutation module on the (unordered) k-element subsets I = {i_{1}, …, i_{k}} ⊂ {1, …, n}. Define the F Σ_{n}-homomorphism

f : M^{(n−k,k)} → M^{(n−k+1,k−1)}, {i_{1}, …, i_{k}} ↦ ∑_{j=1}^{k}{i_{1}, …, î_{j}, …, i_{k}}  (4)

for {i_{1}, …, i_{k}} ⊂ {1, …, n}.

Lemma 3.1. Let p > k ≥ 1 and n ≥ 2k. Then f is surjective.

Proof. Let I be a (k − 1)-element subset of {1, …, n}. We show that the corresponding basis element {I} ∈ M^{(n−k+1,k−1)} belongs to im f. As n ≥ 2k, there exist a subset J ⊆ {1, …, n}\I of order k. Put

v := ∑_{A⊆I, B⊆J, |A|+|B|=k}(−1)^{|B|}\binom{k − 1}{k − |B|}^{−1}\{A ∪ B\}.

We claim that f(−\frac{1}{k}v) = \{I\}. Indeed, if A ⊆ I, B ⊆ J, and |A| + |B| = k then the basis element {I} appears in f({A ∪ B}) only if A = I and |B| = 1. So the coefficient of {I} in f(v) is −k. On the other hand, if {K} appears in f(v) and {K} ≠ {I} then {K} = {C ∪ D} where C ⫋ I, D ⫋ J, |C| + |D| = k − 1. Pick any such {K} = {C ∪ D} and set l = |C|.

Then {C ∪ D} appears in f({A ∪ B}) only if A = C ∪ {a} for some a ∈ I \ C and B = D or A = C and B = D ∪ {b} for some b ∈ J \ D. So the coefficient of {K} in f(v) is

(k − 1 − l)(−1)^{k−1−l}\binom{k − 1}{k − (k − 1 − l)}^{−1} + (k − (k − 1 − l))(−1)^{k−l}\binom{k − 1}{k − (k − l)}^{−1} = 0.

We are now going to use the explicit construction of the Specht module S^{λ} as the submodule of M^{λ} generated by the vector e_{t} = κ_{t}{t} ∈ M^{λ}, see [17], section 4, for the precise definitions.

Lemma 3.2. Let p > k ≥ 1 and n ≥ 2k. Then ker f = S^{(n−k,k)}.

Proof. By [17], Lemma 4.10, S^{(n−k,k)} ⊆ ker f. So the result follows from the equality dim M^{(n−k,k)} = dim S^{(n−k,k)} + dim M^{(n−k+1,k−1)}, see e.g. [17], Example 17.17.

Lemma 3.3. Let p > k ≥ 1 and n ≥ 2k. Denote by Y^{(n−k,k)} the block component of M^{(n−k,k)} containing S^{(n−k,k)}. If there exists r with 0 < r ≤ k and n ≡ 2k − 1 − r (mod p), then S^{(n−k,k)} = D^{(n−k+r,k−r)}|D^{(n−k,k)}, Y^{(n−k,k)} = D^{(n−k+r,k−r)}|D^{(n−k,k)}|D^{(n−k+r,k−r)}, and Y^{(n−k,k)}/D^{(n−k,k)} ≅ (S^{(n−k,k)})^{∗}. Otherwise Y^{(n−k,k)} = S^{(n−k,k)} = D^{(n−k,k)}.

Proof. By [17], Example 17.17, the permutation module M^{(n−k,k)} has a filtration with factors S^{(n−k,k)}, S^{(n−k+1,k−1)}, …, S^{(n)}, with S^{(n−k,k)} being a submodule. Hence Y^{(n−k,k)} has a filtration whose factors are all Specht modules S^{(n−j,j)} with 0 ≤ j ≤ k and S^{(n−j,j)} ∼ S^{(n−k,k)}. If there is no r as in the assumption then it follows from the Nakayama Conjecture that S^{(n−k,k)} is the only such Specht module. Moreover, in this case S^{(n−k,k)} ≅ D^{(n−k,k)} thanks to Lemma 1.10. So we may assume that there is r with 0 < r ≤ k and n ≡ 2k − 1 − r (mod p). Then Y^{(n−k,k)} has a filtration with factors S^{(n−k,k)}, S^{(n−k+r,k−r)}. By Lemma 1.10, S^{(n−k,k)} = D^{(n−k+r,k−r)}|D^{(n−k,k)} and S^{(n−k+r,k−r)} ≅ D^{(n−k+r,k−r)}. Now the lemma follows from the fact that Y^{(n−k,k)} is self-dual, see Lemma 1.2.

The following proposition will be a principal tool in the proof of the Main Theorem.

Proposition 3.4. Let p > k ≥ 1, n ≥ 2k, and D^{λ} be an irreducible F Σ_{n}-module such that

dim Hom_{F Σ_{n}}(M^{(n−k,k)}, End_{F} (D^{λ})) > dim Hom_{F Σ_{n}}(M^{(n−k+1,k−1)}, End_{F} (D^{λ})).

If G is a subgroup of Σ_{n} such that

dim (M^{(n−k,k)})^{G} > dim (M^{(n−k+1,k−1)})^{G}

then the restriction D^{λ}↓_{G} is reducible.

Proof. By Lemmas 3.1 and 3.2, we have an exact sequence

0 ⟶ S^{(n−k,k)} ⟶ M^{(n−k,k)} ⟶^{f} M^{(n−k+1,k−1)} ⟶ 0.

So dim (M^{(n−k,k)})^{G} > dim (M^{(n−k+1,k−1)})^{G} implies

(S^{(n−k,k)})^{G} ≠ 0.  (5)

By dualizing, we also have an exact sequence

0 ⟶ (M^{(n−k+1,k−1)})^{∗} ⟶ (M^{(n−k,k)})^{∗}⟶(S^{(n−k,k)})^{∗} ⟶ 0,

so, using the fact that permutation modules M^{λ} are self-dual and the assumption, we have

((S^{(n−k,k)})^{∗})^{G} ≠ 0.  (6)

Moreover, it follows from the assumption that there exists a homomorphism θ from M^{(n−k,k)} to End_{F} (D^{λ}) which does not factor through f. In other words the restriction θ|S^{(n−k,k)} is a non-zero homomorphism.

Assume first that there does not exist r with 0 < r ≤ k and n ≡ 2k − 1 − r (mod p). Then by Lemma 3.3, S^{(n−k,k)} = D^{(n−k,k)} is a direct summand of M^{(n−k,k)}. So θ induces an embedding of S^{(n−k,k)} into End_{F} (D^{λ}). As 1_{Σ_{n}} is always a submodule of End_{F} (D^{λ}), we have 1_{Σ_{n}} ⊕ S^{(n−k,k)} ⊂ End_{F} (D^{λ}). Now it follows from (5) that

dim End_{F G}(D^{λ}↓_{G}) = dim End_{F} (D^{λ})^{G} ≥ 2,

whence D^{λ}↓_{G} is reducible.

Next, assume that n ≡ 2k − 1 − r (mod p) for some r with 0 < r ≤ k. As in Lemma 3.3, denote the block component of M^{(n−k,k)} containing S^{(n−k,k)} by Y^{(n−k,k)}, so that Y^{(n−k,k)} = D^{(n−k+r,k−r)}|D^{(n−k,k)}|D^{(n−k+r,k−r)}. If θ|Y^{(n−k,k)} is not injective then, using the fact that θ|S^{(n−k,k)} is non-zero and Y^{(n−k,k)} is uniserial, we see that θ induces an embedding of (S^{(n−k,k)})^{∗} ≅ Y^{(n−k,k)}/D^{(n−k+r,k−r)} into End_{F} (D^{λ}). As (S^{(n−k,k)})^{∗} = D^{(n−k,k)}|D^{(n−k+r,k−r)}, its socle is different from 1_{Σ_{n}}, hence 1_{Σ_{n}} ⊕ (S^{(n−k,k)})^{∗} ⊂ End_{F} (D^{λ}). So (6) implies that dim End_{F G}(D^{λ}↓_{G}) ≥ 2 as above.

Finally assume that θ|Y^{(n−k,k)} is an embedding. If r ≠ k then D^{(n−k+r,k−r)} ≅ 1̸_{Σ_{n}} so we have that 1_{Σ_{n}} ⊕ S^{(n−k,k)} ⊂ End_{F} (D^{λ}) and hence dim End_{F G}(D^{λ}↓_{G}) ≥ 2. Now let k = r, when Y^{(n−k,k)} = 1_{Σ_{n}}|D^{(n−k,k)}|1_{Σ_{n}}. Hence M^{(n−k,k)}/Y^{(n−k,k)} ≅ M^{(n−k+1,k−1)}/1_{Σ_{n}}. As 1_{Σ_{n}} is a summand of M^{(n−k+1,k−1)},

dim (M^{(n−k,k)}/Y^{(n−k,k)})^{G} < dim (M^{(n−k+1,k−1)})^{G}.

On the other hand, by assumption,

dim (M^{(n−k+1,k−1)})^{G} < dim (M^{(n−k,k)})^{G} = dim (M^{(n−k,k)}/Y^{(n−k,k)})^{G} + dim (Y^{(n−k,k)})^{G}.

Therefore dim (Y^{(n−k,k)})^{G} ≥ 2. As the restriction θ|Y^{(n−k,k)} is an embedding, it follows that dim End_{F G}(D^{λ}↓_{G}) ≥ 2.

We turn next to the problem of verifying the conditions of the proposition above, when Lemma 3.1 turns out to again be useful. Fix now an irreducible F Σ_{n}-module D^{λ} and define the linear map

f^{∗} : Hom_{F Σ_{n}}(M^{(n−k+1,k−1)}, End_{F} (D^{λ})) → Hom_{F Σ_{n}}(M^{(n−k,k)}, End_{F} (D^{λ})), ψ ↦ ψ ∘ f.

By Lemma 3.1, we immediately have

Lemma 3.5. Let p > k ≥ 1 and n ≥ 2k. Then f^{∗} is injective.

We are going to prove that im f^{∗} is properly contained in Hom_{F Σ_{n}}(M^{(n−k,k)}, End_{F} (D^{λ})), by exhibiting a homomorphism φ which does not factor through f. For a subset I = {i_{1}, …, i_{k}} ⊂ {1, …, n} we denote by Σ_{I} = Σ_{{i_{1},...,i_{k}}} the subgroup of Σ_{n} which consists of all permutations fixing the elements of {1, …, n} \ I. Clearly Σ_{I} ≅ Σ_{k}. We define φ ∈ Hom_{F Σ_{n}}(M^{(n−k,k)}, End_{F} (D^{λ})) by setting for any {I} ∈ M^{(n−k,k)}:

φ({I})(v) = ∑_{g∈Σ_{I}} gv.  (7)

Denote by C = C_{t} the column stabilizer of the tableau

t = \begin{matrix}k + 1 & k + 2 & … & 2k & 2k + 1 & … & n \\ 1 & 2 & … & k \end{matrix} .

This is a subgroup of Σ_{n} isomorphic to the elementary abelian group \underbrace{ℤ/2ℤ × ⋯ × ℤ/2ℤ}_{k}.

Henceforth we will use James’ notation {t} for the element {1, …, k} ∈ M^{(n−k,k)}.

Proposition 3.6. Let p > k ≥ 1 and n ≥ 2k. Let Σ_{k} be embedded into Σ_{n} as Σ_{{1,...,k}}. Assume that φ ∈ im f^{∗}. Then

( ∑_{g∈Σ_{k}, σ∈C}(sign σ)σgσ^{−1})D^{λ} = 0.  (8)

Proof. By assumption, there is ψ ∈ Hom_{F Σ_{n}}(M^{(n−k+1,k−1)}, End_{F} (D^{λ})) with φ = f^{∗}(ψ) = ψ ∘ f. Let

e_{t} = ∑_{σ∈C}(sign σ)σ{t} ∈ S^{(n−k,k)} ⊂ M^{(n−k,k)}

be the polytabloid corresponding to the tableau t defined above, see [17], Definition 4.3. By [17], Lemma 4.10, f(e_{t}) = 0. Hence φ(e_{t}) = 0. On the other hand, we calculate φ(e_{t}) using (7):

φ(e_{t})(v) = ∑_{σ∈C}(sign σ) ∑_{g∈Σ_{{σ(1),...,σ(k)}}} gv = ∑_{g∈Σ_{k}, σ∈C}(sign σ)σgσ^{−1}v.

So φ(e_{t}) ≡ 0 is equivalent to (8).

Lemma 3.7. Let p > k > 1, and D^{λ} be an irreducible F Σ_{2k}-module. If the equation (8) holds then h(λ) < k or h(λ^{M}) < k.

Proof. Set x := ∑_{g∈Σ_{k}, σ∈C}(sign σ)σgσ^{−1} ∈ F Σ_{n}. Assume h(λ) ≥ k and h(λ^{M}) ≥ k. Denote by a_{1} the length of the p-rim of λ, see section 1. Then h(λ^{M}) = a_{1} + ε_{1} − h(λ) ≥ k by Corollary 2.4. Hence a_{1} ≥ 2k − 1, and so λ is one of the following three partitions: (k, 1^{k}), (k + 1, 1^{k−1}) = (k, 1^{k})^{t} or (k, 2, 1^{k−2}). Moreover, (k, 2, 1^{k−2}) should be excluded if p = 2k − 1 (as then h(λ^{M}) = k − 1). In all these cases λ is a core partition so D^{λ} is projective by the Nakayama conjecture, and D^{λ} = S^{λ}. So we have to prove that xS^{λ} ≠ 0.

We first prove that xS^{(k,2,1^{k−2})} ≠ 0. Define the tableau

s = 1 k + 1 k + 3 … 2k ; 2 k + 2 ; 3 ; ⋮ ; k .

Let e_{s} = κ_{s}{s} ∈ S^{(k,2,1^{k−2})} be the corresponding polytabloid. We claim that the coefficient of {s} in xe_{s} is (−1)^{k−1}2(k − 1)!, which shows that xS^{(k,2,1^{k−2})} ≠ 0. Let σ ∈ C = C_{t}, where C is the column stabilizer defined above. Note that (1 + (i, j))κ_{s} = 0 if i, j ∈ {1, 2, …, k} or {i, j} = {k + 1, k + 2}, as in this case we have (i, j) ∈ C_{s}. So if any such (i, j) belongs to σΣ_{k}σ^{−1} we have ∑_{g∈Σ_{k}} σgσ^{−1}κ_{s} = 0. So we just need to consider σ = ∏_{i=2}^{k}(i, k + i) and σ = (1, k + 1) ∏_{i=3}^{k}(i, k + i). In these cases we have sign σ = (−1)^{k−1}. Moreover, if c ∈ C_{s}, g ∈ Σ_{k} and σ is as above, then σgσ^{−1}c{s} = {s} only if c = 1. Hence the coefficient of {s} in xe_{s} is equal to the coefficient of {s} in

(−1)^{k−1} ∑_{g∈Σ_{{1,k+2,k+3,...,2k}}} g{s} + (−1)^{k−1} ∑_{g∈Σ_{{k+1,2,k+3,...,2k}}} g{s},

which is (−1)^{k−1}2(k − 1)!.

Next, we prove that xS^{(k,1^{k})} ≠ 0. Let

s = 1 k + 2 k + 3 … 2k ; 2 ; ⋮ ; k + 1 .

As above, it suffices to prove that the coefficient of {s} in xe_{s} is non-zero. Let σ ∈ C. Note that (1 + (i, j))κ_{s} = 0 if i, j ∈ {1, 2, …, k + 1}, as in this case we have (i, j) ∈ C_{s}. So if any such (i, j) belongs to σΣ_{k}σ^{−1} we have ∑_{g∈Σ_{k}} σgσ^{−1}κ_{s} = 0. So we just need to consider σ = ∏_{i=1}^{k}(i, k + i) and σ = ∏_{i=2}^{k}(i, k + i). In these cases, if c ∈ C_{s} and g ∈ Σ_{k}, then σgσ^{−1}c{s} = {s} only if c = 1. Hence the coefficient of {s} in xe_{s} is equal to the coefficient of {s} in

(−1)^{k} ∑_{g∈Σ_{{k+1,k+2,...,2k}}} g{s} + (−1)^{k−1} ∑_{g∈Σ_{{1,k+2,k+3,...,2k}}} g{s},

which is (−1)^{k}k! + (−1)^{k−1}(k − 1)! = (−1)^{k}(k − 1)!(k − 1).

Finally, we prove that xS^{(k+1,1^{k−1})} ≠ 0. Let

s = 1 k + 1 k + 2 … 2k ; 2 ; ⋮ ; k .

As above, it suffices to prove that the coefficient of {s} in xe_{s} is non-zero. Let σ ∈ C. Note that (1 + (i, j))κ_{s} = 0 if i, j ∈ {1, 2, …, k}, as in this case we have (i, j) ∈ C_{s}. Hence if any such (i, j) belongs to σΣ_{k}σ^{−1} we have ∑_{g∈Σ_{k}} σgσ^{−1}κ_{s} = 0. So we just need to consider σ = ∏_{i=1}^{k}(i, k + i) and σ = ∏_{i∈{1,...,j−1,j+1,...,k}}(i, k + i) for j = 1, 2, …, k. In these cases, if c ∈ C_{s} and g ∈ Σ_{k}, then σgσ^{−1}c{s} = {s} only if c = 1. Hence the coefficient of {s} in xe_{s} is equal to the coefficient of {s} in

(−1)^{k} ∑_{g∈Σ_{{k+1,k+2,...,2k}}} g{s} + ∑_{j=1}^{k}(−1)^{k−1} ∑_{g∈Σ_{{k+1,...,k+j−1,j,k+j+1,...,2k}}} g{s},

which is (−1)^{k}k! + (−1)^{k−1}k! + (k − 1)(−1)^{k−1}(k − 1)! = (−1)^{k−1}(k − 1)!(k − 1).

Proposition 3.8. Let p > k > 1, n ≥ 2k, and D^{λ} be an irreducible F Σ_{n}-module. If the equation (8) holds then h(λ) < k or h(λ^{M}) < k.

Proof. Set x := ∑_{g∈Σ_{k}, σ∈C}(sign σ)σgσ^{−1} ∈ F Σ_{2k} < F Σ_{n}. Apply induction on n. For n = 2k see Lemma 3.7. Let n > 2k. If xD^{λ} = 0 then x annihilates every composition factor of D^{λ}↓_{Σ_{n−1}}. Now use the inductive hypothesis and Theorem 2.8.

Corollary 3.9. Let p > k > 1, n ≥ 2k, and D^{λ} be an irreducible F Σ_{n}-module with h(λ) ≥ k, h(λ^{M}) ≥ k. Then

dim Hom_{F Σ_{n}}(M^{(n−k,k)}, End_{F} (D^{λ})) > dim Hom_{F Σ_{n}}(M^{(n−k+1,k−1)}, End_{F} (D^{λ})).

Proof. This follows from Propositions 3.6 and 3.8.

Take k = 3 in Corollary 3.9. We see that f^{∗} is a proper injection unless the height of λ or λ^{M} is ≤ 2. To get a more complete result for k = 3, we need to treat the 2-row partitions separately.

4. Two-row partitions

Throughout this section we fix a 2-row partition λ = (m, k) of n. Let α ≡ k − 2 (mod p) and β ≡ m − 1 (mod p). So α (resp. β if m > k) is the residue of the bottom (resp. the top) removable node of λ. Set

d_{j} := dim End_{Σ_{n−j}×Σ_{j}}(D^{λ}↓_{Σ_{n−j}×Σ_{j}}).

All composition factors of D^{λ}↓_{Σ_{n−1}} are known: for α ≠ β, one can apply Theorem 1.12, and for α = β see Sheth [42] (the result also follows from [15], Theorems 2.2,2.3, and [8], Theorem D(iii)). We will need the following special case: Lemma 4.1. [42] Let p > 2.

(i) If α ≠ β then D^{λ}↓_{Σ_{n−1}} ≅ (1 − δ_{mk})D^{(m−1,k)} ⊕ (1 − δ_{α,β+1})D^{(m,k−1)}. (ii) If α = β, then D^{λ}↓_{Σ_{n−1}} = D^{(m−1,k)} + 2D^{(m,k−1)} + (∗), where (∗) stands for terms of

the form D^{(m−1+p^{i},k−p^{i})} with i > 0. Corollary 4.2. Assume that p > 3 and k ≥ 2.

(i) If m − k ≥ 2 and α ≠ β + 1, β + 2 then

D^{λ}↓_{Σ_{n−2}} = D^{(m−2,k)} + (2 + δ_{α,β−1})D^{(m−1,k−1)} + (1 + δ_{αβ})D^{(m,k−2)} + (∗),

where (∗) stands for the terms of the form D^{(i,j)} with (i, j) ⊳ (m, k − 2).

(ii) If m = k + 1 then D^{λ}↓_{Σ_{n−2}} ≅ 2D^{(m−1,k−1)} ⊕ D^{(m,k−2)}. (iii) If m = k then D^{λ}↓_{Σ_{n−2}} ≅ D^{(m−1,k−1)} ⊕ D^{(m,k−2)}. (iv) If α = β + 1 then D^{λ}↓_{Σ_{n−2}} ≅ D^{(m−2,k)} ⊕ D^{(m−1,k−1)}.

(v) If α = β + 2 then D^{λ}↓_{Σ_{n−2}} ≅ D^{(m−2,k)} ⊕ 2D^{(m−1,k−1)}. Corollary 4.3. Assume that p > 3, m ≥ 4, and k ≥ 2.

(i) If m − k ≥ 2 and α ≠ β + 1, β + 2 then

D^{λ}↓_{Σ_{n−3}} = (1 − δ_{m,k+2})D^{(m−3,k)} + (3 + δ_{α,β−2})D^{(m−2,k−1)}
+(3 + δ_{α,β−1} + 3δ_{α,β})D^{(m−1,k−2)}
+(1 − δ_{k,2})(1 + δ_{α,β} − δ_{α,β+3})D^{(m,k−3)} + (∗),

where (∗) stands for the terms of the form D^{(i,j)} with (i, j) ⊳ (m, k − 3).

(ii) If m = k + 1 then D^{λ}↓_{Σ_{n−3}} ≅ 2D^{(m−2,k−1)} ⊕ 3D^{(m−1,k−2)} ⊕ (1 − δ_{p,5})D^{(m,k−3)}. (iii) If m = k then D^{λ}↓_{Σ_{n−3}} ≅ 2D^{(m−1,k−2)} ⊕ D^{(m,k−3)}. (iv) If α = β + 1 then D^{λ}↓_{Σ_{n−3}} ≅ D^{(m−3,k)} ⊕ 2D^{(m−2,k−1)}.

(v) If α = β + 2 then D^{λ}↓_{Σ_{n−3}} ≅ (1 − δ_{m−k,2})D^{(m−3,k)} ⊕ 3D^{(m−2,k−1)} ⊕ 2D^{(m−1,k−2)}.

Next we prove some results on restrictions D^{λ}↓_{Σ_{n−2}×Σ_{2}} and D^{λ}↓_{Σ_{n−3}×Σ_{3}}. We consider Specht modules first. Lemma 4.4. Let k ≥ 2. The restriction S^{λ}↓_{Σ_{n−j}×Σ_{j}} has a Specht filtration with factors:

(i) (1 − δ_{m,k+1} − δ_{m,k})S^{(m−2,k)} ⊗ 1_{Σ_{2}}, (1 − δ_{m,k})S^{(m−1,k−1)} ⊗ 1_{Σ_{2}}, S^{(m−1,k−1)} ⊗ sgn_{Σ_{2}}, and S^{(m,k−2)} ⊗ 1_{Σ_{2}}, if j = 2;

(ii) (1 − δ_{m,k+2} − δ_{m,k+1} − δ_{mk}) S^{(m−3,k)} ⊗ 1_{Σ_{3}}, (1 − δ_{m,k+1} − δ_{mk}) S^{(m−2,k−1)} ⊗ 1_{Σ_{3}}, (1 − δ_{mk}) S^{(m−2,k−1)} ⊗ S^{(2,1)}, (1 − δ_{mk}) S^{(m−1,k−2)} ⊗ 1_{Σ_{3}}, S^{(m−1,k−2)} ⊗ S^{(2,1)}, and (1 − δ_{k,2})S^{(m,k−3)} ⊗ 1_{Σ_{3}}, if j = 3. Proof. Follows immediately from Theorem 1.8 and the Littlewood-Richardson Rule. Lemma 4.5. Let p > 3 and k ≥ 2.

(i) If m = k + 1 then D^{λ}↓_{Σ_{n−2}×Σ_{2}} ≅ D^{(m−1,k−1)} ⊗ 1_{Σ_{2}} ⊕ D^{(m−1,k−1)} ⊗ sgn_{Σ_{2}} ⊕ D^{(m,k−2)} ⊗ 1_{Σ_{2}}.

(ii) If m = k then

D^{λ}↓_{Σ_{n−2}×Σ_{2}} ≅ D^{(m−1,k−1)} ⊗ sgn_{Σ_{2}} ⊕ D^{(m,k−2)} ⊗ 1_{Σ_{2}}.

(iii) If α = β + 1 then D^{λ}↓_{Σ_{n−2}×Σ_{2}} ≅ D^{(m−2,k)} ⊗ 1_{Σ_{2}} ⊕ D^{(m−1,k−1)} ⊗ sgn_{Σ_{2}}.

(iv) If α = β + 2 then

D^{λ}↓_{Σ_{n−2}×Σ_{2}} ≅ D^{(m−2,k)} ⊗ 1_{Σ_{2}} ⊕ D^{(m−1,k−1)} ⊗ 1_{Σ_{2}} ⊕ D^{(m−1,k−1)} ⊗ sgn_{Σ_{2}}.

Proof. (i) Let m = k+1. By Lemma 4.4, S^{λ}↓_{Σ_{n−2}×Σ_{2}} has a filtration with factors S^{(m−1,k−1)}⊗ 1_{Σ_{2}}, S^{(m−1,k−1)} ⊗ sgn_{Σ_{2}}, S^{(m,k−2)} ⊗ 1_{Σ_{2}}. Moreover, by Lemma 1.10, we have S^{(m−1,k−1)} = D^{(m−1,k−1)} + (∗), S^{(m,k−2)} = D^{(m,k−2)} + (∗), where (∗) stands for irreducible modules D^{μ} with μ ⊳ (m, k − 2). So

S^{λ}↓_{Σ_{n−2}×Σ_{2}} = D^{(m−1,k−1)} ⊗ 1_{Σ_{2}} + D^{(m−1,k−1)} ⊗ sgn_{Σ_{2}} + D^{(m,k−2)} ⊗ 1_{Σ_{2}} + (∗),

where (∗) stands for terms of the form D^{μ} ⊗ D^{ν} with μ ⊳ (m, k − 2). As D^{λ}↓_{Σ_{n−2}×Σ_{2}} is a quotient of S^{λ}↓_{Σ_{n−2}×Σ_{2}}, it now follows from Corollary 4.2(ii) that

D^{λ}↓_{Σ_{n−2}×Σ_{2}} = D^{(m−1,k−1)} ⊗ 1_{Σ_{2}} + D^{(m−1,k−1)} ⊗ sgn_{Σ_{2}} + D^{(m,k−2)} ⊗ 1_{Σ_{2}}.

The complete reducibility follows from the fact that the restriction D^{λ}↓_{Σ_{n−2}} is completely reducible (by Corollary 4.2(ii)) and Lemma 1.6.

(iii) If α = β + 1, it follows from the assumption p > 3 that m − k ≥ 3. So, by Lemma 4.4, S^{λ}↓_{Σ_{n−2}×Σ_{2}} has a filtration with factors S^{(m−2,k)} ⊗ 1_{Σ_{2}}, S^{(m−1,k−1)} ⊗ 1_{Σ_{2}}, S^{(m−1,k−1)} ⊗ sgn_{Σ_{2}}, S^{(m,k−2)} ⊗ 1_{Σ_{2}}. By Lemma 1.10, we have S^{(m−2,k)} = D^{(m−2,k)} + (∗), S^{(m−1,k−1)} = D^{(m−1,k−1)} + D^{(m,k−2)} + (∗), S^{(m,k−2)} = D^{(m,k−2)} + (∗), where (∗) stands for irreducible modules D^{μ} with μ ⊳ (m, k − 2). So

S^{λ}↓_{Σ_{n−2}×Σ_{2}} = D^{(m−2,k)} ⊗ 1_{Σ_{2}} + D^{(m−1,k−1)} ⊗ 1_{Σ_{2}} + D^{(m−1,k−1)} ⊗ sgn_{Σ_{2}}
+2D^{(m,k−2)} ⊗ 1_{Σ_{2}} + D^{(m,k−2)} ⊗ sgn_{Σ_{2}} + (∗),

where (∗) stands for terms of the form D^{μ} ⊗ D^{ν} with μ ⊳ (m, k − 2). As D^{λ}↓_{Σ_{n−2}×Σ_{2}} is a quotient of S^{λ}↓_{Σ_{n−2}×Σ_{2}}, it now follows from Corollary 4.2(iv) and Lemma 1.5 that

D^{λ}↓_{Σ_{n−2}×Σ_{2}} = D^{(m−2,k)} ⊗ 1_{Σ_{2}} + D^{(m−1,k−1)} ⊗ sgn_{Σ_{2}}.

The complete reducibility follows from the fact that the restriction D^{λ}↓_{Σ_{n−2}} is completely reducible and Lemma 1.6.

The remaining parts of the lemma are proved similarly.

Lemma 4.6. Let p > 3, k ≥ 2. Then d_{2} ≤ 4. Moreover, (i) d_{2} = 2 if α = β + 1 or m = k; (ii) d_{2} = 3 if α = β + 2 or m = k + 1; (iii) d_{2} = 4 if m = 4, k = 2, and p > 5.

Proof. By Lemma 4.4, the restriction S^{λ}↓_{Σ_{n−2}×Σ_{2}} has a filtration with factors

S^{(m−2,k)} ⊗ 1_{Σ_{2}}, S^{(m−1,k−1)} ⊗ 1_{Σ_{2}}, S^{(m−1,k−1)} ⊗ sgn_{Σ_{2}}, S^{(m,k−2)} ⊗ 1_{Σ_{2}},

each appearing at most once. Now, since dim Hom_{F Σ_{m}}(S^{λ}, (S^{μ})^{∗}) = δ_{λμ} for any p-regular partitions λ and μ of m by Lemma 1.4, we conclude that

dim Hom_{F [Σ_{n−2}×Σ_{2}]}(S^{λ}↓_{Σ_{n−2}×Σ_{2}}, (S^{λ}↓_{Σ_{n−2}×Σ_{2}})^{∗}) ≤ 4.

But D^{λ}↓_{Σ_{n−2}×Σ_{2}} is a quotient of S^{λ}↓_{Σ_{n−2}×Σ_{2}} and a submodule of (S^{λ}↓_{Σ_{n−2}×Σ_{2}})^{∗}, so the first claim follows. Finally, (i) and (ii) follow from Lemma 4.5, and (iii) follows from Lemma 4.4(i) as in this case the group algebra F Σ_{6} is semisimple.

Lemma 4.7. Let p > 3, m ≥ 4, and k ≥ 2.

(i) If m − k ≥ 2 and α ≠ β + 1, β + 2 then

D^{λ}↓_{Σ_{n−3}×Σ_{3}} = (1 − δ_{m,k+2})D^{(m−3,k)} ⊗ 1_{Σ_{3}}
+ (1 + δ_{α,β−2})D^{(m−2,k−1)} ⊗ 1_{Σ_{3}} + D^{(m−2,k−1)} ⊗ D^{(2,1)}
+ (1 + δ_{α,β−1} + δ_{αβ})D^{(m−1,k−2)} ⊗ 1_{Σ_{3}}
+ (1 + δ_{αβ})D^{(m−1,k−2)} ⊗ D^{(2,1)}
+ (1 − δ_{k,2})(1 + δ_{αβ} − δ_{α,β+3})D^{(m,k−3)} ⊗ 1_{Σ_{3}} + (∗),

where (∗) stands for terms of the form D^{μ} ⊗ D^{ν} with μ ⊳ (m, k − 3).

(ii) If m = k + 1 then

D^{λ}↓_{Σ_{n−3}×Σ_{3}} ≅ D^{(m−2,k−1)} ⊗ D^{(2,1)} ⊕ D^{(m−1,k−2)} ⊗ 1_{Σ_{3}}
⊕ D^{(m−1,k−2)} ⊗ D^{(2,1)} ⊕ (1 − δ_{p,5})D^{(m,k−3)} ⊗ 1_{Σ_{3}}.

(iii) If m = k then D^{λ}↓_{Σ_{n−3}×Σ_{3}} ≅ D^{(m−1,k−2)} ⊗ D^{(2,1)} ⊕ D^{(m,k−3)} ⊗ 1_{Σ_{3}}.

(iv) If α = β + 1 then D^{λ}↓_{Σ_{n−3}×Σ_{3}} ≅ D^{(m−3,k)} ⊗ 1_{Σ_{3}} ⊕ D^{(m−2,k−1)} ⊗ D^{(2,1)}.

(v) If α = β + 2 then

D^{λ}↓_{Σ_{n−3}×Σ_{3}} ≅ (1 − δ_{m,k+2})D^{(m−3,k)} ⊗ 1_{Σ_{3}} ⊕ D^{(m−2,k−1)} ⊗ 1_{Σ_{3}}
⊕ D^{(m−2,k−1)} ⊗ D^{(2,1)} ⊕ D^{(m−1,k−2)} ⊗ D^{(2,1)}.

Proof. (i) By Lemma 4.4, S^{λ}↓_{Σ_{n−3}×Σ_{3}} has a filtration with factors

(1 − δ_{m,k+2})S^{(m−3,k)} ⊗ 1_{Σ_{3}}, S^{(m−2,k−1)} ⊗ 1_{Σ_{3}}, S^{(m−2,k−1)} ⊗ D^{(2,1)},
S^{(m−1,k−2)} ⊗ 1_{Σ_{3}}, S^{(m−1,k−2)} ⊗ D^{(2,1)}, (1 − δ_{k,2})S^{(m,k−3)} ⊗ 1_{Σ_{3}}.

Moreover, by Lemma 1.10,

S^{(m−3,k)} = D^{(m−3,k)} + δ_{α,β−2}D^{(m−2,k−1)} + δ_{α,β−1}D^{(m−1,k−2)}
+ (1 − δ_{k,2})δ_{αβ}D^{(m,k−3)} + (∗),
S^{(m−2,k−1)} = D^{(m−2,k−1)} + δ_{αβ}D^{(m−1,k−2)} + (∗),
S^{(m−1,k−2)} = D^{(m−1,k−2)} + (∗),
S^{(m,k−3)} = D^{(m,k−3)} + (∗),

where (∗) stands for irreducible modules D^{μ} with μ ⊳ (m, k − 3). So

S^{λ}↓_{Σ_{n−3}×Σ_{3}} = (1 − δ_{m,k+2})D^{(m−3,k)} ⊗ 1_{Σ_{3}} + (1 + δ_{α,β−2})D^{(m−2,k−1)} ⊗ 1_{Σ_{3}}
+D^{(m−2,k−1)} ⊗ D^{(2,1)} + (1 + δ_{α,β−1} + δ_{α,β})D^{(m−1,k−2)} ⊗ 1_{Σ_{3}}
+(1 + δ_{α,β})D^{(m−1,k−2)} ⊗ D^{(2,1)}
+(1 − δ_{k,2})(1 + δ_{α,β})D^{(m,k−3)} ⊗ 1_{Σ_{3}} + (∗),

where (∗) stands for terms of the form D^{μ} ⊗ D^{ν} with μ ⊳ (m, k − 3). Now (i) follows from Corollary 4.3(i) as D^{λ}↓_{Σ_{n−3}×Σ_{3}} is a quotient of S^{λ}↓_{Σ_{n−3}×Σ_{3}}.

(iv) If α = β + 1 then m − k ≥ 3, and, by Corollary 4.3(iv), D^{λ}↓_{Σ_{n−3}} ≅ D^{(m−3,k)} ⊕ 2D^{(m−2,k−1)}. Now the result follows from Lemmas 4.4, 1.10, and 1.6 similarly to the corresponding case in Lemma 4.5.

The remaining parts of the lemma are proved similarly.

Lemma 4.8. Let p > 3, m ≥ 4, and k ≥ 2. Then d_{3} > 4, except in the following situations:

(1) m = k + 1, in which case d_{3} = 4 − δ_{p,5}; (2) m = k, in which case d_{3} = 2; (3) α = β + 1, in which case d_{3} = 2; (4) α = β + 2, in which case d_{3} = 4 − δ_{m,k+2}; and (5) p > 5, m = 4, k = 2, when d_{4} = 4.

Proof. In the exceptional cases (1)-(4) the result follows from Lemma 4.7(ii)-(v). For (5), F Σ_{6} is semisimple, and so we may use Lemma 4.4. So assume that α ≠ β + 1, β + 2, m − k ≥ 2 and n > 6. As the group algebra F Σ_{3} is semisimple, the modules 1_{Σ_{3}} and D^{(2,1)} are in different blocks. Moreover, assume α ≠ β, β − 1, β − 2. Then by the Nakayama Conjecture, all modules in

Δ := {D^{(m−3,k)}, D^{(m−2,k−1)}, D^{(m−1,k−2)}, D^{(m,k−3)}}

are in different blocks (if m − k = 2, disregard the first module, and if k = 2, disregard the last one). Now, by Lemma 4.7(i), the restriction D^{λ}↓_{Σ_{n−3}×Σ_{3}} has composition factors in at least 5 different blocks. Hence it has at least 5 indecomposable components, whence d_{3} ≥ 5.

Let α = β. Then m − k ≥ 4 as p ≥ 5. Moreover, D^{(m−3,k)} ∼ D^{(m,k−3)} ≁ D^{(m−2,k−1)} ∼ D^{(m−1,k−2)}, with D^{(m,k−3)} omitted if k = 2. By Lemma 4.7(i), the restriction D^{λ}↓_{Σ_{n−3}×Σ_{3}} has at least 3 blocks components, at least 2 of which are reducible. Now, by Lemmas 1.2 and 1.3, we have d_{3} ≥ 5.

If α = β − 1, we have m − k ≥ 5. If k > 2 then the modules of Δ belong to 3 different blocks, with D^{(m−3,k)} ∼ D^{(m−1,k−2)}. Then, by Lemma 4.7(i), D^{λ}↓_{Σ_{n−3}×Σ_{3}} has at least 5 blocks components, whence d_{3} ≥ 5. If k = 2, we get at least 4 block components, but at least one of them is reducible, so we may apply Lemmas 1.2 and 1.3.

Finally, let α = β −2. As p ≥ 5 and m−k > 1, we have m−k ≥ 6. If k > 2, the modules of Δ belong to 3 different blocks, with D^{(m−3,k)} ∼ D^{(m−2,k−1)}. By Lemma 4.7(i), D^{λ}↓_{Σ_{n−3}×Σ_{3}} has at least 5 blocks components, unless k = 2 or p = 5 (in which case β − 2 = β + 3). If k = 2 or p = 5, we get at least 4 block components, but at least one of them is reducible, so we may apply Lemmas 1.2 and 1.3 again.

Corollary 4.9. Let p > 3, m ≥ 4, and k ≥ 2. Then d_{3} > d_{2}, except in the following cases:

(1) α = β + 1 or m = k, in which case d_{2} = d_{3} = 2; (2) p = 5 and m = k + 1 or k + 2, in which case d_{2} = d_{3} = 3; (3) p > 5, m = 4, k = 2 in which case d_{2} = d_{3} = 4.

Proof. Follows from Lemmas 4.8 and 4.6.

For the exceptional case (1) above we will need to consider d_{4}.

Lemma 4.10. Let p > 3, k ≥ 2, and α = β + 1. Then d_{4} = 3 unless p = 5 and m = k + 3.

Proof. Note that α = β + 1 implies m − k ≥ p − 2 ≥ 3. It follows from Corollary 4.3 and Lemma 4.1 that

D^{λ}↓_{Σ_{n−4}} ≅ (1 − δ_{m,k+3})D^{(m−4,k)} ⊕ 3D^{(m−3,k−1)} ⊕ 2D^{(m−2,k−2)}.

By Theorem 1.8 and Lemma 1.10,

S^{λ}↓_{Σ_{n−4}×Σ_{4}} = (1 − δ_{m,k+3})D^{(m−4,k)} ⊗ 1_{Σ_{4}} + D^{(m−3,k−1)} ⊗ 1_{Σ_{4}} + D^{(m−3,k−1)} ⊗ D^{(3,1)}

+D^{(m−2,k−2)} ⊗ 1_{Σ_{4}} + D^{(m−2,k−2)} ⊗ D^{(3,1)} + D^{(m−2,k−2)} ⊗ D^{(2,2)} + (∗),

where (∗) stands for the composition factors of the form D^{λ} ⊗ D^{μ} with λ ⊳ (m − 2, k − 2). As dim D^{(3,1)} = 3 and dim D^{(2,2)} = 2, it follows from above and Lemma 1.6 that D^{λ}↓_{Σ_{n−4}×Σ_{4}} is semisimple with at least 3 composition factors, unless m − k = 3, which is only possible if p = 5. The result follows.

Lemma 4.11. Let p > 3 and m = k > 3. Then d_{4} = 3 unless p = 5.

Proof. The argument as in the proof of Lemma 4.10 shows that

D^{λ}↓_{Σ_{n−4}×Σ_{4}} ≅ D^{(m−2,k−2)} ⊗ D^{(2,2)} ⊕ D^{(m−1,k−3)} ⊗ D^{(3,1)} ⊕ (1 − δ_{p,5})D^{(m,k−4)} ⊗ 1_{Σ_{4}},

which implies the desired result.

In the following corollary we gather the most important information obtained in this section.

Corollary 4.12. Let n ≥ 7, and λ = (m, k) be a two row partition with k ≥ 2. (i) Let p > 5. If m ≢ k − 2 (mod p) and m ≠ k then

dim Hom_{F Σ_{n}}(M^{(n−3,3)}, End_{F} (D^{λ})) > dim Hom_{F Σ_{n}}(M^{(n−2,2)}, End_{F} (D^{λ})).

(ii) Let p > 5. If m ≡ k − 2 (mod p) or m = k then

dim Hom_{F Σ_{n}}(M^{(n−4,4)}, End_{F} (D^{λ})) > dim Hom_{F Σ_{n}}(M^{(n−3,3)}, End_{F} (D^{λ})).

(iii) Let p = 5. If m ≢ k − 2 (mod p) and m − k > 3 then

dim Hom_{F Σ_{n}}(M^{(n−3,3)}, End_{F} (D^{λ})) > dim Hom_{F Σ_{n}}(M^{(n−2,2)}, End_{F} (D^{λ})).

(iv) Let p = 5. If m ≡ k − 2 (mod p) and m − k > 3 then

dim Hom_{F Σ_{n}}(M^{(n−4,4)}, End_{F} (D^{λ})) > dim Hom_{F Σ_{n}}(M^{(n−3,3)}, End_{F} (D^{λ})).

Proof. This follows from (3), Corollary 4.9, and Lemmas 4.10, 4.11.

5. Proof of the Main Theorem

Let G < Σ_{n} be a proper subgroup, and D^{λ} be an irreducible F Σ_{n}-module which is irreducible over G. In view of Theorem 0.7, the main thing is to consider the 2-transitive groups. Furthermore, the case where G is 2-transitive and D^{λ} ∈ R_{n}(1) (see (2)) has been studied by Mortimer, see Example 0.5. To deal with 2-transitive groups on all other modules we will use Proposition 5.1 below and some ad hoc methods.

We denote by r_{k} the number of G-orbits on the (unordered) k-element subsets of {1, …, n}. If G is 2-transitive, it is 2-homogeneous, and so we will always have r_{2} = 1.

Proposition 5.1. Let n ≥ 8, G < Σ_{n} be a 2-homogeneous subgroup, and D^{λ} be an irreducible F Σ_{n}-module with D^{λ} ∉ R_{n}(1).

(i) Assume that p > 5 and G is not 3-homogeneous. Then D^{λ}↓_{G} is reducible, except possibly in the cases where D^{λ} ≅ D^{(m,k)} or D^{(m,k)} ⊗ sgn and m ≡ k − 2 (mod p) or m = k. If, additionally, r_{4} > r_{3} then D^{λ}↓_{G} is reducible in the exceptional cases too.

(ii) Assume that p = 5 and G is not 3-homogeneous. Then D^{λ}↓_{G} is reducible, except possibly in the cases where D^{λ} ≅ D^{(m,k)} or D^{(m,k)}⊗sgn and m ≡ k−2 (mod p) or m−k ≤ 3. If, additionally, r_{4} > r_{3} then D^{λ}↓_{G} is reducible in the exceptional cases too, unless m−k ≤ 3.

(iii) Assume that p > 3 and G is 3-homogeneous but not 4-homogeneous. Then D^{λ}↓_{G} is reducible, except possibly in the cases where h(λ) ≤ 3 or h(λ^{M}) ≤ 3.

Proof. (i), (ii). If neither D^{λ} nor D^{λ} ⊗sgn is isomorphic to D^{(m,k)} for some two row partition (m, k), then the result follows from Lemma 1.1, Corollary 3.9, and Proposition 3.4. Otherwise we use Corollary 4.12 and Lemma 1.1, Proposition 3.4 again.

(iii) This follows from Lemma 1.1, Corollary 3.9, and Proposition 3.4.

Now let p > 3, as in the Main Theorem. If n < 5 then F Σ_{n} is semisimple, and the result follows e.g. from [39]. So from now on we assume that n ≥ 5. First, we suppose that D^{λ} ∉ R_{n}(1) unless otherwise stated, and go through the list of 2-transitive groups from [25]. After that we will complete the proof by considering the case G < Σ_{n−1}.

Alternating groups. This case has been considered in Example 0.3.

Groups with a regular normal subgroup. A group G in this class is always a subgroup of the group AΓL(m, q) of all non-degenerate semilinear affine transformations of the affine space V = 𝔽_{q}^{m} acting naturally on the q^{m} points of V. We have n = q^{m}, and | AΓL(m, q)| = fq^{m(m+1)/2}(q^{m} − 1)(q^{m−1} − 1) … (q − 1), if q = ℓ^{f} for a prime ℓ.

Assume first that m = 1. Then q = n ≥ 5. We have |G| ≤ | AΓL(1, q)| = fq(q − 1), where q = ℓ^{f} for a prime ℓ. By (1), if D^{λ} is irreducible over G then dim D^{λ} ≤ \sqrt{fq(q − 1)}. If p > 5, this contradicts Lemma 1.18(i) when q ≥ 9 and Lemma 1.22(i)–(iv) otherwise. If p = 5 we can use Lemma 1.18(i) when q ≥ 11, and Lemma 1.22(i)–(v) otherwise. So from now on we assume that m ≥ 2.

If p = 5, q > 2, and D^{λ} is D^{(r,s)} or D^{(r,s)} ⊗ sgn for r − s ≤ 3, then it follows from Theorem 1.16 that dim D^{λ} = f_{n} or f_{n−1}. By (1), we must have

f_{n−1} ≤ \sqrt{|G|} ≤ (fq^{m(m+1)/2}(q^{m} − 1)(q^{m−1} − 1) … (q − 1))^{1/2} .

But an elementary argument using Lemma 1.17 shows that this is impossible. So from now on we exclude the case where p = 5, q > 2, and D^{λ} is D^{(r,s)} or D^{(r,s)} ⊗ sgn with r − s ≤ 3.

Assume now that G = AΓL(m, q) and q > 2. In this case G is not 3-homogeneous. Indeed, it can not move 3 points lying on an affine line to 3 points in a general position. Bearing in mind Proposition 5.1, we want to prove that r_{4} > r_{3}. It is well known that G is transitive on the triples of points in general position. So r_{3} = 1 + s, where s is the number of orbits of G on the triples {a, b, c} such that a, b, c lie on an affine line. Similarly, G is transitive on 4-tuples of points in general position. Note that 4 points in general position exist only if m ≥ 3. Set t = 1 if m ≥ 3 and t = 0 if m = 2. Also denote by u (resp. v, resp. w) the number of G-orbits on 4-tuples {a, b, c, d} such that a, b, c, d lie in the same affine plane, but no 3 of them lie on a line (resp., exactly 3 of the points a, b, c, d lie on a line, resp. all four of the points lie on a line). Then we can write r_{4}=t+u+v+w. Note that w > 0 if and only if q > 3. Moreover, assume that {a_{1}, b_{1}, c_{1}, d_{1}} and {a_{2}, b_{2}, c_{2}, d_{2}} are in the same G-orbit, and that exactly three points in each of these 4-tuples belong to a line. Assume without loss of generality that a_{i}, b_{i}, and c_{i} lie on the same line (i = 1, 2). Then {a_{1}, b_{1}, c_{1}} and {a_{2}, b_{2}, c_{2}} must be in the same G-orbit (on triples). This implies that v ≥ s. As u > 0, we conclude that r_{4} > r_{3} unless G = AΓL(2, 3). In all the other cases we may use Proposition 5.1.

Now let G = AΓL(2, 3). Then |G| = 432, and so every irreducible F G-module has dimension ≤ 20, thanks to (1). By Lemma 1.22(v), we may assume that p = 7 and dim D^{λ} = 19. But G = V ⋊ GL(2, 3). As D^{λ} is faithful and (p, q) = 1, it follows from [41], section 8.2, Proposition 25 that D^{λ}↓_{G} is induced from a certain proper subgroup of G containing V. As 19 is prime, this implies that G has a subgroup of index 19, which is absurd.

Thus, if m = 1, or if m ≥ 2 and q > 2, then the restriction of D^{λ} to AΓL(m, q), and hence to any of its subgroups, is reducible. It just remains to treat the case G ≤ AΓL(m, 2). This is rather harder than the others so we consider it as a separate case as follows.

Groups G with G ≤ AΓL(m, 2). Note m ≥ 3, as we have assumed n ≥ 5. First, let G = AΓL(m, 2) = AGL(m, 2). Then G is 3-homogeneous but not 4-homogeneous. By Proposition 5.1(iii), we may assume that h(λ) ≤ 3. We will prove that D^{λ}↓_{G} is irreducible if and only if D^{λ} or D^{λ} ⊗ sgn is 1, D^{(n−1,1)}, D^{(n−2,1^{2})}, or D^{(5,3)} (p = 5).

Set M_{l} = {D^{(l)}, D^{(l−1,1)}, D^{(l−2,1,1)}}. We prove the following intermediate fact on branching.

Lemma 5.2. Let l ≥ 6, and D^{γ} be an irreducible F Σ_{l}-module. If every composition factor of the restriction D^{γ}↓_{Σ_{l−1}} belongs to M_{l−1}, then D^{γ} ∈ M_{l}.

Proof. At least one of the modules D^{(l−1)}, D^{(l−2,1)}, D^{(l−3,1,1)} must be in the socle of D^{γ}↓_{Σ_{l−1}}. If this is D^{(l−1)}, then γ = (l) or (l − 1, 1) by Theorem 1.12(i). If this is D^{(l−2,1)} then by Theorem 1.12(i), γ = (l−1, 1), (l−2, 1, 1) or (l−2, 2). But by Theorem 1.12(ii), D^{(l−2,2)}↓_{Σ_{l−1}} contains D^{(l−3,2)} as a composition factor, which leads to a contradiction. Finally, if D^{(l−3,1,1)} appears in the socle of D^{γ}↓_{Σ_{l−1}} then again by Theorem 1.12(i), γ = (l − 2, 1, 1), (l − 3, 2, 1) or (l −3, 1, 1, 1). But D^{(l−3,2,1)}↓_{Σ_{l−1}} contains D^{(l−4,2,1)} and D^{(l−3,1,1,1)}↓_{Σ_{l−1}} contains D^{(l−4,1,1,1)}, in view of Theorem 1.12(ii). This leads to a contradiction again.

We now develop a technical result on restrictions from Σ_{2^{m}} to Σ_{2^{m−1}} × Σ_{2^{m−1}}, which will turn out to be precisely what is needed in the later argument.

Lemma 5.3. Let m ≥ 3, p > 3, n = 2^{m} and λ be a p-regular partition of n with h(λ) ≤ 3. Set k = n/2. Assume that every composition factor of D^{λ} ↓_{Σ_{k}×Σ_{k}} has one of the following forms:

(1) D^{μ} ⊗ D^{μ} with μ ∈ {(k), (k − 1, 1), (2, 1, 1)}.

(2) D^{μ} ⊗D^{ν} or D^{ν} ⊗D^{μ} for μ ∈ {(k −1, 1), (k −2, 1^{2}), (3, 3, 2), (5, 3)} and ν ∈ {(k), (2, 2)}. Then, λ ∈ {(n), (n − 1, 1), (n − 2, 1^{2}), (3, 3, 2), (5, 3)}.

Proof. If m > 4, it follows from the assumption that any composition factor of D^{λ}↓_{Σ_{k}} belongs to M_{k}. So D^{λ} ∈ M_{n} by Lemma 5.2.

Now, let m = 4. If D^{λ} is as in the assumption then all composition factors of the restriction D^{λ}↓_{Σ_{8}} belong to M_{8} ∪ {D^{(5,3)}, D^{(3,3,2)}}. We will deduce from this that D^{λ} ∈ M_{16}. In view of Lemma 5.2, it suffices to prove that every composition factor of D^{λ}↓_{Σ_{10}} belongs to M_{10}.

To verify that, we first prove that every composition factor of D^{λ}↓_{Σ_{9}} belongs to M_{9} ∪ {D^{(3,3,3)}, D^{(6,3)}}, with D^{(6,3)} only needed if p = 5. Indeed, let D^{β} be a composition factor of D^{λ}↓_{Σ_{9}}. Then all composition factors of D^{β}↓_{Σ_{8}} belong to M_{8} ∪ {D^{(5,3)}, D^{(3,3,2)}}. If an element of M_{8} appears in the socle of D^{β}↓_{Σ_{8}} then arguing as in the proof of Lemma 5.2, we conclude that D^{β} ∈ M_{9}. Assume that D^{(5,3)} appears in the socle. By Theorem 1.12(i), β ∈ {(6, 3), (5, 4), (5, 3, 1)}. But by Theorem 1.12(ii), D^{(6,3)}↓_{Σ_{8}} has a composition factor D^{(6,2)}, unless p = 5, which leads to a contradiction. Similarly we get a contradiction in the remaining two cases, as D^{(5,4)}↓_{Σ_{8}} has a composition factor D^{(4,4)}, and D^{(5,3,1)}↓_{Σ_{8}} has a composition factor D^{(4,3,1)}. Finally, if D^{(3,3,2)} appears in the socle of D^{β}↓_{Σ_{8}}, then a similar argument implies that β = (3, 3, 3).

Now, let D^{γ} be a composition factor of D^{λ}↓_{Σ_{10}}. By the previous paragraph, the composition factors of D^{γ}↓_{Σ_{9}} belong to M_{9} ∪ {D^{(3,3,3)}, D^{(6,3)}}, with D^{(6,3)} omitted unless p = 5. If an element of M_{9} appears in the socle of D^{γ}↓_{Σ_{9}} then we argue as in the proof of Lemma 5.2 to conclude that D^{γ} ∈ M_{10}. If p = 5 and D^{(6,3)} appears in the socle of D^{γ}↓_{Σ_{9}} then by Theorem 1.12(i), γ ∈ {(7, 3), (6, 4), (6, 3, 1)}. All of these lead to a contradiction because, in view of Theorem 1.12(ii), D^{(7,3)}↓_{Σ_{9}} contains D^{(7,2)}, D^{(6,4)}↓_{Σ_{9}} contains D^{(5,4)}, and D^{(6,3,1)}↓_{Σ_{9}} contains D^{(5,3,1)}, as composition factors. The case where D^{(3,3,3)} appears in the socle of D^{γ}↓_{Σ_{9}} is considered similarly.

Finally, let m = 3. In this case it suffices to show that a composition factor from the set {D^{(4)} ⊗ D^{(2,2)}, D^{(2,2)} ⊗ D^{(2,2)}, D^{(3,1)} ⊗ D^{(2,1,1)}} appears in the restriction D^{λ}↓_{Σ_{4}×Σ_{4}} for every λ ∈ {(6, 2), (4, 4), (5, 2, 1), (4, 2, 2), (4, 3, 1)}. For the first three λ’s this follows from Lemma 1.11. If λ = (4, 3, 1) or λ = (4, 2, 2) and p > 5 then λ is a p-core, and the required fact follows from the Littlewood-Richardson rule. Finally, if λ = (4, 2, 2) and p = 5, we have λ^{M} = (4, 4), and the result is obtained by tensoring with sgn.

Write G = V_{m}.G_{m}, where G_{m} = GL(m, 2) and V_{m} = 𝔽_{2}^{m} is its natural module. Let {e_{1}, e_{2}, …, e_{m}} be a basis of V_{m}. Then any element of V_{m} can be written in the form a_{1}e_{1} + ⋯ + a_{m}e_{m} where a_{i} = 0 or 1. A formal expression of the form χ = b_{1}ε_{1} + ⋯ + b_{m}ε_{m}, with b_{i} = 0 or 1, will be identified with the linear character

χ : V_{m} → F^{∗}, (a_{1}e_{1} + ⋯ + a_{m}e_{m}) ↦ (−1)^{a_{1}b_{1}+···+a_{m}b_{m}}.

Any irreducible representation of V_{m} over F looks like this, and we denote the set of all irreducible F V_{m}-modules by Irr V_{m}. Note that χ ⊗ χ ≅ 1_{V_{m}} for any χ ∈ Irr V_{m}. Write ¯0 = 0ε_{1} + ⋯ + 0ε_{m} for the trivial representation 1_{V_{m}}. The group G acts on Irr V_{m} via

g · χ(a) = χ(g^{−1}ag), g ∈ G, a ∈ V_{m}, χ ∈ Irr V_{m}.

Under this action G has two orbits on Irr G: {¯0} and Irr G \ {¯0}. We denote the stabilizer of χ ∈ Irr V_{m} in G by G_{χ}. Note that G_{ε_{m}} ≅ V_{m}. AGL(m − 1, 2). Now we explain how the irreducible F G-modules can be parametrized by the irreducible F G_{i}-modules for all 0 ≤ i ≤ m (where G_{0} is the trivial group), see [9], section 5.1, or [44], section 13, for more details. Let L be any irreducible F G-module. Consider the restriction L↓_{V_{m}}. If V_{m} acts trivially on L, the action of G factors through the surjection G → G_{m} to give an irreducible G_{m}-module. If this module is X, we denote L by L(X; 0). Next, assume that V_{m} does not act trivially on L. Then it is not hard to see that

L↓_{V_{m}} ≅ ⨁_{χ∈Irr V_{m}\{¯0}} L_{χ},

where L_{χ} is the χ-isotypic component of L↓_{V_{m}}. Moreover, L_{χ} is an (irreducible) F G_{χ}-module in a natural way, and L ≅ (L_{χ})↑_{V_{m}G_{χ}}^{G} for any χ ∈ Irr V_{m} \ {¯0}. Taking χ to be ε_{m}, we see in particular that L_{ε_{m}} is an irreducible F [V_{m}. AGL(m − 1, 2)]-module, which factors through to give an irreducible F [AGL(m − 1, 2)]-module. Now, AGL(m − 1, 2) = V_{m−1}.G_{m−1}. If V_{m−1} acts trivially on L_{ε_{m}}, this module factors through to give an irreducible F G_{m−1}-module, say X. In this case we denote our L by L(X; 1). Otherwise iterate by taking (L_{ε_{m}})_{ε_{m−1}}, etc. In this way the module L will be labelled by L(X; j), where 0 ≤ j ≤ m − 1 and X is an irreducible F G_{m−j}-module. Note that V_{m} does not have non-trivial fixed points on L(X; j), unless j = 0, when V_{m} in fact acts trivially on the whole module. In particular, the construction implies that dim L(X; j) = dim X(2^{m} − 1)(2^{m−1} − 1) … (2^{m−j+1} − 1).

Now let n = 2^{m} and embed G = V_{m}.G_{m} into Σ_{n} via the natural permutation representation. First, we show that D^{(n−1,1)} and D^{(n−2,1^{2})} are irreducible over G. In fact, we show more:

Lemma 5.4. D^{(n−1,1)} ↓_{G}≅ L(1_{G_{m−1}}; 1) and D^{(n−2,1^{2})} ↓_{G}≅ L(1_{G_{m−2}}; 2).

Proof. By Lemma 1.9, we have S^{(n−1,1)} = D^{(n−1,1)} and S^{(n−2,1^{2})} = D^{(n−2,1^{2})}. Hence the dimensions of the modules are n − 1 = 2^{m} − 1 and (n − 1)(n − 2)/2 = (2^{m} − 1)(2^{m−1} − 1).

Consider the natural permutation module M^{(n−1,1)}. Its elements are the formal F -linear combinations of the form ∑_{v∈V_{m}} f_{v}v. As p does not divide n = 2^{m}, we have M^{(n−1,1)} ≅ S^{(n−1,1)} ⊕ 1. As V_{m} acts regularly on M^{(n−1,1)}, we conclude that

S^{(n−1,1)} ↓_{V_{m}} ≅ ⨁_{χ∈Irr V_{m}\{¯0}} S_{χ}^{(n−1,1)},

where each S_{χ}^{(n−1,1)} is 1-dimensional. So now to prove that S^{(n−1,1)}↓_{G} ≅ L(1_{G_{m−1}}; 1) we just need to check that G_{m−1} acts trivially on S_{ε_{m}}^{(n−1,1)}. To prove this it is enough to show that G_{m−1} acts trivially on M_{ε_{m}}^{(n−1,1)}. Note that for any χ, M_{χ}^{(n−1,1)} is spanned by the vector

x_{χ} := ∑_{v∈V_{m}} χ(v)v.

As ε_{m}(b_{1}e_{1} + ⋯ + b_{m}e_{m}) = (−1)^{b_{m}}, one can easily see that G_{m−1} fixes x_{ε_{m}}.

To prove that S^{(n−2,1^{2})}↓_{G} ≅ L(1_{G_{m−2}}; 2), we show that (S_{ε_{m}}^{(n−2,1^{2})})_{ε_{m−1}} contains a vector fixed by G_{m−2}. This will imply that S^{(n−2,1^{2})}↓_{G} ⊇ L(1_{G_{m−2}}; 2), whence the two are equal by dimension. By Lemma 1.7, S^{(n−2,1^{2})} ≅ ⋀^{2} S^{(n−1,1)}. Note that M^{(n−1,1)} ≅ S^{(n−1,1)} ⊕ 1 implies ⋀^{2} M^{(n−1,1)} ≅ ⋀^{2} S^{(n−1,1)} ⊕ S^{(n−1,1)}. As we already know that (S_{ε_{m}}^{(n−1,1)})_{ε_{m−1}} = 0, it suffices to observe that ((⋀^{2} M^{(n−1,1)})_{ε_{m}})_{ε_{m−1}} contains a G_{m−2}-fixed vector, namely, the vector x_{ε_{m−1}+ε_{m}} ∧ x_{ε_{m−1}}.

Lemma 5.5. Let m ≥ 2, p > 3 and λ be a p-regular partition of n. Then, (D^{λ})^{V_{m}} = D^{λ} if and only if D^{λ} is one of 1_{Σ_{n}}, sgn_{Σ_{n}} or D^{(2,2)}.

Proof. If m > 2 then D^{λ} is a faithful representation of Σ_{n} unless D^{λ} = 1_{Σ_{n}} or sgn_{Σ_{n}}, so only these two cases arise. For m = 2, the natural module D^{(3,1)} clearly has points moved by V_{m}, so D^{(2,1^{2})} = D^{(3,1)} ⊗ sgn does too. It just remains to observe that all of D^{(2,2)} is fixed by V_{m}. Indeed, dim D^{(2,2)} = 2, while Irr V_{m} contains three non-trivial characters permuted transitively by G_{m}, so none of these can arise as constituents of D^{(2,2)} ↓_{V_{m}}.

Lemma 5.6. Let m ≥ 2, p > 3 and λ be a p-regular partition of n with h(λ) ≤ 3. Then, (D^{λ})^{V_{m}} = 0 if and only if D^{λ} is one of D^{(n−1,1)}, D^{(n−2,1^{2})}, D^{(5,3)} or D^{(3,3,2)}.

Proof. (⇐). Lemma 5.4 gives that (D^{λ})^{V_{m}} = 0 for λ = (n − 1, 1) or (n − 2, 1^{2}). For D^{(3,3,2)}, it suffices (since p ∤ |V_{3}|) to prove that S^{(3,3,2)} does not have V_{3}-fixed points, which in turn will follow from (S_{ℂ}^{(3,3,2)})^{V_{3}} = 0. Note that G < A_{8} < Σ_{8}, the module S_{ℂ}^{(3,3,2)} splits into two irreducible ℂA_{n}-modules S^{+} and S^{−} under the restriction to A_{n}, and those modules remain irreducible over G by [39, Theorem 2(iv)]. By faithfulness of the A_{8}-modules, we now conclude that V_{m} does not have fixed points on S^{+} and S^{−}, and hence on S_{ℂ}^{(3,3,2)}. Finally, to prove that D^{(5,3)} does not have V_{3}-invariants, it suffices to prove the same for S^{(5,3)}. Moreover, it is enough to prove that (S^{(5,3)})^{V_{3}} = 0 in any fixed characteristic different from 2, for example in characteristic 5. But in characteristic 5 we have S^{(5,3)} = D^{(5,3)} + D^{(7,1)} by virtue of Lemma 1.10. Moreover, (D^{(7,1)})^{V_{3}} = 0, and D^{(5,3)} ≅ D^{(3,3,2)} ⊗ sgn, whence (D^{(5,3)})^{V_{3}} = 0, and so (S^{(5,3)})^{V_{3}} = 0.

(⇒). Denote our embedding V_{m} ↪ Σ_{n} by φ_{m}. Note that V_{m} = Y ⊔ Z, where Y (resp. Z) is the set of all vectors in V_{m} whose e_{m}-coordinate is 0 (resp. 1). Then Y is a subgroup of V_{m}, isomorphic to V_{m−1}. Let us identify V_{m−1} with Y. Note that V_{m−1} acts regularly on both Y and Z. Denote the symmetric groups on the sets Y and Z by Σ(Y) and Σ(Z), respectively. Of course, Σ(Y) ≅ Σ(Z) ≅ Σ_{n/2}. Moreover H := Σ(Y) × Σ(Z) ≅ Σ_{n/2} × Σ_{n/2} is a Young subgroup of Σ_{n} in a natural way, and V_{m−1} is embedded into H diagonally via φ_{m−1} × φ_{m−1}. Set c = φ_{m}(e_{m}). Then c is an element of order 2 which permutes Y and Z. Let C = {1, c} < Σ_{n} be the subgroup of order 2 generated by c. Then V_{m} = V_{m−1}.C < H.C ≅ Σ_{n/2} wr C.

We next describe the irreducible representations of H.C. Let μ and ν be p-regular partitions of n/2. For any F H-module of the form D^{μ}⊗D^{ν} define a map τ : D^{μ}⊗D^{ν} → D^{ν}⊗D^{μ}, d⊗f ↦ f ⊗ d (for d ∈ D^{μ}, f ∈ D^{ν}). For μ ≠ ν, we consider an F H-module D^{μ} ⊗ D^{ν} ⊕ D^{ν} ⊗ D^{μ}. Let c act on it via c · (x + y) = τ(y) + τ(x) (for x ∈ D^{μ} ⊗ D^{ν}, y ∈ D^{ν} ⊗ D^{μ}). One can easily see that this defines an irreducible F [H.C]-module, which we denote by D(μ, ν). Also consider an F H-module D^{μ} ⊗D^{μ}. Let c act on it via c·x = τ(x) (resp. c·x = −τ(x)). This defines an irreducible F [H.C]-module, which we denote by D(μ, +1) (resp. D(μ, −1)). It follows from the Clifford theory (see e.g. [12], section III.2) that any F [H.C]-module looks like D(μ, ±1) or D(μ, ν) ≅ D(ν, μ) for μ ≠ ν.

For m ≥ 2 and any p-regular partitions μ, ν of n/2, we claim that

(a) D(μ, +1)^{V_{m}} ≠ 0;

(b) D(μ, −1)^{V_{m}} ≠ 0, unless D^{μ} ∈ R_{n/2}(1) = {1, sgn, D^{(n/2−1,1)}, D^{(n/2−1,1)} ⊗ sgn};

(c) D(μ, ν)^{V_{m}} ≠ 0 unless (D^{μ})^{V_{m−1}} = 0, (D^{ν})^{V_{m−1}} = D^{ν} or (D^{μ})^{V_{m−1}} = D^{μ}, (D^{ν})^{V_{m−1}} = 0. Indeed, for (a), pick any non-zero vector v ∈ D_{χ}^{μ} for any χ ∈ Irr V_{m−1}. Then, as χ⊗χ ≅ 1_{V_{m−1}}, v ⊗ v ∈ D(μ, +1) is a non-trivial V_{m}-invariant. For (b), assume that dim D_{χ}^{μ} ≥ 2 for some χ ∈ Irr V_{m−1}. Pick a pair f, d of linearly independent vectors in D_{χ}^{μ}. Then, as χ ⊗ χ ≅ 1_{V_{m−1}}, f ⊗ d − d ⊗ f ∈ D(μ, −1) is a non-trivial V_{m}-invariant. The only problem arises if dim D_{χ}^{μ} ≤ 1 for all χ ∈ Irr V_{m−1}. But this implies that the irreducible F Σ_{n/2}-module D^{μ} has dim D^{μ} ≤ 2^{m−1} = n/2. It follows from Lemmas 1.18 and 1.22 that D^{μ} ∈ R_{n/2}(1), as required. For (c), under the assumption there, we can find χ ∈ Irr V_{m−1} such that both D_{χ}^{μ} and D_{χ}^{ν} are non-zero. Let d ∈ D_{χ}^{μ}, f ∈ D_{χ}^{ν} be non-zero vectors. Then, this time, d ⊗ f + f ⊗ d ∈ D(μ, ν) is a non-trivial V_{m}-invariant.

Now we can complete the proof of the lemma by induction on m = 2, 3, …, the conclusion being immediate for m = 2 by Lemma 5.5. So suppose that m > 2 and that we have proved the result for all smaller m. Take λ with (D^{λ})^{V_{m}} = 0. Then, using the fact that p ∤ |V_{m}|, we must have that D^{V_{m}} = 0 for all composition factors D of (D^{λ}) ↓_{H.C}. Combining this with (a)–(c) above, we deduce that all composition factors of D^{λ} ↓_{H.C} have one of the following forms:

(1) D(μ, −1), with D^{μ} ∈ R_{n/2}(1);

(2) D(μ, ν) with (D^{μ})^{V_{m−1}} = 0, (D^{ν})^{V_{m−1}} = D^{ν}. Also note by a Specht module argument that all composition factors of D^{λ} ↓_{H} must be of the form D^{μ} ⊗ D^{ν} with h(μ), h(ν) ≤ h(λ) ≤ 3. So, applying Lemma 5.5 and the induction hypothesis, all composition factors of D^{λ} ↓_{H} are of the form D^{μ} ⊗ D^{ν} or D^{ν} ⊗ D^{μ} with either μ = ν ∈ {(k), (k − 1, 1), (2, 1, 1)} or μ ∈ {(k), (2, 2)} and ν ∈ {(k − 1, 1), (k − 2, 1, 1), (5, 3), (3, 3, 2)}, where k = n/2. We conclude from Lemma 5.3 that λ ∈ {(n), (n − 1, 1), (n − 2, 1, 1), (5, 3), (3, 3, 2)}. But λ = (n) clearly has (D^{λ})^{V_{m}} ≠ 0 so does not arise.

Now we can complete the analysis in this case. Suppose that D^{λ} is an irreducible F Σ_{n}-module of dimension > 1, with h(λ) ≤ 3, and that D^{λ} ↓_{G} is irreducible. As n ≥ 5, D^{λ} is faithful, so V_{m} cannot fix all of the module D^{λ}. Hence, D^{λ} ↓_{G} contains a composition factor of the form L(X; j) for j > 0, so by the irreducibility D^{λ} ↓_{G}≅ L(X; j) for j > 0. This shows that in fact V_{m} has no non-trivial fixed points on D^{λ}. So by Lemma 5.6, D^{λ} is D^{(n−1,1)}, D^{(n−2,1^{2})}, D^{(5,3)} or D^{(3,3,2)}. It just remains to consider the modules D^{(3,3,2)}, D^{(5,3)}, and D^{(n−2,1^{2})} on the 2-transitive subgroups G with G ≤ AGL(m, 2) < Σ_{n}.

Note that AGL(3, 2) < A_{8} < Σ_{8}, and D^{(3,3,2)} ≅ D^{(3,3,2)}⊗sgn if p > 5, so in this case already the restriction D^{(3,3,2)}↓_{A_{8}} is reducible. As for D^{(5,3)} in characteristic p > 5, it has dimension 28, and so if D^{(5,3)}↓_{AGL(3,2)} was irreducible then GL(2, 2) would have an irreducible module of dimension 4, which is false. Let p = 5. Note that (3, 3, 2)^{M} = (5, 3). By Theorem 1.16, dim D^{(3,3,2)} = dim D^{(5,3)} = 21. It follows by dimensions that the Specht module S^{(3,3,2)} has exactly two composition factors: D^{(3,3,2)} and D^{(5,3)}. It follows from [39, Theorem 2(iv)] that the restriction S_{ℂ}^{(3,3,2)}↓_{AGL(3,2)} has only two composition factors, and since 5 does not divide | AGL(3, 2)|, the reductions of these modules modulo 5 are irreducible. It follows that the restrictions of D^{(3,3,2)} and D^{(5,3)} to AGL(3, 2) are also irreducible. Finally, by (1), no proper subgroup of AGL(3, 2) can be irreducible on these modules.

Now, we consider D^{(n−2,1^{2})}. By Lemma 1.9, D^{(n−2,1^{2})} is an irreducible reduction modulo p of the corresponding Specht module S_{ℂ}^{(n−2,1^{2})}. By [39], the only subgroup of AGL(m, 2) for which the restriction S_{ℂ}^{(n−2,1^{2})}↓_{G} is irreducible is V_{4}.A_{7} < AGL(4, 2). So in characteristic p this is the only possibility, too. We saw above that dim D^{(14,1^{2})} = 105, and

D^{(n−2,1^{2})}↓_{V_{4}} ≅ ⊕_{χ∈Irr V_{4}\{¯0}}7χ

(where 7χ stands for \underbrace{χ ⊕ ⋯ ⊕ χ}_{7 times}). A similar decomposition holds for S_{ℂ}^{(n−2,1^{2})}↓_{V_{4}}. As we

know (from [39]) that the restriction S_{ℂ}^{(n−2,1^{2})}↓_{V_{4}.A_{7}} is irreducible, it follows that A_{7} acts transitively on Irr V_{4} \ {¯0}. The stabilizer H in A_{7} of a non-zero weight χ is then a subgroup of index 15. ¿From the description of the maximal subgroups of A_{7} (see e.g. [10], p. 10), H ≅ GL(2, 7) ≅ GL(3, 2). Moreover, by [41], section 8.2, Proposition 25,

S_{ℂ}^{(n−2,1^{2})}↓_{V_{4}.A_{7}} ≅ (S_{ℂ}^{(n−2,1^{2})})_{χ}↑_{V_{4}.H}^{V_{4}.A_{7}},

and (S_{ℂ}^{(n−2,1^{2})})_{χ} is irreducible as a ℂH-module. But the only irreducible 7-dimensional ℂ GL(3, 2)-module remains irreducible under reduction modulo p > 2, see [10], p. 3, and [23], p. 3. Let L be the reduction. As induction commutes with reduction modulo p, we see that D^{(n−2,1^{2})}↓_{V_{4}.A_{7}}, which is reduction modulo p of S_{ℂ}^{(n−2,1^{2})}↓_{V_{4}.A_{7}}, is isomorphic to L↑_{V_{4}.H}^{V_{4}.A_{7}}, which is irreducible by [41], section 8.2, Proposition 25.

Groups G with PSL(m, q) ≤ G ≤ PΓL(m, q), m ≥ 2. Recall that PΓL(m, q) is the projective semi-linear group. In view of the Fundamental Theorem of Projective Geometry [1], Theorem 2.26, this group is isomorphic to the automorphism group of the projective geometry P G_{m−1}(𝔽_{q}), provided m ≥ 3. In any case, PΓL(m, q) is generated by PGL(m, q) and automorphisms

ℙ^{m−1}(𝔽_{q}) → ℙ^{m−1}(𝔽_{q}), [x_{1} : x_{2} : ⋯: x_{m}] ↦ [x_{1}^{σ} : x_{2}^{σ} : ⋯: x_{m}^{σ}]

where σ is a field automorphism of 𝔽_{q}. The group PΓL(m, q) acts naturally on the points of ℙ^{m−1}(𝔽_{q}) (which we call lines), yielding a permutation representation of degree n = (q^{m} − 1)/(q − 1). If we twist this permutation representation with an outer automorphism of G, we get another permutation representation, non-equivalent to the first one. However, in both cases we get the same subgroup of Σ_{n}, and for the problem we are considering we may assume without loss of generality that the action is natural.

We consider the case m = 2 first, when n = q +1. We have |G| ≤ | PΓL(2, q)| = fq(q^{2} −1), where q = ℓ^{f} for a prime ℓ. Using (1) and Lemma 1.18(i) as above, we may assume that q = 16 or q ≤ 9.

If q = 16 then PΓL(2, 16) ≅ PSL(2, 16).4, and irreducible F PSL(2, 16)-modules have dimensions at most 17, see [10], p. 12, [23], p. 20. It follows that dimensions of irreducible F PΓL(2, 16)-modules do not exceed 17 · 4. Now, if D^{λ} is irreducible over PΓL(2, 16), we get a contradiction to Lemma 1.18(i), which claims that the dimension of an irreducible F Σ_{17}-module outside of R_{n}(1) is at least 103.

If q = 9 we have PΓL(2, 9) ≅ PSL(2, 9) .2^{2}, see [10]. By [10, 23], the dimensions of irreducible F PSL(2, 9)-modules are among 1, 5, 8, 9, 10. It follows that the dimensions of irreducible F PΓL(2, 9)-modules, exceeding 27, can only be 4·10 = 40, 4·9 = 36 and 4·8 = 32. By Lemma 1.22(vi), it remains to consider the restriction D^{(8,1^{2})}↓_{G} if p > 5. But D^{(8,1^{2})} is a reduction modulo p of S_{ℂ}^{(8,1^{2})}. By [39], S_{ℂ}^{(8,1^{2})}↓_{G} is reducible, hence D^{(8,1^{2})}↓_{G} is.

If q = 8, we may assume that p = 5 or 7 as otherwise we are in the ‘characteristic 0’ situation: both F Σ_{n} and F G are semisimple, and the result follows from [39]. Assume first that p = 5. As 5 does not divide | PΓL(2, 8)|, reduction modulo 5 of any irreducible F PΓL(2, 8)- module is irreducible. We know from [39] that S_{ℂ}^{(n−2,2)} is irreducible over PΓL(2, 8), whence D^{(n−2,2)}↓_{PΓL(2,8)} is also irreducible in characteristic 5. Moreover, it follows from [10] that irreducible F PΓL(2, 8)-modules have dimensions 1, 7, 8, 21, and 27. By Lemma 1.22(v), it remains to consider the module D^{(6,3)} of dimension 21. By Lemma 1.10, S^{(6,3)} = D^{(6,3)}+D^{(7,2)}. As we already know that D^{(7,2)}↓_{PΓL(2,8)} is irreducible and dim S^{(6,3)} = 48, D^{(6,3)}↓_{PΓL(2,8)} can

be irreducible only if S_{ℂ}^{(6,3)}↓_{PΓL(2,8)} has two composition factors of dimensions 21 and 27.

However, the group PΓL(2, 8) is 2-homogeneous but not 3-homogeneous, hence M_{ℂ}^{(6,3)}↓_{PΓL(2,8)}

has more invariants than M_{ℂ}^{(7,2)}↓_{PΓL(2,8)} by Lemma 1.1. So in view of [17], Exa! mple 17.17,

we see that S_{ℂ}^{(6,3)}↓_{PΓL(2,8)} has non-trivial invariants. Now, let p = 7. In this case the irreducible F PΓL(2, 8)-modules have dimensions 1, 7, 8, 21, see [23]. Now, if D^{λ} is irreducible over PΓL(2, 8), we get a contradiction with Lemma 1.22(v), which claims that the dimension of an irreducible F Σ_{9}-module in characteristic 7 is either 19 or at least 27.

Let q = 7. Then dimensions of irreducible F PΓL(2, 7)-modules do not exceed 8, see [10], p. 3, and [23], p. 3. Now we can apply Lemma 1.22(iv).

Let q = 5. We only need to consider the case p = 5 as otherwise we are in the ‘characteristic 0’ situation. By Lemma 1.22(ii) and [23], p. 2, we only have to worry about D^{(3^{2})}↓_{PΓL(2,5)}. But D^{(3^{2})} = D^{(3^{2})} by Lemma 1.10, so the result follows from [39] again. Finally, the case q = 4 does not arise as PΓL(2, 4) = Σ_{5}.

¿From now on we assume that m ≥ 3. Let m = 3 and q = 2. The irreducible modules over PΓL(3, 2) ≅ GL_{3}(2) have dimension at most 8, [10, 23], and we can apply Lemma 1.22(iii) to deduce that D^{λ}↓_{G} is reducible if p > 5. If p = 5 we just need to consider the restriction of D^{(5,2)}. A character calculation using [10], pp. 10, 3, shows that S_{ℂ}^{(5,2)}↓_{GL_{3}(2)} is a sum of two irreducible modules of dimensions 8 and 6. On the other hand a reduction modulo 5 of S^{(5,2)} has composition factors D^{(5,2)} and D^{(6,1)} of dimensions 8 and 6, respectively. As reduction modulo p commutes with restriction to a subgroup, and 5 does not divide | GL_{3}(2)|, we conclude that D^{(5,2)}↓_{GL_{3}(2)} is irreducible. From now on we assume that (m, q) ≠ (3, 2).

If p = 5, and D^{λ} is D^{(r,s)} or D^{(r,s)} ⊗ sgn for r − s ≤ 3, then it follows from Theorem 1.16 that dim D^{λ} = f_{n} or f_{n−1}. By (1), we must have

f_{n−1} ≤ \sqrt{|G|} ≤ (fq^{m(m−1)/2}(q^{m} − 1)(q^{m−1} − 1) … (q^{2} − 1))^{1/2} .

But an elementary argument using Lemma 1.17 shows that this is impossible (as we already have m ≥ 3, and (m, q) ≠ (3, 2)). So in what follows we exclude the case where p = 5 and D^{λ} is D^{(r,s)} or D^{(r,s)} ⊗ sgn for r − s ≤ 3.

Under the assumptions which we have now made, the group G = PΓL(m, q) is not 3homogeneous, as it cannot move three lines in a general position to three lines in one plane. In view of Proposition 5.1, it remains to prove that r_{4} > r_{3}. It is well known that G is transitive on the triples of lines in general position. Moreover, it is not difficult to see that it is also transitive on the triples of lines lying in one plane. Thus r_{3} = 2. For the 4-tuples of lines, we have an orbit of lines in general position (if m ≥ 4), an orbit of lines such that no three of them lie in one plane, an orbit of lines such that exactly three of them lie in one plane, and an orbit of lines all four of which lie in one plane (if q ≥ 3). Thus r_{4} > r_{3} (as we already have (m, q) ≠ (3, 2)).

Groups G with PSU(3, q) ≤ G ≤ PΓU(3, q), q > 2. The group PΓU(3, q) is isomorphic to Aut(PSU(3, q)) ≅ PSU(3, q).H, where H is of order (3, q+1)f if q^{2} = ℓ^{f} for a prime ℓ. Thus |G| ≤ | PΓU(3, q)| = fq^{3}(q^{2} − 1)(q^{3} + 1). The group G has a 2-transitive representation of degree n = q^{3} + 1. If D^{λ} is irreducible over G, we get from (1) and Lemma 1.18(i) that

\sqrt{fq^{3}(q^{2} − 1)(q^{3} + 1)} ≥ ((q^{3} + 1)^{2} − 5(q^{3} + 1) + 2)/2,

which is impossible.

Groups G with Sz(q) ≤ G ≤ Aut(Sz(q)), q > 2. We have q = 2^{f} for an odd f, and |G| ≤ | Aut(Sz(q))| = fq^{2}(q^{2} + 1)(q − 1). The group G has a 2-transitive representation of degree n = q^{2} + 1. So we get from (1) and Lemmas 1.18(i), 1.22(vi) that

\sqrt{fq^{2}(q^{2} + 1)(q − 1)} ≥ ((q^{2} + 1)^{2} − 5(q^{2} + 1) + 2)/2,

which is impossible.

Groups G with Re(q)′ ≤ G ≤ Aut(Re(q)′). We have q = 3^{f} for an odd f, and |G| ≤ | Aut(Re(q))| = fq^{3}(q^{3} + 1)(q − 1). The group G has a 2-transitive representation of degree n = q^{3} + 1. So as above we get

\sqrt{fq^{3}(q^{3} + 1)(q − 1)} ≥ ((q^{3} + 1)^{2} − 5(q^{3} + 1) + 2)/2,

which is impossible.

The group G = Sp(2m, 2), m ≥ 3. In this case G has two 2-transitive representations Ω^{0} and Ω^{1} of degrees 2^{m−1}(2^{m} + 1) and 2^{m−1}(2^{m} − 1), respectively.

If p = 5, and D^{λ} is D^{(m,k)} or D^{(m,k)} ⊗ sgn for m − k ≤ 3, then by Theorem 1.16 and (1), we must have

f_{n−1} ≤ \sqrt{|G|} = (2^{m^{2}}(2^{2m} − 1)(2^{2(m−1)} − 1) … (2^{2} − 1))^{1/2} .

But an elementary argument using Lemma 1.17 shows that this is impossible. So from now on we exclude the case where p = 5 and D^{λ} is D^{(m,k)} or D^{(m,k)} ⊗ sgn for m − k ≤ 3.

We shortly describe the construction of the representations Ω^{0} and Ω^{1} referring the reader, for example, to [11], section 7.7, for more details. Let V be a 2m-dimensional 𝔽_{2}-vector space endowed with a non-degenerate symplectic form (·, ·), and {e_{1}, …, e_{n}, f_{1}, …, f_{n}} be a corresponding symplectic basis of V. For a ∈ V and ε ∈ 𝔽_{2} set

L(a, ε) := {v ∈ V | (a, v) = ε}.

We consider the quadratic form Q_{0} on V defined by

Q_{0}( ∑_{i=1}^{m}(a_{i}e_{i} + b_{i}f_{i})) := ∑_{i=1}^{m} a_{i}b_{i}.

The following technical results on Q_{0} will be of importance:

Lemma 5.7. [11, Lemma 7.7B] Let ε, δ ∈ 𝔽_{2}, and a, b be distinct vectors of V. Then Q_{0} is not constant on L(a, ε) ∩ L(b, δ).

Lemma 5.8. [11, section 7.7] For every a, b ∈ V we have

Q_{0}(a + b) − Q_{0}(a) − Q_{0}(b) = (a, b).  (9)

Moreover, if Q is another quadratic form such that Q(a + b) − Q(a) − Q(b) = (a, b) for all a, b ∈ V, then Q(−) = Q_{0}(−) + (c, −) for some c ∈ V.

For any c ∈ V define the quadratic form Q_{c} by setting

Q_{c}(−) := Q_{0}(−) + (c, −).

Set Ω := {Q_{c} | c ∈ V }. Then, in view of Lemma 5.8, G acts on Ω by (g · Q)(v) := Q(g^{−1} · v), Q ∈ Ω, v ∈ V. For a vector b ∈ V define the transvection t_{b} ∈ G as the map

t_{b} : v ↦ v + (v, b)b (v ∈ V).

Lemma 5.9. [11, Lemma 7.7A] Let a, b ∈ V. Then

t_{b} · Q_{a} = \begin{cases}Q_{a}, & if Q_{a}(b) = 1; \\ Q_{a+b} & if Q_{a}(b) = 0. \end{cases}

For ε = 0 or 1 we set Ω^{ε} := {Q_{c} | Q_{0}(c) = ε}.

Lemma 5.10. [11, Theorem 7.7A] Ω^{0} and Ω^{1} are the orbits of G on Ω. Moreover, G is 2-transitive on each of them and |Ω^{0}| = 2^{m−1}(2^{m} + 1), |Ω^{1}| = 2^{m−1}(2^{m} − 1).

We prove that for both Ω^{0} and Ω^{1} one has r_{3} = 2 and r_{4} > 2. Then Proposition 5.1 will show that D^{λ}↓_{G} is reducible.

Fix ε ∈ 𝔽_{2}. For δ ∈ 𝔽_{2}, we denote by Γ_{δ} the set of all 3-element subsets {Q_{a}, Q_{b}, Q_{c}} of Ω^{ε} such that (a, b) + (a, c) + (b, c) = δ.

Lemma 5.11. Γ_{0} and Γ_{1} are the orbits of G on 3-element subsets of Ω^{ε}. In particular, r_{3} = 2.

Proof. First, we prove that Γ_{δ} is G-invariant. Let {Q_{a}, Q_{b}, Q_{c}} ∈ Γ_{δ} . As G is generated by transvections, it is enough to prove that {t_{x} · Q_{a}, t_{x} · Q_{b}, t_{x} · Q_{c}} ∈ Γ_{δ} for every x ∈ V. By Lemma 5.9, we have t_{x} · Q_{d} = Q_{d+(1−Q_{d}(x))x} for any d, x ∈ V. Now,

(a + (1 − Q_{a}(x))x, b + (1 − Q_{b}(x))x)
+(a + (1 − Q_{a}(x))x, c + (1 − Q_{c}(x))x)
+(b + (1 − Q_{b}(x))x, c + (1 − Q_{c}(x))x)
= (a, b) + (1 − Q_{b}(x))(a, x) + (1 − Q_{a}(x))(b, x)
+(a, c) + (1 − Q_{c}(x))(a, x) + (1 − Q_{a}(x))(c, x)
+(b, c) + (1 − Q_{c}(x))(b, x) + (1 − Q_{b}(x))(c, x)
= (a, b) + (a, c) + (b, c) − (Q_{b}(x) + Q_{c}(x))(a, x)
−(Q_{a}(x) + Q_{c}(x))(b, x) − (Q_{a}(x) + Q_{b}(x))(c, x)
= (a, b) + (a, c) + (b, c) − (Q_{0}(x) + (b, x) + Q_{0}(x) + (c, x))(a, x)

−(Q_{0}(x) + (a, x) + Q_{0}(x) + (c, x))(b, x) − (Q_{0}(x) + (a, x) + Q_{0}(x) + (b, x))(c, x)

= (a, b) + (a, c) + (b, c) = δ,

which proves that Γ_{δ} is G-invariant.

It remains to prove that G is transitive on Γ_{δ}. As G is 2-transitive, it is enough to prove that for {Q_{a}, Q_{b}, Q_{c}}, {Q_{a}, Q_{b}, Q_{d}} ∈ Γ_{δ} with c ≠ d, there exists g ∈ G with g · Q_{a} = Q_{a}, g · Q_{b} = Q_{b}, and g · Q_{c} = Q_{d}.

First of all, for a triple {Q_{a}, Q_{b}, Q_{y}} ∈ Γ_{δ} we find a condition for

t_{c+y} · Q_{a} = Q_{a}, t_{c+y} · Q_{b} = Q_{b}, t_{c+y} · Q_{c} = Q_{y}.  (10)

In view of Lemma 5.9 and (9), the conditions (10) are equivalent to

Q_{0}(y) = ε, (a + c, y) = 1 + (a, c), (b + c, y) = 1 + (b, c).  (11)

The equality (a, b) + (a, c) + (b, c) = (a, b) + (a, y) + (b, y) implies that (a + c, y) = 1 + (a, c) is equivalent to (b + c, y) = 1 + (b, c). Thus, (11) is equivalent to

Q_{0}(y) = ε, (a + c, y) = 1 + (a, c).

So if we have (a + c, d) = 1 + (a, c) then the transvection t_{c+d} will move {Q_{a}, Q_{b}, Q_{c}} to {Q_{a}, Q_{b}, Q_{d}}. Otherwise, we have (a + c, d) = (a, c). In this case we wish to use the product t_{y+d}t_{c+y} for some choice of y. By the previous paragraph, this product will move {Q_{a}, Q_{b}, Q_{c}} to {Q_{a}, Q_{b}, Q_{d}}, providing y satisfies

Q_{0}(y) = ε, (a + c, y) = 1 + (a, c), (a + y, d) = 1 + (a, y),

which is equivalent to

Q_{0}(y) = ε, (a + c, y) = 1 + (a, c), (a + d, y) = 1 + (a, d).

By Lemma 5.7, there always exists y satisfying these conditions.

Lemma 5.12. Let ε ∈ 𝔽_{2}. For the action of G on Ω^{ε} we have r_{4} > 2.

Proof. Let {e_{1}, …, e_{m}, f_{1}, …, f_{m}} be a symplectic basis in V. We claim that

{Q_{e_{1}}, Q_{f_{1}}, Q_{e_{1}+f_{1}+e_{2}}, Q_{e_{1}+f_{1}+f_{2}}}, {Q_{e_{1}+e_{2}+e_{3}}, Q_{f_{1}}, Q_{f_{2}}, Q_{f_{3}}}, and {Q_{e_{1}}, Q_{f_{1}}, Q_{f_{1}+e_{2}}, Q_{e_{3}}}

belong to distinct G-orbits on 4-element sets. Indeed, it follows from Lemma 5.11 that {Q_{a_{1}}, Q_{b_{1}}, Q_{c_{1}}, Q_{d_{1}}} and {Q_{a_{2}}, Q_{b_{2}}, Q_{c_{2}}, Q_{d_{2}}} are in the same orbit only if the sets

{(a_{1}, b_{1}) + (a_{1}, c_{1}) + (b_{1}, c_{1}), (a_{1}, b_{1}) + (a_{1}, d_{1}) + (b_{1}, d_{1}),
(a_{1}, c_{1}) + (a_{1}, d_{1}) + (c_{1}, d_{1}), (b_{1}, c_{1}) + (b_{1}, d_{1}) + (c_{1}, d_{1})}

and

{(a_{2}, b_{2}) + (a_{2}, c_{2}) + (b_{2}, c_{2}), (a_{2}, b_{2}) + (a_{2}, d_{2}) + (b_{2}, d_{2}),
(a_{2}, c_{2}) + (a_{2}, d_{2}) + (c_{2}, d_{2}), (b_{2}, c_{2}) + (b_{2}, d_{2}) + (c_{2}, d_{2})}

are the same. But the corresponding sets for the 4-tuples above are {1, 1, 1, 1}, {0, 0, 0, 0}, and {0, 1, 1, 0}.

The group PSL(2, 11) in representations of degree 11. By [10, 23], dimensions of the irreducible modules over PSL(2, 11) do not exceed 12, and one can use Lemma 1.22(vii).

The group M_{11} in representations of degrees 11 and 12. The Mathieu group M_{11} has a 4-transitive representation of degree 11 and a 3-transitive representation of degree 12. We only have to consider the cases p = 11, 7, and 5, because otherwise we are in the ‘characteristic 0’ situation.

If p = 11 it follows from [16] or [23], p. 34, that the irreducible modules over M_{11} have dimensions 1, 9, 10, 11, 16, 44, 55. In view of Lemmas 1.22(vii),(viii), only the modules D^{(9,2)}, D^{(9,2)} ⊗ sgn and D^{(10,1^{2})}, D^{(10,1^{2})} ⊗ sgn over Σ_{11} and Σ_{12}, respectively, may be irreducible over M_{11}. We will work with modules D^{(9,2)} and D^{(10,1^{2})} only, as the result for D^{λ} ⊗ sgn follows from that for D^{λ}. In characteristic 0 the corresponding Specht modules S_{ℂ}^{(9,2)} and S_{ℂ}^{(10,1^{2})} are irreducible over M_{11}, see [39]. Moreover, by Lemmas 1.10, 1.9 and the character information available from [10, 23], we conclude that reductions modulo 11 of these modules are irreducible for both Σ_{11} and M_{11}. This shows that the restrictions are irreducible in characteristic 11, too.

Let p = 7. Then the irreducible modules over M_{11} have dimensions 1, 10, 11, 16, 44, 45, 55. By Lemma 1.22(vii),(viii), we have to consider the restrictions

D^{(9,2)}↓_{M_{11}}, D^{(9,1^{2})}↓_{M_{11}}, and D^{(10,1^{2})}↓_{M_{11}}.

By [39], the corresponding Specht modules in characteristic 0 are irreducible over M_{11}. Moreover, by Lemma 1.9 their reductions modulo 7 are irreducible. Finally, because 7 does not divide |M_{11}|, their restrictions to M_{11} are irreducible.

Let p = 5. Then the irreducible modules over M_{11} have dimensions 1, 10, 11, 16, 45, 55. By Lemma 1.22(vii),(viii), we have to consider the restrictions

D^{(7,4)}↓_{M_{11}}, D^{(9,1^{2})}↓_{M_{11}} and D^{(10,1^{2})}↓_{M_{11}}.

By [39], the restrictions S_{ℂ}^{(9,1^{2})}↓_{M_{11}} and S_{ℂ}^{(10,1^{2})}↓_{M_{11}} are irreducible over M_{11}. Moreover, by Lemma 1.9 their reductions modulo 5 for Σ_{11} and Σ_{12} are irreducible. By [10], p. 18, M_{11} has only one complex representation of dimension 45, and only one complex representation of dimension 55, and these representations remain irreducible modulo 5, thanks to [10], p. 18, and [23], p. 34. This implies that D^{(9,1^{2})}↓_{M_{11}} and D^{(10,1^{2})}↓_{M_{11}} are irreducible. Finally, it follows from the character information contained in [10, 23] that D^{(7,4)}↓_{M_{11}} is reducible (in fact it has composition factors of dimensions 10 and 45).

The group M_{12}. This group has two 5-transitive representation of degree 12. We only have to consider the cases p = 11, 7, and 5.

If p = 11 it follows from [16] or [23], p. 77, that the irreducible modules over M_{12} have dimensions 1, 11, 16, 29, 53, 55, 66, 91, 99, 176. In view of Lemma 1.22(viii), only the modules D^{(10,2)}, D^{(10,1^{2})} (and the corresponding D^{λ} ⊗ sgn) may be irreducible over M_{11}. In fact, in characteristic 0 the corresponding Specht modules S_{ℂ}^{(10,2)} and S_{ℂ}^{(10,1^{2})} are irreducible over M_{12}, see [39]. Moreover, by Lemma 1.9 and the character information available from [10, 23], we conclude that the reduction modulo 11 of the module S^{(10,1^{2})} is irreducible for both Σ_{11} and M_{11}. This shows that the restriction D^{(10,1^{2})}↓_{M_{11}} is irreducible in characteristic 11. Now we consider the module D^{(10,2)}. By Lemma 1.10, for the corresponding Specht module we have S^{(10,2)} = D^{(10,2)} + 1. Moreover, by a character calculation using [10, 23], S^{(10,2)}↓_{M_{11}} = L + 1, where L is a 53-dimensional irreducible F M_{11}-module. It follows that D^{(10,2)}↓_{M_{11}} = L.

Let p = 7. Then the irreducible modules over M_{12} have dimensions 1, 11, 16, 45, 54, 55, 66, 99, 120, 144, 176. In view of Lemma 1.22(viii), only the modules D^{(10,2)} and D^{(10,1^{2})} need to be considered. The corresponding Specht modules S_{ℂ}^{(10,2)} and S_{ℂ}^{(10,1^{2})} are irreducible over M_{12}, see [39]. Moreover, by Lemmas 1.10, 1.9 and because 7 does not divide |M_{12}|, we conclude that reductions modulo 7 of these Specht modules are irreducible for both Σ_{12} and M_{12}. This shows that the restrictions to M_{12} are irreducible in characteristic 7, too.

Finally, let p = 5. Then the irreducible modules over M_{12} have dimensions 1, 11, 16, 45, 55, 66, 78, 98, 120. In view of Lemma 1.22(viii), only the module D^{(10,1^{2})} needs to be considered, which is done as for other characteristics above.

The case M_{22} ≤ G ≤ M_{22}.2. Such a group has a 3-transitive representation of degree 22. By comparing dimensions with the use of [10], p. 40, [23], pp. 96–100, and Lemmas 1.23, 1.21, we only have to worry about the restriction of D^{(20,1^{2})}. By [39], S_{ℂ}^{(20,1^{2})}↓_{G} is irreducible. If p ≠ 11, it follows from Lemma 1.9 and [10, 23], that reductions modulo p of the module S_{ℂ}^{(20,1^{2})} are irreducible for both Σ_{22} and G. This shows that the restriction D^{(20,1^{2})}↓_{G} is irreducible in characteristic p. Let p = 11. By Lemma 1.9 and [10, 23], we have S^{(20,1^{2})} = D^{(20,1^{2})} +D^{(21,1)}, and S^{(20,1^{2})}↓_{G} = L_{1} + L_{2} where dim L_{1} = dim D^{(20,1^{2})} = 190 and dim L_{2} = dim D^{(21,1)} = 20. So D^{(20,1^{2})}↓_{G} is irreducible in characteristic 11.

The group M_{23}. This group has a 4-transitive representation of degree 23. By [10], p. 71, [23], pp. 178–179, and Lemmas 1.23, 1.21, we only have to consider the restrictions of D^{(21,1^{2})} and D^{(21,2)}. In characteristic 0, the restrictions of the corresponding Specht modules are irreducible over M_{23} by [39].

Let λ = (21, 2). Assume first that p ≠ 7, 11. Then reductions modulo p of the Specht module S_{ℂ}^{λ} are irreducible, thanks to Lemma 1.10. To prove that D^{λ} is irreducible over M_{23}, we have to show that the reduction modulo p of S_{ℂ}^{λ}↓_{M_{23}} is also irreducible. If p ≠ 5, 23 this follows from the fact that p does not divide |M_{23}|. Let p = 5 or 23. Note that dim S_{ℂ}^{λ} = 230. So the restriction S_{ℂ}^{λ}↓_{M_{23}} is an irreducible module of dimension 230. But M_{23} has only one irreducible module of dimension 230 in characteristic 0. Now one can use the information on characters available in [10, 23] to deduce that the reduction modulo p of S_{ℂ}^{λ}↓_{M_{23}} is irreducible. Let p = 11. Then by Lemma 1.10, S^{λ} = D^{λ} + 1. Moreover, S^{λ}↓_{M_{23}} has composition factors of dimensions 229 and 1, thanks to [10, 23]. This implies that D^{λ}↓_{M_{23}} is irreducible. The case p = 7 is similar to the case p = 11.

Let λ = (21, 1^{2}). Then dim S^{λ} = 231. Assume first that p = 23. Then by Lemma 1.9, S^{λ} = D^{λ} + D^{(22,1)}. This implies that S^{λ}↓_{M_{23}} is reducible. But, by [10, 23], the only ordinary irreducible character of M_{23} of dimension 231, which is reducible modulo 23, is χ_{6} (in the notation of [10]), and the reduction has composition factors of dimensions 210 and 21. As dim D^{(22,1)} = 21, we conclude that D^{λ} is irreducible over M_{23}. Now let p ≠ 23. Then the reduction modulo p of the Specht module S_{ℂ}^{λ} is irreducible, thanks to Lemma 1.9. To prove that D^{λ} is irreducible over M_{23}, it remains to note that the reduction modulo p of χ_{6} is irreducible, see [10, 23].

The group M_{24}. This group has a 5-transitive representation of degree 24. By [10], p. 96, [23], pp. 268–271, Lemmas 1.23, 1.21, we only have to consider the restrictions of D^{(22,1^{2})}, D^{(22,2)}, D^{(21,2,1)}, and D^{(21,1^{3})}. Note that in characteristic 0, the restrictions of the corresponding Specht modules are irreducible over M_{24} by [39]. All four cases are treated similarly, so we consider only one of them (the hardest).

Let λ = (21, 2, 1). Assume first that p ≠ 7, 23. Then the reduction modulo p of the Specht module S_{ℂ}^{λ} is irreducible, thanks to Lemma 1.21(iv). To prove that D^{λ} is irreducible over M_{24}, we have to show that the reduction modulo p of S_{ℂ}^{λ}↓_{M_{24}} is also irreducible, which follows from [10, 23]. Let p = 23. Then it is easy to deduce from [32], Theorem 1.10, and Lemma 1.21 that S^{λ} = D^{λ} + D^{(n−2,2)}. Moreover, S^{λ}↓_{M_{24}} has composition factors of dimensions 3269 and 251, thanks to [10, 23]. This implies that D^{λ}↓_{M_{24}} is irreducible. The case p = 7 is similar.

The group A_{7} in representations of degree 15. This case is considered using (1) and Lemma 1.18(i).

The Higman-Sims group HS in representations of degree 176. This case is considered using (1) and Lemma 1.18(i).

The Conway group Co_{3} in representation of degree 276. By Lemmas 1.20,1.21, Theorem 1.16 and (1), we only have to consider the modules D^{(274,2)} and D^{(274,1^{2})}. Moreover, by Lemma 1.21(i),(ii), we have

dim D^{(274,2)} = \begin{cases}37399, & if p = 137; \\ 37673, & if p = 11 or 5; \\ 37674, & otherwise; \end{cases}

dim D^{(274,1^{2})} = \begin{cases}37401, & if p = 23; \\ 37675, & otherwise. \end{cases}

Using the Gap library, we find that Co_{3} does not have irreducible modules with such dimensions.

Completion of the proof. We have now completed the analysis of the 2-transitive groups. Now we can finish the proof of the main theorem by considering the case G < Σ_{n−1} < Σ_{n}.

In view of Example 0.4, we may assume that λ is a Jantzen-Seitz partition. In this case Theorem 1.12 implies that D^{λ}↓_{Σ_{n−1}} = D^{λ_{A}}, where A is the top removable node of λ. By Theorem 0.7, we need to consider the cases where G ≤ Σ_{n−2} or G is 2-transitive (as a subgroup of Σ_{n−1}). It follows easily from Theorem 1.12 that D^{λ}↓_{Σ_{n−2}} is reducible, so we only need to worry about the latter case. By what has already been proved in this section, we need to consider the cases where λ_{A} ≠ (λ_{A})^{M} or λ_{A} = (n − 2, 1), (n − 3, 2), (n − 3, 1^{2}), (5, 3) for p = 5, (21, 2, 1), (21, 1^{3}) (for example, the first case will give irreducible restrictions if G = A_{n−1} and the last two are only needed for the case G = M_{24}). By [6], Theorem 5.10, if λ is a Jantzen-Seitz partition with λ ≠ λ^{M}, then λ_{A} ≠ (λ_{A})^{M}, which yields the case (vii) of the Main Theorem. On the other hand, if λ = λ^{M} then already the restriction D^{λ}↓_{A_{n}} is reducible. For the other cases, we just need to observe that (n − 1, 1) (resp. (n − 2, 2), (n−2, 1^{2}), (22, 1^{3}), (6, 3)) is Jantzen-Seitz if and only if n ≡ 0 (mod p) (resp. n ≡ 2 (mod p), n ≡ 0 (mod p), p = 5, p = 5), and (22, 2, 1) is not Jantzen-Seitz. Thus, for example in the case (ix), the exceptions from Example 0.5 do not appear because for the exceptions (i), (iii)-(v) from 0.5 we would need to have p|(n − 1), and for the exception (ii) to appear p must be 2.

References

[1] E. Artin, Geometric Algebra, Interscience publishers, New York, London, 1957.

[2] M. Aschbacher, On the maximal subgroups of the finite classical groups, Invent. Math. 76(1984), 469– 514.

[3] D. Benson, Spin modules for symmetric groups, J. Lond. Math. Soc.(2) 38(1988), 250–262.

[4] C. Bessenrodt and J.B. Olsson, On residue symbols and the Mullineux conjecture, J. Algebraic Combin. 7 (1998), 227–251.

[5] C. Bessenrodt and J.B. Olsson, Residue symbols and Jantzen-Seitz partitions, J. Combin. Theory, Ser. A 81 (1998), 201–230.

[6] C. Bessenrodt and J.B. Olsson, Branching of modular representations of the alternating groups, J. Algebra 209 (1998), 143–174.

[7] R. Brandl, The remaining hearts of Ree(q), preprint.

[8] J. Brundan and A. S. Kleshchev, Modular Littlewood-Richardson coefficients, Math. Z., to appear.

[9] J. Brundan, R. Dipper, and A. S. Kleshchev, Quantum linear groups and representations of GL_{n}(𝔽_{q}), preprint, University of Oregon, 1999.

[10] J.H. Conway, R. T. Curtis, S. P. Norton, R. A Parker, R. A. Wilson, Atlas of Finite Groups, Clarendon Press, Oxford, 1985.

[11] J. D. Dixon and B. Mortimer, Permutation Groups, Graduate Texts in Mathematics 163, Springer-Verlag, Berlin, Heidelberg, New York, 1996.

[12] W. Feit, The representation theory of finite groups, North-Holland Publ. Comp., Amsterdam, New York, Oxford, 1982.

[13] B. Ford, Irreducible restrictions of representations of the symmetric groups, Bull. Lond. Math. Soc. 27(1995), 453–459.

[14] B. Ford and A. S. Kleshchev, A proof of the Mullineux conjecture, Math. Z. 226(1997), 267–308.

[15] K. Erdmann, Tensor products and dimensions of simple modules for symmetric groups, Manuscripta Math. 88 (1995), 357–386.

[16] G. D. James, The modular characters of the Mathieu groups, J. Algebra, 27 (1973), 57–111.

[17] G. D. James, The representation theory of the symmetric groups, Springer Lecture Notes 682, Berlin, Heidelberg, New York, 1978.

[18] G. D. James, On the minimal dimensions of irreducible representations of symmetric groups, Math. Proc. Camb. Phil. Soc., 94 (1983), 417–424.

[19] G. D. James, The representation theory of the symmetric groups, in The Arcata Conference on Representations of Finite Groups, Proc. Symp. Pure Math. 47, Amer. Math. Soc., 1987, 111–126.

[20] G. James and A. Kerber, The representation theory of the symmetric group, Addison-Wesley, London, 1981.

[21] G. D. James and G. E. Murphy, The determinant of the Gram matrix for a Specht module, J. Algebra, 59 (1979), 222–235.

[22] G. D. James and M. H. Peel, Specht series for skew representations of symmetric groups, J. Algebra, 56 (1979), 343–364.

[23] C. Jansen, K. Lux, R. Parker, R. Wilson, An Atlas of Brauer Characters, Clarendon Press, Oxford, 1995.

[24] J. C. Jantzen and G. M. Seitz, On the representation theory of the symmetric groups, Proc. London Math. Soc. 65 (1992), 475–504.

[25] W. M. Kantor, Homogeneous designs and geometric lattices, J. Combin. Theory Ser. A 38 (1985), 66–74.

[26] P. Kleidman and M. Liebeck, The subgroup structure of the finite classical groups. London Mathematical Society Lecture Note Series, 129, Cambridge University Press.

[27] A. S. Kleshchev. On restrictions of irreducible modular representations of semisimple algebraic groups and symmetric groups to some natural subgroups I, Proc. London Math. Soc. 69 (1994), 515–540.

[28] A. S. Kleshchev, Branching rules for modular representations of symmetric groups, I, J. Algebra 178 (1995), 493–511.

[29] A. S. Kleshchev, Branching rules for modular representations of symmetric groups, II, J. reine angew. Math. 459 (1995), 163–212.

[30] A. S. Kleshchev, Branching rules for modular representations of symmetric groups III: some corollaries and a problem of Mullineux, J. London Math. Soc. 54 (1996), 25–38.

[31] A. S. Kleshchev, Completely splittable representations of symmetric groups, J. Algebra 181 (1996), 584–592.

[32] A. S. Kleshchev, On decomposition numbers and branching coefficients for symmetric and special linear groups, Proc. London Math. Soc. (3) 75 (1997), 497–558.

[33] A. S. Kleshchev and J. Sheth, Representations of the symmetric group are reducible over singly transitive subgroups, preprint, University of Oregon, 1999.

[34] O. Mathieu, On the dimension of some modular irreducible representations of the symmetric group, Lett. Math. Phys. 38 (1996), 23–32.

[35] B. Mortimer, The modular permutation representations of the known doubly transitive groups, Proc. Lond. Math. Soc(3) 41(1980), 1–20.

[36] G. Mullineux, Bijections of p-regular partitions and p-modular irreducibles of the symmetric groups. J. London Math. Soc. (2), 20(1979), 60–66.

[37] M. H. Peel, Hook representations of the symmetric groups, Glasgow Math. J., 12(1971), 136–149.

[38] A. Ryba, Fibonacci representations of the symmetric groups, J. Algebra 170 (1994), 678–686.

[39] J. Saxl, Irreducible characters of the symmetric groups that remain irreducible in subgroups, J. Algebra, 111(1987), 210–219.

[40] M. Schönert et. al., Gap: groups, algorithms and programming, 3.4.3, RWTH Aachen, 1996.

[41] J.-P. Serre, Linear representations of finite groups, Graduate Texts in Mathematics 42, Springer-Verlag, Berlin, Heidelberg, New York, 1977.

[42] J. Sheth, Branching rules for two row partitions and applications to the inductive systems for symmetric groups, Commun. Algebra 27(7) (1999), 3303–3316.

[43] H. Wenzl, Hecke algebras of type A_{n} and subfactors, Invent. Math. 92 (1988), 349–383.

[44] A. Zelevinsky, Representations of finite classical groups, Lecture Notes in Mathematics 869, Springer-Verlag, Berlin, Heidelberg, New York, 1981.

Department of Mathematics, University of Oregon, Eugene, OR 97403, USA

 E-mail address: brundan@darkwing.uoregon.edu

Department of Mathematics, University of Oregon, Eugene, OR 97403, USA

 E-mail address: klesh@math.uoregon.edu