# Semiclassical states for non-cooperative singularly perturbed fractional Schrödinger systems – Boundary Value Problems

May 12, 2022

### For the original problem (2.1)

Now we temporarily come back to our original problem (2.1). Note that using the similar arguments as Sect. 3, (J_{varepsilon}) also possess the linking structure, and thus there exist ((mathit{PS})_{c_{varepsilon}}) sequences with

$$0< c_{varepsilon}leqslant sup_{E^{-}oplus mathbb{R}^{+}e}J_{varepsilon}, quad e=(e_{1},e_{2})in E, e_{1}>0,e_{2}>0.$$

(4.1)

On the other hand, according to the assumption ((A_{0})), we may fix (overline{a}in mathbb{R}) such that

$$0< a(0)=min_{xin mathbb{R}^{N}}a(x)< overline{a}< liminf _{lvert x rvert rightarrow infty}a(x).$$

We denote by (J_{0}) and (c_{0}) the energy functional defined in (2.5) with (a(0)) in place of (a(varepsilon x)) and least energy, respectively. Let ((U_{0},V_{0})) be a ground-state for (J_{0}), it is easy to check that

$$J_{varepsilon}(U_{0},V_{0})=J_{0}(U_{0},V_{0})+o_{varepsilon}(1)=c_{0}+o(1),$$

(4.2)

and

$$J^{prime }_{varepsilon}(U_{0},V_{0}) ( Phi ,Psi )=J^{prime }_{0}(U_{0},V_{0}) ( Phi ,Psi )+o(1),$$

uniformly for bounded Φ, (Psi in X). Applying Theorem 3.1 and Lemma 3.2 to the functionals (J_{varepsilon}), we deduce that

$$sup_{E^{-}oplus mathbb{R}^{+}(U_{0},V_{0})}J_{varepsilon}=J_{varepsilon}(U_{0},V_{0})=c_{0}+o(1)< c( overline{a})+o(1),$$

which complies with (4.1) to conclude that (0< c_{varepsilon}< c(overline{a})).

### Some auxiliary problems

To apply “Indefinite functional theorem” to get the positive solutions of (2.1), we only need to show that ((mathit{PS})_{c_{varepsilon}}) condition holds for (0< c_{varepsilon}< c(overline{a})). For this purpose, in this subsection, we will study some auxiliary problems and establish several auxiliary results employing the ideas from [16].

Consider the nonlocal system as follows:

$$textstylebegin{cases} (-Delta )^{s}u+b(x)u=g(v), & text{in } mathbb{R}^{N}, \ (-Delta )^{s}v+b(x)v=f(u), & text{in } mathbb{R}^{N}, end{cases}$$

(4.3)

where (b(x)in C(mathbb{R}^{N})), (b(x)geqslant overline{b}>0) for any (xin mathbb{R}^{N}) and (lim_{lvert x rvert rightarrow infty}b(x)=b_{infty}in mathbb{R}).

The associated energy functional to the extension problem of (4.3) is defined by

begin{aligned} J_{b}(U,V)={}&k_{s} int _{mathbb{R}_{+}^{N+1}}y^{1-2s}langle nabla U, nabla Vrangle ,dz+ int _{mathbb{R}^{N}times {y=0}}b(x)UV,dx \ & {}- int _{mathbb{R}^{N}times {y=0}}bigl(F(U)+G(V)bigr),dx. end{aligned}

We denote by (J_{infty}) the corresponding functional with (b_{infty}) in place of (b(x)). Of course, here we work in the space (E:=Xtimes X), (X={Uin X^{s}(mathbb{R}_{+}^{N+1}): int _{ mathbb{R}^{N}times {y=0}}b(x)U^{2},dx<+infty }).

Now we prove the following Lemma.

### Lemma 4.1

Under the assumptions ((A_{1}))((A_{4})), the ((mathit{PS})) condition holds for (J_{b}) at critical level (0< c< c(b_{infty})). Moreover, (J_{b}) admits a groundstate critical level (c_{b}) and (c_{b}geqslant c(overline{b})).

### Proof

Let ((U_{n},V_{n})) be such that (J_{b}(U_{n},V_{n})rightarrow cin (0,c(b_{infty}))) and (J^{prime }_{b}(U_{n},V_{n})rightarrow 0) as (nrightarrow infty ). It follows from Lemma 3.2 that ((U_{n},V_{n})) is bounded in E and assumes that ((U_{n},V_{n})rightharpoonup (U,V)) in E, clearly, (J_{b}^{prime }(U,V)=0). In particular,

begin{aligned} 2J_{b}(U,V)&=2J_{b}(U,V)-J^{prime }_{b}(U,V) (U,V) \ &= int _{mathbb{R}^{N}}bigl(f(u)u-2F(u)+g(v)v-2G(v)bigr),dxgeqslant 0. end{aligned}

(4.4)

Putting (overline{U}_{n}=U_{n}-U), (overline{V}_{n}=V_{n}-V), we next show that (overline{U}_{n}rightarrow 0), (overline{V}_{n}rightarrow 0) in X.

Indeed, one has

begin{aligned} J_{b}(overline{U}_{n}, overline{V}_{n})={}&k_{s} int _{mathbb{R}_{+}^{N+1}}y^{1-2s} bigllangle nabla (U_{n}-U),nabla (V_{n}-V)bigrrangle ,dz \ &{}+ int _{mathbb{R}^{N}times {y=0}}b(x) (U_{n}-U) (V_{n}-V) ,dx \ & {}- int _{mathbb{R}^{N}times {y=0}}bigl(F(U_{n}-U)+G(V_{n}-V) bigr),dx \ ={}&J_{b}(U_{n},V_{n})-J_{b}(U,V)+ int _{mathbb{R}^{N}times {y=0}}bigl(F(U_{n})-F(U)-F(U_{n}-U) \ &{}+ G(V_{n})-G(V)-G(V_{n}-V)bigr),dx, end{aligned}

(4.5)

and

begin{aligned}& J^{prime }_{b}( overline{U}_{n},overline{V}_{n}) (Phi ,Psi ) \& quad = k_{s} int _{mathbb{R}_{+}^{N+1}}y^{1-2s}bigl[bigllangle nabla Phi , nabla (V_{n}-V)bigrrangle +bigllangle nabla (U_{n}-U),nabla Psi bigrrangle bigr] ,dz \& qquad {} + int _{mathbb{R}^{N}times {y=0}}b(x)bigl[Phi (x,0) (V_{n}-V)+ Psi (x,0) (U_{n}-U)bigr],dx \& qquad {} – int _{mathbb{R}^{N}times {y=0}}bigl[f(U_{n}-U)Phi (x,0)+g(V_{n}-V) Psi (x,0)bigr],dx \& quad = J^{prime }_{b}(U_{n},V_{n}) ( Phi ,Psi )-J^{prime }_{b}(U,V) (Phi ,Psi ) \& qquad {} + int _{mathbb{R}^{N}}bigl[bigl(f(u_{n}-u)-f(u_{n})+f(u) bigr)Phi (x,0) \& qquad {} +bigl(g(v_{n}-v)-g(v_{n})+g(v)bigr)Psi (x,0) bigr],dx, end{aligned}

(4.6)

for any bounded functions (Phi ,Psi in X).

Now we compute the third term of (4.5) and (4.6), respectively,

begin{aligned}& biggllvert int _{mathbb{R}^{N}}bigl(F(u_{n})-F(u)-F(u_{n}-u)+G(v_{n})-G(v)-G(v_{n}-v) bigr),dx biggrrvert \& quad = int _{lvert x rvert leqslant R}+ int _{lvert x rvert >R} \& quad = o_{n}(1)+C int _{lvert x rvert >R}bigl(lvert u_{n} rvert ^{2}+ lvert u_{n} rvert ^{p}+lvert u rvert ^{2}+lvert u rvert ^{p} \& qquad {} +lvert v_{n}rvert ^{2}+lvert v_{n} rvert ^{p}+lvert vrvert ^{2}+ lvert v rvert ^{p}bigr),dx \& quad leqslant o_{n}(1)+o_{R}(1). end{aligned}

Similarly,

begin{aligned} & int _{mathbb{R}^{N}}bigl[bigl(f(u_{n}-u)-f(u_{n})+f(u) bigr)Phi (x,0) \ & qquad {}+bigl(g(v_{n}-v)-g(v_{n})+g(v)bigr)Psi (x,0) bigr],dx \ &quad leqslant o_{n}(1)+o_{R}(1), end{aligned}

uniformly for any bounded Φ and Ψ in X.

Combining these estimates with (4.4)–(4.6) gives

$$J_{b}(overline{U}_{n},overline{V}_{n})=J_{b}(U_{n},V_{n})-J_{b}(U,V)+o(1) leqslant c+o(1),$$

(4.7)

and

$$sup bigl{ bigllvert J_{b}^{prime }( overline{U}_{n},overline{V}_{n}) (Phi , Psi )bigrrvert , Vert Phi Vert _{X}+ Vert Psi Vert _{X}leqslant 1bigr} =o(1).$$

(4.8)

Since (overline{U}_{n}rightharpoonup 0), (overline{V}_{n}rightharpoonup 0) in X and (b_{infty}=lim_{lvert x rvert rightarrow infty}b(x)), a similar conclusion as (4.7) and (4.8) holds for (J_{infty}(overline{U}_{n},overline{V}_{n})) and (J^{prime }_{infty}(overline{U}_{n},overline{V}_{n})); if (liminf_{nrightarrow infty}) (J_{infty}(overline{U}_{n},overline{V}_{n})>0), we deduce from Lemma 3.2 that

$$c(b_{infty})leqslant sup_{E^{-}oplus mathbb{R}^{+}(overline{U}_{n}, overline{V}_{n})}J_{infty}=J_{infty}(overline{U}_{n},overline{V}_{n}) leqslant c+o(1),$$

which contradicts with the assumption (c< c(b_{infty})). Consequently,

$$liminf_{nrightarrow infty}J_{infty}(overline{U}_{n}, overline{V}_{n})leqslant 0.$$

This implies that

begin{aligned} &liminf_{nrightarrow infty} bigl[2J_{infty}(overline{U}_{n}, overline{V}_{n})-J^{prime }_{infty}(overline{U}_{n},overline{V}_{n}) ( overline{U}_{n},overline{V}_{n})bigr] \ &quad = liminf_{nrightarrow infty} int _{mathbb{R}^{N}times {y=0}}bigl[f(overline{U}_{n}) overline{U}_{n}-2F(overline{U}_{n})+g( overline{V}_{n})overline{V}_{n} \ & qquad {} -2G(overline{V}_{n})bigr],dx \ &quad leqslant 0, end{aligned}

and thus

$$liminf_{nrightarrow infty} int _{mathbb{R}^{N}times {y=0 }}bigl(F(overline{U}_{n})+G( overline{V}_{n})bigr),dxleqslant 0,$$

which turns to be

$$liminf_{nrightarrow infty}bigl( Vert overline{U}_{n} Vert _{X}+ Vert overline{V}_{n} Vert _{X}bigr)=0.$$

Therefore, (U_{n}rightarrow U), (V_{n}rightarrow V) as (nrightarrow infty ); that is the ((mathit{PS})_{c}) condition holds for (0< c< c(b_{infty})). Applying “Indefinite functional theorem”, we can derive that there exists a nontrivial critical point ((U,V)) for the functional (J_{b}).

We set

$$c_{b}:=inf bigl{ J_{b}(U,V),(U,V)neq (0,0),J_{b}^{prime }(U,V)=0bigr} .$$

By the standard arguments, the infimum is actually a minimum, and it follows that (J_{b}) admits a ground-state critical level (c_{b}).

Finally, we compare the least energy levels of (J_{b}) with the ones of (J_{bar{b}}); this estimate is crucial for the proof of Theorem 1.1.

Assume that (c(bar{b})>c_{b}). For (tin [0,1]), let (b_{t}(x):=(1-t)b(x)+tbar{b}) and denote by (c’), (c_{t}) the corresponding ground-state level and linking level, respectively. Since (b_{t}(x)=b(x)+t(bar{b}-b(x))leqslant b(x)), (c_{t}leqslant c’leqslant c). Consequently, (c_{t}leqslant c_{b}). It follows from the assumption (c(bar{b})>c_{b}) and the fact ((1-t)b_{infty}+tbar{b}geqslant bar{b}) and Lemma 3.2 that

$$c_{t}leqslant c_{b}< c(bar{b})leqslant c bigl((1-t)b_{infty}+tbar{b}bigr),$$

for every (tin [0,1]). This is a contradiction for (t=1), since (c_{1}=c(bar{b})), which ends the proof. □

Next, utilizing Lemma 4.1, we prove an important auxiliary result that is directly applied to the proof of Theorem 1.1.

### Lemma 4.2

Let (b(x)geqslant overline{b}>0) and suppose ({(U_{n},V_{n})}) is a ((mathit{PS})_{c}) sequence for (J_{b}) with (liminf_{nrightarrow infty}J_{b}(U_{n},V_{n})>0), then (c(overline{b})leqslant J_{b}(U_{n},V_{n})+o(1)).

### Proof

By Lemma 4.1, we can choose M so large that

$$sup_{E^{-}oplus mathbb{R}^{+}e}J_{b}=J_{b}(U_{n},V_{n})+o(1)< c(M).$$

(4.9)

Define the truncated functions

$$b_{n}(x)= textstylebegin{cases} b(x), & lvert xrvert leqslant n, \ M , &lvert xrvert > n, end{cases}$$

then (b_{n}(x)geqslant overline{b}>0) and (b_{n}(x)rightarrow M) as (lvert xrvert rightarrow infty ). Take n so large that

$$J_{b_{n}}(U_{n},V_{n})=J_{b}(U_{n},V_{n})+o(1)=c+o(1),$$

(4.10)

and

$$J^{prime }_{b_{n}}(U_{n},V_{n}) (Phi ,Psi )=o(1),$$

(4.11)

uniformly for all bounded functions Φ and Ψ in X. From (4.9)–(4.11), we can employ Lemma 4.1 to get (J_{b_{n}}) admitting a ground state critical level (c_{b_{n}}>0) and

$$c(overline{b})leqslant c_{b_{n}}leqslant sup_{E^{-}oplus mathbb{R}^{+}(U_{n},V_{n})}J_{b_{n}}=J_{b_{n}}(U_{n},V_{n})+o(1)=J_{b}(U_{n},V_{n})+o(1).$$

This completes the proof. □

### The proof of Theorem 1.1

In this subsection, we prove Theorem 1.1 divided into three results. First of all, we give the existence result.

### Theorem 1.1(i)

Assume that f, g satisfy ((A_{1}))((A_{4})), then for all small (varepsilon >0), (J_{varepsilon}) have ground states ((U_{varepsilon},V_{varepsilon})in E) with critical values (0< c_{varepsilon}< c(overline{a})) and (c_{varepsilon}rightarrow c_{0}) as (varepsilon rightarrow 0).

### Proof

Recall that there exists a ((mathit{PS})_{c_{varepsilon}}) sequence ({(U_{n},V_{n})}subset E) for (J_{varepsilon}), assume that ((U_{n},V_{n})rightharpoonup (U,V)in E), and for sufficiently small (varepsilon >0), (0< c_{varepsilon}< c(overline{a})). And there exists a constant (R_{0}>0) such that for some fixed (varepsilon >0),

$$a(varepsilon x)geqslant overline{a} quad text{for all } lvert x rvert geqslant R_{0}.$$

According to “Indefinite functional theorem” [15], we only need to show that for fixed (varepsilon >0), the ((mathit{PS})_{c_{varepsilon}}) condition holds for (J_{varepsilon}) at critical levels (0< c_{varepsilon}< c(overline{a})).

Introduce the following truncated function

$$b(x)= textstylebegin{cases} a(varepsilon x), & lvert xrvert geqslant R_{0}, \ geqslant overline{a} , & lvert xrvert < R_{0}, end{cases}$$

then (b(x)geqslant overline {a}) for all (xin mathbb{R}^{N}). Remark that (overline{U}_{n}=U_{n}-U), (overline{V}_{n}=V_{n}-V) and (overline{U}_{n}rightharpoonup 0), (overline{V}_{n}rightharpoonup 0) in X.

Observe that

begin{aligned} int _{mathbb{R}^{N}times {y=0}}b(x)overline{U}_{n} overline{V}_{n},dx&= int _{lvert xrvert geqslant R_{0}}+ int _{ lvert xrvert < R_{0}} \ &= int _{mathbb{R}^{N}times {y=0}}a(varepsilon x)overline{U}_{n} overline{V}_{n},dx+o_{n}(1). end{aligned}

Using the similar arguments as Lemma 4.1, it is easy to check that

begin{aligned}& J_{b}(overline{U}_{n},overline{V}_{n})= J_{varepsilon}(overline{U}_{n}, overline{V}_{n})+o_{n}(1) leqslant c_{varepsilon}+o(1), \& J^{prime }_{b}(overline{U}_{n}, overline{V}_{n})=o(1). end{aligned}

If (liminf_{nrightarrow infty}J_{b}(overline{U}_{n}, overline{V}_{n})>0), we can derive applying Lemma 4.2 that

$$c(overline{a})leqslant J_{b}(overline{U}_{n}, overline{V}_{n}) leqslant c_{varepsilon}+o(1),$$

which is a contradiction. As a result, (liminf_{nrightarrow infty}J_{b}(overline{U}_{n}, overline{V}_{n})leqslant 0). This gives (overline{U}_{n}rightarrow 0), (overline{V}_{n}rightarrow 0) in X.

Next, we show (c_{varepsilon}rightarrow c_{0}) as (varepsilon rightarrow 0).

By (4.2), we obtain that (limsup_{varepsilon rightarrow 0}c_{varepsilon}leqslant c_{0}). On the other hand, let ((U_{varepsilon},V_{varepsilon})) be the ground states for (J_{varepsilon}), then ((U_{varepsilon}, V_{varepsilon})) are bounded in E. In particular, we have

$$J_{0}(U_{varepsilon},V_{varepsilon})=J_{varepsilon}(U_{varepsilon},V_{varepsilon})+ int _{mathbb{R}^{N}times {y=0}}bigl(a(0)-a( varepsilon x)bigr)U_{varepsilon}V_{varepsilon},dx=c_{varepsilon}+o(1),$$

and for any (Phi ,Psi in X),

begin{aligned} bigllvert J_{0}^{prime }(U_{varepsilon},V_{varepsilon}) (Phi ,Psi ) bigrrvert &= int _{mathbb{R}^{N}times {y=0}}bigllvert a(0)-a( varepsilon x)bigrrvert lvert U_{varepsilon}Phi +V_{varepsilon}Psi rvert ,dx \ &leqslant o(1) bigl( Vert Phi Vert _{X}+ Vert Psi Vert _{X}bigr). end{aligned}

By Lemma 3.2, we conclude that

$$c_{0}leqslant sup_{E^{-}oplus mathbb{R}^{+}(U_{varepsilon},V_{varepsilon})}J_{0}=J_{0}(U_{varepsilon},V_{varepsilon})=c_{varepsilon}+o(1).$$

Hence, (c_{0}leqslant liminf_{varepsilon rightarrow 0}c_{varepsilon}). This completes the proof. □

From now, we consider the positive functions (U_{varepsilon}>0), (V_{varepsilon}>0) given by Theorem 1.1(i), which satisfy (u_{varepsilon}=U_{varepsilon}(x,0)in H), (v_{varepsilon}=V_{varepsilon}(x,0)in H) and

$$textstylebegin{cases} (-Delta )^{s}u_{varepsilon}+a(varepsilon x)u_{varepsilon}=g(v_{varepsilon}), & text{in } mathbb{R}^{N}, \ (-Delta )^{s}v_{varepsilon}+a(varepsilon x)v_{varepsilon}=f(u_{varepsilon}), & text{in } mathbb{R}^{N}. end{cases}$$

Next, we study the concentration behavior of this family of solution ((u_{varepsilon},v_{varepsilon})) as (varepsilon rightarrow 0).

### Theorem 1.1(ii)

Suppose that ((u_{varepsilon},v_{varepsilon})) is the ground states of problem (2.1) for all ε sufficiently small, then ((u_{varepsilon},v_{varepsilon})) attain their maximum value at some unique and common points (x_{varepsilon}in mathbb{R}^{N}) such that

$$lim_{varepsilon rightarrow 0}a(varepsilon x_{varepsilon})=a(0)= min _{xin mathbb{R}^{N}}a(x).$$

### Proof

Since (u_{varepsilon}>0), (v_{varepsilon}>0), for each small (varepsilon >0), there exist (x_{varepsilon}, y_{varepsilon}in mathbb{R}^{N}) such that, respectively,

$$u_{varepsilon}(varepsilon x_{varepsilon})=max_{xin mathbb{R}^{N}}u_{varepsilon}, qquad v_{varepsilon}(varepsilon y_{varepsilon})=max _{yin mathbb{R}^{N}}v_{varepsilon}.$$

We split the proof into several steps.

Step 1. ({varepsilon x_{varepsilon}}) and ({varepsilon y_{varepsilon}}) are bounded.

Suppose by contradiction that there exists a subsequence ({varepsilon _{j}x_{varepsilon _{j}}}) such that (lvert varepsilon _{j}x_{varepsilon _{j}}rvert rightarrow + infty ) as (varepsilon _{j}rightarrow 0). Define the functions (widetilde{u}_{j}(x)=u_{varepsilon _{j}}(x+x_{varepsilon _{j}})) and (widetilde{v}_{j}(x)=v_{varepsilon _{j}}(x+x_{varepsilon _{j}})). Observe that these functions satisfy the following system:

$$textstylebegin{cases} (-Delta )^{s}widetilde{u}_{j}+a_{j}(x)widetilde{u}_{j}=g( widetilde{v}_{j}), \ (-Delta )^{s}widetilde{v}_{j}+a_{j}(x)widetilde{v}_{j}=f( widetilde{u}_{j}), end{cases}$$

here (a_{j}(x)=a(varepsilon _{j}x+varepsilon _{j}x_{varepsilon _{j}})). Denote by (I_{j}) the corresponding energy functional,

$$I_{j}(widetilde{u}_{j},widetilde{v}_{j}) = int _{mathbb{R}^{N}}(- Delta )^{s}widetilde{u}_{j} widetilde{v}_{j},dx + int _{mathbb{R}^{N}}a_{j}(x) widetilde{u}_{j} widetilde{v}_{j},dx – int _{mathbb{R}^{N}}bigl(F( widetilde{u}_{j})+G( widetilde{v}_{j})bigr),dx.$$

(4.12)

From Lemma 3.2, the families ({(widetilde{u}_{j},widetilde{v}_{j})}) are bounded in E, and let ((widetilde{u}_{j},widetilde{v}_{j})rightharpoonup (u,v)neq (0,0) in E). Recall from Theorem 1.1(i) that

$$0< liminf_{jrightarrow infty}I_{j}(widetilde{u}_{j}, widetilde{v}_{j}) leqslant limsup_{jrightarrow infty}I_{j}( widetilde{u}_{j},widetilde{v}_{j})< c(overline{a}).$$

(4.13)

Let (b_{j}(x):=max {overline{a},a_{j}(x)}) and denote by (overline{I}_{j}) the corresponding energy functional defined as in (4.12) with (a_{j}(x)) replaced by (b_{j}(x)). By the assumption (lvert varepsilon _{j}x_{varepsilon _{j}}rvert rightarrow + infty ), it holds that ((widetilde{u}_{j},widetilde{v}_{j})) is a ((mathit{PS})) sequence for (overline{I}_{j}) and

$$overline{I}_{j}(widetilde{u}_{j}, widetilde{v}_{j})=I_{j}( widetilde{u}_{j}, widetilde{v}_{j})+o_{varepsilon _{j}}(1).$$

Since (b_{j}(x)geqslant overline{a}) for every (xin mathbb{R}^{N}), it follows then from Lemma 4.2 that

$$c(overline{a})leqslant overline{I}_{j}(widetilde{u}_{j}, widetilde{v}_{j})+o(1)=I_{j}(widetilde{u}_{j} ,widetilde{v}_{j})+o(1),$$

which contradicts with (4.13). Hence ({varepsilon x_{varepsilon}}) is bounded, the same is ({varepsilon y_{varepsilon}}).

Furthermore, we have that there is a subsequence ({widetilde{u}_{j_{n}}}) and ({widetilde{v}_{j_{n}}}) such that for any (eta >0), there exists (r_{eta}>0) satisfying

$$limsup_{nrightarrow infty} int _{B_{N}(0, n)setminus B_{N}(0,r)}bigl( widetilde{u}^{q}_{j_{n}}+ widetilde{v}^{q}_{j_{n}}bigr),dxleqslant eta$$

for all (rgeqslant r_{eta}) (see an argument to [24, Lemma 5.7]). Here (qin [2, 2^{*}_{s})), which implies together with the assumptions ((A_{1})(A_{2})) that for any (eta >0), there exists (r_{eta}>0),

$$limsup_{nrightarrow infty} int _{B_{N}(0, n)setminus B_{N}(0,r)}bigl(f( widetilde{u}_{j_{n}}) widetilde{u}_{j_{n}}+g(widetilde{v}_{j_{n}}) widetilde{v}_{j_{n}}bigr),dxleqslant eta ,$$

(4.14)

for all (rgeqslant r_{eta}).

According to the above arguments, assume that (varepsilon x_{varepsilon}rightarrow x_{0}), (varepsilon y_{varepsilon}rightarrow y_{0}) as (varepsilon rightarrow 0); now we can conclude from (a(varepsilon x+varepsilon x_{varepsilon})rightarrow a(x_{0})), (a( varepsilon x+varepsilon y_{varepsilon})rightarrow a(y_{0})) pointwise and (4.14) that (c_{varepsilon}rightarrow c(a(x_{0}))) and (c_{varepsilon}rightarrow c(a(y_{0}))), which combine with Lemma 4.3 that (a(x_{0})=a(y_{0})=a(0)=min_{xin mathbb{R}^{N}}a(x)). In conclusion, any maximum points (varepsilon x_{varepsilon}), (varepsilon y_{varepsilon}) of (u_{varepsilon}), (v_{varepsilon}), respectively, have the concentration point as (varepsilon rightarrow 0).

Step 2. (x_{varepsilon}=y_{varepsilon}) as (varepsilon rightarrow 0) and the maximum point of (u_{varepsilon}) (and of (v_{varepsilon}) as well) is unique if (varepsilon >0) is sufficiently small.

Let us prove that there exists (C>0) such that, for every subsequence ({varepsilon _{j}}) and every large j,

$$lvert x_{varepsilon _{j}}-y_{varepsilon _{j}} rvert leqslant C.$$

(4.15)

For this purpose, let (overline{u}_{j}=u_{varepsilon _{j}}(x+y_{varepsilon _{j}})), (overline{v}_{j}=v_{varepsilon _{j}}(x+y_{varepsilon _{j}})), we can follow Step 1 to assume ((overline{u}_{j},overline{v}_{j})rightharpoonup (overline{u}, overline{v})neq (0,0)); it follows from (4.14) that there is a subsequence ({overline{u}_{j_{n}}}) and ({overline{v}_{j_{n}}}) such that, for any (eta >0) there exists (R>0) such that, for every large n,

$$int _{{B_{N}(0,R)}^{c}}bigl(f(overline{u}_{j_{n}}) overline{u}_{j_{n}}+g( overline{v}_{j_{n}}) overline{v}_{j_{n}}bigr),dxleqslant eta .$$

Denoting (w_{j_{n}}:=y_{varepsilon _{j_{n}}}-x_{varepsilon _{j_{n}}}), the above inequality reads as

$$int _{{B_{N}(w_{j_{n}},R)}^{c}}bigl(f(widetilde{u}_{j_{n}}) widetilde{u}_{j_{n}}+g(widetilde{v}_{j_{n}}) widetilde{v}_{j_{n}}bigr),dx leqslant eta .$$

So, if (lvert w_{j_{n}}rvert rightarrow +infty ), we conclude from (4.14) that for every (eta >0), for every large n,

$$int _{mathbb{R}^{N}}bigl(f(widetilde{u}_{j_{n}}) widetilde{u}_{j_{n}}+g( widetilde{v}_{j_{n}}) widetilde{v}_{j_{n}}bigr),dxleqslant 2eta ,$$

which conclude that (int _{mathbb{R}^{N}}(f(u)u+g(v)v),dx=0), and thus (u=v=0). This is a contradiction.

Now, according to (4.15), let (w_{0}in mathbb{R}^{N}) be such that (y_{varepsilon _{j}}-x_{varepsilon _{j}}rightarrow w_{0}( varepsilon _{j}rightarrow 0)). Since (overline{u}_{j}(x)=widetilde{u}_{j}(x+y_{varepsilon _{j}}-x_{ varepsilon _{j}})rightharpoonup u(x+w_{0})), combining with the fact that both (widetilde{u}_{j}(x)) and (overline{u}_{j}(x)) have the same limit function u (and similarly for (widetilde{v}_{j}(x)) and (overline{v}_{j}(x))), we can derive (u(x)=u(x+w_{0})) (and similarly for v). u has 0 and (w_{0}) as maximum points, so that (w_{0}=0). This establishes (y_{varepsilon}=x_{varepsilon}) as (varepsilon rightarrow 0).

Similarly, if (z_{varepsilon}in mathbb{R}^{N}) is also a maximum point of (u_{varepsilon}), then the preceding arguments yield that (z_{varepsilon}=x_{varepsilon}(=y_{varepsilon})) for small values of ε.

Step 3. (u_{varepsilon}, v_{varepsilon}(x)rightarrow 0) as (lvert x rvert rightarrow infty ) uniformly for all small ε.

Indeed, assume by contradiction that there exist (delta >0) and (xi _{varepsilon}in mathbb{R}^{N}) with (lvert xi _{varepsilon}rvert rightarrow infty ) such that

$$delta leqslant bigllvert u_{varepsilon}(xi _{varepsilon})bigrrvert leqslant C int _{B_{N}(xi _{varepsilon},1)}bigllvert u_{varepsilon}(y) bigrrvert ,dy.$$

Assume that (u_{varepsilon}rightharpoonup u) in H, we obtain, as (varepsilon rightarrow 0),

begin{aligned} delta &leqslant Cbiggl( int _{B_{N}(xi _{varepsilon},1)}bigllvert u_{varepsilon}(y)bigrrvert ^{2},dybiggr)^{frac{1}{2}} \ &leqslant Cbiggl( int _{B_{N}(xi _{varepsilon},1)}bigllvert u_{varepsilon}(y)-u(y) bigrrvert ^{2},dybiggr)^{frac{1}{2}} +Cbiggl( int _{B_{N}(xi _{varepsilon},1)} bigllvert u(y)bigrrvert ^{2},dy biggr)^{frac{1}{2}}rightarrow 0, end{aligned}

which is a contradiction completing the proof. □

Define

$$omega _{varepsilon}(x)=u_{varepsilon}biggl(frac{x}{varepsilon} biggr),qquad xi _{varepsilon}(x)=v_{varepsilon}biggl( frac{x}{varepsilon}biggr) quad text{and}quad z_{varepsilon}=varepsilon x_{varepsilon}.$$

Then ((omega _{varepsilon},xi _{varepsilon})) is a solution to (1.2) for all small (varepsilon >0). Since (z_{varepsilon}) is a maximum point of (omega _{varepsilon}) and (xi _{varepsilon}), we have

$$lim_{varepsilon rightarrow 0}a(z_{varepsilon})=a(0).$$

Now, we study the decay behavior of this family of solution ((omega _{varepsilon},xi _{varepsilon})).

### Theorem 1.1(iii)

There exist (0< C_{1}leqslant C_{2}) and (R>1) such that for all small (varepsilon >0),

$$frac{C_{1}varepsilon ^{N+2s}}{lvert x-z_{varepsilon}rvert ^{N+2s}} leqslant omega _{varepsilon}(x), qquad xi _{varepsilon}(x)leqslant frac{C_{2}varepsilon ^{N+2s}}{lvert x-z_{varepsilon}rvert ^{N+2s}}$$

for all (lvert x rvert geqslant R).

### Proof

Firstly, we use the following Claims, according to [25].

(i) There is a continuous function (w_{1}) in (mathbb{R}^{N}) satisfying

$$(-Delta )^{s}w_{1}(x)+mu w_{1}(x)=0,quad text{if } lvert x rvert >1,$$

and

$$w_{1}(x)geqslant frac{C_{1}}{lvert x rvert ^{N+2s}},$$

for an appropriate (C_{1}>0), where (mu :=sup a(varepsilon x));

(ii) There is a continuous function (w_{2}) in (mathbb{R}^{N}) satisfying

$$(-Delta )^{s}w_{2}(x)+tau w_{2}(x)=0,quad text{if } lvert x rvert >1.$$

and

$$w_{2}(x)leqslant frac{C_{2}}{lvert x rvert ^{N+2s}},$$

for an appropriate (C_{2}>0), where (0<tau <frac{1}{2}inf a(varepsilon x)).

By (u_{varepsilon}(x),v_{varepsilon}(x)rightarrow 0) as (lvert x rvert rightarrow infty ) uniformly for all small ε and the condition ((A_{0})), ((A_{1})), we conclude that there is a large (R_{1}>0) such that

$$(-Delta )^{s}(u_{varepsilon}+v_{varepsilon})+ frac{a(varepsilon x)}{2}(u_{varepsilon}+v_{varepsilon})=g(v_{varepsilon})+f(u_{varepsilon})-frac{a(varepsilon x)}{2}(u_{varepsilon}+v_{varepsilon})leqslant 0 quad text{in } {B_{R_{1}}}^{c}.$$

Moreover, by the continuity of solution ((u_{varepsilon},v_{varepsilon})) and (w_{2}), for all small (varepsilon >0), there exists (C>0) such that

$$u_{varepsilon}(x)+v_{varepsilon}(x)-Cw_{2}(x)leqslant 0 quad text{in } overline{B_{R_{1}}}.$$

Therefore,

$$(-Delta )^{s}(u_{varepsilon}+v_{varepsilon}-Cw_{2})+tau (u_{varepsilon}+v_{varepsilon}-Cw_{2})leqslant 0quad text{in } {B_{R_{1}}}^{c}.$$

Using comparison arguments, we get

$$u_{varepsilon}+v_{varepsilon}leq Cw_{2}leq frac{C_{2}}{lvert x rvert ^{N+2s}} quad text{for } lvert x rvert geq R_{1}.$$

Since ((u_{varepsilon},v_{varepsilon})) is a positive solution, then

$$u_{varepsilon}(x)leqslant frac{C}{lvert x rvert ^{N+2s}},qquad v_{varepsilon}(x) leqslant frac{C}{lvert x rvert ^{N+2s}},quad text{for } lvert x rvert geq R_{1}.$$

Hence, by rescaling, it follows that there exists (C_{2}>0) such that

$$omega _{varepsilon}(x),xi _{varepsilon}(x)leqslant frac{C_{2}varepsilon ^{N+2s}}{lvert x-z_{varepsilon}rvert ^{N+2s}}.$$

On the other hand, by the continuity of ((u_{varepsilon},v_{varepsilon})) and (w_{1}), there exist constants (C_{2}, C_{3}>0) such that, respectively,

begin{aligned}& u_{varepsilon}(x)-C_{2}w_{1}(x)geqslant 0quad text{in } overline{B_{1}}, \& v_{varepsilon}(x)-C_{3}w_{1}(x)geqslant 0quad text{in } overline{B_{1}}, end{aligned}

which imply that

begin{aligned}& (-Delta )^{s}(u_{varepsilon}-C_{2}w_{1})+ mu (u_{varepsilon}-C_{2}w_{1}) geqslant 0quad text{in } {overline{B_{1}}}^{c}, \& (-Delta )^{s}(v_{varepsilon}-C_{3}w_{1})+ mu (v_{varepsilon}-C_{3}w_{1}) geqslant 0quad text{in } {overline{B_{1}}}^{c}. end{aligned}

By the similar comparison arguments, we conclude the second inequality, which ends the proof of Theorem 1.1. □