Xamarin.Android APK crashes following migration to

Question

We've got a mobile application (Xamarin.Forms for iOS and Android) that we've been in the progress of migrating from our on-premise Team Foundation Server (TFS) to Visual Studio Team Services (VSTS). For the last couple of months now we've hosted our code in VSTS but have performed the builds and releases to the respective stores from our on-premise TFS. Late last week I migrated both the build and release definitions to VSTS as well, so that now we have the entire process in VSTS.

Or so I thought... For, while the iOS build and release works fine, and Android builds and releases as well, the Android application now crashes when you attempt to run it. No major (read: no) changes have taken place in the code and neither HockeyApp nor our App's own error log are able to record the reason for the crash - nor does the Google Developer Console provide us with any insight for that matter. After a couple of tries I was able to get the following exception out of a Logcat:

I/MonoDroid(21970): UNHANDLED EXCEPTION:
I/MonoDroid(21970): System.ArgumentException: element is not of type Xamarin.Forms.View
I/MonoDroid(21970): Parameter name: element
I/MonoDroid(21970):   at Xamarin.Forms.Platform.Android.VisualElementRenderer`1[TElement].Xamarin.Forms.Platform.Android.IVisualElementRenderer.SetElement (Xamarin.Forms.VisualElement element) [0x0001c] in <fa196d4afd1b4356b38d6cf3bb1e4df8>:0 
I/MonoDroid(21970):   at Xamarin.Forms.Platform.Android.Platform.CreateRenderer (Xamarin.Forms.VisualElement element, Android.Support.V4.App.FragmentManager fragmentManager, Android.Content.Context context) [0x00031] in <fa196d4afd1b4356b38d6cf3bb1e4df8>:0 
I/MonoDroid(21970):   at Xamarin.Forms.Platform.Android.AppCompat.FragmentContainer.OnCreateView (Android.Views.LayoutInflater inflater, Android.Views.ViewGroup container, Android.OS.Bundle savedInstanceState) [0x0001b] in <fa196d4afd1b4356b38d6cf3bb1e4df8>:0 
I/MonoDroid(21970):   at Android.Support.V4.App.Fragment.n_OnCreateView_Landroid_view_LayoutInflater_Landroid_view_ViewGroup_Landroid_os_Bundle_ (System.IntPtr jnienv, System.IntPtr native__this, System.IntPtr native_inflater, System.IntPtr native_container, System.IntPtr native_savedInstanceState) [0x00020] in <20045e457c414d37bc46e447a5bb3bb5>:0 
I/MonoDroid(21970):   at (wrapper dynamic-method) System.Object.b09cff61-6f2e-471d-b43a-8d2949513eb7(intptr,intptr,intptr,intptr,intptr)

Now there's only one place in our code where we use fragments, which is in a custom renderer we use to extend the default implementation of the BottomBarPage we use instead of Xamarin's default TabbedPage. I therefore attempted a build with this feature turned off (using the default TabbedPage instead of a BottomBarPage and removing the ExportRenderer-attribute), but this did not solve the problem. Of course I also had a look online to see if I could match the error message with any problem anyone else might have experienced. Results seemed rather meager and not particularly applicable to my case: install the latest version of Xamarin.Forms (which we cannot do at the moment - besides things were working on our private build server) or do a clean and rebuild (I actually tried turning on the Clean-option in our Xamarin.Android-task, but to no effect).

I therefore proceeded to compare the last build log from TFS with a recent one from VSTS to see if I could find any major difference. This turned out to be rather tricky as the output between the two build definitions differs markedly. Superficially, they both look the same, though, performing the same tasks. I did notice, however, that our own Mac build server used newer versions of both MS Build and Android Build Tools than does the VSTS Hosted Mac (15.6.0.0 versus 15.4.0.0 and 27.0.1 versus 23.0.0 respectively), so I decided to change the build environment, as there seems to be no way to influence these parts of the build-environment and the highest installed version of the Android Build Tools will be used when none is explicitly specified anyway (according to https://docs.microsoft.com/en-us/xamarin/android/deploy-test/building-apps/build-process). Running on VS2017 Hosted now at least gives me Android Build Tools version 27.0.3. The JDK-version is the same on all environments, by the way.

With the switch in build-environment also not resolving the issue I next tried the following:

• Turned off AOT-compilation - though a comparison of the log showed that what differed mostly were IDs and compression levels/final file sizes.
• Verified that I deselected to use the shared runtime (as suggested here) and that both "armeabi" and "armeabi-v7a" architectures are selected for compilation (as proposed here).
• Switched from using the Xamarin.Android build-task to using the shell-script which I had built for our on-premise TFS, as the latter's agent had lacked the Xamarin.Android-capability and the script was meant as a work-around.
• Compared the list of Android resources reported by the log as included in the build, which was equal.
• Compared the list of intermediate assemblies reported by both build logs as being packaged and found them to be more or less equal (with the new VSTS-build adding unfamiliar Mono.Data.Tds.dll- and System.Runtime.InteropServices.RuntimeInformation.dll-assemblies).
• Performed a per-project comparison of copied files reported by the build logs - something I found of little informative value, since I considered additional assemblies & files reported by VSTS harmless, but am otherwise unable to evaluate the impact of files reported by TFS but not reported by VSTS (in other words, I'm unsure of the informative value of comparing the lists of copied files).
• Verified the list of files reported by both logs as having been signed and zip-aligned (the configuration between TFS and VSTS is completely the same, therefore this task has received little attention in my investigation otherwise).
• Made a build on VSTS from the same branch and commit as the last-known working one made on TFS and compared these using Android Studio's APK analysis/package comparison tool.

Only the latter - the package comparison - I've found to be informative, yet of what I do not know. That is to say, one would expect packages containing exactly the same code and resource files to have little difference among them except for maybe the version numbers in the Android-manifest. And, indeed, this is what I'm seeing for the assets and resources. However, significantly (at least in my opinion), the META-INF, lib, classes.dex, assemblies and typemap-files all differ in size. It is unclear to me what this means, though...

Now about 1.5 days in, the only promising result so far is another post here on Stack Overflow that seems to describe the same symptoms as we're experiencing, yet reports a different exception message in their logcat. No resolution has been posted, however.

---

Edit 1

One thing I've noticed just now when comparing the Android-manifest between builds made on TFS versus those made on VSTS is that the following metadata is lacking from the application-node when building on VSTS (yet is present in local [debug-]builds):

<meta-data
    android:name="android.support.VERSION"
    android:value="26.1.0" />

I'm unsure what package adds this node and what it does, however, but feel it might be significant and will be investigating this further.

---

Edit 2

After a long day of frustration trying all kinds of different combinations of libraries (some of which don't seem to want to install on the .NetStandard projects VS2017 creates for mobile Apps by default instead of the old PCL), I managed to narrow down where the above entry in the Android-manifest comes from: apparently this is something that the version 26.1.0 Xamarin Android support libraries add. It turns out that one of the Xamarin Android support libraries referenced in the CS-project-file of one of the three Android-projects our App uses was still pointing towards an old 26.1.0.1 version, whereas all other references had been updated to 27.0.2. Changing this entry manually ensured that entry in the Android-manifest went away. This, however, did, unfortunately, not solve our build issues with the Android-version of our App...

---

Edit 3

Building on a hosted VS2017 environment now results in the below exception messages:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(2374):C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(2374,3): Error : error XA5101: Missing Android NDK toolchains directory '\toolchains'. Please install the Android NDK. at Xamarin.Android.Tasks.NdkUtil.GetNdkToolchainDirectories(String toolchainsPath, AndroidTargetArch arch) at Xamarin.Android.Tasks.NdkUtil.GetNdkToolchainPath(String androidNdkPath, AndroidTargetArch arch) at Xamarin.Android.Tasks.NdkUtil.GetNdkTool(String androidNdkPath, AndroidTargetArch arch, String tool) at Xamarin.Android.Tasks.NdkUtil.GetNdkToolPrefix(String androidNdkPath, AndroidTargetArch arch) at Xamarin.Android.Tasks.Aot.<GetAotConfigs>d__73.MoveNext() at System.Collections.Concurrent.Partitioner.DynamicPartitionerForIEnumerable`1.InternalPartitionEnumerable.GrabChunk_Buffered(KeyValuePair`2[] destArray, Int32 requestedChunkSize, Int32& actualNumElementsGrabbed) at System.Collections.Concurrent.Partitioner.DynamicPartitionerForIEnumerable`1.InternalPartitionEnumerator.GrabNextChunk(Int32 requestedChunkSize) at System.Collections.Concurrent.Partitioner.DynamicPartitionEnumerator_Abstract`2.MoveNext() at System.Threading.Tasks.Parallel.<>c__DisplayClass42_0`2.<PartitionerForEachWorker>b__1() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask) at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object )
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(2374):C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(2374,3): Error XA3001: System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Java.Interop.Tools.Diagnostics.XamarinAndroidException: error XA5101: Missing Android NDK toolchains directory '\toolchains'. Please install the Android NDK. at Xamarin.Android.Tasks.NdkUtil.GetNdkToolchainDirectories(String toolchainsPath, AndroidTargetArch arch) at Xamarin.Android.Tasks.NdkUtil.GetNdkToolchainPath(String androidNdkPath, AndroidTargetArch arch) at Xamarin.Android.Tasks.NdkUtil.GetNdkTool(String androidNdkPath, AndroidTargetArch arch, String tool) at Xamarin.Android.Tasks.NdkUtil.GetNdkToolPrefix(String androidNdkPath, AndroidTargetArch arch) at Xamarin.Android.Tasks.Aot.<GetAotConfigs>d__73.MoveNext() at System.Collections.Concurrent.Partitioner.DynamicPartitionerForIEnumerable`1.InternalPartitionEnumerable.GrabChunk_Buffered(KeyValuePair`2[] destArray, Int32 requestedChunkSize, Int32& actualNumElementsGrabbed) at System.Collections.Concurrent.Partitioner.DynamicPartitionerForIEnumerable`1.InternalPartitionEnumerator.GrabNextChunk(Int32 requestedChunkSize) at System.Collections.Concurrent.Partitioner.DynamicPartitionEnumerator_Abstract`2.MoveNext() at System.Threading.Tasks.Parallel.<>c__DisplayClass42_0`2.<PartitionerForEachWorker>b__1() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask) at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at System.Threading.Tasks.Task.Wait() at System.Threading.Tasks.Parallel.PartitionerForEachWorker[TSource,TLocal](Partitioner`1 source, ParallelOptions parallelOptions, Action`1 simpleBody, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally) at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally) at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body) at Xamarin.Android.Tasks.Aot.RunParallelAotCompiler(List`1 nativeLibs) at Xamarin.Android.Tasks.Aot.<>c__DisplayClass71_0.<DoExecute>b__0() at System.Threading.Tasks.Task`1.InnerInvoke() at System.Threading.Tasks.Task.Execute() --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) at System.Threading.Tasks.Task`1.get_Result() at Xamarin.Android.Tasks.Aot.DoExecute() at Xamarin.Android.Tasks.Aot.Execute() ---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> Java.Interop.Tools.Diagnostics.XamarinAndroidException: error XA5101: Missing Android NDK toolchains directory '\toolchains'. Please install the Android NDK. at Xamarin.Android.Tasks.NdkUtil.GetNdkToolchainDirectories(String toolchainsPath, AndroidTargetArch arch) at Xamarin.Android.Tasks.NdkUtil.GetNdkToolchainPath(String androidNdkPath, AndroidTargetArch arch) at Xamarin.Android.Tasks.NdkUtil.GetNdkTool(String androidNdkPath, AndroidTargetArch arch, String tool) at Xamarin.Android.Tasks.NdkUtil.GetNdkToolPrefix(String androidNdkPath, AndroidTargetArch arch) at Xamarin.Android.Tasks.Aot.<GetAotConfigs>d__73.MoveNext() at System.Collections.Concurrent.Partitioner.DynamicPartitionerForIEnumerable`1.InternalPartitionEnumerable.GrabChunk_Buffered(KeyValuePair`2[] destArray, Int32 requestedChunkSize, Int32& actualNumElementsGrabbed) at System.Collections.Concurrent.Partitioner.DynamicPartitionerForIEnumerable`1.InternalPartitionEnumerator.GrabNextChunk(Int32 requestedChunkSize) at System.Collections.Concurrent.Partitioner.DynamicPartitionEnumerator_Abstract`2.MoveNext() at System.Threading.Tasks.Parallel.<>c__DisplayClass42_0`2.<PartitionerForEachWorker>b__1() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask) at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at System.Threading.Tasks.Task.Wait() at System.Threading.Tasks.Parallel.PartitionerForEachWorker[TSource,TLocal](Partitioner`1 source, ParallelOptions parallelOptions, Action`1 simpleBody, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally) at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally) at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body) at Xamarin.Android.Tasks.Aot.RunParallelAotCompiler(List`1 nativeLibs) at Xamarin.Android.Tasks.Aot.<>c__DisplayClass71_0.<DoExecute>b__0() at System.Threading.Tasks.Task`1.InnerInvoke() at System.Threading.Tasks.Task.Execute() ---> (Inner Exception #0) Java.Interop.Tools.Diagnostics.XamarinAndroidException: error XA5101: Missing Android NDK toolchains directory '\toolchains'. Please install the Android NDK. at Xamarin.Android.Tasks.NdkUtil.GetNdkToolchainDirectories(String toolchainsPath, AndroidTargetArch arch) at Xamarin.Android.Tasks.NdkUtil.GetNdkToolchainPath(String androidNdkPath, AndroidTargetArch arch) at Xamarin.Android.Tasks.NdkUtil.GetNdkTool(String androidNdkPath, AndroidTargetArch arch, String tool) at Xamarin.Android.Tasks.NdkUtil.GetNdkToolPrefix(String androidNdkPath, AndroidTargetArch arch) at Xamarin.Android.Tasks.Aot.<GetAotConfigs>d__73.MoveNext() at System.Collections.Concurrent.Partitioner.DynamicPartitionerForIEnumerable`1.InternalPartitionEnumerable.GrabChunk_Buffered(KeyValuePair`2[] destArray, Int32 requestedChunkSize, Int32& actualNumElementsGrabbed) at System.Collections.Concurrent.Partitioner.DynamicPartitionerForIEnumerable`1.InternalPartitionEnumerator.GrabNextChunk(Int32 requestedChunkSize) at System.Collections.Concurrent.Partitioner.DynamicPartitionEnumerator_Abstract`2.MoveNext() at System.Threading.Tasks.Parallel.<>c__DisplayClass42_0`2.<PartitionerForEachWorker>b__1() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask) at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object )<--- <--- 
Process 'msbuild.exe' exited with code '1'.

I'm not - as far as I'm aware - using the Android NDK, so I don't quite understand why I'm getting this exception message now. However, the VSTS hosted Mac environment doesn't seem to experience these problems (yet), so I'm switching back to building on Mac.

---

Edit 4

Did some searching, and apparently the above exception messages are due to problems with the automated update of Visual Studio from version 15.6 to version 15.7, which recently took place (source). Always nice to see these things being checked global roll-out.

---

Edit 5

So here are some further things I've tried since the last update: explicitly setting the JDK-version used by Xamarin.Android to version 8 and the JDK Architecture to x64 (as described here); and removing all custom renderers for Android, since the original exception seems to point at the VisualElementRenderer<TElement>, which is the base class for almost all (if not all) default and custom renderers. All of this to no avail, however, so that I suspect the exception thrown derives from within Xamarin.Forms itself.

---

Edit 6

We finally tried adding the Mac we use for our internal build flow in TFS as a private build agent to VSTS today, which was a surprisingly easy thing to do! We didn't even need to change our firewall configuration, as it seems that the agent makes connection with the server (across HTTP and HTTPS), not the other way around (we followed the steps described here). I would that all installations were this easy!

Notwithstanding the ease with which we configured our private build agent, we still experienced the same crash following builds, irrespective of whether we built with the Xamarin.Android build-task or our custom build script (which we had previously used on the very same machine from within TFS with success).

We did eventually bump into the solution by following up on a suggestion made here, however, something that seemed somewhat unrelated and not pertinent before. Yet, as I had coincidentally just updated the version of our Xamarin.Forms from version 2.5.1.444934 to version 2.5.1.527436 the day before, I decided to try running a build with this code. Which turned out to work! I've got no clue why though... But I guess that's less relevant than being able to build again!

Answer 1

As it turns out, updating Xamarin.Forms is the solution.

Though suggested as resolution to the same error as ours on the Xamarin forums, and something I had come across before, the age of the post and the old version of Xamarin.Forms involved made me disregard this option as less relevant to our problem. What didn't help either was that I thought we were already on the highest version of Xamarin.Forms 2.5.1, and are not quite ready to move to Xamarin.Forms 3.0 yet. Yesterday, however, I discovered there was still a revision we could upgrade within the 2.5.1-branch, which is what I did. When I then tried a build with this code, surprisingly the resulting APK turned out to run without any issue!

The reason why updating Xamarin.Forms works is unclear to us, though. We did, however, observe that this particular version of Xamarin.Forms, 2.5.1.527436, was released just about a week before I migrated our build definition from TFS to VSTS. It might therefore be that there's some correlation there.

---

Edit 1

Ran into some trouble with Xamarin.Forms 2.5.1.527436 today, which has eventually led me to the most likely cause for our problems creating an Android build of our App. That is, in order to resolve a crash caused by Xamarin.Forms, I tried to revert Forms to the last version I knew still worked. However, the problem with Xamarin.Forms persisted, even after the downgrade. This triggered me to inspect the project-files and compare the working version of the App to that which had been downgraded, yet should have ended up the same notwithstanding.

As it turns out some project files had references to two different versions of Xamarin.Forms: the one that I had intended to use, and the one we had been using before that. Now this mixture didn't cause any problems while running on a local development machine, since this machine would have both old and new versions of Xamarin.Forms installed and there would therefore not be any missing assemblies. However, I can imagine that this would not be the case when running a VSTS-build on a build-server. Moreover, the project files containing the incorrect assembly references were both Android projects - which explains why iOS had no trouble building...!